2024-03-28T17:45:37.914 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-28T17:45:37.914 DEBUG:teuthology.run:Teuthology command: teuthology --owner mchangir --verbose --archive /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625380 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} --name mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi -- /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625380/orig.config.yaml 2024-03-28T17:45:37.969 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T17:45:38.066 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625380 branch: wip-mchangir-testing-main-20240318.032620 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} email: mchangir@redhat.com first_in_suite: false job_id: '7625380' kernel: client: branch: testing flavor: default kdb: true sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-testing-main-20240318.032620 ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow metadata IO - SLOW_OPS - slow request sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing-main-20240318.032620 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 owner: mchangir priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3645 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 sleep_before_teardown: 0 subset: 132/256 suite: fs suite_branch: wip-mchangir-testing-main-20240318.032620 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi037.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD671zWe5hybXF0KyHI5OHfR+Dyqaeb/05OwJeCiWHFaKLqBB1OtTLf6XS3DbvDmElLofT7bTDjQ90wkz+S0qMg= smithi059.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG+bhcY6Mt2Aj9RVFa5FiGYdcKB7aLKAmoBR0TxF4kjqWEUyIcgBCwMgojh2JWjOqrcvGZWJQE2Da3EiOqkvZrA= smithi170.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF1xqvzBeqkDqWmTNLpr7tUxXWzbkqsRRbYHEbEsh9rYvEo1O91posCLpyy6cGDZPWtC9anPTVbnaDfpvIE8DTM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/fsx.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-27_03:45:23 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-28T17:45:38.067 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa; will attempt to use it 2024-03-28T17:45:38.067 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks 2024-03-28T17:45:38.067 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-28T17:45:38.069 INFO:teuthology.task.internal:Checking packages... 2024-03-28T17:45:38.092 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3be9a5c9bc793e11e2800a8c0c696e8b46742033' 2024-03-28T17:45:38.092 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-28T17:45:38.092 INFO:teuthology.packaging:ref: None 2024-03-28T17:45:38.092 INFO:teuthology.packaging:tag: None 2024-03-28T17:45:38.092 INFO:teuthology.packaging:branch: wip-mchangir-testing-main-20240318.032620 2024-03-28T17:45:38.092 INFO:teuthology.packaging:sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:45:38.092 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-testing-main-20240318.032620 2024-03-28T17:45:38.233 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2145.g3be9a5c9 2024-03-28T17:45:38.234 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-28T17:45:38.241 INFO:teuthology.task.internal:no buildpackages task found 2024-03-28T17:45:38.242 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-28T17:45:38.251 INFO:teuthology.task.internal:Saving configuration 2024-03-28T17:45:38.272 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-28T17:45:38.278 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T17:45:38.297 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625380', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 17:40:38.444984', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD671zWe5hybXF0KyHI5OHfR+Dyqaeb/05OwJeCiWHFaKLqBB1OtTLf6XS3DbvDmElLofT7bTDjQ90wkz+S0qMg='} 2024-03-28T17:45:38.315 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625380', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 17:40:38.446456', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG+bhcY6Mt2Aj9RVFa5FiGYdcKB7aLKAmoBR0TxF4kjqWEUyIcgBCwMgojh2JWjOqrcvGZWJQE2Da3EiOqkvZrA='} 2024-03-28T17:45:38.335 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625380', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 17:40:38.447541', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF1xqvzBeqkDqWmTNLpr7tUxXWzbkqsRRbYHEbEsh9rYvEo1O91posCLpyy6cGDZPWtC9anPTVbnaDfpvIE8DTM='} 2024-03-28T17:45:38.335 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-28T17:45:38.341 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-28T17:45:38.341 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-28T17:45:38.341 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-28T17:45:38.341 INFO:teuthology.run_tasks:Running task console_log... 2024-03-28T17:45:38.442 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7d3805a0d0>, signals=[15]) 2024-03-28T17:45:38.442 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-28T17:45:38.449 INFO:teuthology.task.internal:Opening connections... 2024-03-28T17:45:38.449 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi037.front.sepia.ceph.com 2024-03-28T17:45:38.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T17:45:38.559 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2024-03-28T17:45:38.559 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T17:45:38.662 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T17:45:38.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T17:45:38.750 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-28T17:45:38.757 DEBUG:teuthology.orchestra.run.smithi037:> uname -m 2024-03-28T17:45:38.775 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64 2024-03-28T17:45:38.776 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release 2024-03-28T17:45:38.833 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="CentOS Stream" 2024-03-28T17:45:38.833 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="9" 2024-03-28T17:45:38.833 INFO:teuthology.orchestra.run.smithi037.stdout:ID="centos" 2024-03-28T17:45:38.833 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE="rhel fedora" 2024-03-28T17:45:38.833 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="9" 2024-03-28T17:45:38.833 INFO:teuthology.orchestra.run.smithi037.stdout:PLATFORM_ID="platform:el9" 2024-03-28T17:45:38.833 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T17:45:38.833 INFO:teuthology.orchestra.run.smithi037.stdout:ANSI_COLOR="0;31" 2024-03-28T17:45:38.833 INFO:teuthology.orchestra.run.smithi037.stdout:LOGO="fedora-logo-icon" 2024-03-28T17:45:38.833 INFO:teuthology.orchestra.run.smithi037.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T17:45:38.834 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://centos.org/" 2024-03-28T17:45:38.834 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T17:45:38.834 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T17:45:38.834 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T17:45:38.834 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2024-03-28T17:45:38.860 DEBUG:teuthology.orchestra.run.smithi059:> uname -m 2024-03-28T17:45:38.879 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2024-03-28T17:45:38.879 DEBUG:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2024-03-28T17:45:38.937 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="CentOS Stream" 2024-03-28T17:45:38.937 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="9" 2024-03-28T17:45:38.937 INFO:teuthology.orchestra.run.smithi059.stdout:ID="centos" 2024-03-28T17:45:38.937 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE="rhel fedora" 2024-03-28T17:45:38.937 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="9" 2024-03-28T17:45:38.937 INFO:teuthology.orchestra.run.smithi059.stdout:PLATFORM_ID="platform:el9" 2024-03-28T17:45:38.937 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T17:45:38.937 INFO:teuthology.orchestra.run.smithi059.stdout:ANSI_COLOR="0;31" 2024-03-28T17:45:38.937 INFO:teuthology.orchestra.run.smithi059.stdout:LOGO="fedora-logo-icon" 2024-03-28T17:45:38.937 INFO:teuthology.orchestra.run.smithi059.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T17:45:38.937 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://centos.org/" 2024-03-28T17:45:38.937 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T17:45:38.937 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T17:45:38.938 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T17:45:38.938 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2024-03-28T17:45:38.960 DEBUG:teuthology.orchestra.run.smithi170:> uname -m 2024-03-28T17:45:38.978 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2024-03-28T17:45:38.978 DEBUG:teuthology.orchestra.run.smithi170:> cat /etc/os-release 2024-03-28T17:45:39.038 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream" 2024-03-28T17:45:39.038 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="9" 2024-03-28T17:45:39.038 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos" 2024-03-28T17:45:39.038 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora" 2024-03-28T17:45:39.038 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="9" 2024-03-28T17:45:39.038 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el9" 2024-03-28T17:45:39.039 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T17:45:39.039 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2024-03-28T17:45:39.039 INFO:teuthology.orchestra.run.smithi170.stdout:LOGO="fedora-logo-icon" 2024-03-28T17:45:39.039 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T17:45:39.039 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/" 2024-03-28T17:45:39.039 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T17:45:39.039 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T17:45:39.039 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T17:45:39.039 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2024-03-28T17:45:39.060 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-28T17:45:39.070 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-28T17:45:39.091 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-28T17:45:39.091 DEBUG:teuthology.orchestra.run.smithi037:> test '!' -e /home/ubuntu/cephtest 2024-03-28T17:45:39.094 DEBUG:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2024-03-28T17:45:39.096 DEBUG:teuthology.orchestra.run.smithi170:> test '!' -e /home/ubuntu/cephtest 2024-03-28T17:45:39.112 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-28T17:45:39.118 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-28T17:45:39.118 DEBUG:teuthology.orchestra.run.smithi037:> test -z $(ls -A /var/lib/ceph) 2024-03-28T17:45:39.151 DEBUG:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2024-03-28T17:45:39.153 DEBUG:teuthology.orchestra.run.smithi170:> test -z $(ls -A /var/lib/ceph) 2024-03-28T17:45:39.183 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-28T17:45:39.252 INFO:teuthology.run_tasks:Running task kernel... 2024-03-28T17:45:39.267 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'} 2024-03-28T17:45:39.267 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-28T17:45:39.267 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-28T17:45:39.267 DEBUG:teuthology.orchestra.run.smithi037:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T17:45:39.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T17:45:39.284 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-28T17:45:39.479 DEBUG:teuthology.task.kernel:sha1 for client.0 is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T17:45:39.479 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-03-28T17:45:39.479 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-03-28T17:45:39.498 INFO:teuthology.orchestra.run.smithi037.stdout:5.14.0-361.el9.x86_64 2024-03-28T17:45:39.498 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T17:45:39.498 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-28T17:45:39.499 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum upgrade -y linux-firmware 2024-03-28T17:45:41.728 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-28T17:45:43.403 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-28T17:45:45.003 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.0 MB 00:01 2024-03-28T17:45:48.278 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - AppStream 8.7 MB/s | 19 MB 00:02 2024-03-28T17:45:59.553 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - CRB 824 kB/s | 6.1 MB 00:07 2024-03-28T17:46:01.246 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-28T17:46:05.913 INFO:teuthology.orchestra.run.smithi037.stdout:Extra Packages for Enterprise Linux 4.5 MB/s | 21 MB 00:04 2024-03-28T17:46:11.039 INFO:teuthology.orchestra.run.smithi037.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-03-28T17:46:13.347 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-28T17:46:13.362 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-28T17:46:13.362 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-28T17:46:13.362 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-28T17:46:13.362 INFO:teuthology.orchestra.run.smithi037.stdout:Upgrading: 2024-03-28T17:46:13.362 INFO:teuthology.orchestra.run.smithi037.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-28T17:46:13.362 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:46:13.362 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-28T17:46:13.362 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-28T17:46:13.362 INFO:teuthology.orchestra.run.smithi037.stdout:Upgrade 1 Package 2024-03-28T17:46:13.362 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:46:13.362 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 375 M 2024-03-28T17:46:13.362 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-28T17:46:24.714 INFO:teuthology.orchestra.run.smithi037.stdout:linux-firmware-20240219-143.el9.noarch.rpm 33 MB/s | 375 MB 00:11 2024-03-28T17:46:24.715 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-28T17:46:24.716 INFO:teuthology.orchestra.run.smithi037.stdout:Total 33 MB/s | 375 MB 00:11 2024-03-28T17:46:24.716 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-28T17:46:24.813 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-28T17:46:24.813 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-28T17:46:27.320 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-28T17:46:27.320 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-28T17:46:33.883 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-28T17:46:34.154 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-28T17:46:34.250 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T17:46:35.473 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T17:46:35.474 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-28T17:46:35.952 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T17:46:35.975 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:46:35.975 INFO:teuthology.orchestra.run.smithi037.stdout:Upgraded: 2024-03-28T17:46:35.975 INFO:teuthology.orchestra.run.smithi037.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-28T17:46:35.975 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:46:35.975 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-28T17:46:36.291 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-03-28T17:46:36.292 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T17:46:36.517 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ 2024-03-28T17:46:36.518 DEBUG:teuthology.orchestra.run.smithi037:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-28T17:46:36.520 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi037.front.sepia.ceph.com to complete... 2024-03-28T17:47:10.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28 17:47:10 URL:https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm [336317861/336317861] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-28T17:47:10.347 INFO:teuthology.orchestra.run.smithi037.stderr:FINISHED --2024-03-28 17:47:10-- 2024-03-28T17:47:10.347 INFO:teuthology.orchestra.run.smithi037.stderr:Total wall clock time: 34s 2024-03-28T17:47:10.347 INFO:teuthology.orchestra.run.smithi037.stderr:Downloaded: 1 files, 321M in 33s (9.65 MB/s) 2024-03-28T17:47:10.351 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-03-28T17:47:10.352 DEBUG:teuthology.orchestra.run.smithi037:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-28T17:47:12.382 INFO:teuthology.orchestra.run.smithi037.stdout:Verifying... ######################################## 2024-03-28T17:47:12.491 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing... ######################################## 2024-03-28T17:47:12.576 INFO:teuthology.orchestra.run.smithi037.stdout:Updating / installing... 2024-03-28T17:47:20.408 INFO:teuthology.orchestra.run.smithi037.stdout:kernel-6.8.0_g1df0c7e8a235-1 ######################################## 2024-03-28T17:47:34.374 INFO:teuthology.orchestra.run.smithi037.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T17:47:57.840 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi037.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-28T17:47:57.840 DEBUG:teuthology.orchestra.run.smithi037:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-28T17:47:57.929 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235 2024-03-28T17:47:57.929 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/System.map 2024-03-28T17:47:57.929 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/config 2024-03-28T17:47:57.929 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel 2024-03-28T17:47:57.930 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch 2024-03-28T17:47:57.930 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86 2024-03-28T17:47:57.930 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto 2024-03-28T17:47:57.930 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-28T17:47:57.930 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-28T17:47:57.930 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-28T17:47:57.930 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-28T17:47:57.930 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-28T17:47:57.930 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-28T17:47:57.930 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-28T17:47:57.930 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-28T17:47:57.930 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-28T17:47:57.930 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-28T17:47:57.931 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-28T17:47:57.931 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-28T17:47:57.931 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-28T17:47:57.931 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-28T17:47:57.931 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-28T17:47:57.931 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-28T17:47:57.931 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-28T17:47:57.931 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-28T17:47:57.931 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-28T17:47:57.931 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-28T17:47:57.931 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel 2024-03-28T17:47:57.931 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu 2024-03-28T17:47:57.931 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce 2024-03-28T17:47:57.931 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-28T17:47:57.931 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm 2024-03-28T17:47:57.932 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-28T17:47:57.932 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-28T17:47:57.932 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-03-28T17:47:57.932 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto 2024-03-28T17:47:57.932 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-03-28T17:47:57.932 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-03-28T17:47:57.932 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-03-28T17:47:57.932 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx 2024-03-28T17:47:57.932 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-03-28T17:47:57.932 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-03-28T17:47:57.932 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-28T17:47:57.932 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-03-28T17:47:57.932 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-03-28T17:47:57.932 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-03-28T17:47:57.933 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-03-28T17:47:57.933 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-03-28T17:47:57.933 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-03-28T17:47:57.933 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-03-28T17:47:57.933 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-03-28T17:47:57.933 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-03-28T17:47:57.933 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-03-28T17:47:57.933 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-03-28T17:47:57.933 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-03-28T17:47:57.933 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-03-28T17:47:57.933 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-03-28T17:47:57.933 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-03-28T17:47:57.933 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-03-28T17:47:57.934 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-03-28T17:47:57.934 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-03-28T17:47:57.934 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-03-28T17:47:57.934 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-03-28T17:47:57.934 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-03-28T17:47:57.934 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-03-28T17:47:57.934 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-03-28T17:47:57.934 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-03-28T17:47:57.934 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-03-28T17:47:57.934 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-03-28T17:47:57.934 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-03-28T17:47:57.934 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-03-28T17:47:57.934 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-03-28T17:47:57.935 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-03-28T17:47:57.935 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-03-28T17:47:57.935 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-03-28T17:47:57.935 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-03-28T17:47:57.935 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-03-28T17:47:57.935 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-03-28T17:47:57.935 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-03-28T17:47:57.935 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-03-28T17:47:57.935 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-03-28T17:47:57.935 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-03-28T17:47:57.935 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-03-28T17:47:57.935 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-03-28T17:47:57.935 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-03-28T17:47:57.935 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers 2024-03-28T17:47:57.935 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi 2024-03-28T17:47:57.936 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-28T17:47:57.936 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-03-28T17:47:57.936 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/custom_method.ko 2024-03-28T17:47:57.936 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-03-28T17:47:57.936 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-03-28T17:47:57.936 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-03-28T17:47:57.936 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-03-28T17:47:57.936 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-03-28T17:47:57.936 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata 2024-03-28T17:47:57.936 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-03-28T17:47:57.936 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-03-28T17:47:57.936 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ata_generic.ko 2024-03-28T17:47:57.936 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-03-28T17:47:57.936 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-03-28T17:47:57.936 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-03-28T17:47:57.937 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-03-28T17:47:57.937 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-03-28T17:47:57.937 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-03-28T17:47:57.937 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-03-28T17:47:57.937 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-03-28T17:47:57.937 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-03-28T17:47:57.937 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-03-28T17:47:57.937 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-03-28T17:47:57.937 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-03-28T17:47:57.937 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-03-28T17:47:57.937 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-28T17:47:57.937 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-28T17:47:57.937 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-03-28T17:47:57.937 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-03-28T17:47:57.937 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-03-28T17:47:57.938 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-03-28T17:47:57.938 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-03-28T17:47:57.938 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-03-28T17:47:57.938 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-03-28T17:47:57.938 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-03-28T17:47:57.938 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-03-28T17:47:57.938 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-03-28T17:47:57.938 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-03-28T17:47:57.938 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-03-28T17:47:57.938 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-03-28T17:47:57.938 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-28T17:47:57.938 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-28T17:47:57.938 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-03-28T17:47:57.938 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-03-28T17:47:57.939 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-03-28T17:47:57.939 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-03-28T17:47:57.939 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-03-28T17:47:57.939 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sis.ko 2024-03-28T17:47:57.939 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-03-28T17:47:57.939 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-03-28T17:47:57.939 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-03-28T17:47:57.939 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-03-28T17:47:57.939 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-03-28T17:47:57.939 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-03-28T17:47:57.939 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-03-28T17:47:57.939 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-03-28T17:47:57.939 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-03-28T17:47:57.939 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-03-28T17:47:57.940 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-03-28T17:47:57.940 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-03-28T17:47:57.940 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-03-28T17:47:57.940 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-03-28T17:47:57.940 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-03-28T17:47:57.940 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-03-28T17:47:57.940 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-03-28T17:47:57.940 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay 2024-03-28T17:47:57.940 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-03-28T17:47:57.940 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base 2024-03-28T17:47:57.940 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap 2024-03-28T17:47:57.940 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-28T17:47:57.940 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-28T17:47:57.940 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma 2024-03-28T17:47:57.941 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-03-28T17:47:57.941 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block 2024-03-28T17:47:57.941 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe 2024-03-28T17:47:57.941 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-03-28T17:47:57.941 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/brd.ko 2024-03-28T17:47:57.941 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd 2024-03-28T17:47:57.941 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-03-28T17:47:57.941 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-03-28T17:47:57.941 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/loop.ko 2024-03-28T17:47:57.941 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx 2024-03-28T17:47:57.941 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-28T17:47:57.941 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-03-28T17:47:57.941 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk 2024-03-28T17:47:57.941 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-03-28T17:47:57.941 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-03-28T17:47:57.941 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-03-28T17:47:57.942 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/virtio_blk.ko 2024-03-28T17:47:57.942 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback 2024-03-28T17:47:57.942 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-28T17:47:57.942 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkfront.ko 2024-03-28T17:47:57.942 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram 2024-03-28T17:47:57.942 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-03-28T17:47:57.942 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char 2024-03-28T17:47:57.942 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-03-28T17:47:57.942 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random 2024-03-28T17:47:57.942 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-28T17:47:57.942 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-28T17:47:57.942 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-28T17:47:57.942 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-03-28T17:47:57.942 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-28T17:47:57.943 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi 2024-03-28T17:47:57.943 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-28T17:47:57.943 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-28T17:47:57.943 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-28T17:47:57.943 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-28T17:47:57.943 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-28T17:47:57.943 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-28T17:47:57.943 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-03-28T17:47:57.943 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave 2024-03-28T17:47:57.943 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-03-28T17:47:57.943 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-03-28T17:47:57.943 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-03-28T17:47:57.943 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm 2024-03-28T17:47:57.943 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm.ko 2024-03-28T17:47:57.943 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-28T17:47:57.943 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-28T17:47:57.944 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-28T17:47:57.944 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-28T17:47:57.944 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-28T17:47:57.944 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/uv_mmtimer.ko 2024-03-28T17:47:57.944 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/virtio_console.ko 2024-03-28T17:47:57.944 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq 2024-03-28T17:47:57.944 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-28T17:47:57.944 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-28T17:47:57.944 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-28T17:47:57.944 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-28T17:47:57.944 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-28T17:47:57.944 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-28T17:47:57.944 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto 2024-03-28T17:47:57.944 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp 2024-03-28T17:47:57.944 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-28T17:47:57.945 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-03-28T17:47:57.945 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel 2024-03-28T17:47:57.945 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat 2024-03-28T17:47:57.945 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common 2024-03-28T17:47:57.945 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-28T17:47:57.945 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-28T17:47:57.945 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-28T17:47:57.945 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-03-28T17:47:57.945 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-03-28T17:47:57.945 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio 2024-03-28T17:47:57.945 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-28T17:47:57.945 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca 2024-03-28T17:47:57.945 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-03-28T17:47:57.945 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq 2024-03-28T17:47:57.945 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-28T17:47:57.946 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma 2024-03-28T17:47:57.946 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw 2024-03-28T17:47:57.946 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-28T17:47:57.946 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat 2024-03-28T17:47:57.946 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-28T17:47:57.946 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac 2024-03-28T17:47:57.946 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-03-28T17:47:57.946 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-03-28T17:47:57.946 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-03-28T17:47:57.946 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-03-28T17:47:57.946 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-03-28T17:47:57.946 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-03-28T17:47:57.946 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-03-28T17:47:57.946 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-03-28T17:47:57.947 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-03-28T17:47:57.947 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-03-28T17:47:57.947 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-03-28T17:47:57.947 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-03-28T17:47:57.947 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-03-28T17:47:57.947 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware 2024-03-28T17:47:57.947 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/edd.ko 2024-03-28T17:47:57.947 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-28T17:47:57.947 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio 2024-03-28T17:47:57.947 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-03-28T17:47:57.947 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-28T17:47:57.947 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid 2024-03-28T17:47:57.947 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-03-28T17:47:57.947 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-03-28T17:47:57.948 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-03-28T17:47:57.948 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-03-28T17:47:57.948 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-03-28T17:47:57.948 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-03-28T17:47:57.948 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-03-28T17:47:57.948 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-03-28T17:47:57.948 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-03-28T17:47:57.948 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-03-28T17:47:57.948 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-28T17:47:57.948 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-28T17:47:57.948 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-03-28T17:47:57.948 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-03-28T17:47:57.948 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-03-28T17:47:57.948 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-03-28T17:47:57.948 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-03-28T17:47:57.949 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-03-28T17:47:57.949 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-03-28T17:47:57.949 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-03-28T17:47:57.949 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-28T17:47:57.949 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-28T17:47:57.949 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-03-28T17:47:57.949 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-03-28T17:47:57.949 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-03-28T17:47:57.949 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-03-28T17:47:57.949 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-03-28T17:47:57.949 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-03-28T17:47:57.949 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-03-28T17:47:57.949 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-03-28T17:47:57.949 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-03-28T17:47:57.949 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-28T17:47:57.949 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-03-28T17:47:57.949 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-28T17:47:57.950 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-28T17:47:57.950 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-28T17:47:57.950 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-28T17:47:57.950 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-28T17:47:57.950 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-28T17:47:57.950 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-28T17:47:57.950 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-28T17:47:57.950 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-28T17:47:57.950 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-03-28T17:47:57.950 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-03-28T17:47:57.950 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-03-28T17:47:57.950 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-28T17:47:57.950 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-03-28T17:47:57.951 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-03-28T17:47:57.951 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-03-28T17:47:57.951 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-03-28T17:47:57.951 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-03-28T17:47:57.951 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-28T17:47:57.951 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-03-28T17:47:57.951 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-03-28T17:47:57.951 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-03-28T17:47:57.951 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-03-28T17:47:57.951 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-03-28T17:47:57.951 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-03-28T17:47:57.951 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-03-28T17:47:57.951 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-03-28T17:47:57.951 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-03-28T17:47:57.951 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-03-28T17:47:57.952 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-03-28T17:47:57.952 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-03-28T17:47:57.952 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv 2024-03-28T17:47:57.952 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-03-28T17:47:57.952 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-03-28T17:47:57.952 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-03-28T17:47:57.952 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon 2024-03-28T17:47:57.952 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-03-28T17:47:57.952 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-03-28T17:47:57.952 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-28T17:47:57.952 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-03-28T17:47:57.952 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-03-28T17:47:57.952 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-03-28T17:47:57.952 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-03-28T17:47:57.953 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-03-28T17:47:57.953 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-03-28T17:47:57.953 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-03-28T17:47:57.953 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-03-28T17:47:57.953 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-03-28T17:47:57.953 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-03-28T17:47:57.953 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-03-28T17:47:57.953 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-03-28T17:47:57.953 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-03-28T17:47:57.953 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-03-28T17:47:57.953 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-03-28T17:47:57.953 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-03-28T17:47:57.953 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-03-28T17:47:57.953 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-03-28T17:47:57.953 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-03-28T17:47:57.953 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-28T17:47:57.954 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-03-28T17:47:57.954 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-03-28T17:47:57.954 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-28T17:47:57.954 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-03-28T17:47:57.954 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-03-28T17:47:57.954 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-03-28T17:47:57.954 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-03-28T17:47:57.954 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-03-28T17:47:57.954 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-03-28T17:47:57.954 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-03-28T17:47:57.954 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-03-28T17:47:57.954 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-03-28T17:47:57.954 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-03-28T17:47:57.954 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-03-28T17:47:57.954 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-03-28T17:47:57.954 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-03-28T17:47:57.955 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-03-28T17:47:57.955 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-28T17:47:57.955 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-03-28T17:47:57.955 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-03-28T17:47:57.955 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-03-28T17:47:57.955 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-03-28T17:47:57.955 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-03-28T17:47:57.955 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-03-28T17:47:57.955 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-03-28T17:47:57.955 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-03-28T17:47:57.955 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-03-28T17:47:57.955 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-03-28T17:47:57.955 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-03-28T17:47:57.955 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-03-28T17:47:57.956 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-03-28T17:47:57.956 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-03-28T17:47:57.956 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-03-28T17:47:57.956 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-03-28T17:47:57.956 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-03-28T17:47:57.956 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-03-28T17:47:57.956 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-03-28T17:47:57.956 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-03-28T17:47:57.956 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-03-28T17:47:57.956 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-03-28T17:47:57.956 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-03-28T17:47:57.956 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-03-28T17:47:57.956 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-03-28T17:47:57.956 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-03-28T17:47:57.956 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-03-28T17:47:57.956 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-03-28T17:47:57.957 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-03-28T17:47:57.957 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-03-28T17:47:57.957 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-03-28T17:47:57.957 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-03-28T17:47:57.957 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-03-28T17:47:57.957 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-03-28T17:47:57.957 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-03-28T17:47:57.957 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-03-28T17:47:57.957 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-03-28T17:47:57.957 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-03-28T17:47:57.957 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-03-28T17:47:57.957 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-03-28T17:47:57.957 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-03-28T17:47:57.957 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-03-28T17:47:57.957 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-03-28T17:47:57.958 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-03-28T17:47:57.958 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-03-28T17:47:57.958 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-03-28T17:47:57.958 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-03-28T17:47:57.958 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus 2024-03-28T17:47:57.958 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-28T17:47:57.958 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-28T17:47:57.958 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-28T17:47:57.958 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-28T17:47:57.958 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-28T17:47:57.958 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-28T17:47:57.958 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-28T17:47:57.958 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-28T17:47:57.959 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-28T17:47:57.959 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-28T17:47:57.959 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-28T17:47:57.959 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-28T17:47:57.959 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-03-28T17:47:57.959 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-03-28T17:47:57.959 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-03-28T17:47:57.959 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-28T17:47:57.959 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-03-28T17:47:57.959 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-03-28T17:47:57.959 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-03-28T17:47:57.959 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-03-28T17:47:57.959 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-03-28T17:47:57.959 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-03-28T17:47:57.960 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-03-28T17:47:57.960 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-03-28T17:47:57.960 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-03-28T17:47:57.960 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-03-28T17:47:57.960 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-03-28T17:47:57.960 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-03-28T17:47:57.960 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-03-28T17:47:57.960 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-03-28T17:47:57.960 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-03-28T17:47:57.960 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-03-28T17:47:57.960 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-03-28T17:47:57.960 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-03-28T17:47:57.960 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-03-28T17:47:57.960 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-03-28T17:47:57.960 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-03-28T17:47:57.961 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-03-28T17:47:57.961 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-03-28T17:47:57.961 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-03-28T17:47:57.961 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-03-28T17:47:57.961 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c 2024-03-28T17:47:57.961 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos 2024-03-28T17:47:57.961 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-28T17:47:57.961 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-28T17:47:57.961 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses 2024-03-28T17:47:57.961 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-28T17:47:57.961 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-28T17:47:57.961 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-28T17:47:57.961 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-28T17:47:57.961 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-28T17:47:57.961 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-28T17:47:57.961 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-28T17:47:57.961 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-28T17:47:57.962 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-28T17:47:57.962 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-28T17:47:57.962 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-28T17:47:57.962 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-28T17:47:57.962 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-28T17:47:57.962 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-28T17:47:57.962 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-28T17:47:57.962 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-28T17:47:57.962 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-28T17:47:57.962 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-28T17:47:57.962 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-28T17:47:57.962 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-28T17:47:57.962 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-28T17:47:57.962 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-28T17:47:57.962 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-28T17:47:57.963 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-dev.ko 2024-03-28T17:47:57.963 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-03-28T17:47:57.963 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-28T17:47:57.963 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-03-28T17:47:57.963 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-03-28T17:47:57.963 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband 2024-03-28T17:47:57.963 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core 2024-03-28T17:47:57.963 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-28T17:47:57.963 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-03-28T17:47:57.963 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-28T17:47:57.963 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-28T17:47:57.963 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-28T17:47:57.963 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-28T17:47:57.963 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-28T17:47:57.963 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw 2024-03-28T17:47:57.964 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4 2024-03-28T17:47:57.964 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-28T17:47:57.964 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4 2024-03-28T17:47:57.964 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-28T17:47:57.964 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5 2024-03-28T17:47:57.964 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-28T17:47:57.964 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca 2024-03-28T17:47:57.964 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-28T17:47:57.964 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp 2024-03-28T17:47:57.964 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib 2024-03-28T17:47:57.964 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-28T17:47:57.964 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser 2024-03-28T17:47:57.964 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-28T17:47:57.964 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert 2024-03-28T17:47:57.964 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-28T17:47:57.965 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp 2024-03-28T17:47:57.965 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-28T17:47:57.965 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt 2024-03-28T17:47:57.965 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-28T17:47:57.965 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input 2024-03-28T17:47:57.965 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport 2024-03-28T17:47:57.965 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-28T17:47:57.965 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-28T17:47:57.965 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-03-28T17:47:57.965 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-03-28T17:47:57.965 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard 2024-03-28T17:47:57.965 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-28T17:47:57.965 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse 2024-03-28T17:47:57.965 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-03-28T17:47:57.965 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-03-28T17:47:57.966 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-03-28T17:47:57.966 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-28T17:47:57.966 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-03-28T17:47:57.966 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-28T17:47:57.966 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-28T17:47:57.966 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-28T17:47:57.966 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4 2024-03-28T17:47:57.966 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-28T17:47:57.966 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio 2024-03-28T17:47:57.966 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-03-28T17:47:57.966 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-03-28T17:47:57.966 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-28T17:47:57.966 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-03-28T17:47:57.966 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-03-28T17:47:57.966 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds 2024-03-28T17:47:57.967 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-03-28T17:47:57.967 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-03-28T17:47:57.967 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-03-28T17:47:57.967 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-03-28T17:47:57.967 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger 2024-03-28T17:47:57.967 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-28T17:47:57.967 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-28T17:47:57.967 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-28T17:47:57.967 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-28T17:47:57.967 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-28T17:47:57.967 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-28T17:47:57.967 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-28T17:47:57.967 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-28T17:47:57.967 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-28T17:47:57.967 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md 2024-03-28T17:47:57.968 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache 2024-03-28T17:47:57.968 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-03-28T17:47:57.968 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-03-28T17:47:57.968 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-03-28T17:47:57.968 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-03-28T17:47:57.968 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-03-28T17:47:57.968 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-03-28T17:47:57.968 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-03-28T17:47:57.968 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-03-28T17:47:57.968 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-03-28T17:47:57.968 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-03-28T17:47:57.968 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-03-28T17:47:57.968 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-03-28T17:47:57.968 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-03-28T17:47:57.969 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-03-28T17:47:57.969 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-03-28T17:47:57.969 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-03-28T17:47:57.969 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data 2024-03-28T17:47:57.969 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-28T17:47:57.969 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-03-28T17:47:57.969 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-03-28T17:47:57.969 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-03-28T17:47:57.969 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-03-28T17:47:57.969 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message 2024-03-28T17:47:57.969 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion 2024-03-28T17:47:57.969 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-03-28T17:47:57.969 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-03-28T17:47:57.969 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-03-28T17:47:57.969 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-03-28T17:47:57.970 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-03-28T17:47:57.970 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-03-28T17:47:57.970 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-03-28T17:47:57.970 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd 2024-03-28T17:47:57.970 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-03-28T17:47:57.970 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-03-28T17:47:57.970 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mfd-core.ko 2024-03-28T17:47:57.970 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-03-28T17:47:57.970 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-03-28T17:47:57.970 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-03-28T17:47:57.970 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-03-28T17:47:57.970 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc 2024-03-28T17:47:57.970 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl 2024-03-28T17:47:57.970 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-28T17:47:57.970 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-03-28T17:47:57.971 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-03-28T17:47:57.971 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-03-28T17:47:57.971 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710 2024-03-28T17:47:57.971 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-03-28T17:47:57.971 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom 2024-03-28T17:47:57.971 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-03-28T17:47:57.971 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-28T17:47:57.971 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-03-28T17:47:57.971 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-03-28T17:47:57.971 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-03-28T17:47:57.971 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-03-28T17:47:57.971 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-03-28T17:47:57.971 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d 2024-03-28T17:47:57.971 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-28T17:47:57.971 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-28T17:47:57.971 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei 2024-03-28T17:47:57.972 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-03-28T17:47:57.972 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-03-28T17:47:57.972 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-03-28T17:47:57.972 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru 2024-03-28T17:47:57.972 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-28T17:47:57.972 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp 2024-03-28T17:47:57.972 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-28T17:47:57.972 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-28T17:47:57.972 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-28T17:47:57.972 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-03-28T17:47:57.972 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-03-28T17:47:57.972 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-03-28T17:47:57.972 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-03-28T17:47:57.972 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci 2024-03-28T17:47:57.973 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-28T17:47:57.973 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net 2024-03-28T17:47:57.973 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding 2024-03-28T17:47:57.973 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-03-28T17:47:57.973 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa 2024-03-28T17:47:57.973 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53 2024-03-28T17:47:57.973 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-28T17:47:57.973 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-28T17:47:57.973 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-28T17:47:57.973 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx 2024-03-28T17:47:57.973 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-28T17:47:57.973 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-03-28T17:47:57.973 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-03-28T17:47:57.973 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet 2024-03-28T17:47:57.973 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com 2024-03-28T17:47:57.974 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-28T17:47:57.974 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-28T17:47:57.974 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-28T17:47:57.974 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-28T17:47:57.974 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390 2024-03-28T17:47:57.974 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-28T17:47:57.974 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-28T17:47:57.974 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-28T17:47:57.974 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-28T17:47:57.974 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec 2024-03-28T17:47:57.974 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-28T17:47:57.974 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere 2024-03-28T17:47:57.974 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-28T17:47:57.974 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon 2024-03-28T17:47:57.975 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-28T17:47:57.975 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera 2024-03-28T17:47:57.975 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-28T17:47:57.975 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd 2024-03-28T17:47:57.975 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-28T17:47:57.975 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-28T17:47:57.975 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-28T17:47:57.975 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros 2024-03-28T17:47:57.975 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx 2024-03-28T17:47:57.975 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-28T17:47:57.975 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-28T17:47:57.975 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-28T17:47:57.975 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-28T17:47:57.975 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-28T17:47:57.975 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-28T17:47:57.977 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio 2024-03-28T17:47:57.977 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-28T17:47:57.977 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-28T17:47:57.977 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-28T17:47:57.977 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-28T17:47:57.977 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-28T17:47:57.977 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-28T17:47:57.977 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-28T17:47:57.977 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-28T17:47:57.977 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-28T17:47:57.977 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-28T17:47:57.977 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco 2024-03-28T17:47:57.977 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic 2024-03-28T17:47:57.977 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-28T17:47:57.977 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec 2024-03-28T17:47:57.978 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip 2024-03-28T17:47:57.978 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-28T17:47:57.978 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-28T17:47:57.978 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-28T17:47:57.978 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-28T17:47:57.978 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-28T17:47:57.978 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-28T17:47:57.978 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink 2024-03-28T17:47:57.978 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-28T17:47:57.978 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-28T17:47:57.978 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-03-28T17:47:57.978 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex 2024-03-28T17:47:57.978 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet 2024-03-28T17:47:57.978 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox 2024-03-28T17:47:57.981 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-28T17:47:57.981 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-28T17:47:57.981 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-28T17:47:57.981 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-28T17:47:57.981 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-28T17:47:57.981 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-28T17:47:57.981 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel 2024-03-28T17:47:57.981 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-28T17:47:57.981 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom 2024-03-28T17:47:57.981 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-28T17:47:57.981 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-28T17:47:57.981 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi 2024-03-28T17:47:57.981 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-28T17:47:57.981 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-28T17:47:57.982 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion 2024-03-28T17:47:57.982 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-28T17:47:57.982 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia 2024-03-28T17:47:57.982 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-28T17:47:57.982 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines 2024-03-28T17:47:57.982 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-28T17:47:57.982 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-28T17:47:57.982 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic 2024-03-28T17:47:57.982 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-28T17:47:57.982 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-28T17:47:57.982 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-28T17:47:57.982 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-28T17:47:57.982 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-28T17:47:57.982 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc 2024-03-28T17:47:57.983 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-28T17:47:57.983 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek 2024-03-28T17:47:57.983 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-28T17:47:57.983 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-28T17:47:57.983 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-28T17:47:57.983 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-28T17:47:57.983 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker 2024-03-28T17:47:57.983 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-28T17:47:57.983 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc 2024-03-28T17:47:57.983 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-28T17:47:57.983 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan 2024-03-28T17:47:57.983 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-28T17:47:57.983 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis 2024-03-28T17:47:57.983 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-28T17:47:57.986 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154 2024-03-28T17:47:57.986 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-28T17:47:57.986 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-03-28T17:47:57.986 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan 2024-03-28T17:47:57.986 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-28T17:47:57.986 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-03-28T17:47:57.986 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-03-28T17:47:57.986 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio 2024-03-28T17:47:57.986 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-03-28T17:47:57.986 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-28T17:47:57.986 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-28T17:47:57.986 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-28T17:47:57.986 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-03-28T17:47:57.986 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/net_failover.ko 2024-03-28T17:47:57.987 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-03-28T17:47:57.987 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-03-28T17:47:57.987 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs 2024-03-28T17:47:57.987 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-28T17:47:57.987 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-28T17:47:57.987 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy 2024-03-28T17:47:57.987 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-03-28T17:47:57.987 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-28T17:47:57.987 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-28T17:47:57.987 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-28T17:47:57.987 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-03-28T17:47:57.987 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-03-28T17:47:57.987 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-03-28T17:47:57.987 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-03-28T17:47:57.988 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/dp83640.ko 2024-03-28T17:47:57.988 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-03-28T17:47:57.988 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-03-28T17:47:57.988 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-03-28T17:47:57.988 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-03-28T17:47:57.988 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-03-28T17:47:57.988 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-03-28T17:47:57.988 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-03-28T17:47:57.988 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-03-28T17:47:57.988 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-03-28T17:47:57.988 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-03-28T17:47:57.988 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-03-28T17:47:57.988 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-03-28T17:47:57.988 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp 2024-03-28T17:47:57.988 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-28T17:47:57.989 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-03-28T17:47:57.989 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-28T17:47:57.989 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-28T17:47:57.989 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-28T17:47:57.989 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-28T17:47:57.989 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-03-28T17:47:57.989 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-03-28T17:47:57.989 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-03-28T17:47:57.989 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip 2024-03-28T17:47:57.989 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slhc.ko 2024-03-28T17:47:57.989 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-03-28T17:47:57.989 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-03-28T17:47:57.989 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-03-28T17:47:57.989 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team 2024-03-28T17:47:57.989 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tun.ko 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/virtio_net.ko 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netfront.ko 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-28T17:47:57.995 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-28T17:47:57.995 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-03-28T17:47:57.995 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-03-28T17:47:57.995 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp 2024-03-28T17:47:57.995 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-03-28T17:47:57.995 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-03-28T17:47:57.995 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc 2024-03-28T17:47:57.995 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-28T17:47:57.995 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-28T17:47:57.995 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-28T17:47:57.995 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-28T17:47:57.995 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-28T17:47:57.995 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-28T17:47:57.995 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-28T17:47:57.995 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-28T17:47:57.996 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-28T17:47:57.996 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-28T17:47:57.996 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-03-28T17:47:57.996 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-28T17:47:57.996 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-28T17:47:57.996 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-28T17:47:57.996 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-28T17:47:57.996 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-28T17:47:57.996 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-28T17:47:57.996 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-03-28T17:47:57.996 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-28T17:47:57.996 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-28T17:47:57.996 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-28T17:47:57.996 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-28T17:47:57.996 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-28T17:47:57.997 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-28T17:47:57.997 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-28T17:47:57.997 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-28T17:47:57.997 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-28T17:47:57.997 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-28T17:47:57.997 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-28T17:47:57.997 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-28T17:47:57.997 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-03-28T17:47:57.997 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi 2024-03-28T17:47:57.997 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-03-28T17:47:57.997 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-03-28T17:47:57.997 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-03-28T17:47:57.997 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-03-28T17:47:57.997 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-03-28T17:47:57.998 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid 2024-03-28T17:47:57.998 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-28T17:47:57.998 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-03-28T17:47:57.998 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx 2024-03-28T17:47:57.998 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-28T17:47:57.998 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-28T17:47:57.998 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-03-28T17:47:57.998 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr 2024-03-28T17:47:57.998 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-28T17:47:57.998 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-03-28T17:47:57.998 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi 2024-03-28T17:47:57.998 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-28T17:47:57.998 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa 2024-03-28T17:47:57.998 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-03-28T17:47:57.998 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc 2024-03-28T17:47:57.999 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-28T17:47:57.999 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i 2024-03-28T17:47:57.999 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-28T17:47:57.999 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-03-28T17:47:57.999 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor 2024-03-28T17:47:57.999 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-28T17:47:57.999 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi 2024-03-28T17:47:57.999 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-28T17:47:57.999 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-28T17:47:57.999 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-28T17:47:57.999 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-28T17:47:57.999 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-28T17:47:57.999 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-03-28T17:47:57.999 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler 2024-03-28T17:47:57.999 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-28T17:47:58.000 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-28T17:47:58.000 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-28T17:47:58.000 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-28T17:47:58.000 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-03-28T17:47:58.000 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r 2024-03-28T17:47:58.000 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-28T17:47:58.000 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-03-28T17:47:58.000 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe 2024-03-28T17:47:58.000 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-28T17:47:58.000 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-28T17:47:58.000 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic 2024-03-28T17:47:58.000 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-03-28T17:47:58.000 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-03-28T17:47:58.000 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-03-28T17:47:58.000 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-03-28T17:47:58.000 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-03-28T17:47:58.001 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-03-28T17:47:58.001 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-03-28T17:47:58.001 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci 2024-03-28T17:47:58.001 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-03-28T17:47:58.001 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-28T17:47:58.001 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-28T17:47:58.001 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc 2024-03-28T17:47:58.001 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-03-28T17:47:58.001 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-03-28T17:47:58.001 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-28T17:47:58.001 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas 2024-03-28T17:47:58.001 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-03-28T17:47:58.001 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc 2024-03-28T17:47:58.001 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-28T17:47:58.001 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid 2024-03-28T17:47:58.001 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-03-28T17:47:58.001 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-28T17:47:58.002 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-28T17:47:58.002 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-28T17:47:58.002 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas 2024-03-28T17:47:58.002 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-28T17:47:58.002 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas 2024-03-28T17:47:58.002 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-28T17:47:58.002 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-03-28T17:47:58.002 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001 2024-03-28T17:47:58.002 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-28T17:47:58.002 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-03-28T17:47:58.002 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-03-28T17:47:58.002 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx 2024-03-28T17:47:58.002 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-28T17:47:58.002 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-28T17:47:58.003 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx 2024-03-28T17:47:58.003 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-28T17:47:58.003 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-03-28T17:47:58.003 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-03-28T17:47:58.003 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-28T17:47:58.003 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-28T17:47:58.003 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-28T17:47:58.003 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-28T17:47:58.003 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-28T17:47:58.003 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-03-28T17:47:58.003 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-03-28T17:47:58.003 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-03-28T17:47:58.003 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2 2024-03-28T17:47:58.003 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-28T17:47:58.003 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-03-28T17:47:58.003 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-28T17:47:58.004 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-03-28T17:47:58.004 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb 2024-03-28T17:47:58.004 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-03-28T17:47:58.004 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target 2024-03-28T17:47:58.004 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi 2024-03-28T17:47:58.004 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-28T17:47:58.004 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback 2024-03-28T17:47:58.004 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-28T17:47:58.004 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-03-28T17:47:58.004 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-03-28T17:47:58.004 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-03-28T17:47:58.004 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-03-28T17:47:58.004 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-03-28T17:47:58.004 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc 2024-03-28T17:47:58.005 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-28T17:47:58.005 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal 2024-03-28T17:47:58.005 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel 2024-03-28T17:47:58.005 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal 2024-03-28T17:47:58.005 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-28T17:47:58.005 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-28T17:47:58.005 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-28T17:47:58.005 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-28T17:47:58.005 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-28T17:47:58.005 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-28T17:47:58.005 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-28T17:47:58.005 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-28T17:47:58.005 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-28T17:47:58.005 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-28T17:47:58.005 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-28T17:47:58.005 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-28T17:47:58.006 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-28T17:47:58.006 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-28T17:47:58.006 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-28T17:47:58.006 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-28T17:47:58.006 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-28T17:47:58.006 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-28T17:47:58.006 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty 2024-03-28T17:47:58.006 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless 2024-03-28T17:47:58.006 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-28T17:47:58.006 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-03-28T17:47:58.006 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-03-28T17:47:58.006 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-03-28T17:47:58.006 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial 2024-03-28T17:47:58.006 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250 2024-03-28T17:47:58.006 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-28T17:47:58.007 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-03-28T17:47:58.007 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm 2024-03-28T17:47:58.007 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-28T17:47:58.007 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs 2024-03-28T17:47:58.009 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core 2024-03-28T17:47:58.009 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-28T17:47:58.009 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host 2024-03-28T17:47:58.009 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-28T17:47:58.009 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio 2024-03-28T17:47:58.009 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-03-28T17:47:58.009 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-03-28T17:47:58.009 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-03-28T17:47:58.009 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-03-28T17:47:58.009 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-03-28T17:47:58.009 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb 2024-03-28T17:47:58.010 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class 2024-03-28T17:47:58.010 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-03-28T17:47:58.010 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-28T17:47:58.010 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-03-28T17:47:58.010 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-03-28T17:47:58.010 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host 2024-03-28T17:47:58.010 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-28T17:47:58.010 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image 2024-03-28T17:47:58.010 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-03-28T17:47:58.010 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-03-28T17:47:58.010 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc 2024-03-28T17:47:58.010 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-03-28T17:47:58.010 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-03-28T17:47:58.011 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-03-28T17:47:58.011 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-03-28T17:47:58.011 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-03-28T17:47:58.011 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-03-28T17:47:58.011 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-03-28T17:47:58.011 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-28T17:47:58.011 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-03-28T17:47:58.011 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-03-28T17:47:58.011 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga 2024-03-28T17:47:58.011 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-28T17:47:58.011 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-28T17:47:58.011 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-03-28T17:47:58.011 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-03-28T17:47:58.011 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-28T17:47:58.012 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-03-28T17:47:58.012 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-03-28T17:47:58.012 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy 2024-03-28T17:47:58.012 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-03-28T17:47:58.012 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial 2024-03-28T17:47:58.012 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-03-28T17:47:58.012 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-03-28T17:47:58.012 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-28T17:47:58.012 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-03-28T17:47:58.012 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-03-28T17:47:58.012 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-03-28T17:47:58.012 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-28T17:47:58.012 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-28T17:47:58.012 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-03-28T17:47:58.013 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-28T17:47:58.013 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-28T17:47:58.013 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-28T17:47:58.013 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-03-28T17:47:58.013 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-03-28T17:47:58.013 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-03-28T17:47:58.013 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-03-28T17:47:58.013 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-28T17:47:58.013 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-03-28T17:47:58.013 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-28T17:47:58.013 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-28T17:47:58.013 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-28T17:47:58.013 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-03-28T17:47:58.013 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-03-28T17:47:58.014 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-03-28T17:47:58.014 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-03-28T17:47:58.014 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-03-28T17:47:58.014 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-03-28T17:47:58.014 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-03-28T17:47:58.014 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-03-28T17:47:58.014 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-03-28T17:47:58.014 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-03-28T17:47:58.014 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-03-28T17:47:58.014 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-03-28T17:47:58.014 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-03-28T17:47:58.014 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-03-28T17:47:58.014 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-28T17:47:58.014 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio.ko 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_balloon.ko 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_mmio.ko 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci.ko 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_ring.ko 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs 2024-03-28T17:47:58.024 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-03-28T17:47:58.024 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat 2024-03-28T17:47:58.024 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/fat.ko 2024-03-28T17:47:58.024 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-03-28T17:47:58.024 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/vfat.ko 2024-03-28T17:47:58.024 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse 2024-03-28T17:47:58.024 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-03-28T17:47:58.024 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/fuse.ko 2024-03-28T17:47:58.024 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2 2024-03-28T17:47:58.024 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-03-28T17:47:58.024 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs 2024-03-28T17:47:58.024 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-03-28T17:47:58.024 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus 2024-03-28T17:47:58.024 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-03-28T17:47:58.024 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs 2024-03-28T17:47:58.025 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-03-28T17:47:58.025 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs 2024-03-28T17:47:58.025 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-03-28T17:47:58.025 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd 2024-03-28T17:47:58.025 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-03-28T17:47:58.025 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix 2024-03-28T17:47:58.025 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-03-28T17:47:58.025 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs 2024-03-28T17:47:58.025 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-03-28T17:47:58.025 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs 2024-03-28T17:47:58.025 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout 2024-03-28T17:47:58.025 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-28T17:47:58.025 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout 2024-03-28T17:47:58.025 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-28T17:47:58.025 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout 2024-03-28T17:47:58.026 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-28T17:47:58.026 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-03-28T17:47:58.026 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-03-28T17:47:58.026 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-03-28T17:47:58.026 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common 2024-03-28T17:47:58.026 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-03-28T17:47:58.026 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-03-28T17:47:58.026 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd 2024-03-28T17:47:58.026 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-03-28T17:47:58.026 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2 2024-03-28T17:47:58.026 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-03-28T17:47:58.026 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls 2024-03-28T17:47:58.026 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-03-28T17:47:58.026 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-03-28T17:47:58.027 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-03-28T17:47:58.027 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-03-28T17:47:58.027 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-03-28T17:47:58.027 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-03-28T17:47:58.027 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-03-28T17:47:58.027 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-03-28T17:47:58.027 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-03-28T17:47:58.027 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-03-28T17:47:58.027 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-03-28T17:47:58.027 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-03-28T17:47:58.027 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-03-28T17:47:58.027 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-03-28T17:47:58.027 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-03-28T17:47:58.027 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-03-28T17:47:58.027 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-03-28T17:47:58.028 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-03-28T17:47:58.028 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-03-28T17:47:58.028 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-03-28T17:47:58.028 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-03-28T17:47:58.028 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-03-28T17:47:58.028 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-03-28T17:47:58.028 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-03-28T17:47:58.028 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-03-28T17:47:58.028 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-03-28T17:47:58.028 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-03-28T17:47:58.028 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-03-28T17:47:58.028 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-03-28T17:47:58.028 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-03-28T17:47:58.028 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-03-28T17:47:58.028 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-03-28T17:47:58.029 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-03-28T17:47:58.029 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-03-28T17:47:58.029 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-03-28T17:47:58.029 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-03-28T17:47:58.029 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-03-28T17:47:58.029 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-03-28T17:47:58.029 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-03-28T17:47:58.029 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-03-28T17:47:58.029 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-03-28T17:47:58.029 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-03-28T17:47:58.029 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-03-28T17:47:58.029 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-03-28T17:47:58.029 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-03-28T17:47:58.029 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-03-28T17:47:58.029 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-03-28T17:47:58.030 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-03-28T17:47:58.030 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-28T17:47:58.030 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-03-28T17:47:58.030 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2 2024-03-28T17:47:58.030 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster 2024-03-28T17:47:58.030 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-28T17:47:58.030 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm 2024-03-28T17:47:58.030 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-28T17:47:58.030 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs 2024-03-28T17:47:58.030 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-28T17:47:58.030 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-03-28T17:47:58.030 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-28T17:47:58.030 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-28T17:47:58.030 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-28T17:47:58.031 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs 2024-03-28T17:47:58.031 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-03-28T17:47:58.031 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore 2024-03-28T17:47:58.031 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-03-28T17:47:58.031 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs 2024-03-28T17:47:58.031 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-03-28T17:47:58.031 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs 2024-03-28T17:47:58.031 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-03-28T17:47:58.031 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb 2024-03-28T17:47:58.031 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client 2024-03-28T17:47:58.031 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-03-28T17:47:58.031 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common 2024-03-28T17:47:58.031 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-03-28T17:47:58.031 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-03-28T17:47:58.031 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs 2024-03-28T17:47:58.032 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-03-28T17:47:58.032 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv 2024-03-28T17:47:58.032 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-03-28T17:47:58.032 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf 2024-03-28T17:47:58.032 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-03-28T17:47:58.032 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs 2024-03-28T17:47:58.032 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-03-28T17:47:58.032 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs 2024-03-28T17:47:58.032 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-03-28T17:47:58.032 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel 2024-03-28T17:47:58.032 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu 2024-03-28T17:47:58.032 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu/rcutorture.ko 2024-03-28T17:47:58.032 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-03-28T17:47:58.032 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace 2024-03-28T17:47:58.032 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-28T17:47:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib 2024-03-28T17:47:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/asn1_encoder.ko 2024-03-28T17:47:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-03-28T17:47:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-03-28T17:47:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto 2024-03-28T17:47:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-03-28T17:47:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-03-28T17:47:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-03-28T17:47:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-03-28T17:47:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4 2024-03-28T17:47:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-03-28T17:47:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-03-28T17:47:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math 2024-03-28T17:47:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-03-28T17:47:58.034 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6 2024-03-28T17:47:58.034 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-03-28T17:47:58.034 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon 2024-03-28T17:47:58.034 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-28T17:47:58.034 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-03-28T17:47:58.034 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-03-28T17:47:58.034 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-03-28T17:47:58.034 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd 2024-03-28T17:47:58.034 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd/zstd_compress.ko 2024-03-28T17:47:58.034 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm 2024-03-28T17:47:58.034 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-03-28T17:47:58.034 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net 2024-03-28T17:47:58.034 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan 2024-03-28T17:47:58.034 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-03-28T17:47:58.034 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-03-28T17:47:58.035 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-03-28T17:47:58.035 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-03-28T17:47:58.035 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-28T17:47:58.035 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-03-28T17:47:58.035 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-03-28T17:47:58.035 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-03-28T17:47:58.035 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802 2024-03-28T17:47:58.035 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-03-28T17:47:58.035 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-03-28T17:47:58.035 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-03-28T17:47:58.035 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-03-28T17:47:58.035 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-03-28T17:47:58.035 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q 2024-03-28T17:47:58.035 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-03-28T17:47:58.035 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p 2024-03-28T17:47:58.036 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-03-28T17:47:58.036 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-03-28T17:47:58.036 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-03-28T17:47:58.036 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-03-28T17:47:58.036 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv 2024-03-28T17:47:58.036 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-03-28T17:47:58.036 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge 2024-03-28T17:47:58.036 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-03-28T17:47:58.036 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-03-28T17:47:58.036 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter 2024-03-28T17:47:58.036 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-28T17:47:58.036 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-28T17:47:58.036 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-28T17:47:58.036 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-28T17:47:58.036 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-28T17:47:58.037 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-28T17:47:58.037 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-28T17:47:58.037 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-28T17:47:58.037 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-28T17:47:58.037 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-28T17:47:58.037 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-28T17:47:58.037 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-28T17:47:58.037 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-28T17:47:58.037 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-28T17:47:58.037 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-28T17:47:58.037 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-28T17:47:58.037 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-28T17:47:58.037 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-28T17:47:58.037 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-28T17:47:58.037 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-28T17:47:58.038 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-03-28T17:47:58.038 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-28T17:47:58.038 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-28T17:47:58.038 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-28T17:47:58.038 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph 2024-03-28T17:47:58.038 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-03-28T17:47:58.038 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core 2024-03-28T17:47:58.038 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/failover.ko 2024-03-28T17:47:58.038 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-03-28T17:47:58.038 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp 2024-03-28T17:47:58.038 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-03-28T17:47:58.038 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-03-28T17:47:58.038 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-03-28T17:47:58.038 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-03-28T17:47:58.038 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver 2024-03-28T17:47:58.039 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver/dns_resolver.ko 2024-03-28T17:47:58.039 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa 2024-03-28T17:47:58.039 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-03-28T17:47:58.039 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-03-28T17:47:58.039 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-03-28T17:47:58.039 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-03-28T17:47:58.039 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-03-28T17:47:58.039 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154 2024-03-28T17:47:58.039 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan 2024-03-28T17:47:58.039 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-28T17:47:58.039 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-03-28T17:47:58.039 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-28T17:47:58.039 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4 2024-03-28T17:47:58.039 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-03-28T17:47:58.039 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-03-28T17:47:58.040 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-03-28T17:47:58.040 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-03-28T17:47:58.040 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-03-28T17:47:58.040 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-03-28T17:47:58.040 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-03-28T17:47:58.040 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-03-28T17:47:58.040 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-03-28T17:47:58.040 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-03-28T17:47:58.040 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter 2024-03-28T17:47:58.040 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-28T17:47:58.040 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-28T17:47:58.040 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-28T17:47:58.040 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-28T17:47:58.040 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-28T17:47:58.041 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-28T17:47:58.041 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-28T17:47:58.041 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-28T17:47:58.041 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-28T17:47:58.041 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-28T17:47:58.041 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-28T17:47:58.041 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-28T17:47:58.041 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-28T17:47:58.041 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-28T17:47:58.041 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-28T17:47:58.041 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-28T17:47:58.041 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-28T17:47:58.041 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-28T17:47:58.041 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-28T17:47:58.041 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-28T17:47:58.042 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-28T17:47:58.042 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-03-28T17:47:58.042 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-03-28T17:47:58.042 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-03-28T17:47:58.042 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-03-28T17:47:58.042 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-03-28T17:47:58.042 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-03-28T17:47:58.042 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-03-28T17:47:58.042 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-03-28T17:47:58.042 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-03-28T17:47:58.042 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-03-28T17:47:58.042 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-03-28T17:47:58.042 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-03-28T17:47:58.042 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-03-28T17:47:58.042 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-03-28T17:47:58.043 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-03-28T17:47:58.043 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-03-28T17:47:58.043 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-28T17:47:58.043 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6 2024-03-28T17:47:58.043 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-03-28T17:47:58.043 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-03-28T17:47:58.043 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-03-28T17:47:58.043 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-03-28T17:47:58.043 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-28T17:47:58.043 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-03-28T17:47:58.043 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-03-28T17:47:58.043 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter 2024-03-28T17:47:58.043 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-28T17:47:58.043 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-28T17:47:58.044 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-28T17:47:58.044 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-28T17:47:58.044 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-28T17:47:58.044 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-28T17:47:58.044 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-28T17:47:58.044 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-28T17:47:58.044 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-28T17:47:58.044 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-28T17:47:58.044 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-28T17:47:58.044 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-28T17:47:58.044 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-28T17:47:58.044 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-28T17:47:58.044 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-28T17:47:58.044 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-28T17:47:58.044 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-28T17:47:58.045 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-28T17:47:58.045 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-28T17:47:58.045 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-28T17:47:58.045 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-28T17:47:58.045 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-28T17:47:58.045 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-28T17:47:58.045 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-28T17:47:58.045 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-03-28T17:47:58.045 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-03-28T17:47:58.045 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-28T17:47:58.045 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key 2024-03-28T17:47:58.045 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-03-28T17:47:58.045 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp 2024-03-28T17:47:58.045 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-03-28T17:47:58.046 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-28T17:47:58.046 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-03-28T17:47:58.046 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-03-28T17:47:58.046 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-03-28T17:47:58.046 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-03-28T17:47:58.046 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-03-28T17:47:58.046 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc 2024-03-28T17:47:58.046 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-03-28T17:47:58.046 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154 2024-03-28T17:47:58.046 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-03-28T17:47:58.046 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls 2024-03-28T17:47:58.046 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-03-28T17:47:58.046 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter 2024-03-28T17:47:58.046 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset 2024-03-28T17:47:58.046 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-03-28T17:47:58.047 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-28T17:47:58.047 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-28T17:47:58.047 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-28T17:47:58.047 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-28T17:47:58.047 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-28T17:47:58.047 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-28T17:47:58.047 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-28T17:47:58.047 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-28T17:47:58.047 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-28T17:47:58.047 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-28T17:47:58.047 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-28T17:47:58.047 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-28T17:47:58.047 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-28T17:47:58.047 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-28T17:47:58.048 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-28T17:47:58.048 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs 2024-03-28T17:47:58.048 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-28T17:47:58.048 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-28T17:47:58.048 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-28T17:47:58.048 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-28T17:47:58.048 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-28T17:47:58.048 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-28T17:47:58.048 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-28T17:47:58.048 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-28T17:47:58.048 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-28T17:47:58.048 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-28T17:47:58.048 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-28T17:47:58.048 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-28T17:47:58.048 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-28T17:47:58.049 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-28T17:47:58.049 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-03-28T17:47:58.049 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-03-28T17:47:58.049 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-28T17:47:58.049 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-28T17:47:58.049 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-28T17:47:58.049 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-28T17:47:58.049 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-28T17:47:58.049 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-28T17:47:58.049 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-28T17:47:58.049 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-28T17:47:58.049 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-28T17:47:58.049 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-28T17:47:58.049 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-28T17:47:58.049 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-28T17:47:58.049 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-28T17:47:58.050 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-03-28T17:47:58.050 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-28T17:47:58.050 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-03-28T17:47:58.050 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-03-28T17:47:58.050 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-28T17:47:58.050 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-28T17:47:58.050 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-03-28T17:47:58.050 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-03-28T17:47:58.050 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-28T17:47:58.050 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-28T17:47:58.050 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-03-28T17:47:58.050 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-03-28T17:47:58.050 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-28T17:47:58.050 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-03-28T17:47:58.050 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-28T17:47:58.051 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-28T17:47:58.051 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-03-28T17:47:58.051 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-03-28T17:47:58.051 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-03-28T17:47:58.051 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-28T17:47:58.051 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-03-28T17:47:58.051 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-28T17:47:58.051 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-03-28T17:47:58.051 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-03-28T17:47:58.051 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-03-28T17:47:58.051 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-03-28T17:47:58.051 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-03-28T17:47:58.051 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-03-28T17:47:58.051 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-03-28T17:47:58.051 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_quota.ko 2024-03-28T17:47:58.051 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-03-28T17:47:58.052 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-03-28T17:47:58.052 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-03-28T17:47:58.052 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-03-28T17:47:58.052 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tunnel.ko 2024-03-28T17:47:58.052 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-03-28T17:47:58.052 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-28T17:47:58.052 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-28T17:47:58.052 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-28T17:47:58.052 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-03-28T17:47:58.052 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-03-28T17:47:58.052 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-03-28T17:47:58.052 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-03-28T17:47:58.052 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-28T17:47:58.052 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-03-28T17:47:58.052 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-03-28T17:47:58.052 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-28T17:47:58.052 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-03-28T17:47:58.053 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-03-28T17:47:58.053 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-28T17:47:58.053 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-03-28T17:47:58.053 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-28T17:47:58.053 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-03-28T17:47:58.053 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-28T17:47:58.053 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-28T17:47:58.053 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-03-28T17:47:58.053 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-03-28T17:47:58.053 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-03-28T17:47:58.053 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-03-28T17:47:58.053 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-03-28T17:47:58.053 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-03-28T17:47:58.053 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-03-28T17:47:58.053 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-03-28T17:47:58.053 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-03-28T17:47:58.053 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-03-28T17:47:58.054 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-03-28T17:47:58.054 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-03-28T17:47:58.054 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-03-28T17:47:58.054 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-03-28T17:47:58.054 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-03-28T17:47:58.054 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-03-28T17:47:58.054 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-03-28T17:47:58.054 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-03-28T17:47:58.054 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-03-28T17:47:58.054 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-03-28T17:47:58.054 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-03-28T17:47:58.054 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-03-28T17:47:58.054 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-03-28T17:47:58.054 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-03-28T17:47:58.054 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-03-28T17:47:58.055 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-03-28T17:47:58.055 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-03-28T17:47:58.055 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-03-28T17:47:58.055 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-03-28T17:47:58.055 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-03-28T17:47:58.055 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-03-28T17:47:58.055 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-03-28T17:47:58.055 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-03-28T17:47:58.055 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-03-28T17:47:58.055 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-03-28T17:47:58.055 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-03-28T17:47:58.055 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-03-28T17:47:58.055 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-03-28T17:47:58.055 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-03-28T17:47:58.055 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-03-28T17:47:58.055 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-03-28T17:47:58.056 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-03-28T17:47:58.056 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-03-28T17:47:58.056 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-03-28T17:47:58.056 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-03-28T17:47:58.056 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-03-28T17:47:58.056 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-03-28T17:47:58.056 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-03-28T17:47:58.056 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-03-28T17:47:58.056 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-03-28T17:47:58.056 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-03-28T17:47:58.056 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink 2024-03-28T17:47:58.056 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-03-28T17:47:58.056 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh 2024-03-28T17:47:58.056 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-03-28T17:47:58.056 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch 2024-03-28T17:47:58.056 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-03-28T17:47:58.057 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-03-28T17:47:58.057 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-03-28T17:47:58.057 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-03-28T17:47:58.057 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet 2024-03-28T17:47:58.057 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-03-28T17:47:58.057 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds 2024-03-28T17:47:58.057 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-03-28T17:47:58.057 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-03-28T17:47:58.057 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-03-28T17:47:58.057 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched 2024-03-28T17:47:58.057 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-03-28T17:47:58.057 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-03-28T17:47:58.057 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-03-28T17:47:58.057 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-03-28T17:47:58.057 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-03-28T17:47:58.058 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-03-28T17:47:58.058 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-03-28T17:47:58.058 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-03-28T17:47:58.058 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-03-28T17:47:58.058 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-03-28T17:47:58.058 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-03-28T17:47:58.058 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-03-28T17:47:58.058 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-03-28T17:47:58.058 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-03-28T17:47:58.058 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-03-28T17:47:58.058 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-03-28T17:47:58.058 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-03-28T17:47:58.058 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-03-28T17:47:58.058 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-03-28T17:47:58.058 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-03-28T17:47:58.059 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-03-28T17:47:58.059 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-03-28T17:47:58.059 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-03-28T17:47:58.059 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-03-28T17:47:58.059 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-03-28T17:47:58.059 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-03-28T17:47:58.059 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-03-28T17:47:58.059 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-03-28T17:47:58.059 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-03-28T17:47:58.059 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-03-28T17:47:58.059 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-03-28T17:47:58.059 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-03-28T17:47:58.059 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-03-28T17:47:58.059 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-03-28T17:47:58.059 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-03-28T17:47:58.060 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-03-28T17:47:58.060 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-03-28T17:47:58.060 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-03-28T17:47:58.060 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-03-28T17:47:58.060 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-03-28T17:47:58.060 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-03-28T17:47:58.060 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-03-28T17:47:58.060 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-03-28T17:47:58.060 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp 2024-03-28T17:47:58.060 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-03-28T17:47:58.060 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-03-28T17:47:58.060 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc 2024-03-28T17:47:58.060 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss 2024-03-28T17:47:58.060 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-28T17:47:58.060 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-28T17:47:58.060 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-03-28T17:47:58.061 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma 2024-03-28T17:47:58.061 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-28T17:47:58.061 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc 2024-03-28T17:47:58.061 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-03-28T17:47:58.061 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-03-28T17:47:58.061 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix 2024-03-28T17:47:58.061 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-03-28T17:47:58.061 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock 2024-03-28T17:47:58.061 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-28T17:47:58.061 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-28T17:47:58.061 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-03-28T17:47:58.061 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-28T17:47:58.061 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-28T17:47:58.061 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm 2024-03-28T17:47:58.061 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-28T17:47:58.062 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security 2024-03-28T17:47:58.062 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys 2024-03-28T17:47:58.062 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys 2024-03-28T17:47:58.062 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-28T17:47:58.062 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys 2024-03-28T17:47:58.062 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys/trusted.ko 2024-03-28T17:47:58.062 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt 2024-03-28T17:47:58.062 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib 2024-03-28T17:47:58.062 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-03-28T17:47:58.062 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-03-28T17:47:58.062 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-03-28T17:47:58.062 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-03-28T17:47:58.062 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.alias.bin 2024-03-28T17:47:58.062 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-03-28T17:47:58.062 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-03-28T17:47:58.063 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-03-28T17:47:58.063 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-03-28T17:47:58.063 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-03-28T17:47:58.063 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-03-28T17:47:58.063 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-03-28T17:47:58.063 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-03-28T17:47:58.063 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-03-28T17:47:58.063 INFO:teuthology.orchestra.run.smithi037.stdout:/lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-28T17:47:58.067 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-03-28T17:47:58.067 DEBUG:teuthology.orchestra.run.smithi037:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-28T17:47:58.089 INFO:teuthology.orchestra.run.smithi037.stdout:postinstall scriptlet (using /bin/sh): 2024-03-28T17:47:58.089 INFO:teuthology.orchestra.run.smithi037.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-28T17:47:58.089 INFO:teuthology.orchestra.run.smithi037.stdout: /usr/bin/kernel-install add 6.8.0-g1df0c7e8a235 /lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-28T17:47:58.090 INFO:teuthology.orchestra.run.smithi037.stdout:fi 2024-03-28T17:47:58.090 INFO:teuthology.orchestra.run.smithi037.stdout:for file in vmlinuz System.map config; do 2024-03-28T17:47:58.090 INFO:teuthology.orchestra.run.smithi037.stdout: if ! cmp --silent "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235"; then 2024-03-28T17:47:58.090 INFO:teuthology.orchestra.run.smithi037.stdout: cp "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235" 2024-03-28T17:47:58.090 INFO:teuthology.orchestra.run.smithi037.stdout: fi 2024-03-28T17:47:58.090 INFO:teuthology.orchestra.run.smithi037.stdout:done 2024-03-28T17:47:58.090 INFO:teuthology.orchestra.run.smithi037.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-28T17:47:58.090 INFO:teuthology.orchestra.run.smithi037.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-28T17:47:58.090 INFO:teuthology.orchestra.run.smithi037.stdout:new-kernel-pkg --remove 6.8.0-g1df0c7e8a235 --rminitrd --initrdfile=/boot/initramfs-6.8.0-g1df0c7e8a235.img 2024-03-28T17:47:58.090 INFO:teuthology.orchestra.run.smithi037.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-28T17:47:58.090 INFO:teuthology.orchestra.run.smithi037.stdout:kernel-install remove 6.8.0-g1df0c7e8a235 2024-03-28T17:47:58.090 INFO:teuthology.orchestra.run.smithi037.stdout:fi 2024-03-28T17:47:58.090 INFO:teuthology.orchestra.run.smithi037.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-28T17:47:58.091 INFO:teuthology.orchestra.run.smithi037.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-28T17:47:58.091 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/update-bootloader --remove 6.8.0-g1df0c7e8a235 2024-03-28T17:47:58.091 INFO:teuthology.orchestra.run.smithi037.stdout:fi 2024-03-28T17:47:58.092 DEBUG:teuthology.orchestra.run.smithi037:> sudo rpm -qi grub2-tools 2024-03-28T17:47:58.170 INFO:teuthology.orchestra.run.smithi037.stdout:Name : grub2-tools 2024-03-28T17:47:58.170 INFO:teuthology.orchestra.run.smithi037.stdout:Epoch : 1 2024-03-28T17:47:58.170 INFO:teuthology.orchestra.run.smithi037.stdout:Version : 2.06 2024-03-28T17:47:58.170 INFO:teuthology.orchestra.run.smithi037.stdout:Release : 68.el9 2024-03-28T17:47:58.170 INFO:teuthology.orchestra.run.smithi037.stdout:Architecture: x86_64 2024-03-28T17:47:58.170 INFO:teuthology.orchestra.run.smithi037.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T17:47:58.170 INFO:teuthology.orchestra.run.smithi037.stdout:Group : Unspecified 2024-03-28T17:47:58.170 INFO:teuthology.orchestra.run.smithi037.stdout:Size : 8264254 2024-03-28T17:47:58.170 INFO:teuthology.orchestra.run.smithi037.stdout:License : GPLv3+ 2024-03-28T17:47:58.170 INFO:teuthology.orchestra.run.smithi037.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T17:47:58.170 INFO:teuthology.orchestra.run.smithi037.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T17:47:58.170 INFO:teuthology.orchestra.run.smithi037.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T17:47:58.170 INFO:teuthology.orchestra.run.smithi037.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T17:47:58.171 INFO:teuthology.orchestra.run.smithi037.stdout:Packager : builder@centos.org 2024-03-28T17:47:58.171 INFO:teuthology.orchestra.run.smithi037.stdout:Vendor : CentOS 2024-03-28T17:47:58.171 INFO:teuthology.orchestra.run.smithi037.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T17:47:58.171 INFO:teuthology.orchestra.run.smithi037.stdout:Summary : Support tools for GRUB. 2024-03-28T17:47:58.171 INFO:teuthology.orchestra.run.smithi037.stdout:Description : 2024-03-28T17:47:58.171 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:47:58.171 INFO:teuthology.orchestra.run.smithi037.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T17:47:58.171 INFO:teuthology.orchestra.run.smithi037.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T17:47:58.171 INFO:teuthology.orchestra.run.smithi037.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T17:47:58.171 INFO:teuthology.orchestra.run.smithi037.stdout:hardware devices. 2024-03-28T17:47:58.171 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:47:58.171 INFO:teuthology.orchestra.run.smithi037.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T17:47:58.172 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T17:47:58.172 INFO:teuthology.task.kernel:Updating grub on smithi037 to boot 6.8.0-g1df0c7e8a235 2024-03-28T17:47:58.172 DEBUG:teuthology.orchestra.run.smithi037:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T17:47:58.812 INFO:teuthology.orchestra.run.smithi037.stderr:Generating grub configuration file ... 2024-03-28T17:48:00.182 INFO:teuthology.orchestra.run.smithi037.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T17:48:00.214 INFO:teuthology.orchestra.run.smithi037.stderr:done 2024-03-28T17:48:00.217 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-03-28T17:48:00.231 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.wFnx7Sqawu 2024-03-28T17:48:00.232 DEBUG:teuthology.orchestra.run.smithi037:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wFnx7Sqawu 2024-03-28T17:48:00.304 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.wFnx7Sqawu 2024-03-28T17:48:00.442 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.wFnx7Sqawu is 6KB 2024-03-28T17:48:00.491 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.wFnx7Sqawu 2024-03-28T17:48:00.505 DEBUG:teuthology.orchestra.run.smithi037:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T17:48:00.574 INFO:teuthology.orchestra.run.smithi037.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T17:48:00.574 INFO:teuthology.orchestra.run.smithi037.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T17:48:00.575 INFO:teuthology.orchestra.run.smithi037.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T17:48:00.575 INFO:teuthology.orchestra.run.smithi037.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235.conf 2024-03-28T17:48:00.576 DEBUG:teuthology.orchestra.run.smithi037:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235 2024-03-28T17:48:00.756 DEBUG:teuthology.orchestra.run.smithi037:> sudo shutdown -r now 2024-03-28T17:48:30.759 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-28T17:48:30.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-03-28T17:48:30.761 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T17:49:30.764 DEBUG:teuthology.orchestra.remote:timed out 2024-03-28T17:49:39.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-03-28T17:49:39.766 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T17:49:40.263 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-03-28T17:49:40.892 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-03-28T17:49:40.893 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-g1df0c7e8a235"... 2024-03-28T17:49:40.893 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-03-28T17:49:40.932 INFO:teuthology.orchestra.run.smithi037.stdout:6.8.0-g1df0c7e8a235 2024-03-28T17:49:40.932 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-03-28T17:49:40.932 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T17:49:40.932 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T17:49:41.935 DEBUG:teuthology.parallel:result is None 2024-03-28T17:49:41.936 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-28T17:49:41.943 INFO:teuthology.task.internal:Creating test directory... 2024-03-28T17:49:41.943 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T17:49:41.951 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T17:49:41.953 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T17:49:41.997 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-28T17:49:42.004 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-28T17:49:42.013 INFO:teuthology.task.internal:Creating archive directory... 2024-03-28T17:49:42.013 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T17:49:42.018 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T17:49:42.020 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T17:49:42.076 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-28T17:49:42.087 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-28T17:49:42.087 DEBUG:teuthology.orchestra.run.smithi059:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T17:49:42.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T17:49:42.104 DEBUG:teuthology.orchestra.run.smithi170:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T17:49:42.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T17:49:42.121 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T17:49:42.127 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T17:49:42.147 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T17:49:42.197 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T17:49:42.219 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T17:49:42.221 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T17:49:42.239 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T17:49:42.295 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T17:49:42.361 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T17:49:42.366 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-28T17:49:42.375 INFO:teuthology.task.internal:Configuring sudo... 2024-03-28T17:49:42.375 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T17:49:42.380 DEBUG:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T17:49:42.382 DEBUG:teuthology.orchestra.run.smithi170:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T17:49:42.455 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-28T17:49:42.469 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-28T17:49:42.469 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T17:49:42.473 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T17:49:42.475 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T17:49:42.507 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T17:49:42.626 DEBUG:teuthology.orchestra.run.smithi037:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T17:49:42.703 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T17:49:42.770 DEBUG:teuthology.orchestra.run.smithi037:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T17:49:42.857 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-28T17:49:42.857 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T17:49:42.924 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T17:49:42.967 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T17:49:43.026 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T17:49:43.104 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T17:49:43.160 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-28T17:49:43.160 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T17:49:43.233 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T17:49:43.278 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T17:49:43.334 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T17:49:43.418 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T17:49:43.464 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T17:49:43.464 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T17:49:43.534 DEBUG:teuthology.orchestra.run.smithi037:> sudo service rsyslog restart 2024-03-28T17:49:43.538 DEBUG:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2024-03-28T17:49:43.540 DEBUG:teuthology.orchestra.run.smithi170:> sudo service rsyslog restart 2024-03-28T17:49:43.580 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T17:49:43.616 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T17:49:43.630 INFO:teuthology.orchestra.run.smithi037.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T17:49:44.110 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-28T17:49:44.120 INFO:teuthology.task.internal:Starting timer... 2024-03-28T17:49:44.120 INFO:teuthology.run_tasks:Running task pcp... 2024-03-28T17:49:44.135 INFO:teuthology.run_tasks:Running task selinux... 2024-03-28T17:49:44.150 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-28T17:49:44.151 DEBUG:teuthology.orchestra.run.smithi037:> sudo service auditd rotate 2024-03-28T17:49:44.281 INFO:teuthology.orchestra.run.smithi037.stdout:Rotating logs: 2024-03-28T17:49:44.285 DEBUG:teuthology.orchestra.run.smithi059:> sudo service auditd rotate 2024-03-28T17:49:44.355 INFO:teuthology.orchestra.run.smithi059.stdout:Rotating logs: 2024-03-28T17:49:44.358 DEBUG:teuthology.orchestra.run.smithi170:> sudo service auditd rotate 2024-03-28T17:49:44.423 INFO:teuthology.orchestra.run.smithi170.stdout:Rotating logs: 2024-03-28T17:49:44.425 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-28T17:49:44.426 DEBUG:teuthology.orchestra.run.smithi037:> /usr/sbin/getenforce 2024-03-28T17:49:44.470 INFO:teuthology.orchestra.run.smithi037.stdout:Permissive 2024-03-28T17:49:44.470 DEBUG:teuthology.orchestra.run.smithi059:> /usr/sbin/getenforce 2024-03-28T17:49:44.494 INFO:teuthology.orchestra.run.smithi059.stdout:Permissive 2024-03-28T17:49:44.494 DEBUG:teuthology.orchestra.run.smithi170:> /usr/sbin/getenforce 2024-03-28T17:49:44.540 INFO:teuthology.orchestra.run.smithi170.stdout:Permissive 2024-03-28T17:49:44.541 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi037.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi059.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi170.front.sepia.ceph.com': 'permissive'} 2024-03-28T17:49:44.541 DEBUG:teuthology.orchestra.run.smithi037:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T17:49:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T17:49:44.617 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T17:49:44.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T17:49:44.649 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T17:49:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T17:49:44.679 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-28T17:49:44.679 DEBUG:teuthology.orchestra.run.smithi037:> sudo /usr/sbin/setenforce permissive 2024-03-28T17:49:44.786 DEBUG:teuthology.orchestra.run.smithi059:> sudo /usr/sbin/setenforce permissive 2024-03-28T17:49:44.830 DEBUG:teuthology.orchestra.run.smithi170:> sudo /usr/sbin/setenforce permissive 2024-03-28T17:49:44.873 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-28T17:49:44.883 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-28T17:49:44.899 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-28T17:49:44.900 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi037.front.sepia.ceph.com,smithi059.front.sepia.ceph.com,smithi170.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-28T17:55:50.865 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi037.front.sepia.ceph.com'), Remote(name='ubuntu@smithi059.front.sepia.ceph.com'), Remote(name='ubuntu@smithi170.front.sepia.ceph.com')] 2024-03-28T17:55:50.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-03-28T17:55:50.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T17:55:51.024 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-03-28T17:55:51.184 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-03-28T17:55:51.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-28T17:55:51.185 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T17:55:51.287 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-03-28T17:55:51.358 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-28T17:55:51.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-28T17:55:51.359 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T17:55:51.446 DEBUG:teuthology.orchestra.run.smithi170:> true 2024-03-28T17:55:51.518 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-28T17:55:51.519 INFO:teuthology.run_tasks:Running task clock... 2024-03-28T17:55:51.530 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-28T17:55:51.530 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T17:55:51.530 DEBUG:teuthology.orchestra.run.smithi037:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T17:55:51.538 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T17:55:51.538 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T17:55:51.540 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T17:55:51.540 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T17:55:51.576 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T17:55:51.591 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T17:55:51.597 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T17:55:51.608 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T17:55:51.635 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: ntpd: command not found 2024-03-28T17:55:51.639 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: ntpd: command not found 2024-03-28T17:55:51.641 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T17:55:51.653 INFO:teuthology.orchestra.run.smithi170.stdout:506 Cannot talk to daemon 2024-03-28T17:55:51.654 INFO:teuthology.orchestra.run.smithi059.stdout:506 Cannot talk to daemon 2024-03-28T17:55:51.669 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T17:55:51.675 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T17:55:51.686 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T17:55:51.695 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T17:55:51.698 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T17:55:51.730 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-03-28T17:55:51.734 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T17:55:51.734 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-03-28T17:55:51.734 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:55:51.734 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:55:51.734 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:55:51.734 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:55:51.743 INFO:teuthology.orchestra.run.smithi059.stderr:bash: line 1: ntpq: command not found 2024-03-28T17:55:51.747 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T17:55:51.747 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2024-03-28T17:55:51.747 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:55:51.747 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:55:51.747 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:55:51.747 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:55:51.801 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: ntpd: command not found 2024-03-28T17:55:51.847 INFO:teuthology.orchestra.run.smithi037.stdout:506 Cannot talk to daemon 2024-03-28T17:55:51.903 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T17:55:51.958 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T17:55:52.093 INFO:teuthology.orchestra.run.smithi037.stderr:bash: line 1: ntpq: command not found 2024-03-28T17:55:52.102 INFO:teuthology.orchestra.run.smithi037.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T17:55:52.102 INFO:teuthology.orchestra.run.smithi037.stdout:=============================================================================== 2024-03-28T17:55:52.102 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:55:52.102 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:55:52.102 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:55:52.103 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:55:52.104 INFO:teuthology.run_tasks:Running task install... 2024-03-28T17:55:52.112 DEBUG:teuthology.task.install:project ceph 2024-03-28T17:55:52.112 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}} 2024-03-28T17:55:52.112 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'} 2024-03-28T17:55:52.112 INFO:teuthology.task.install:Using flavor: default 2024-03-28T17:55:52.119 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-28T17:55:52.120 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-28T17:55:52.120 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T17:55:52.120 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:55:52.122 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T17:55:52.122 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:55:52.123 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T17:55:52.123 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:55:52.266 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T17:55:52.266 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T17:55:52.269 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T17:55:52.270 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T17:55:52.337 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T17:55:52.337 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T17:55:52.437 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T17:55:52.437 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-28T17:55:52.437 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T17:55:52.444 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T17:55:52.444 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T17:55:52.444 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T17:55:52.475 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T17:55:52.475 DEBUG:teuthology.orchestra.run.smithi170:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T17:55:52.506 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T17:55:52.507 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-28T17:55:52.507 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T17:55:52.539 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T17:55:52.540 DEBUG:teuthology.orchestra.run.smithi059:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T17:55:52.542 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T17:55:52.542 DEBUG:teuthology.orchestra.run.smithi037:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T17:55:52.545 DEBUG:teuthology.orchestra.run.smithi170:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T17:55:52.612 DEBUG:teuthology.orchestra.run.smithi059:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T17:55:52.627 DEBUG:teuthology.orchestra.run.smithi170:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T17:55:52.680 DEBUG:teuthology.orchestra.run.smithi037:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T17:55:52.694 INFO:teuthology.orchestra.run.smithi170.stdout:check_obsoletes = 1 2024-03-28T17:55:52.696 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum clean all 2024-03-28T17:55:52.698 DEBUG:teuthology.orchestra.run.smithi059:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T17:55:52.728 INFO:teuthology.orchestra.run.smithi059.stdout:check_obsoletes = 1 2024-03-28T17:55:52.729 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum clean all 2024-03-28T17:55:52.815 DEBUG:teuthology.orchestra.run.smithi037:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T17:55:52.914 INFO:teuthology.orchestra.run.smithi037.stdout:check_obsoletes = 1 2024-03-28T17:55:52.919 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum clean all 2024-03-28T17:55:52.930 INFO:teuthology.orchestra.run.smithi170.stdout:74 files removed 2024-03-28T17:55:52.958 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T17:55:52.976 INFO:teuthology.orchestra.run.smithi059.stdout:74 files removed 2024-03-28T17:55:53.004 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T17:55:53.279 INFO:teuthology.orchestra.run.smithi037.stdout:74 files removed 2024-03-28T17:55:53.319 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T17:55:53.646 INFO:teuthology.orchestra.run.smithi170.stdout:ceph packages for x86_64 208 kB/s | 83 kB 00:00 2024-03-28T17:55:53.781 INFO:teuthology.orchestra.run.smithi059.stdout:ceph packages for x86_64 177 kB/s | 83 kB 00:00 2024-03-28T17:55:54.080 INFO:teuthology.orchestra.run.smithi170.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-28T17:55:54.198 INFO:teuthology.orchestra.run.smithi059.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-28T17:55:54.271 INFO:teuthology.orchestra.run.smithi037.stdout:ceph packages for x86_64 191 kB/s | 83 kB 00:00 2024-03-28T17:55:54.430 INFO:teuthology.orchestra.run.smithi170.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-03-28T17:55:54.540 INFO:teuthology.orchestra.run.smithi059.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-28T17:55:54.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph noarch packages 24 kB/s | 10 kB 00:00 2024-03-28T17:55:54.807 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - AppStream 19 MB/s | 6.7 MB 00:00 2024-03-28T17:55:55.001 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-28T17:55:55.431 INFO:teuthology.orchestra.run.smithi037.stdout:ceph source packages 3.3 kB/s | 1.9 kB 00:00 2024-03-28T17:55:56.140 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-28T17:55:56.266 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2024-03-28T17:55:56.443 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-28T17:55:57.115 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-28T17:55:57.994 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-28T17:55:58.975 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-28T17:56:00.064 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.0 MB 00:01 2024-03-28T17:56:01.009 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - BaseOS 1.9 MB/s | 8.0 MB 00:04 2024-03-28T17:56:02.082 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-03-28T17:56:03.060 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-28T17:56:03.084 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 9.9 MB/s | 6.1 MB 00:00 2024-03-28T17:56:04.551 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-03-28T17:56:06.320 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-28T17:56:06.466 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-03-28T17:56:07.427 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-28T17:56:08.400 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 16 kB 00:00 2024-03-28T17:56:09.203 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-03-28T17:56:10.152 INFO:teuthology.orchestra.run.smithi037.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-28T17:56:11.411 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-28T17:56:12.032 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 7.6 MB/s | 21 MB 00:02 2024-03-28T17:56:13.773 INFO:teuthology.orchestra.run.smithi170.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T17:56:13.774 INFO:teuthology.orchestra.run.smithi170.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T17:56:13.786 INFO:teuthology.orchestra.run.smithi170.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T17:56:13.787 INFO:teuthology.orchestra.run.smithi170.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T17:56:13.788 INFO:teuthology.orchestra.run.smithi170.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T17:56:13.789 INFO:teuthology.orchestra.run.smithi170.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T17:56:13.790 INFO:teuthology.orchestra.run.smithi170.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T17:56:13.793 INFO:teuthology.orchestra.run.smithi170.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T17:56:13.794 INFO:teuthology.orchestra.run.smithi170.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T17:56:13.876 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-28T17:56:13.888 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-03-28T17:56:13.888 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-28T17:56:13.888 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-03-28T17:56:13.888 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-28T17:56:13.888 INFO:teuthology.orchestra.run.smithi170.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T17:56:13.888 INFO:teuthology.orchestra.run.smithi170.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T17:56:13.888 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T17:56:13.888 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T17:56:13.888 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T17:56:13.889 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T17:56:13.889 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T17:56:13.889 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T17:56:13.889 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T17:56:13.889 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T17:56:13.890 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T17:56:13.890 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T17:56:13.890 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T17:56:13.890 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T17:56:13.890 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T17:56:13.890 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T17:56:13.890 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T17:56:13.891 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T17:56:13.891 INFO:teuthology.orchestra.run.smithi170.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T17:56:13.891 INFO:teuthology.orchestra.run.smithi170.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T17:56:13.891 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T17:56:13.891 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T17:56:13.891 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T17:56:13.891 INFO:teuthology.orchestra.run.smithi170.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T17:56:13.891 INFO:teuthology.orchestra.run.smithi170.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T17:56:13.891 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T17:56:13.892 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T17:56:13.892 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T17:56:13.892 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T17:56:13.892 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T17:56:13.892 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T17:56:13.892 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T17:56:13.892 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T17:56:13.892 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T17:56:13.892 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T17:56:13.892 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T17:56:13.892 INFO:teuthology.orchestra.run.smithi170.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T17:56:13.893 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrading: 2024-03-28T17:56:13.893 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T17:56:13.893 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T17:56:13.893 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T17:56:13.893 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T17:56:13.893 INFO:teuthology.orchestra.run.smithi170.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T17:56:13.893 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T17:56:13.893 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T17:56:13.893 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T17:56:13.893 INFO:teuthology.orchestra.run.smithi170.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T17:56:13.893 INFO:teuthology.orchestra.run.smithi170.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T17:56:13.893 INFO:teuthology.orchestra.run.smithi170.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T17:56:13.894 INFO:teuthology.orchestra.run.smithi170.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T17:56:13.894 INFO:teuthology.orchestra.run.smithi170.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T17:56:13.894 INFO:teuthology.orchestra.run.smithi170.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T17:56:13.894 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T17:56:13.894 INFO:teuthology.orchestra.run.smithi170.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T17:56:13.894 INFO:teuthology.orchestra.run.smithi170.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T17:56:13.894 INFO:teuthology.orchestra.run.smithi170.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T17:56:13.894 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T17:56:13.894 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T17:56:13.894 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T17:56:13.894 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T17:56:13.894 INFO:teuthology.orchestra.run.smithi170.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T17:56:13.895 INFO:teuthology.orchestra.run.smithi170.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T17:56:13.895 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-28T17:56:13.895 INFO:teuthology.orchestra.run.smithi170.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T17:56:13.895 INFO:teuthology.orchestra.run.smithi170.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T17:56:13.895 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T17:56:13.895 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T17:56:13.895 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T17:56:13.895 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T17:56:13.895 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T17:56:13.895 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T17:56:13.895 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T17:56:13.895 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T17:56:13.895 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T17:56:13.895 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T17:56:13.895 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T17:56:13.895 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T17:56:13.896 INFO:teuthology.orchestra.run.smithi170.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T17:56:13.896 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T17:56:13.896 INFO:teuthology.orchestra.run.smithi170.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T17:56:13.896 INFO:teuthology.orchestra.run.smithi170.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T17:56:13.896 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T17:56:13.896 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T17:56:13.896 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T17:56:13.896 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T17:56:13.896 INFO:teuthology.orchestra.run.smithi170.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T17:56:13.896 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T17:56:13.896 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T17:56:13.896 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T17:56:13.896 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T17:56:13.897 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T17:56:13.897 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T17:56:13.897 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T17:56:13.897 INFO:teuthology.orchestra.run.smithi170.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T17:56:13.897 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T17:56:13.897 INFO:teuthology.orchestra.run.smithi170.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T17:56:13.897 INFO:teuthology.orchestra.run.smithi170.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T17:56:13.897 INFO:teuthology.orchestra.run.smithi170.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T17:56:13.897 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T17:56:13.897 INFO:teuthology.orchestra.run.smithi170.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T17:56:13.897 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T17:56:13.897 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T17:56:13.897 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T17:56:13.897 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T17:56:13.897 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T17:56:13.897 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T17:56:13.898 INFO:teuthology.orchestra.run.smithi170.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T17:56:13.898 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T17:56:13.898 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T17:56:13.898 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T17:56:13.898 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T17:56:13.898 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T17:56:13.898 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T17:56:13.898 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T17:56:13.898 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T17:56:13.898 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T17:56:13.898 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T17:56:13.898 INFO:teuthology.orchestra.run.smithi170.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T17:56:13.898 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T17:56:13.898 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T17:56:13.899 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T17:56:13.899 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T17:56:13.899 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T17:56:13.899 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T17:56:13.899 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T17:56:13.899 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T17:56:13.899 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T17:56:13.899 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T17:56:13.899 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T17:56:13.899 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T17:56:13.899 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T17:56:13.899 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T17:56:13.899 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T17:56:13.899 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T17:56:13.899 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T17:56:13.899 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T17:56:13.900 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T17:56:13.900 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T17:56:13.900 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T17:56:13.900 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T17:56:13.900 INFO:teuthology.orchestra.run.smithi170.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T17:56:13.900 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T17:56:13.900 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T17:56:13.900 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T17:56:13.900 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T17:56:13.900 INFO:teuthology.orchestra.run.smithi170.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T17:56:13.900 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T17:56:13.900 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T17:56:13.900 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T17:56:13.900 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T17:56:13.900 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T17:56:13.900 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T17:56:13.901 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T17:56:13.901 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T17:56:13.901 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T17:56:13.901 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T17:56:13.901 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T17:56:13.901 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T17:56:13.901 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T17:56:13.901 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T17:56:13.901 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T17:56:13.901 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T17:56:13.901 INFO:teuthology.orchestra.run.smithi170.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T17:56:13.901 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T17:56:13.901 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T17:56:13.901 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T17:56:13.901 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T17:56:13.901 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T17:56:13.902 INFO:teuthology.orchestra.run.smithi170.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T17:56:13.902 INFO:teuthology.orchestra.run.smithi170.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T17:56:13.902 INFO:teuthology.orchestra.run.smithi170.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T17:56:13.902 INFO:teuthology.orchestra.run.smithi170.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T17:56:13.902 INFO:teuthology.orchestra.run.smithi170.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T17:56:13.902 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T17:56:13.902 INFO:teuthology.orchestra.run.smithi170.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T17:56:13.902 INFO:teuthology.orchestra.run.smithi170.stdout:Installing weak dependencies: 2024-03-28T17:56:13.902 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T17:56:13.902 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T17:56:13.902 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-28T17:56:13.902 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-03-28T17:56:13.902 INFO:teuthology.orchestra.run.smithi170.stdout:Install 150 Packages 2024-03-28T17:56:13.903 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrade 24 Packages 2024-03-28T17:56:13.903 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T17:56:13.903 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 225 M 2024-03-28T17:56:13.903 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-28T17:56:14.579 INFO:teuthology.orchestra.run.smithi170.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 30 kB/s | 6.6 kB 00:00 2024-03-28T17:56:15.254 INFO:teuthology.orchestra.run.smithi170.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.3 MB/s | 890 kB 00:00 2024-03-28T17:56:15.387 INFO:teuthology.orchestra.run.smithi170.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.1 MB/s | 145 kB 00:00 2024-03-28T17:56:15.979 INFO:teuthology.orchestra.run.smithi170.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 3.9 MB/s | 2.3 MB 00:00 2024-03-28T17:56:16.185 INFO:teuthology.orchestra.run.smithi037.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-03-28T17:56:16.254 INFO:teuthology.orchestra.run.smithi170.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 5.4 MB/s | 1.5 MB 00:00 2024-03-28T17:56:16.363 INFO:teuthology.orchestra.run.smithi170.stdout:(6/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 2.7 MB/s | 5.4 MB 00:02 2024-03-28T17:56:16.913 INFO:teuthology.orchestra.run.smithi170.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 7.1 MB/s | 4.7 MB 00:00 2024-03-28T17:56:17.204 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-28T17:56:18.146 INFO:teuthology.orchestra.run.smithi170.stdout:(8/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 8.0 MB/s | 9.9 MB 00:01 2024-03-28T17:56:18.314 INFO:teuthology.orchestra.run.smithi170.stdout:(9/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x86 8.6 MB/s | 17 MB 00:01 2024-03-28T17:56:18.339 INFO:teuthology.orchestra.run.smithi170.stdout:(10/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el 131 kB/s | 25 kB 00:00 2024-03-28T17:56:18.406 INFO:teuthology.orchestra.run.smithi170.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 3.2 MB/s | 220 kB 00:00 2024-03-28T17:56:18.481 INFO:teuthology.orchestra.run.smithi170.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 360 kB/s | 27 kB 00:00 2024-03-28T17:56:18.556 INFO:teuthology.orchestra.run.smithi170.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 418 kB/s | 31 kB 00:00 2024-03-28T17:56:18.715 INFO:teuthology.orchestra.run.smithi170.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 4.6 MB/s | 746 kB 00:00 2024-03-28T17:56:18.807 INFO:teuthology.orchestra.run.smithi170.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.7 MB/s | 163 kB 00:00 2024-03-28T17:56:18.882 INFO:teuthology.orchestra.run.smithi037.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T17:56:18.883 INFO:teuthology.orchestra.run.smithi037.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T17:56:18.890 INFO:teuthology.orchestra.run.smithi170.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.5 MB/s | 127 kB 00:00 2024-03-28T17:56:18.903 INFO:teuthology.orchestra.run.smithi037.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T17:56:18.905 INFO:teuthology.orchestra.run.smithi037.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T17:56:18.906 INFO:teuthology.orchestra.run.smithi037.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T17:56:18.908 INFO:teuthology.orchestra.run.smithi037.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T17:56:18.910 INFO:teuthology.orchestra.run.smithi037.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T17:56:18.913 INFO:teuthology.orchestra.run.smithi037.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T17:56:18.915 INFO:teuthology.orchestra.run.smithi037.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T17:56:19.009 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-28T17:56:19.040 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================================ 2024-03-28T17:56:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-28T17:56:19.040 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================================ 2024-03-28T17:56:19.040 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-28T17:56:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T17:56:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T17:56:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T17:56:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T17:56:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T17:56:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T17:56:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T17:56:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T17:56:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T17:56:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T17:56:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T17:56:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T17:56:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T17:56:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T17:56:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T17:56:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T17:56:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T17:56:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T17:56:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T17:56:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T17:56:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T17:56:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T17:56:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T17:56:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T17:56:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T17:56:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T17:56:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T17:56:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T17:56:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T17:56:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T17:56:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T17:56:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T17:56:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T17:56:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T17:56:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T17:56:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T17:56:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T17:56:19.042 INFO:teuthology.orchestra.run.smithi037.stdout:Upgrading: 2024-03-28T17:56:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T17:56:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T17:56:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T17:56:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T17:56:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T17:56:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T17:56:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T17:56:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T17:56:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T17:56:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T17:56:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T17:56:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T17:56:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T17:56:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T17:56:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T17:56:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T17:56:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T17:56:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T17:56:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T17:56:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T17:56:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T17:56:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T17:56:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T17:56:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T17:56:19.044 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-28T17:56:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T17:56:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T17:56:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T17:56:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T17:56:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T17:56:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T17:56:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T17:56:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T17:56:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T17:56:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T17:56:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T17:56:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T17:56:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T17:56:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T17:56:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T17:56:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T17:56:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T17:56:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T17:56:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T17:56:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T17:56:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T17:56:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T17:56:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T17:56:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T17:56:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T17:56:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T17:56:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T17:56:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T17:56:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T17:56:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T17:56:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T17:56:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T17:56:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T17:56:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T17:56:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T17:56:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T17:56:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T17:56:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T17:56:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T17:56:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T17:56:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T17:56:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T17:56:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T17:56:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T17:56:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T17:56:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T17:56:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T17:56:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T17:56:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T17:56:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T17:56:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T17:56:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T17:56:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T17:56:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T17:56:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T17:56:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T17:56:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T17:56:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T17:56:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T17:56:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T17:56:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T17:56:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T17:56:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T17:56:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T17:56:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T17:56:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T17:56:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T17:56:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T17:56:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T17:56:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T17:56:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T17:56:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T17:56:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T17:56:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T17:56:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T17:56:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T17:56:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T17:56:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T17:56:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T17:56:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T17:56:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T17:56:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T17:56:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T17:56:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T17:56:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T17:56:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T17:56:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T17:56:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T17:56:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T17:56:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T17:56:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T17:56:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T17:56:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T17:56:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T17:56:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T17:56:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T17:56:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T17:56:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T17:56:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T17:56:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T17:56:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T17:56:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T17:56:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T17:56:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T17:56:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T17:56:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T17:56:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T17:56:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T17:56:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T17:56:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T17:56:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T17:56:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T17:56:19.051 INFO:teuthology.orchestra.run.smithi037.stdout:Installing weak dependencies: 2024-03-28T17:56:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T17:56:19.052 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:56:19.052 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-28T17:56:19.052 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================================ 2024-03-28T17:56:19.052 INFO:teuthology.orchestra.run.smithi037.stdout:Install 150 Packages 2024-03-28T17:56:19.052 INFO:teuthology.orchestra.run.smithi037.stdout:Upgrade 24 Packages 2024-03-28T17:56:19.052 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:56:19.052 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 225 M 2024-03-28T17:56:19.052 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-28T17:56:19.099 INFO:teuthology.orchestra.run.smithi170.stdout:(17/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 4.3 MB/s | 20 MB 00:04 2024-03-28T17:56:19.132 INFO:teuthology.orchestra.run.smithi170.stdout:(18/174): libradosstriper1-19.0.0-2145.g3be9a5c 2.0 MB/s | 501 kB 00:00 2024-03-28T17:56:19.216 INFO:teuthology.orchestra.run.smithi170.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 540 kB/s | 45 kB 00:00 2024-03-28T17:56:19.300 INFO:teuthology.orchestra.run.smithi170.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.5 MB/s | 131 kB 00:00 2024-03-28T17:56:19.392 INFO:teuthology.orchestra.run.smithi170.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.7 MB/s | 162 kB 00:00 2024-03-28T17:56:19.483 INFO:teuthology.orchestra.run.smithi170.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 3.4 MB/s | 321 kB 00:00 2024-03-28T17:56:19.584 INFO:teuthology.orchestra.run.smithi170.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 2.9 MB/s | 298 kB 00:00 2024-03-28T17:56:19.668 INFO:teuthology.orchestra.run.smithi170.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.2 MB/s | 99 kB 00:00 2024-03-28T17:56:19.693 INFO:teuthology.orchestra.run.smithi059.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T17:56:19.694 INFO:teuthology.orchestra.run.smithi059.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T17:56:19.708 INFO:teuthology.orchestra.run.smithi059.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T17:56:19.709 INFO:teuthology.orchestra.run.smithi059.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T17:56:19.710 INFO:teuthology.orchestra.run.smithi059.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T17:56:19.712 INFO:teuthology.orchestra.run.smithi059.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T17:56:19.714 INFO:teuthology.orchestra.run.smithi059.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T17:56:19.717 INFO:teuthology.orchestra.run.smithi059.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T17:56:19.718 INFO:teuthology.orchestra.run.smithi059.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T17:56:19.720 INFO:teuthology.orchestra.run.smithi037.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-03-28T17:56:19.759 INFO:teuthology.orchestra.run.smithi170.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 933 kB/s | 85 kB 00:00 2024-03-28T17:56:19.803 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-28T17:56:19.815 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================================ 2024-03-28T17:56:19.815 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-28T17:56:19.815 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================================ 2024-03-28T17:56:19.815 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-28T17:56:19.815 INFO:teuthology.orchestra.run.smithi059.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T17:56:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T17:56:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T17:56:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T17:56:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T17:56:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T17:56:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T17:56:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T17:56:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T17:56:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T17:56:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T17:56:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T17:56:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T17:56:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T17:56:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T17:56:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T17:56:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T17:56:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T17:56:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T17:56:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T17:56:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T17:56:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T17:56:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T17:56:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T17:56:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T17:56:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T17:56:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T17:56:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T17:56:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T17:56:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T17:56:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T17:56:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T17:56:19.818 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T17:56:19.818 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T17:56:19.818 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T17:56:19.818 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T17:56:19.818 INFO:teuthology.orchestra.run.smithi059.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T17:56:19.818 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrading: 2024-03-28T17:56:19.818 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T17:56:19.818 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T17:56:19.818 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T17:56:19.818 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T17:56:19.818 INFO:teuthology.orchestra.run.smithi059.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T17:56:19.818 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T17:56:19.818 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T17:56:19.818 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T17:56:19.819 INFO:teuthology.orchestra.run.smithi059.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T17:56:19.819 INFO:teuthology.orchestra.run.smithi059.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T17:56:19.819 INFO:teuthology.orchestra.run.smithi059.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T17:56:19.819 INFO:teuthology.orchestra.run.smithi059.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T17:56:19.819 INFO:teuthology.orchestra.run.smithi059.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T17:56:19.819 INFO:teuthology.orchestra.run.smithi059.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T17:56:19.819 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T17:56:19.819 INFO:teuthology.orchestra.run.smithi059.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T17:56:19.819 INFO:teuthology.orchestra.run.smithi059.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T17:56:19.819 INFO:teuthology.orchestra.run.smithi059.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T17:56:19.819 INFO:teuthology.orchestra.run.smithi059.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T17:56:19.819 INFO:teuthology.orchestra.run.smithi059.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T17:56:19.819 INFO:teuthology.orchestra.run.smithi059.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T17:56:19.819 INFO:teuthology.orchestra.run.smithi059.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T17:56:19.819 INFO:teuthology.orchestra.run.smithi059.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T17:56:19.820 INFO:teuthology.orchestra.run.smithi059.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T17:56:19.820 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-28T17:56:19.820 INFO:teuthology.orchestra.run.smithi059.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T17:56:19.820 INFO:teuthology.orchestra.run.smithi059.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T17:56:19.820 INFO:teuthology.orchestra.run.smithi059.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T17:56:19.820 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T17:56:19.820 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T17:56:19.820 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T17:56:19.820 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T17:56:19.820 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T17:56:19.820 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T17:56:19.820 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T17:56:19.820 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T17:56:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T17:56:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T17:56:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T17:56:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T17:56:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T17:56:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T17:56:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T17:56:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T17:56:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T17:56:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T17:56:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T17:56:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T17:56:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T17:56:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T17:56:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T17:56:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T17:56:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T17:56:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T17:56:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T17:56:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T17:56:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T17:56:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T17:56:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T17:56:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T17:56:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T17:56:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T17:56:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T17:56:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T17:56:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T17:56:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T17:56:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T17:56:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T17:56:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T17:56:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T17:56:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T17:56:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T17:56:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T17:56:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T17:56:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T17:56:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T17:56:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T17:56:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T17:56:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T17:56:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T17:56:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T17:56:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T17:56:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T17:56:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T17:56:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T17:56:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T17:56:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T17:56:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T17:56:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T17:56:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T17:56:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T17:56:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T17:56:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T17:56:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T17:56:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T17:56:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T17:56:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T17:56:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T17:56:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T17:56:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T17:56:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T17:56:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T17:56:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T17:56:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T17:56:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T17:56:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T17:56:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T17:56:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T17:56:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T17:56:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T17:56:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T17:56:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T17:56:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T17:56:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T17:56:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T17:56:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T17:56:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T17:56:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T17:56:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T17:56:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T17:56:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T17:56:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T17:56:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T17:56:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T17:56:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T17:56:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T17:56:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T17:56:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T17:56:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T17:56:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T17:56:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T17:56:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T17:56:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T17:56:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T17:56:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T17:56:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T17:56:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T17:56:19.828 INFO:teuthology.orchestra.run.smithi059.stdout:Installing weak dependencies: 2024-03-28T17:56:19.828 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T17:56:19.828 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-28T17:56:19.828 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-28T17:56:19.828 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================================ 2024-03-28T17:56:19.828 INFO:teuthology.orchestra.run.smithi059.stdout:Install 150 Packages 2024-03-28T17:56:19.828 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrade 24 Packages 2024-03-28T17:56:19.828 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-28T17:56:19.829 INFO:teuthology.orchestra.run.smithi170.stdout:(26/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 7.1 MB/s | 5.1 MB 00:00 2024-03-28T17:56:19.829 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 225 M 2024-03-28T17:56:19.829 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-28T17:56:19.935 INFO:teuthology.orchestra.run.smithi170.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 1.5 MB/s | 170 kB 00:00 2024-03-28T17:56:20.019 INFO:teuthology.orchestra.run.smithi170.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 407 kB/s | 34 kB 00:00 2024-03-28T17:56:20.127 INFO:teuthology.orchestra.run.smithi170.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 1.3 MB/s | 139 kB 00:00 2024-03-28T17:56:20.186 INFO:teuthology.orchestra.run.smithi170.stdout:(30/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 7.2 MB/s | 3.1 MB 00:00 2024-03-28T17:56:20.470 INFO:teuthology.orchestra.run.smithi037.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.2 MB/s | 890 kB 00:00 2024-03-28T17:56:20.572 INFO:teuthology.orchestra.run.smithi059.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 17 kB/s | 6.6 kB 00:00 2024-03-28T17:56:20.586 INFO:teuthology.orchestra.run.smithi170.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 3.9 MB/s | 1.8 MB 00:00 2024-03-28T17:56:20.629 INFO:teuthology.orchestra.run.smithi037.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 923 kB/s | 145 kB 00:00 2024-03-28T17:56:20.695 INFO:teuthology.orchestra.run.smithi170.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 2.2 MB/s | 248 kB 00:00 2024-03-28T17:56:20.812 INFO:teuthology.orchestra.run.smithi170.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 422 kB/s | 49 kB 00:00 2024-03-28T17:56:20.912 INFO:teuthology.orchestra.run.smithi170.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 165 kB/s | 16 kB 00:00 2024-03-28T17:56:21.029 INFO:teuthology.orchestra.run.smithi170.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 2.2 MB/s | 264 kB 00:00 2024-03-28T17:56:21.229 INFO:teuthology.orchestra.run.smithi170.stdout:(36/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 3.7 MB/s | 748 kB 00:00 2024-03-28T17:56:21.296 INFO:teuthology.orchestra.run.smithi170.stdout:(37/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 273 kB/s | 18 kB 00:00 2024-03-28T17:56:21.430 INFO:teuthology.orchestra.run.smithi170.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.8 MB/s | 662 kB 00:00 2024-03-28T17:56:21.480 INFO:teuthology.orchestra.run.smithi170.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 18 MB/s | 920 kB 00:00 2024-03-28T17:56:21.513 INFO:teuthology.orchestra.run.smithi170.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-28T17:56:21.556 INFO:teuthology.orchestra.run.smithi170.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-28T17:56:21.639 INFO:teuthology.orchestra.run.smithi170.stdout:(42/174): ceph-mgr-diskprediction-local-19.0.0- 5.1 MB/s | 7.4 MB 00:01 2024-03-28T17:56:21.664 INFO:teuthology.orchestra.run.smithi170.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 291 kB/s | 32 kB 00:00 2024-03-28T17:56:21.740 INFO:teuthology.orchestra.run.smithi170.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-28T17:56:21.765 INFO:teuthology.orchestra.run.smithi170.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 175 kB/s | 17 kB 00:00 2024-03-28T17:56:21.807 INFO:teuthology.orchestra.run.smithi170.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 150 kB/s | 10 kB 00:00 2024-03-28T17:56:21.832 INFO:teuthology.orchestra.run.smithi170.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 159 kB 00:00 2024-03-28T17:56:21.847 INFO:teuthology.orchestra.run.smithi059.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 698 kB/s | 890 kB 00:01 2024-03-28T17:56:21.857 INFO:teuthology.orchestra.run.smithi170.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2024-03-28T17:56:21.888 INFO:teuthology.orchestra.run.smithi170.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 12 MB/s | 661 kB 00:00 2024-03-28T17:56:21.921 INFO:teuthology.orchestra.run.smithi170.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.0 MB/s | 578 kB 00:00 2024-03-28T17:56:21.947 INFO:teuthology.orchestra.run.smithi170.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 803 kB/s | 46 kB 00:00 2024-03-28T17:56:21.972 INFO:teuthology.orchestra.run.smithi170.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-28T17:56:21.989 INFO:teuthology.orchestra.run.smithi059.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.0 MB/s | 145 kB 00:00 2024-03-28T17:56:21.997 INFO:teuthology.orchestra.run.smithi170.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2024-03-28T17:56:22.023 INFO:teuthology.orchestra.run.smithi170.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-28T17:56:22.045 INFO:teuthology.orchestra.run.smithi037.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 1.6 MB/s | 2.3 MB 00:01 2024-03-28T17:56:22.048 INFO:teuthology.orchestra.run.smithi170.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 684 kB/s | 34 kB 00:00 2024-03-28T17:56:22.149 INFO:teuthology.orchestra.run.smithi170.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-28T17:56:22.182 INFO:teuthology.orchestra.run.smithi170.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 7.7 MB/s | 1.0 MB 00:00 2024-03-28T17:56:22.207 INFO:teuthology.orchestra.run.smithi170.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-28T17:56:22.333 INFO:teuthology.orchestra.run.smithi170.stdout:(59/174): python3-babel-2.9.1-2.el9.noarch.rpm 38 MB/s | 5.8 MB 00:00 2024-03-28T17:56:22.358 INFO:teuthology.orchestra.run.smithi170.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 165 kB/s | 24 kB 00:00 2024-03-28T17:56:22.383 INFO:teuthology.orchestra.run.smithi170.stdout:(61/174): python3-jsonpointer-2.0-4.el9.noarch. 361 kB/s | 18 kB 00:00 2024-03-28T17:56:22.408 INFO:teuthology.orchestra.run.smithi170.stdout:(62/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-28T17:56:22.434 INFO:teuthology.orchestra.run.smithi170.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 630 kB/s | 32 kB 00:00 2024-03-28T17:56:22.534 INFO:teuthology.orchestra.run.smithi170.stdout:(64/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-28T17:56:22.551 INFO:teuthology.orchestra.run.smithi170.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.9 MB/s | 217 kB 00:00 2024-03-28T17:56:22.571 INFO:teuthology.orchestra.run.smithi037.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 2.9 MB/s | 1.5 MB 00:00 2024-03-28T17:56:22.576 INFO:teuthology.orchestra.run.smithi170.stdout:(66/174): python3-oauthlib-3.1.1-5.el9.noarch.r 4.1 MB/s | 174 kB 00:00 2024-03-28T17:56:22.601 INFO:teuthology.orchestra.run.smithi170.stdout:(67/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-28T17:56:22.618 INFO:teuthology.orchestra.run.smithi170.stdout:(68/174): python3-prettytable-0.7.2-27.el9.noar 981 kB/s | 41 kB 00:00 2024-03-28T17:56:22.644 INFO:teuthology.orchestra.run.smithi170.stdout:(69/174): python3-protobuf-3.14.0-13.el9.noarch 5.5 MB/s | 238 kB 00:00 2024-03-28T17:56:22.669 INFO:teuthology.orchestra.run.smithi170.stdout:(70/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-28T17:56:22.694 INFO:teuthology.orchestra.run.smithi170.stdout:(71/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-28T17:56:22.711 INFO:teuthology.orchestra.run.smithi170.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-28T17:56:22.728 INFO:teuthology.orchestra.run.smithi170.stdout:(73/174): python3-requests-oauthlib-1.3.0-12.el 1.2 MB/s | 43 kB 00:00 2024-03-28T17:56:23.088 INFO:teuthology.orchestra.run.smithi037.stdout:(6/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 1.5 MB/s | 5.4 MB 00:03 2024-03-28T17:56:23.270 INFO:teuthology.orchestra.run.smithi170.stdout:(74/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 9.6 MB/s | 48 MB 00:04 2024-03-28T17:56:23.287 INFO:teuthology.orchestra.run.smithi170.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 69 kB/s | 39 kB 00:00 2024-03-28T17:56:23.454 INFO:teuthology.orchestra.run.smithi170.stdout:(76/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-03-28T17:56:23.479 INFO:teuthology.orchestra.run.smithi170.stdout:(77/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 197 kB/s | 41 kB 00:00 2024-03-28T17:56:23.504 INFO:teuthology.orchestra.run.smithi170.stdout:(78/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-03-28T17:56:23.514 INFO:teuthology.orchestra.run.smithi059.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 1.5 MB/s | 2.3 MB 00:01 2024-03-28T17:56:23.530 INFO:teuthology.orchestra.run.smithi170.stdout:(79/174): uuid-1.6.2-55.el9.x86_64.rpm 758 kB/s | 56 kB 00:00 2024-03-28T17:56:23.555 INFO:teuthology.orchestra.run.smithi170.stdout:(80/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-28T17:56:23.580 INFO:teuthology.orchestra.run.smithi170.stdout:(81/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 587 kB/s | 44 kB 00:00 2024-03-28T17:56:23.614 INFO:teuthology.orchestra.run.smithi170.stdout:(82/174): fuse-2.9.9-15.el9.x86_64.rpm 943 kB/s | 79 kB 00:00 2024-03-28T17:56:23.639 INFO:teuthology.orchestra.run.smithi170.stdout:(83/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-28T17:56:23.673 INFO:teuthology.orchestra.run.smithi170.stdout:(84/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-28T17:56:23.698 INFO:teuthology.orchestra.run.smithi170.stdout:(85/174): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-28T17:56:23.723 INFO:teuthology.orchestra.run.smithi170.stdout:(86/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-28T17:56:23.748 INFO:teuthology.orchestra.run.smithi170.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-28T17:56:23.756 INFO:teuthology.orchestra.run.smithi059.stdout:(5/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 1.5 MB/s | 5.4 MB 00:03 2024-03-28T17:56:23.796 INFO:teuthology.orchestra.run.smithi037.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 3.8 MB/s | 4.7 MB 00:01 2024-03-28T17:56:23.998 INFO:teuthology.orchestra.run.smithi170.stdout:(88/174): jq-1.6-16.el9.x86_64.rpm 624 kB/s | 187 kB 00:00 2024-03-28T17:56:24.024 INFO:teuthology.orchestra.run.smithi170.stdout:(89/174): libquadmath-11.4.1-3.el9.x86_64.rpm 711 kB/s | 195 kB 00:00 2024-03-28T17:56:24.057 INFO:teuthology.orchestra.run.smithi170.stdout:(90/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-03-28T17:56:24.083 INFO:teuthology.orchestra.run.smithi170.stdout:(91/174): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-03-28T17:56:24.183 INFO:teuthology.orchestra.run.smithi170.stdout:(92/174): elfutils-libelf-devel-0.190-2.el9.x86 135 kB/s | 21 kB 00:00 2024-03-28T17:56:24.208 INFO:teuthology.orchestra.run.smithi170.stdout:(93/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 169 kB/s | 25 kB 00:00 2024-03-28T17:56:24.298 INFO:teuthology.orchestra.run.smithi059.stdout:(6/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 1.9 MB/s | 1.5 MB 00:00 2024-03-28T17:56:24.442 INFO:teuthology.orchestra.run.smithi170.stdout:(94/174): postgresql-contrib-13.14-1.el9.x86_64 3.8 MB/s | 911 kB 00:00 2024-03-28T17:56:24.471 INFO:teuthology.orchestra.run.smithi037.stdout:(8/174): ceph-common-19.0.0-2145.g3be9a5c9.el9. 4.1 MB/s | 20 MB 00:05 2024-03-28T17:56:24.484 INFO:teuthology.orchestra.run.smithi170.stdout:(95/174): postgresql-13.14-1.el9.x86_64.rpm 5.3 MB/s | 1.6 MB 00:00 2024-03-28T17:56:24.509 INFO:teuthology.orchestra.run.smithi170.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 1.9 MB/s | 133 kB 00:00 2024-03-28T17:56:24.555 INFO:teuthology.orchestra.run.smithi037.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 306 kB/s | 25 kB 00:00 2024-03-28T17:56:24.567 INFO:teuthology.orchestra.run.smithi170.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 4.1 MB/s | 245 kB 00:00 2024-03-28T17:56:24.659 INFO:teuthology.orchestra.run.smithi170.stdout:(98/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 7.2 MB/s | 4.1 MB 00:00 2024-03-28T17:56:24.685 INFO:teuthology.orchestra.run.smithi170.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.1 MB/s | 248 kB 00:00 2024-03-28T17:56:24.690 INFO:teuthology.orchestra.run.smithi059.stdout:(7/174): ceph-common-19.0.0-2145.g3be9a5c9.el9. 4.5 MB/s | 20 MB 00:04 2024-03-28T17:56:24.718 INFO:teuthology.orchestra.run.smithi170.stdout:(100/174): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-03-28T17:56:24.735 INFO:teuthology.orchestra.run.smithi170.stdout:(101/174): qatlib-service-24.02.0-1.el9.x86_64. 715 kB/s | 36 kB 00:00 2024-03-28T17:56:24.769 INFO:teuthology.orchestra.run.smithi170.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-03-28T17:56:24.795 INFO:teuthology.orchestra.run.smithi170.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.5 MB/s | 94 kB 00:00 2024-03-28T17:56:24.971 INFO:teuthology.orchestra.run.smithi170.stdout:(104/174): postgresql-server-13.14-1.el9.x86_64 12 MB/s | 5.8 MB 00:00 2024-03-28T17:56:25.071 INFO:teuthology.orchestra.run.smithi170.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 5.4 MB/s | 548 kB 00:00 2024-03-28T17:56:25.179 INFO:teuthology.orchestra.run.smithi170.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 5.1 MB/s | 309 kB 00:00 2024-03-28T17:56:25.221 INFO:teuthology.orchestra.run.smithi170.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 465 kB/s | 19 kB 00:00 2024-03-28T17:56:25.355 INFO:teuthology.orchestra.run.smithi170.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 33 MB/s | 4.4 MB 00:00 2024-03-28T17:56:25.380 INFO:teuthology.orchestra.run.smithi170.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-03-28T17:56:25.406 INFO:teuthology.orchestra.run.smithi170.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-03-28T17:56:25.439 INFO:teuthology.orchestra.run.smithi170.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-03-28T17:56:25.456 INFO:teuthology.orchestra.run.smithi170.stdout:(112/174): lua-devel-5.4.4-4.el9.x86_64.rpm 32 kB/s | 22 kB 00:00 2024-03-28T17:56:25.481 INFO:teuthology.orchestra.run.smithi170.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-03-28T17:56:25.523 INFO:teuthology.orchestra.run.smithi170.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 13 MB/s | 548 kB 00:00 2024-03-28T17:56:25.557 INFO:teuthology.orchestra.run.smithi170.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-03-28T17:56:25.574 INFO:teuthology.orchestra.run.smithi059.stdout:(8/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 2.6 MB/s | 4.7 MB 00:01 2024-03-28T17:56:25.590 INFO:teuthology.orchestra.run.smithi170.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 966 kB/s | 32 kB 00:00 2024-03-28T17:56:25.624 INFO:teuthology.orchestra.run.smithi170.stdout:(117/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 838 kB 00:00 2024-03-28T17:56:25.649 INFO:teuthology.orchestra.run.smithi170.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 256 kB/s | 15 kB 00:00 2024-03-28T17:56:25.666 INFO:teuthology.orchestra.run.smithi059.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 275 kB/s | 25 kB 00:00 2024-03-28T17:56:25.675 INFO:teuthology.orchestra.run.smithi170.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-03-28T17:56:25.700 INFO:teuthology.orchestra.run.smithi170.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-03-28T17:56:25.725 INFO:teuthology.orchestra.run.smithi170.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-03-28T17:56:25.766 INFO:teuthology.orchestra.run.smithi170.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 539 kB/s | 35 kB 00:00 2024-03-28T17:56:25.792 INFO:teuthology.orchestra.run.smithi170.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-03-28T17:56:25.826 INFO:teuthology.orchestra.run.smithi170.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 4.2 MB/s | 144 kB 00:00 2024-03-28T17:56:25.860 INFO:teuthology.orchestra.run.smithi170.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 323 kB/s | 11 kB 00:00 2024-03-28T17:56:25.910 INFO:teuthology.orchestra.run.smithi170.stdout:(126/174): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-03-28T17:56:25.930 INFO:teuthology.orchestra.run.smithi037.stdout:(10/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 4.6 MB/s | 9.9 MB 00:02 2024-03-28T17:56:25.935 INFO:teuthology.orchestra.run.smithi170.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 235 kB/s | 18 kB 00:00 2024-03-28T17:56:25.961 INFO:teuthology.orchestra.run.smithi170.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 461 kB/s | 23 kB 00:00 2024-03-28T17:56:25.986 INFO:teuthology.orchestra.run.smithi170.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 386 kB/s | 19 kB 00:00 2024-03-28T17:56:26.011 INFO:teuthology.orchestra.run.smithi170.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-03-28T17:56:26.014 INFO:teuthology.orchestra.run.smithi037.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 2.6 MB/s | 220 kB 00:00 2024-03-28T17:56:26.045 INFO:teuthology.orchestra.run.smithi170.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 17 MB/s | 1.0 MB 00:00 2024-03-28T17:56:26.061 INFO:teuthology.orchestra.run.smithi170.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 923 kB/s | 46 kB 00:00 2024-03-28T17:56:26.087 INFO:teuthology.orchestra.run.smithi170.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.8 MB/s | 79 kB 00:00 2024-03-28T17:56:26.089 INFO:teuthology.orchestra.run.smithi037.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 364 kB/s | 27 kB 00:00 2024-03-28T17:56:26.104 INFO:teuthology.orchestra.run.smithi170.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.3 MB/s | 58 kB 00:00 2024-03-28T17:56:26.129 INFO:teuthology.orchestra.run.smithi170.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 6.4 MB/s | 272 kB 00:00 2024-03-28T17:56:26.146 INFO:teuthology.orchestra.run.smithi170.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 393 kB/s | 16 kB 00:00 2024-03-28T17:56:26.156 INFO:teuthology.orchestra.run.smithi037.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 476 kB/s | 31 kB 00:00 2024-03-28T17:56:26.180 INFO:teuthology.orchestra.run.smithi170.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-03-28T17:56:26.197 INFO:teuthology.orchestra.run.smithi170.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 560 kB/s | 28 kB 00:00 2024-03-28T17:56:26.222 INFO:teuthology.orchestra.run.smithi170.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 733 kB/s | 31 kB 00:00 2024-03-28T17:56:26.224 INFO:teuthology.orchestra.run.smithi059.stdout:(10/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 6.4 MB/s | 9.9 MB 00:01 2024-03-28T17:56:26.247 INFO:teuthology.orchestra.run.smithi170.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-03-28T17:56:26.272 INFO:teuthology.orchestra.run.smithi170.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-28T17:56:26.274 INFO:teuthology.orchestra.run.smithi059.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 4.3 MB/s | 220 kB 00:00 2024-03-28T17:56:26.298 INFO:teuthology.orchestra.run.smithi170.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 716 kB/s | 36 kB 00:00 2024-03-28T17:56:26.306 INFO:teuthology.orchestra.run.smithi037.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 4.9 MB/s | 746 kB 00:00 2024-03-28T17:56:26.316 INFO:teuthology.orchestra.run.smithi059.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 648 kB/s | 27 kB 00:00 2024-03-28T17:56:26.323 INFO:teuthology.orchestra.run.smithi170.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-28T17:56:26.348 INFO:teuthology.orchestra.run.smithi170.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-28T17:56:26.365 INFO:teuthology.orchestra.run.smithi037.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 2.8 MB/s | 163 kB 00:00 2024-03-28T17:56:26.367 INFO:teuthology.orchestra.run.smithi059.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 625 kB/s | 31 kB 00:00 2024-03-28T17:56:26.374 INFO:teuthology.orchestra.run.smithi170.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-03-28T17:56:26.408 INFO:teuthology.orchestra.run.smithi170.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 7.1 MB/s | 427 kB 00:00 2024-03-28T17:56:26.423 INFO:teuthology.orchestra.run.smithi037.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 2.2 MB/s | 127 kB 00:00 2024-03-28T17:56:26.432 INFO:teuthology.orchestra.run.smithi170.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 342 kB/s | 20 kB 00:00 2024-03-28T17:56:26.458 INFO:teuthology.orchestra.run.smithi170.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-03-28T17:56:26.501 INFO:teuthology.orchestra.run.smithi059.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 5.5 MB/s | 746 kB 00:00 2024-03-28T17:56:26.517 INFO:teuthology.orchestra.run.smithi170.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-28T17:56:26.532 INFO:teuthology.orchestra.run.smithi037.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 4.6 MB/s | 501 kB 00:00 2024-03-28T17:56:26.559 INFO:teuthology.orchestra.run.smithi059.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 2.7 MB/s | 163 kB 00:00 2024-03-28T17:56:26.610 INFO:teuthology.orchestra.run.smithi059.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 2.5 MB/s | 127 kB 00:00 2024-03-28T17:56:26.702 INFO:teuthology.orchestra.run.smithi059.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 5.3 MB/s | 501 kB 00:00 2024-03-28T17:56:26.717 INFO:teuthology.orchestra.run.smithi170.stdout:(150/174): protobuf-compiler-3.14.0-13.el9.x86_ 449 kB/s | 863 kB 00:01 2024-03-28T17:56:27.009 INFO:teuthology.orchestra.run.smithi170.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 1.1 MB/s | 321 kB 00:00 2024-03-28T17:56:27.067 INFO:teuthology.orchestra.run.smithi170.stdout:(152/174): librados2-19.0.0-2145.g3be9a5c9.el9. 5.6 MB/s | 3.4 MB 00:00 2024-03-28T17:56:27.393 INFO:teuthology.orchestra.run.smithi170.stdout:(153/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 3.7 MB/s | 3.1 MB 00:00 2024-03-28T17:56:27.543 INFO:teuthology.orchestra.run.smithi059.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 6.1 MB/s | 5.1 MB 00:00 2024-03-28T17:56:27.549 INFO:teuthology.orchestra.run.smithi037.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 5.1 MB/s | 5.1 MB 00:01 2024-03-28T17:56:27.568 INFO:teuthology.orchestra.run.smithi170.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 241 kB/s | 42 kB 00:00 2024-03-28T17:56:27.585 INFO:teuthology.orchestra.run.smithi059.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 1.1 MB/s | 45 kB 00:00 2024-03-28T17:56:27.599 INFO:teuthology.orchestra.run.smithi037.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 926 kB/s | 45 kB 00:00 2024-03-28T17:56:27.636 INFO:teuthology.orchestra.run.smithi059.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 2.6 MB/s | 131 kB 00:00 2024-03-28T17:56:27.672 INFO:teuthology.orchestra.run.smithi037.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.8 MB/s | 131 kB 00:00 2024-03-28T17:56:27.686 INFO:teuthology.orchestra.run.smithi059.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 3.2 MB/s | 162 kB 00:00 2024-03-28T17:56:27.733 INFO:teuthology.orchestra.run.smithi037.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 2.7 MB/s | 162 kB 00:00 2024-03-28T17:56:27.735 INFO:teuthology.orchestra.run.smithi170.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.5 MB/s | 248 kB 00:00 2024-03-28T17:56:27.761 INFO:teuthology.orchestra.run.smithi059.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 4.2 MB/s | 321 kB 00:00 2024-03-28T17:56:27.809 INFO:teuthology.orchestra.run.smithi037.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 4.2 MB/s | 321 kB 00:00 2024-03-28T17:56:27.818 INFO:teuthology.orchestra.run.smithi170.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 858 kB/s | 71 kB 00:00 2024-03-28T17:56:27.836 INFO:teuthology.orchestra.run.smithi059.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 3.9 MB/s | 298 kB 00:00 2024-03-28T17:56:27.877 INFO:teuthology.orchestra.run.smithi170.stdout:(157/174): NetworkManager-libnm-1.47.2-1.el9.x8 2.3 MB/s | 1.8 MB 00:00 2024-03-28T17:56:27.884 INFO:teuthology.orchestra.run.smithi037.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 3.9 MB/s | 298 kB 00:00 2024-03-28T17:56:27.887 INFO:teuthology.orchestra.run.smithi059.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.9 MB/s | 99 kB 00:00 2024-03-28T17:56:27.902 INFO:teuthology.orchestra.run.smithi170.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 436 kB/s | 36 kB 00:00 2024-03-28T17:56:27.934 INFO:teuthology.orchestra.run.smithi037.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 2.0 MB/s | 99 kB 00:00 2024-03-28T17:56:27.945 INFO:teuthology.orchestra.run.smithi059.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-28T17:56:27.977 INFO:teuthology.orchestra.run.smithi170.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.9 MB/s | 192 kB 00:00 2024-03-28T17:56:27.984 INFO:teuthology.orchestra.run.smithi037.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.7 MB/s | 85 kB 00:00 2024-03-28T17:56:28.061 INFO:teuthology.orchestra.run.smithi170.stdout:(160/174): NetworkManager-1.47.2-1.el9.x86_64.r 2.2 MB/s | 2.3 MB 00:01 2024-03-28T17:56:28.086 INFO:teuthology.orchestra.run.smithi170.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 212 kB/s | 23 kB 00:00 2024-03-28T17:56:28.111 INFO:teuthology.orchestra.run.smithi170.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.2 MB/s | 256 kB 00:00 2024-03-28T17:56:28.145 INFO:teuthology.orchestra.run.smithi170.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 108 kB 00:00 2024-03-28T17:56:28.187 INFO:teuthology.orchestra.run.smithi170.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-03-28T17:56:28.212 INFO:teuthology.orchestra.run.smithi170.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 956 kB/s | 64 kB 00:00 2024-03-28T17:56:28.238 INFO:teuthology.orchestra.run.smithi170.stdout:(166/174): libmount-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 136 kB 00:00 2024-03-28T17:56:28.279 INFO:teuthology.orchestra.run.smithi170.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 312 kB/s | 29 kB 00:00 2024-03-28T17:56:28.379 INFO:teuthology.orchestra.run.smithi059.stdout:(26/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 7.1 MB/s | 3.1 MB 00:00 2024-03-28T17:56:28.429 INFO:teuthology.orchestra.run.smithi059.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 3.4 MB/s | 170 kB 00:00 2024-03-28T17:56:28.455 INFO:teuthology.orchestra.run.smithi170.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 5.0 MB/s | 1.2 MB 00:00 2024-03-28T17:56:28.480 INFO:teuthology.orchestra.run.smithi059.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 682 kB/s | 34 kB 00:00 2024-03-28T17:56:28.530 INFO:teuthology.orchestra.run.smithi059.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.7 MB/s | 139 kB 00:00 2024-03-28T17:56:28.564 INFO:teuthology.orchestra.run.smithi170.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 4.2 MB/s | 465 kB 00:00 2024-03-28T17:56:28.577 INFO:teuthology.orchestra.run.smithi037.stdout:(26/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 5.2 MB/s | 3.1 MB 00:00 2024-03-28T17:56:28.614 INFO:teuthology.orchestra.run.smithi170.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 5.7 MB/s | 2.2 MB 00:00 2024-03-28T17:56:28.635 INFO:teuthology.orchestra.run.smithi037.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.9 MB/s | 170 kB 00:00 2024-03-28T17:56:28.656 INFO:teuthology.orchestra.run.smithi170.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 339 kB/s | 31 kB 00:00 2024-03-28T17:56:28.685 INFO:teuthology.orchestra.run.smithi037.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 693 kB/s | 34 kB 00:00 2024-03-28T17:56:28.763 INFO:teuthology.orchestra.run.smithi059.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 7.7 MB/s | 1.8 MB 00:00 2024-03-28T17:56:28.764 INFO:teuthology.orchestra.run.smithi170.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.2 MB/s | 1.1 MB 00:00 2024-03-28T17:56:28.769 INFO:teuthology.orchestra.run.smithi037.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 1.6 MB/s | 139 kB 00:00 2024-03-28T17:56:28.781 INFO:teuthology.orchestra.run.smithi170.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 594 kB/s | 74 kB 00:00 2024-03-28T17:56:28.889 INFO:teuthology.orchestra.run.smithi170.stdout:(174/174): util-linux-2.37.4-18.el9.x86_64.rpm 3.7 MB/s | 2.3 MB 00:00 2024-03-28T17:56:28.898 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-28T17:56:28.898 INFO:teuthology.orchestra.run.smithi170.stdout:Total 15 MB/s | 225 MB 00:14 2024-03-28T17:56:29.261 INFO:teuthology.orchestra.run.smithi037.stdout:(30/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 2.7 MB/s | 17 MB 00:06 2024-03-28T17:56:29.328 INFO:teuthology.orchestra.run.smithi037.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 3.2 MB/s | 1.8 MB 00:00 2024-03-28T17:56:29.411 INFO:teuthology.orchestra.run.smithi037.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 2.9 MB/s | 248 kB 00:00 2024-03-28T17:56:29.470 INFO:teuthology.orchestra.run.smithi037.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 862 kB/s | 49 kB 00:00 2024-03-28T17:56:29.487 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-28T17:56:29.520 INFO:teuthology.orchestra.run.smithi037.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 335 kB/s | 16 kB 00:00 2024-03-28T17:56:29.572 INFO:teuthology.orchestra.run.smithi059.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 9.1 MB/s | 7.4 MB 00:00 2024-03-28T17:56:29.596 INFO:teuthology.orchestra.run.smithi037.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 3.5 MB/s | 264 kB 00:00 2024-03-28T17:56:29.631 INFO:teuthology.orchestra.run.smithi059.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 4.2 MB/s | 248 kB 00:00 2024-03-28T17:56:29.673 INFO:teuthology.orchestra.run.smithi059.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 1.2 MB/s | 49 kB 00:00 2024-03-28T17:56:29.715 INFO:teuthology.orchestra.run.smithi059.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 397 kB/s | 16 kB 00:00 2024-03-28T17:56:29.746 INFO:teuthology.orchestra.run.smithi037.stdout:(36/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 4.9 MB/s | 748 kB 00:00 2024-03-28T17:56:29.760 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-28T17:56:29.761 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-28T17:56:29.773 INFO:teuthology.orchestra.run.smithi059.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 4.4 MB/s | 264 kB 00:00 2024-03-28T17:56:29.796 INFO:teuthology.orchestra.run.smithi037.stdout:(37/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 372 kB/s | 18 kB 00:00 2024-03-28T17:56:29.882 INFO:teuthology.orchestra.run.smithi059.stdout:(36/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 6.8 MB/s | 748 kB 00:00 2024-03-28T17:56:29.918 INFO:teuthology.orchestra.run.smithi059.stdout:(37/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 511 kB/s | 18 kB 00:00 2024-03-28T17:56:29.997 INFO:teuthology.orchestra.run.smithi037.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 3.3 MB/s | 662 kB 00:00 2024-03-28T17:56:30.052 INFO:teuthology.orchestra.run.smithi059.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.9 MB/s | 662 kB 00:00 2024-03-28T17:56:30.063 INFO:teuthology.orchestra.run.smithi037.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 920 kB 00:00 2024-03-28T17:56:30.102 INFO:teuthology.orchestra.run.smithi059.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 18 MB/s | 920 kB 00:00 2024-03-28T17:56:30.105 INFO:teuthology.orchestra.run.smithi037.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.6 MB/s | 106 kB 00:00 2024-03-28T17:56:30.136 INFO:teuthology.orchestra.run.smithi059.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-28T17:56:30.147 INFO:teuthology.orchestra.run.smithi037.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.3 MB/s | 307 kB 00:00 2024-03-28T17:56:30.178 INFO:teuthology.orchestra.run.smithi059.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-28T17:56:30.181 INFO:teuthology.orchestra.run.smithi037.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 977 kB/s | 32 kB 00:00 2024-03-28T17:56:30.212 INFO:teuthology.orchestra.run.smithi059.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 953 kB/s | 32 kB 00:00 2024-03-28T17:56:30.298 INFO:teuthology.orchestra.run.smithi037.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-03-28T17:56:30.320 INFO:teuthology.orchestra.run.smithi059.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-28T17:56:30.340 INFO:teuthology.orchestra.run.smithi037.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 421 kB/s | 17 kB 00:00 2024-03-28T17:56:30.354 INFO:teuthology.orchestra.run.smithi059.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 514 kB/s | 17 kB 00:00 2024-03-28T17:56:30.374 INFO:teuthology.orchestra.run.smithi037.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 311 kB/s | 10 kB 00:00 2024-03-28T17:56:30.388 INFO:teuthology.orchestra.run.smithi059.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 303 kB/s | 10 kB 00:00 2024-03-28T17:56:30.415 INFO:teuthology.orchestra.run.smithi037.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2024-03-28T17:56:30.421 INFO:teuthology.orchestra.run.smithi059.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-28T17:56:30.455 INFO:teuthology.orchestra.run.smithi059.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-28T17:56:30.457 INFO:teuthology.orchestra.run.smithi037.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-03-28T17:56:30.506 INFO:teuthology.orchestra.run.smithi059.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-28T17:56:30.524 INFO:teuthology.orchestra.run.smithi037.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.8 MB/s | 661 kB 00:00 2024-03-28T17:56:30.547 INFO:teuthology.orchestra.run.smithi059.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 578 kB 00:00 2024-03-28T17:56:30.575 INFO:teuthology.orchestra.run.smithi037.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-28T17:56:30.600 INFO:teuthology.orchestra.run.smithi037.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.9 MB/s | 46 kB 00:00 2024-03-28T17:56:30.651 INFO:teuthology.orchestra.run.smithi037.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.6 MB/s | 282 kB 00:00 2024-03-28T17:56:30.692 INFO:teuthology.orchestra.run.smithi037.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 4.5 MB/s | 187 kB 00:00 2024-03-28T17:56:30.706 INFO:teuthology.orchestra.run.smithi059.stdout:(50/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 2.6 MB/s | 17 MB 00:06 2024-03-28T17:56:30.731 INFO:teuthology.orchestra.run.smithi059.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 252 kB/s | 46 kB 00:00 2024-03-28T17:56:30.734 INFO:teuthology.orchestra.run.smithi037.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.2 MB/s | 217 kB 00:00 2024-03-28T17:56:30.757 INFO:teuthology.orchestra.run.smithi059.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-28T17:56:30.768 INFO:teuthology.orchestra.run.smithi037.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-28T17:56:30.790 INFO:teuthology.orchestra.run.smithi059.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 187 kB 00:00 2024-03-28T17:56:30.815 INFO:teuthology.orchestra.run.smithi059.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-03-28T17:56:30.841 INFO:teuthology.orchestra.run.smithi059.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 687 kB/s | 34 kB 00:00 2024-03-28T17:56:30.943 INFO:teuthology.orchestra.run.smithi037.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-03-28T17:56:30.974 INFO:teuthology.orchestra.run.smithi059.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-03-28T17:56:31.010 INFO:teuthology.orchestra.run.smithi037.stdout:(56/174): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-03-28T17:56:31.026 INFO:teuthology.orchestra.run.smithi059.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.4 MB/s | 1.0 MB 00:00 2024-03-28T17:56:31.050 INFO:teuthology.orchestra.run.smithi059.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.0 MB/s | 78 kB 00:00 2024-03-28T17:56:31.061 INFO:teuthology.orchestra.run.smithi037.stdout:(57/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 78 kB 00:00 2024-03-28T17:56:31.234 INFO:teuthology.orchestra.run.smithi059.stdout:(59/174): python3-babel-2.9.1-2.el9.noarch.rpm 28 MB/s | 5.8 MB 00:00 2024-03-28T17:56:31.251 INFO:teuthology.orchestra.run.smithi059.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 124 kB/s | 24 kB 00:00 2024-03-28T17:56:31.278 INFO:teuthology.orchestra.run.smithi037.stdout:(58/174): python3-babel-2.9.1-2.el9.noarch.rpm 27 MB/s | 5.8 MB 00:00 2024-03-28T17:56:31.284 INFO:teuthology.orchestra.run.smithi059.stdout:(61/174): python3-mako-1.1.4-6.el9.noarch.rpm 4.3 MB/s | 144 kB 00:00 2024-03-28T17:56:31.301 INFO:teuthology.orchestra.run.smithi059.stdout:(62/174): python3-jsonpointer-2.0-4.el9.noarch. 363 kB/s | 18 kB 00:00 2024-03-28T17:56:31.318 INFO:teuthology.orchestra.run.smithi059.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 942 kB/s | 32 kB 00:00 2024-03-28T17:56:31.412 INFO:teuthology.orchestra.run.smithi037.stdout:(59/174): ceph-mgr-diskprediction-local-19.0.0- 3.4 MB/s | 7.4 MB 00:02 2024-03-28T17:56:31.437 INFO:teuthology.orchestra.run.smithi037.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 155 kB/s | 24 kB 00:00 2024-03-28T17:56:31.460 INFO:teuthology.orchestra.run.smithi059.stdout:(64/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-28T17:56:31.462 INFO:teuthology.orchestra.run.smithi037.stdout:(61/174): python3-jsonpointer-2.0-4.el9.noarch. 370 kB/s | 18 kB 00:00 2024-03-28T17:56:31.487 INFO:teuthology.orchestra.run.smithi037.stdout:(62/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 144 kB 00:00 2024-03-28T17:56:31.492 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-28T17:56:31.493 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-28T17:56:31.493 INFO:teuthology.orchestra.run.smithi059.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2024-03-28T17:56:31.512 INFO:teuthology.orchestra.run.smithi037.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 657 kB/s | 32 kB 00:00 2024-03-28T17:56:31.519 INFO:teuthology.orchestra.run.smithi059.stdout:(66/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 174 kB 00:00 2024-03-28T17:56:31.544 INFO:teuthology.orchestra.run.smithi059.stdout:(67/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-28T17:56:31.561 INFO:teuthology.orchestra.run.smithi059.stdout:(68/174): python3-prettytable-0.7.2-27.el9.noar 984 kB/s | 41 kB 00:00 2024-03-28T17:56:31.586 INFO:teuthology.orchestra.run.smithi059.stdout:(69/174): python3-protobuf-3.14.0-13.el9.noarch 5.5 MB/s | 238 kB 00:00 2024-03-28T17:56:31.612 INFO:teuthology.orchestra.run.smithi059.stdout:(70/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-28T17:56:31.637 INFO:teuthology.orchestra.run.smithi059.stdout:(71/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-28T17:56:31.655 INFO:teuthology.orchestra.run.smithi037.stdout:(64/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-28T17:56:31.662 INFO:teuthology.orchestra.run.smithi059.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 951 kB/s | 47 kB 00:00 2024-03-28T17:56:31.688 INFO:teuthology.orchestra.run.smithi059.stdout:(73/174): python3-requests-oauthlib-1.3.0-12.el 857 kB/s | 43 kB 00:00 2024-03-28T17:56:31.688 INFO:teuthology.orchestra.run.smithi037.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2024-03-28T17:56:31.730 INFO:teuthology.orchestra.run.smithi037.stdout:(66/174): python3-packaging-20.9-5.el9.noarch.r 1.7 MB/s | 69 kB 00:00 2024-03-28T17:56:31.764 INFO:teuthology.orchestra.run.smithi037.stdout:(67/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.3 MB/s | 174 kB 00:00 2024-03-28T17:56:31.789 INFO:teuthology.orchestra.run.smithi037.stdout:(68/174): python3-prettytable-0.7.2-27.el9.noar 734 kB/s | 41 kB 00:00 2024-03-28T17:56:31.814 INFO:teuthology.orchestra.run.smithi037.stdout:(69/174): python3-protobuf-3.14.0-13.el9.noarch 4.8 MB/s | 238 kB 00:00 2024-03-28T17:56:31.839 INFO:teuthology.orchestra.run.smithi037.stdout:(70/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 132 kB 00:00 2024-03-28T17:56:31.855 INFO:teuthology.orchestra.run.smithi059.stdout:(74/174): python3-toml-0.10.2-6.el9.noarch.rpm 237 kB/s | 39 kB 00:00 2024-03-28T17:56:31.881 INFO:teuthology.orchestra.run.smithi037.stdout:(71/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.2 MB/s | 211 kB 00:00 2024-03-28T17:56:31.907 INFO:teuthology.orchestra.run.smithi037.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 736 kB/s | 47 kB 00:00 2024-03-28T17:56:31.932 INFO:teuthology.orchestra.run.smithi037.stdout:(73/174): python3-requests-oauthlib-1.3.0-12.el 871 kB/s | 43 kB 00:00 2024-03-28T17:56:32.013 INFO:teuthology.orchestra.run.smithi059.stdout:(75/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-28T17:56:32.039 INFO:teuthology.orchestra.run.smithi059.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 225 kB/s | 41 kB 00:00 2024-03-28T17:56:32.072 INFO:teuthology.orchestra.run.smithi059.stdout:(77/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-28T17:56:32.098 INFO:teuthology.orchestra.run.smithi059.stdout:(78/174): uuid-1.6.2-55.el9.x86_64.rpm 968 kB/s | 56 kB 00:00 2024-03-28T17:56:32.123 INFO:teuthology.orchestra.run.smithi059.stdout:(79/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-28T17:56:32.148 INFO:teuthology.orchestra.run.smithi059.stdout:(80/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2024-03-28T17:56:32.174 INFO:teuthology.orchestra.run.smithi059.stdout:(81/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-28T17:56:32.199 INFO:teuthology.orchestra.run.smithi059.stdout:(82/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-28T17:56:32.232 INFO:teuthology.orchestra.run.smithi059.stdout:(83/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-28T17:56:32.257 INFO:teuthology.orchestra.run.smithi037.stdout:(74/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-28T17:56:32.258 INFO:teuthology.orchestra.run.smithi059.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-28T17:56:32.292 INFO:teuthology.orchestra.run.smithi059.stdout:(85/174): python3-pycparser-2.20-6.el9.noarch.r 2.1 MB/s | 124 kB 00:00 2024-03-28T17:56:32.299 INFO:teuthology.orchestra.run.smithi037.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 110 kB/s | 39 kB 00:00 2024-03-28T17:56:32.317 INFO:teuthology.orchestra.run.smithi059.stdout:(86/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.1 MB/s | 187 kB 00:00 2024-03-28T17:56:32.333 INFO:teuthology.orchestra.run.smithi037.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.2 MB/s | 41 kB 00:00 2024-03-28T17:56:32.367 INFO:teuthology.orchestra.run.smithi037.stdout:(77/174): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 300 kB 00:00 2024-03-28T17:56:32.392 INFO:teuthology.orchestra.run.smithi037.stdout:(78/174): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 56 kB 00:00 2024-03-28T17:56:32.417 INFO:teuthology.orchestra.run.smithi037.stdout:(79/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 63 kB 00:00 2024-03-28T17:56:32.443 INFO:teuthology.orchestra.run.smithi037.stdout:(80/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 904 kB/s | 44 kB 00:00 2024-03-28T17:56:32.468 INFO:teuthology.orchestra.run.smithi059.stdout:(87/174): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-28T17:56:32.468 INFO:teuthology.orchestra.run.smithi037.stdout:(81/174): fuse-2.9.9-15.el9.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2024-03-28T17:56:32.502 INFO:teuthology.orchestra.run.smithi037.stdout:(82/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.1 MB/s | 241 kB 00:00 2024-03-28T17:56:32.518 INFO:teuthology.orchestra.run.smithi059.stdout:(88/174): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-03-28T17:56:32.543 INFO:teuthology.orchestra.run.smithi059.stdout:(89/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.5 MB/s | 803 kB 00:00 2024-03-28T17:56:32.560 INFO:teuthology.orchestra.run.smithi037.stdout:(83/174): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.1 MB 00:00 2024-03-28T17:56:32.568 INFO:teuthology.orchestra.run.smithi059.stdout:(90/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-28T17:56:32.585 INFO:teuthology.orchestra.run.smithi037.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-28T17:56:32.611 INFO:teuthology.orchestra.run.smithi037.stdout:(85/174): python3-pycparser-2.20-6.el9.noarch.r 2.5 MB/s | 124 kB 00:00 2024-03-28T17:56:32.644 INFO:teuthology.orchestra.run.smithi037.stdout:(86/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2024-03-28T17:56:32.827 INFO:teuthology.orchestra.run.smithi059.stdout:(91/174): elfutils-libelf-devel-0.190-2.el9.x86 76 kB/s | 21 kB 00:00 2024-03-28T17:56:32.844 INFO:teuthology.orchestra.run.smithi059.stdout:(92/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 92 kB/s | 25 kB 00:00 2024-03-28T17:56:32.961 INFO:teuthology.orchestra.run.smithi037.stdout:(87/174): jq-1.6-16.el9.x86_64.rpm 538 kB/s | 187 kB 00:00 2024-03-28T17:56:33.020 INFO:teuthology.orchestra.run.smithi037.stdout:(88/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-03-28T17:56:33.070 INFO:teuthology.orchestra.run.smithi037.stdout:(89/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 195 kB 00:00 2024-03-28T17:56:33.129 INFO:teuthology.orchestra.run.smithi037.stdout:(90/174): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-03-28T17:56:33.196 INFO:teuthology.orchestra.run.smithi037.stdout:(91/174): elfutils-libelf-devel-0.190-2.el9.x86 173 kB/s | 21 kB 00:00 2024-03-28T17:56:33.221 INFO:teuthology.orchestra.run.smithi037.stdout:(92/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 280 kB/s | 25 kB 00:00 2024-03-28T17:56:33.461 INFO:teuthology.orchestra.run.smithi059.stdout:(93/174): postgresql-13.14-1.el9.x86_64.rpm 2.6 MB/s | 1.6 MB 00:00 2024-03-28T17:56:33.586 INFO:teuthology.orchestra.run.smithi059.stdout:(94/174): postgresql-contrib-13.14-1.el9.x86_64 7.1 MB/s | 911 kB 00:00 2024-03-28T17:56:33.694 INFO:teuthology.orchestra.run.smithi059.stdout:(95/174): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-03-28T17:56:33.722 INFO:teuthology.orchestra.run.smithi037.stdout:(93/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 7.9 MB/s | 4.1 MB 00:00 2024-03-28T17:56:33.771 INFO:teuthology.orchestra.run.smithi037.stdout:(94/174): postgresql-13.14-1.el9.x86_64.rpm 2.9 MB/s | 1.6 MB 00:00 2024-03-28T17:56:33.830 INFO:teuthology.orchestra.run.smithi037.stdout:(95/174): postgresql-private-libs-13.14-1.el9.x 2.3 MB/s | 133 kB 00:00 2024-03-28T17:56:33.872 INFO:teuthology.orchestra.run.smithi037.stdout:(96/174): postgresql-contrib-13.14-1.el9.x86_64 6.0 MB/s | 911 kB 00:00 2024-03-28T17:56:33.931 INFO:teuthology.orchestra.run.smithi037.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 4.2 MB/s | 245 kB 00:00 2024-03-28T17:56:33.998 INFO:teuthology.orchestra.run.smithi037.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 3.7 MB/s | 248 kB 00:00 2024-03-28T17:56:34.056 INFO:teuthology.orchestra.run.smithi037.stdout:(99/174): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-03-28T17:56:34.107 INFO:teuthology.orchestra.run.smithi037.stdout:(100/174): qatlib-service-24.02.0-1.el9.x86_64. 728 kB/s | 36 kB 00:00 2024-03-28T17:56:34.157 INFO:teuthology.orchestra.run.smithi037.stdout:(101/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 965 kB/s | 47 kB 00:00 2024-03-28T17:56:34.216 INFO:teuthology.orchestra.run.smithi037.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-28T17:56:34.266 INFO:teuthology.orchestra.run.smithi037.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 456 kB/s | 22 kB 00:00 2024-03-28T17:56:34.375 INFO:teuthology.orchestra.run.smithi037.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 7.9 MB/s | 863 kB 00:00 2024-03-28T17:56:34.553 INFO:teuthology.orchestra.run.smithi059.stdout:(96/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 2.4 MB/s | 4.1 MB 00:01 2024-03-28T17:56:34.820 INFO:teuthology.orchestra.run.smithi059.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 921 kB/s | 245 kB 00:00 2024-03-28T17:56:34.925 INFO:teuthology.orchestra.run.smithi037.stdout:(105/174): ceph-test-19.0.0-2145.g3be9a5c9.el9. 4.6 MB/s | 48 MB 00:10 2024-03-28T17:56:35.076 INFO:teuthology.orchestra.run.smithi037.stdout:(106/174): postgresql-server-13.14-1.el9.x86_64 4.6 MB/s | 5.8 MB 00:01 2024-03-28T17:56:35.087 INFO:teuthology.orchestra.run.smithi059.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 930 kB/s | 248 kB 00:00 2024-03-28T17:56:35.543 INFO:teuthology.orchestra.run.smithi037.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 48 kB/s | 19 kB 00:00 2024-03-28T17:56:35.579 INFO:teuthology.orchestra.run.smithi059.stdout:(99/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 4.8 MB/s | 48 MB 00:09 2024-03-28T17:56:35.604 INFO:teuthology.orchestra.run.smithi059.stdout:(100/174): qatlib-24.02.0-1.el9.x86_64.rpm 427 kB/s | 221 kB 00:00 2024-03-28T17:56:35.667 INFO:teuthology.orchestra.run.smithi037.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 450 kB/s | 309 kB 00:00 2024-03-28T17:56:35.701 INFO:teuthology.orchestra.run.smithi037.stdout:(109/174): abseil-cpp-20211102.0-3.el9.x86_64.r 413 kB/s | 548 kB 00:01 2024-03-28T17:56:35.713 INFO:teuthology.orchestra.run.smithi059.stdout:(101/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 437 kB/s | 47 kB 00:00 2024-03-28T17:56:35.772 INFO:teuthology.orchestra.run.smithi037.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 252 kB/s | 26 kB 00:00 2024-03-28T17:56:35.805 INFO:teuthology.orchestra.run.smithi037.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 471 kB/s | 49 kB 00:00 2024-03-28T17:56:35.821 INFO:teuthology.orchestra.run.smithi059.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 865 kB/s | 94 kB 00:00 2024-03-28T17:56:35.847 INFO:teuthology.orchestra.run.smithi059.stdout:(103/174): qatlib-service-24.02.0-1.el9.x86_64. 134 kB/s | 36 kB 00:00 2024-03-28T17:56:35.889 INFO:teuthology.orchestra.run.smithi037.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 578 kB/s | 67 kB 00:00 2024-03-28T17:56:35.931 INFO:teuthology.orchestra.run.smithi037.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-03-28T17:56:36.114 INFO:teuthology.orchestra.run.smithi037.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 2.9 MB/s | 548 kB 00:00 2024-03-28T17:56:36.122 INFO:teuthology.orchestra.run.smithi059.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 74 kB/s | 22 kB 00:00 2024-03-28T17:56:36.165 INFO:teuthology.orchestra.run.smithi037.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.0 MB/s | 838 kB 00:00 2024-03-28T17:56:36.224 INFO:teuthology.orchestra.run.smithi037.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 401 kB/s | 43 kB 00:00 2024-03-28T17:56:36.230 INFO:teuthology.orchestra.run.smithi059.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 4.9 MB/s | 548 kB 00:00 2024-03-28T17:56:36.265 INFO:teuthology.orchestra.run.smithi037.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 324 kB/s | 32 kB 00:00 2024-03-28T17:56:36.272 INFO:teuthology.orchestra.run.smithi059.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 7.2 MB/s | 309 kB 00:00 2024-03-28T17:56:36.349 INFO:teuthology.orchestra.run.smithi037.stdout:(118/174): libarrow-9.0.0-10.el9.x86_64.rpm 5.5 MB/s | 4.4 MB 00:00 2024-03-28T17:56:36.390 INFO:teuthology.orchestra.run.smithi059.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 166 kB/s | 19 kB 00:00 2024-03-28T17:56:36.399 INFO:teuthology.orchestra.run.smithi037.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 86 kB/s | 15 kB 00:00 2024-03-28T17:56:36.433 INFO:teuthology.orchestra.run.smithi037.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.0 MB/s | 172 kB 00:00 2024-03-28T17:56:36.483 INFO:teuthology.orchestra.run.smithi037.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-03-28T17:56:36.517 INFO:teuthology.orchestra.run.smithi037.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-03-28T17:56:36.523 INFO:teuthology.orchestra.run.smithi059.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 33 MB/s | 4.4 MB 00:00 2024-03-28T17:56:36.542 INFO:teuthology.orchestra.run.smithi037.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 327 kB/s | 35 kB 00:00 2024-03-28T17:56:36.557 INFO:teuthology.orchestra.run.smithi059.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 784 kB/s | 26 kB 00:00 2024-03-28T17:56:36.590 INFO:teuthology.orchestra.run.smithi059.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-03-28T17:56:36.600 INFO:teuthology.orchestra.run.smithi037.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.7 MB/s | 203 kB 00:00 2024-03-28T17:56:36.624 INFO:teuthology.orchestra.run.smithi059.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-03-28T17:56:36.651 INFO:teuthology.orchestra.run.smithi037.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-03-28T17:56:36.657 INFO:teuthology.orchestra.run.smithi059.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 4.4 MB/s | 151 kB 00:00 2024-03-28T17:56:36.708 INFO:teuthology.orchestra.run.smithi059.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 16 MB/s | 838 kB 00:00 2024-03-28T17:56:36.735 INFO:teuthology.orchestra.run.smithi037.stdout:(126/174): python3-grpcio-1.46.7-10.el9.x86_64. 9.4 MB/s | 2.0 MB 00:00 2024-03-28T17:56:36.758 INFO:teuthology.orchestra.run.smithi059.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 11 MB/s | 548 kB 00:00 2024-03-28T17:56:36.760 INFO:teuthology.orchestra.run.smithi037.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 67 kB/s | 11 kB 00:00 2024-03-28T17:56:36.783 INFO:teuthology.orchestra.run.smithi059.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-03-28T17:56:36.785 INFO:teuthology.orchestra.run.smithi037.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 133 kB/s | 18 kB 00:00 2024-03-28T17:56:36.809 INFO:teuthology.orchestra.run.smithi059.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-03-28T17:56:36.835 INFO:teuthology.orchestra.run.smithi037.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 233 kB/s | 23 kB 00:00 2024-03-28T17:56:36.842 INFO:teuthology.orchestra.run.smithi059.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 450 kB/s | 15 kB 00:00 2024-03-28T17:56:36.861 INFO:teuthology.orchestra.run.smithi037.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 195 kB/s | 19 kB 00:00 2024-03-28T17:56:36.876 INFO:teuthology.orchestra.run.smithi059.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 5.1 MB/s | 172 kB 00:00 2024-03-28T17:56:36.926 INFO:teuthology.orchestra.run.smithi059.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-03-28T17:56:36.936 INFO:teuthology.orchestra.run.smithi037.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 131 kB/s | 20 kB 00:00 2024-03-28T17:56:36.961 INFO:teuthology.orchestra.run.smithi037.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 466 kB/s | 46 kB 00:00 2024-03-28T17:56:37.002 INFO:teuthology.orchestra.run.smithi059.stdout:(120/174): postgresql-server-13.14-1.el9.x86_64 1.7 MB/s | 5.8 MB 00:03 2024-03-28T17:56:37.027 INFO:teuthology.orchestra.run.smithi059.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-03-28T17:56:37.037 INFO:teuthology.orchestra.run.smithi037.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 792 kB/s | 79 kB 00:00 2024-03-28T17:56:37.060 INFO:teuthology.orchestra.run.smithi059.stdout:(122/174): python3-google-auth-2.28.2-1.el9.noa 6.0 MB/s | 203 kB 00:00 2024-03-28T17:56:37.077 INFO:teuthology.orchestra.run.smithi059.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 469 kB/s | 35 kB 00:00 2024-03-28T17:56:37.079 INFO:teuthology.orchestra.run.smithi037.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 4.2 MB/s | 1.0 MB 00:00 2024-03-28T17:56:37.104 INFO:teuthology.orchestra.run.smithi037.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 407 kB/s | 58 kB 00:00 2024-03-28T17:56:37.146 INFO:teuthology.orchestra.run.smithi037.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-03-28T17:56:37.178 INFO:teuthology.orchestra.run.smithi059.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 17 MB/s | 2.0 MB 00:00 2024-03-28T17:56:37.179 INFO:teuthology.orchestra.run.smithi037.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 165 kB/s | 16 kB 00:00 2024-03-28T17:56:37.203 INFO:teuthology.orchestra.run.smithi059.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-03-28T17:56:37.205 INFO:teuthology.orchestra.run.smithi037.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 899 kB/s | 90 kB 00:00 2024-03-28T17:56:37.220 INFO:teuthology.orchestra.run.smithi059.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 256 kB/s | 11 kB 00:00 2024-03-28T17:56:37.237 INFO:teuthology.orchestra.run.smithi059.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 528 kB/s | 18 kB 00:00 2024-03-28T17:56:37.254 INFO:teuthology.orchestra.run.smithi059.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 689 kB/s | 23 kB 00:00 2024-03-28T17:56:37.263 INFO:teuthology.orchestra.run.smithi037.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 242 kB/s | 28 kB 00:00 2024-03-28T17:56:37.271 INFO:teuthology.orchestra.run.smithi059.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 582 kB/s | 19 kB 00:00 2024-03-28T17:56:37.289 INFO:teuthology.orchestra.run.smithi037.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 285 kB/s | 31 kB 00:00 2024-03-28T17:56:37.296 INFO:teuthology.orchestra.run.smithi059.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 467 kB/s | 20 kB 00:00 2024-03-28T17:56:37.322 INFO:teuthology.orchestra.run.smithi037.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-03-28T17:56:37.330 INFO:teuthology.orchestra.run.smithi059.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 17 MB/s | 1.0 MB 00:00 2024-03-28T17:56:37.355 INFO:teuthology.orchestra.run.smithi059.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 794 kB/s | 46 kB 00:00 2024-03-28T17:56:37.356 INFO:teuthology.orchestra.run.smithi037.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 648 kB/s | 59 kB 00:00 2024-03-28T17:56:37.380 INFO:teuthology.orchestra.run.smithi059.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-03-28T17:56:37.381 INFO:teuthology.orchestra.run.smithi037.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 391 kB/s | 36 kB 00:00 2024-03-28T17:56:37.406 INFO:teuthology.orchestra.run.smithi059.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-03-28T17:56:37.414 INFO:teuthology.orchestra.run.smithi037.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 557 kB/s | 51 kB 00:00 2024-03-28T17:56:37.431 INFO:teuthology.orchestra.run.smithi059.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-28T17:56:37.456 INFO:teuthology.orchestra.run.smithi059.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-03-28T17:56:37.482 INFO:teuthology.orchestra.run.smithi037.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-28T17:56:37.482 INFO:teuthology.orchestra.run.smithi059.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-03-28T17:56:37.507 INFO:teuthology.orchestra.run.smithi059.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-03-28T17:56:37.507 INFO:teuthology.orchestra.run.smithi037.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 718 kB/s | 90 kB 00:00 2024-03-28T17:56:37.532 INFO:teuthology.orchestra.run.smithi059.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-03-28T17:56:37.549 INFO:teuthology.orchestra.run.smithi037.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-03-28T17:56:37.566 INFO:teuthology.orchestra.run.smithi059.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-03-28T17:56:37.582 INFO:teuthology.orchestra.run.smithi037.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 200 kB/s | 20 kB 00:00 2024-03-28T17:56:37.591 INFO:teuthology.orchestra.run.smithi059.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-28T17:56:37.616 INFO:teuthology.orchestra.run.smithi037.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-28T17:56:37.616 INFO:teuthology.orchestra.run.smithi059.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 712 kB/s | 36 kB 00:00 2024-03-28T17:56:37.641 INFO:teuthology.orchestra.run.smithi059.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-28T17:56:37.667 INFO:teuthology.orchestra.run.smithi059.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-28T17:56:37.692 INFO:teuthology.orchestra.run.smithi059.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-03-28T17:56:37.717 INFO:teuthology.orchestra.run.smithi059.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 8.3 MB/s | 427 kB 00:00 2024-03-28T17:56:37.742 INFO:teuthology.orchestra.run.smithi059.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-03-28T17:56:37.768 INFO:teuthology.orchestra.run.smithi059.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-03-28T17:56:37.775 INFO:teuthology.orchestra.run.smithi037.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2024-03-28T17:56:37.810 INFO:teuthology.orchestra.run.smithi059.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-03-28T17:56:37.817 INFO:teuthology.orchestra.run.smithi037.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.7 MB/s | 321 kB 00:00 2024-03-28T17:56:37.885 INFO:teuthology.orchestra.run.smithi059.stdout:(150/174): protobuf-compiler-3.14.0-13.el9.x86_ 424 kB/s | 863 kB 00:02 2024-03-28T17:56:37.935 INFO:teuthology.orchestra.run.smithi059.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-28T17:56:38.050 INFO:teuthology.orchestra.run.smithi037.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 9.9 MB/s | 2.3 MB 00:00 2024-03-28T17:56:38.192 INFO:teuthology.orchestra.run.smithi037.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 13 MB/s | 1.8 MB 00:00 2024-03-28T17:56:38.236 INFO:teuthology.orchestra.run.smithi059.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 7.7 MB/s | 2.3 MB 00:00 2024-03-28T17:56:38.259 INFO:teuthology.orchestra.run.smithi037.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 653 kB/s | 42 kB 00:00 2024-03-28T17:56:38.311 INFO:teuthology.orchestra.run.smithi059.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 25 MB/s | 1.8 MB 00:00 2024-03-28T17:56:38.343 INFO:teuthology.orchestra.run.smithi037.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-03-28T17:56:38.361 INFO:teuthology.orchestra.run.smithi059.stdout:(154/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 5.7 MB/s | 3.1 MB 00:00 2024-03-28T17:56:38.378 INFO:teuthology.orchestra.run.smithi059.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 629 kB/s | 42 kB 00:00 2024-03-28T17:56:38.418 INFO:teuthology.orchestra.run.smithi037.stdout:(156/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 3.9 MB/s | 3.1 MB 00:00 2024-03-28T17:56:38.454 INFO:teuthology.orchestra.run.smithi059.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 952 kB/s | 71 kB 00:00 2024-03-28T17:56:38.495 INFO:teuthology.orchestra.run.smithi059.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 874 kB/s | 36 kB 00:00 2024-03-28T17:56:38.518 INFO:teuthology.orchestra.run.smithi037.stdout:(157/174): librados2-19.0.0-2145.g3be9a5c9.el9. 3.6 MB/s | 3.4 MB 00:00 2024-03-28T17:56:38.548 INFO:teuthology.orchestra.run.smithi059.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.6 MB/s | 192 kB 00:00 2024-03-28T17:56:38.568 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-28T17:56:38.569 INFO:teuthology.orchestra.run.smithi037.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 318 kB/s | 71 kB 00:00 2024-03-28T17:56:38.573 INFO:teuthology.orchestra.run.smithi059.stdout:(159/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.1 MB/s | 248 kB 00:00 2024-03-28T17:56:38.599 INFO:teuthology.orchestra.run.smithi059.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 5.0 MB/s | 256 kB 00:00 2024-03-28T17:56:38.616 INFO:teuthology.orchestra.run.smithi059.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 550 kB/s | 23 kB 00:00 2024-03-28T17:56:38.649 INFO:teuthology.orchestra.run.smithi059.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-28T17:56:38.674 INFO:teuthology.orchestra.run.smithi059.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 155 kB 00:00 2024-03-28T17:56:38.693 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T17:56:38.708 INFO:teuthology.orchestra.run.smithi059.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 136 kB 00:00 2024-03-28T17:56:38.719 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T17:56:38.733 INFO:teuthology.orchestra.run.smithi059.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 64 kB 00:00 2024-03-28T17:56:38.759 INFO:teuthology.orchestra.run.smithi059.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 572 kB/s | 29 kB 00:00 2024-03-28T17:56:38.759 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T17:56:38.768 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T17:56:38.808 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T17:56:38.809 INFO:teuthology.orchestra.run.smithi059.stdout:(167/174): librados2-19.0.0-2145.g3be9a5c9.el9. 3.3 MB/s | 3.4 MB 00:01 2024-03-28T17:56:38.836 INFO:teuthology.orchestra.run.smithi037.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 608 kB/s | 192 kB 00:00 2024-03-28T17:56:38.859 INFO:teuthology.orchestra.run.smithi059.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 22 MB/s | 2.2 MB 00:00 2024-03-28T17:56:38.861 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T17:56:38.895 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T17:56:38.901 INFO:teuthology.orchestra.run.smithi059.stdout:(169/174): openssl-3.0.7-27.el9.x86_64.rpm 7.2 MB/s | 1.2 MB 00:00 2024-03-28T17:56:38.921 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T17:56:38.927 INFO:teuthology.orchestra.run.smithi059.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 6.8 MB/s | 465 kB 00:00 2024-03-28T17:56:38.936 INFO:teuthology.orchestra.run.smithi037.stdout:(160/174): elfutils-debuginfod-client-0.190-2.e 70 kB/s | 36 kB 00:00 2024-03-28T17:56:38.951 INFO:teuthology.orchestra.run.smithi059.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 621 kB/s | 31 kB 00:00 2024-03-28T17:56:38.962 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T17:56:38.986 INFO:teuthology.orchestra.run.smithi037.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 309 kB/s | 23 kB 00:00 2024-03-28T17:56:38.994 INFO:teuthology.orchestra.run.smithi059.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 16 MB/s | 1.1 MB 00:00 2024-03-28T17:56:39.046 INFO:teuthology.orchestra.run.smithi037.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.0 MB/s | 108 kB 00:00 2024-03-28T17:56:39.087 INFO:teuthology.orchestra.run.smithi037.stdout:(163/174): elfutils-libs-0.190-2.el9.x86_64.rpm 573 kB/s | 256 kB 00:00 2024-03-28T17:56:39.119 INFO:teuthology.orchestra.run.smithi059.stdout:(173/174): util-linux-2.37.4-18.el9.x86_64.rpm 7.4 MB/s | 2.3 MB 00:00 2024-03-28T17:56:39.121 INFO:teuthology.orchestra.run.smithi037.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 155 kB 00:00 2024-03-28T17:56:39.155 INFO:teuthology.orchestra.run.smithi037.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 136 kB 00:00 2024-03-28T17:56:39.180 INFO:teuthology.orchestra.run.smithi037.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 698 kB/s | 64 kB 00:00 2024-03-28T17:56:39.205 INFO:teuthology.orchestra.run.smithi037.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 346 kB/s | 29 kB 00:00 2024-03-28T17:56:39.227 INFO:teuthology.orchestra.run.smithi059.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 270 kB/s | 74 kB 00:00 2024-03-28T17:56:39.231 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-28T17:56:39.232 INFO:teuthology.orchestra.run.smithi059.stdout:Total 12 MB/s | 225 MB 00:19 2024-03-28T17:56:39.281 INFO:teuthology.orchestra.run.smithi037.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 9.7 MB/s | 1.2 MB 00:00 2024-03-28T17:56:39.284 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T17:56:39.355 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T17:56:39.398 INFO:teuthology.orchestra.run.smithi037.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 10 MB/s | 2.2 MB 00:00 2024-03-28T17:56:39.440 INFO:teuthology.orchestra.run.smithi037.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 2.9 MB/s | 465 kB 00:00 2024-03-28T17:56:39.476 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T17:56:39.506 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T17:56:39.507 INFO:teuthology.orchestra.run.smithi037.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 7.6 MB/s | 2.3 MB 00:00 2024-03-28T17:56:39.523 INFO:teuthology.orchestra.run.smithi037.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 249 kB/s | 31 kB 00:00 2024-03-28T17:56:39.591 INFO:teuthology.orchestra.run.smithi037.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.2 MB/s | 1.1 MB 00:00 2024-03-28T17:56:39.598 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T17:56:39.627 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T17:56:39.666 INFO:teuthology.orchestra.run.smithi037.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 472 kB/s | 74 kB 00:00 2024-03-28T17:56:39.668 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T17:56:39.683 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-28T17:56:39.683 INFO:teuthology.orchestra.run.smithi037.stdout:Total 11 MB/s | 225 MB 00:20 2024-03-28T17:56:39.817 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-28T17:56:39.834 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T17:56:39.883 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T17:56:39.911 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T17:56:40.000 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T17:56:40.023 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T17:56:40.070 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T17:56:40.110 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T17:56:40.115 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-28T17:56:40.115 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-28T17:56:40.132 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T17:56:40.148 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T17:56:40.341 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T17:56:40.372 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T17:56:40.404 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T17:56:40.513 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T17:56:40.607 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T17:56:40.667 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T17:56:40.731 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T17:56:40.801 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T17:56:41.024 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T17:56:41.135 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T17:56:41.237 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T17:56:41.264 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T17:56:41.291 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T17:56:41.327 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T17:56:41.371 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-28T17:56:41.402 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T17:56:41.455 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T17:56:41.494 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T17:56:41.558 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T17:56:41.592 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T17:56:41.661 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T17:56:41.693 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T17:56:41.730 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T17:56:41.773 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-28T17:56:41.773 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-28T17:56:41.799 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T17:56:41.835 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T17:56:41.878 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T17:56:41.907 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T17:56:41.965 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-28T17:56:41.966 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-28T17:56:41.975 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T17:56:42.021 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T17:56:42.095 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T17:56:42.229 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T17:56:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T17:56:42.807 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T17:56:42.886 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T17:56:42.963 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T17:56:43.023 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T17:56:43.033 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T17:56:43.071 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T17:56:43.121 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T17:56:43.158 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T17:56:43.188 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T17:56:43.420 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T17:56:43.591 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T17:56:43.665 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T17:56:43.701 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T17:56:43.841 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T17:56:43.895 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T17:56:43.951 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T17:56:44.023 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T17:56:44.058 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T17:56:44.094 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T17:56:44.142 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T17:56:44.261 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T17:56:44.405 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T17:56:44.421 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-28T17:56:44.422 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-28T17:56:44.447 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T17:56:44.594 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T17:56:44.643 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T17:56:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T17:56:44.715 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T17:56:44.818 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T17:56:44.856 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T17:56:44.897 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T17:56:44.922 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T17:56:44.947 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T17:56:45.018 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T17:56:45.150 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T17:56:45.254 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T17:56:45.297 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T17:56:45.346 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T17:56:45.401 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T17:56:45.585 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T17:56:45.643 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T17:56:45.783 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T17:56:45.824 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T17:56:45.856 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T17:56:45.888 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T17:56:45.915 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T17:56:45.950 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T17:56:45.989 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T17:56:46.021 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T17:56:46.050 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T17:56:46.127 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T17:56:46.161 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T17:56:46.194 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T17:56:46.235 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T17:56:46.270 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T17:56:46.360 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T17:56:46.450 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T17:56:46.502 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T17:56:46.533 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T17:56:46.580 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T17:56:46.678 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T17:56:46.734 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T17:56:47.470 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T17:56:47.515 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T17:56:47.549 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T17:56:47.597 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T17:56:47.686 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T17:56:47.725 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T17:56:47.910 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T17:56:47.939 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T17:56:48.017 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T17:56:48.666 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T17:56:48.779 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T17:56:49.412 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-28T17:56:49.531 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T17:56:49.590 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T17:56:49.621 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T17:56:49.630 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T17:56:49.671 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T17:56:49.733 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T17:56:49.767 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T17:56:49.792 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T17:56:49.834 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T17:56:50.009 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T17:56:50.099 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T17:56:50.129 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T17:56:50.161 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T17:56:50.188 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T17:56:50.238 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T17:56:50.435 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T17:56:50.504 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T17:56:50.535 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T17:56:50.558 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T17:56:50.585 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T17:56:50.609 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T17:56:50.640 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T17:56:50.661 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T17:56:50.676 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T17:56:50.698 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T17:56:50.744 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T17:56:50.931 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T17:56:50.980 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T17:56:51.007 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T17:56:51.104 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T17:56:51.133 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T17:56:51.143 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T17:56:51.171 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T17:56:51.202 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T17:56:51.237 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T17:56:51.242 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T17:56:51.259 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T17:56:51.273 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T17:56:51.276 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T17:56:51.468 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T17:56:51.498 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T17:56:51.539 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T17:56:51.771 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T17:56:51.858 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-28T17:56:51.874 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T17:56:51.918 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T17:56:51.948 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T17:56:51.996 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T17:56:52.002 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T17:56:52.034 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T17:56:52.043 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T17:56:52.079 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T17:56:52.174 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T17:56:52.244 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T17:56:52.351 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T17:56:52.409 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T17:56:52.457 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T17:56:52.469 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T17:56:52.498 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T17:56:52.525 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T17:56:52.569 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T17:56:52.646 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T17:56:52.656 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T17:56:52.706 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T17:56:52.744 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T17:56:52.796 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T17:56:52.810 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T17:56:52.843 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T17:56:52.895 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T17:56:52.936 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T17:56:52.964 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T17:56:53.035 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T17:56:53.202 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T17:56:53.231 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T17:56:53.246 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T17:56:53.299 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T17:56:53.317 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T17:56:53.370 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T17:56:53.413 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T17:56:53.491 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T17:56:53.509 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T17:56:53.528 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T17:56:53.559 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T17:56:53.561 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T17:56:53.636 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T17:56:53.693 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T17:56:53.726 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T17:56:53.900 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T17:56:53.986 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T17:56:54.098 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T17:56:54.139 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T17:56:54.221 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T17:56:54.221 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T17:56:54.221 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T17:56:54.221 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T17:56:54.249 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T17:56:54.261 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T17:56:54.299 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T17:56:54.339 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T17:56:54.357 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T17:56:54.381 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T17:56:54.417 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T17:56:54.464 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T17:56:54.473 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T17:56:54.504 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T17:56:54.562 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T17:56:54.600 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T17:56:54.600 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T17:56:54.629 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T17:56:54.740 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T17:56:54.842 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T17:56:54.867 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T17:56:54.885 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T17:56:54.909 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T17:56:54.974 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T17:56:55.032 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T17:56:55.112 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T17:56:55.157 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T17:56:55.248 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T17:56:55.298 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T17:56:55.351 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T17:56:55.401 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T17:56:55.408 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T17:56:55.450 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T17:56:55.481 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T17:56:55.527 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T17:56:55.563 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T17:56:55.611 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T17:56:55.646 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T17:56:55.690 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T17:56:55.740 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T17:56:55.763 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T17:56:55.908 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T17:56:55.915 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T17:56:55.962 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T17:56:55.992 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T17:56:56.005 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T17:56:56.034 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T17:56:56.134 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T17:56:56.166 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T17:56:56.182 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T17:56:56.207 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T17:56:56.232 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T17:56:56.235 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T17:56:56.265 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T17:56:56.349 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T17:56:56.482 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T17:56:56.595 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T17:56:56.613 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T17:56:56.657 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T17:56:56.716 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T17:56:56.779 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T17:56:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T17:56:56.961 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T17:56:56.975 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T17:56:57.036 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T17:56:57.091 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T17:56:57.160 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T17:56:57.183 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T17:56:57.226 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T17:56:57.257 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T17:56:57.258 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T17:56:57.289 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T17:56:57.317 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T17:56:57.351 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T17:56:57.391 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T17:56:57.393 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T17:56:57.431 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T17:56:57.460 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T17:56:57.462 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T17:56:57.509 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T17:56:57.541 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T17:56:57.579 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T17:56:57.612 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T17:56:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T17:56:57.654 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T17:56:57.664 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T17:56:57.688 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T17:56:57.728 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T17:56:57.778 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T17:56:57.788 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T17:56:57.817 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T17:56:57.878 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T17:56:57.928 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T17:56:57.935 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T17:56:57.959 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T17:56:57.970 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T17:56:58.015 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T17:56:58.032 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T17:56:58.061 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T17:56:58.118 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T17:56:58.164 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T17:56:58.178 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T17:56:58.222 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T17:56:58.349 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T17:56:58.541 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T17:56:58.608 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T17:56:58.953 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T17:56:58.998 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T17:56:59.033 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T17:56:59.082 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T17:56:59.172 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T17:56:59.208 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T17:56:59.333 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T17:56:59.391 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T17:56:59.422 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T17:56:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T17:56:59.501 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T17:56:59.576 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T17:56:59.613 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T17:56:59.645 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T17:56:59.687 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T17:56:59.757 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T17:56:59.807 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T17:56:59.858 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T17:57:00.176 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T17:57:00.290 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T17:57:00.305 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T17:57:00.543 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T17:57:00.620 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T17:57:00.688 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T17:57:00.910 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T17:57:00.964 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T17:57:01.044 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T17:57:01.147 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T17:57:01.180 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T17:57:01.220 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T17:57:01.289 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T17:57:01.328 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T17:57:01.393 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T17:57:01.461 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T17:57:01.583 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T17:57:01.653 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T17:57:01.664 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T17:57:01.703 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T17:57:01.717 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T17:57:01.734 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T17:57:01.755 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T17:57:01.761 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T17:57:01.815 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T17:57:01.959 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T17:57:01.995 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T17:57:02.042 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T17:57:02.089 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T17:57:02.091 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T17:57:02.124 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T17:57:02.128 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T17:57:02.216 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T17:57:02.246 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T17:57:02.258 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T17:57:02.282 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T17:57:02.452 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T17:57:02.596 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T17:57:02.653 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T17:57:02.717 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T17:57:02.769 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T17:57:02.792 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T17:57:02.794 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T17:57:02.866 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T17:57:02.896 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T17:57:03.051 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T17:57:03.115 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T17:57:03.290 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T17:57:03.337 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T17:57:03.386 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T17:57:03.425 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T17:57:03.476 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T17:57:03.523 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T17:57:03.593 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T17:57:03.648 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T17:57:03.713 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T17:57:03.860 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T17:57:03.900 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T17:57:03.943 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T17:57:04.018 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T17:57:04.072 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T17:57:04.230 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T17:57:04.353 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T17:57:04.408 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T17:57:04.444 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T17:57:04.502 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T17:57:04.645 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T17:57:04.719 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T17:57:05.252 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T17:57:05.292 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T17:57:05.752 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T17:57:05.809 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T17:57:05.842 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T17:57:05.858 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T17:57:05.895 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T17:57:05.935 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T17:57:05.994 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T17:57:05.994 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T17:57:05.994 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T17:57:05.994 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-28T17:57:06.021 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T17:57:06.050 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T17:57:06.092 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T17:57:06.341 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T17:57:06.372 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T17:57:06.395 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T17:57:06.395 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /sys 2024-03-28T17:57:06.395 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /proc 2024-03-28T17:57:06.395 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /mnt 2024-03-28T17:57:06.396 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /var/tmp 2024-03-28T17:57:06.396 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /home 2024-03-28T17:57:06.396 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /root 2024-03-28T17:57:06.396 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /tmp 2024-03-28T17:57:06.396 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T17:57:06.469 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T17:57:06.572 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T17:57:06.590 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T17:57:06.590 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:06.590 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T17:57:06.590 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T17:57:07.318 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T17:57:07.457 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T17:57:08.058 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T17:57:08.084 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T17:57:08.084 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:08.084 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T17:57:08.084 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T17:57:08.084 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T17:57:08.084 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T17:57:08.301 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T17:57:08.324 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T17:57:08.324 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:08.324 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T17:57:08.324 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T17:57:08.324 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T17:57:08.324 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T17:57:08.768 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T17:57:08.791 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T17:57:08.791 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:08.791 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T17:57:08.792 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T17:57:08.792 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T17:57:08.792 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T17:57:08.878 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T17:57:09.092 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T17:57:09.201 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T17:57:09.239 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T17:57:09.270 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T17:57:09.296 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T17:57:09.311 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T17:57:09.327 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T17:57:09.358 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T17:57:09.418 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T17:57:09.441 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T17:57:09.674 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T17:57:09.701 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T17:57:09.718 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T17:57:09.866 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T17:57:09.981 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T17:57:10.016 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T17:57:10.088 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T17:57:10.658 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T17:57:10.675 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T17:57:10.684 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T17:57:10.711 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T17:57:10.784 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T17:57:10.795 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T17:57:10.821 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T17:57:10.946 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T17:57:10.970 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T17:57:10.993 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T17:57:10.993 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:10.993 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T17:57:10.993 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T17:57:10.993 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T17:57:10.994 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T17:57:11.037 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T17:57:11.048 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T17:57:11.094 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T17:57:11.970 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T17:57:11.998 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T17:57:12.021 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T17:57:12.021 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:12.021 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T17:57:12.021 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T17:57:12.021 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T17:57:12.021 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T17:57:12.065 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T17:57:12.087 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T17:57:12.087 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:12.087 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T17:57:12.087 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T17:57:12.132 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T17:57:12.155 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T17:57:12.155 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:12.155 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T17:57:12.155 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T17:57:12.155 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T17:57:12.155 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T17:57:12.462 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T17:57:12.484 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T17:57:12.484 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:12.484 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T17:57:12.484 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T17:57:12.484 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T17:57:12.485 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T17:57:13.537 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T17:57:13.585 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T17:57:14.262 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T17:57:14.304 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T17:57:14.459 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T17:57:14.459 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T17:57:14.460 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T17:57:14.460 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:57:14.505 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T17:57:16.937 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T17:57:16.979 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T17:57:17.009 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T17:57:17.036 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T17:57:17.220 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T17:57:17.328 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T17:57:17.368 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T17:57:17.407 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T17:57:17.468 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T17:57:17.511 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T17:57:17.561 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T17:57:17.597 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T17:57:17.633 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T17:57:17.679 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T17:57:17.739 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T17:57:17.774 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T17:57:18.087 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T17:57:18.087 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /sys 2024-03-28T17:57:18.088 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /proc 2024-03-28T17:57:18.088 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /mnt 2024-03-28T17:57:18.088 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /var/tmp 2024-03-28T17:57:18.088 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /home 2024-03-28T17:57:18.088 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /root 2024-03-28T17:57:18.088 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /tmp 2024-03-28T17:57:18.088 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-28T17:57:18.257 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T17:57:18.273 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T17:57:18.273 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:18.274 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T17:57:18.274 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-28T17:57:18.288 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T17:57:18.330 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T17:57:18.350 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T17:57:19.169 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T17:57:19.250 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T17:57:19.291 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T17:57:19.371 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T17:57:19.404 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T17:57:19.429 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T17:57:19.454 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T17:57:19.481 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T17:57:19.481 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T17:57:19.568 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T17:57:19.599 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T17:57:19.599 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T17:57:19.675 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T17:57:19.701 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T17:57:19.742 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T17:57:19.768 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T17:57:19.793 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T17:57:19.802 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T17:57:19.825 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T17:57:19.829 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T17:57:19.829 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:19.829 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T17:57:19.829 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T17:57:19.829 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T17:57:19.830 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-28T17:57:19.874 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T17:57:19.906 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T17:57:19.931 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T17:57:19.956 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T17:57:19.982 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T17:57:20.007 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T17:57:20.014 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T17:57:20.014 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T17:57:20.061 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T17:57:20.084 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T17:57:20.084 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:20.085 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T17:57:20.085 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T17:57:20.085 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T17:57:20.085 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-28T17:57:20.193 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T17:57:20.193 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T17:57:20.268 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T17:57:20.293 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T17:57:20.317 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T17:57:20.344 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T17:57:20.369 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T17:57:20.434 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T17:57:20.550 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T17:57:20.573 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T17:57:20.573 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:20.573 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T17:57:20.574 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T17:57:20.574 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T17:57:20.574 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-28T17:57:20.680 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T17:57:21.032 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T17:57:21.061 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T17:57:21.103 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T17:57:21.164 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T17:57:21.193 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T17:57:21.438 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T17:57:21.477 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T17:57:22.491 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T17:57:22.519 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T17:57:22.623 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T17:57:22.789 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T17:57:22.821 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T17:57:22.845 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T17:57:22.846 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:22.846 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T17:57:22.846 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T17:57:22.846 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T17:57:22.846 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-28T17:57:22.907 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T17:57:22.918 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T17:57:22.962 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T17:57:23.901 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T17:57:23.933 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T17:57:23.956 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T17:57:23.956 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:23.956 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T17:57:23.956 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T17:57:23.957 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T17:57:23.957 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-28T17:57:24.008 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T17:57:24.031 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T17:57:24.031 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:24.031 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T17:57:24.031 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-28T17:57:24.077 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T17:57:24.101 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T17:57:24.101 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:24.101 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T17:57:24.101 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T17:57:24.101 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T17:57:24.102 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-28T17:57:24.419 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T17:57:24.443 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T17:57:24.443 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:24.443 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T17:57:24.443 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T17:57:24.443 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T17:57:24.443 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-28T17:57:25.199 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T17:57:25.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T17:57:25.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T17:57:25.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T17:57:25.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T17:57:25.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T17:57:25.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T17:57:25.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T17:57:25.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T17:57:25.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T17:57:25.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T17:57:25.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T17:57:25.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T17:57:25.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T17:57:25.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T17:57:25.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T17:57:25.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T17:57:25.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T17:57:25.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T17:57:25.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T17:57:25.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T17:57:25.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T17:57:25.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T17:57:25.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T17:57:25.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T17:57:25.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T17:57:25.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T17:57:25.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T17:57:25.202 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T17:57:25.202 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T17:57:25.202 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T17:57:25.219 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T17:57:25.219 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T17:57:25.219 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T17:57:25.219 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T17:57:25.219 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T17:57:25.219 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T17:57:25.220 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T17:57:25.220 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T17:57:25.220 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T17:57:25.220 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T17:57:25.220 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T17:57:25.220 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T17:57:25.220 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T17:57:25.220 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T17:57:25.220 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T17:57:25.220 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T17:57:25.220 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T17:57:25.220 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T17:57:25.220 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T17:57:25.220 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T17:57:25.220 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T17:57:25.220 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T17:57:25.221 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T17:57:25.221 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T17:57:25.221 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T17:57:25.221 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T17:57:25.221 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T17:57:25.221 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T17:57:25.221 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T17:57:25.221 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T17:57:25.221 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T17:57:25.221 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T17:57:25.221 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T17:57:25.221 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T17:57:25.221 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T17:57:25.221 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T17:57:25.221 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T17:57:25.221 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T17:57:25.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T17:57:25.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T17:57:25.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T17:57:25.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T17:57:25.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T17:57:25.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T17:57:25.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T17:57:25.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T17:57:25.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T17:57:25.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T17:57:25.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T17:57:25.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T17:57:25.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T17:57:25.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T17:57:25.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T17:57:25.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T17:57:25.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T17:57:25.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T17:57:25.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T17:57:25.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T17:57:25.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T17:57:25.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T17:57:25.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T17:57:25.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T17:57:25.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T17:57:25.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T17:57:25.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T17:57:25.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T17:57:25.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T17:57:25.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T17:57:25.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T17:57:25.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T17:57:25.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T17:57:25.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T17:57:25.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T17:57:25.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T17:57:25.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T17:57:25.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T17:57:25.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T17:57:25.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T17:57:25.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T17:57:25.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T17:57:25.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T17:57:25.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T17:57:25.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T17:57:25.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T17:57:25.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T17:57:25.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T17:57:25.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T17:57:25.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T17:57:25.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T17:57:25.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T17:57:25.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T17:57:25.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T17:57:25.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T17:57:25.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T17:57:25.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T17:57:25.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T17:57:25.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T17:57:25.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T17:57:25.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T17:57:25.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T17:57:25.226 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T17:57:25.226 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T17:57:25.226 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T17:57:25.226 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T17:57:25.226 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T17:57:25.226 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T17:57:25.226 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T17:57:25.226 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T17:57:25.226 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T17:57:25.226 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T17:57:25.226 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T17:57:25.226 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T17:57:25.227 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T17:57:25.227 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T17:57:25.227 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T17:57:25.227 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T17:57:25.227 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T17:57:25.227 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T17:57:25.227 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T17:57:25.227 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T17:57:25.227 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T17:57:25.227 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T17:57:25.227 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T17:57:25.227 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T17:57:25.227 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T17:57:25.227 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T17:57:25.227 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T17:57:25.228 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T17:57:25.228 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T17:57:25.228 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T17:57:25.228 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T17:57:25.228 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T17:57:25.228 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T17:57:25.228 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T17:57:25.228 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T17:57:25.228 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T17:57:25.228 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T17:57:25.228 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T17:57:25.228 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T17:57:25.228 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T17:57:25.228 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T17:57:25.228 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T17:57:25.229 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T17:57:25.229 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T17:57:25.229 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T17:57:25.229 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T17:57:25.229 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T17:57:25.229 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T17:57:25.229 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T17:57:25.229 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T17:57:25.229 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T17:57:25.229 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T17:57:25.229 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T17:57:25.229 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T17:57:25.229 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T17:57:25.229 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T17:57:25.229 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T17:57:25.229 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T17:57:25.230 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T17:57:25.230 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T17:57:25.230 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T17:57:25.230 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T17:57:25.230 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T17:57:25.230 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T17:57:25.230 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T17:57:25.230 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T17:57:25.230 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T17:57:27.774 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T17:57:27.775 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /sys 2024-03-28T17:57:27.775 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /proc 2024-03-28T17:57:27.775 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /mnt 2024-03-28T17:57:27.775 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /var/tmp 2024-03-28T17:57:27.775 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /home 2024-03-28T17:57:27.775 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /root 2024-03-28T17:57:27.775 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /tmp 2024-03-28T17:57:27.775 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:57:27.994 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T17:57:28.051 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T17:57:28.052 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:28.052 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T17:57:28.052 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:57:28.305 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T17:57:28.305 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T17:57:28.305 INFO:teuthology.orchestra.run.smithi170.stdout:Upgraded: 2024-03-28T17:57:28.305 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T17:57:28.306 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T17:57:28.306 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T17:57:28.306 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T17:57:28.306 INFO:teuthology.orchestra.run.smithi170.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T17:57:28.306 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T17:57:28.306 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T17:57:28.306 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T17:57:28.306 INFO:teuthology.orchestra.run.smithi170.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T17:57:28.306 INFO:teuthology.orchestra.run.smithi170.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T17:57:28.306 INFO:teuthology.orchestra.run.smithi170.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T17:57:28.306 INFO:teuthology.orchestra.run.smithi170.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T17:57:28.306 INFO:teuthology.orchestra.run.smithi170.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T17:57:28.306 INFO:teuthology.orchestra.run.smithi170.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.306 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.307 INFO:teuthology.orchestra.run.smithi170.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T17:57:28.307 INFO:teuthology.orchestra.run.smithi170.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T17:57:28.307 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T17:57:28.307 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T17:57:28.307 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T17:57:28.307 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T17:57:28.307 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T17:57:28.307 INFO:teuthology.orchestra.run.smithi170.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T17:57:28.307 INFO:teuthology.orchestra.run.smithi170.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T17:57:28.307 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-28T17:57:28.307 INFO:teuthology.orchestra.run.smithi170.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T17:57:28.307 INFO:teuthology.orchestra.run.smithi170.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T17:57:28.307 INFO:teuthology.orchestra.run.smithi170.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T17:57:28.307 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T17:57:28.307 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.308 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.308 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.308 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.308 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:28.308 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.308 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.308 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.308 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:28.308 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:28.308 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:28.308 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:28.308 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:28.308 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.308 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.309 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:28.309 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.309 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.309 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.309 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:28.309 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:28.309 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.309 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.309 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:28.309 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T17:57:28.309 INFO:teuthology.orchestra.run.smithi170.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T17:57:28.309 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T17:57:28.309 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T17:57:28.309 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T17:57:28.309 INFO:teuthology.orchestra.run.smithi170.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T17:57:28.310 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T17:57:28.310 INFO:teuthology.orchestra.run.smithi170.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T17:57:28.310 INFO:teuthology.orchestra.run.smithi170.stdout: jq-1.6-16.el9.x86_64 2024-03-28T17:57:28.310 INFO:teuthology.orchestra.run.smithi170.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T17:57:28.310 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T17:57:28.310 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T17:57:28.310 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.310 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.310 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.310 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T17:57:28.310 INFO:teuthology.orchestra.run.smithi170.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T17:57:28.310 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T17:57:28.310 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T17:57:28.311 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T17:57:28.311 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.311 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.311 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T17:57:28.311 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.311 INFO:teuthology.orchestra.run.smithi170.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T17:57:28.311 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T17:57:28.311 INFO:teuthology.orchestra.run.smithi170.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T17:57:28.311 INFO:teuthology.orchestra.run.smithi170.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T17:57:28.311 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T17:57:28.311 INFO:teuthology.orchestra.run.smithi170.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T17:57:28.311 INFO:teuthology.orchestra.run.smithi170.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T17:57:28.311 INFO:teuthology.orchestra.run.smithi170.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T17:57:28.311 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T17:57:28.311 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T17:57:28.312 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T17:57:28.312 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T17:57:28.312 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T17:57:28.312 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T17:57:28.312 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T17:57:28.312 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T17:57:28.312 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T17:57:28.312 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T17:57:28.312 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T17:57:28.312 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T17:57:28.312 INFO:teuthology.orchestra.run.smithi170.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T17:57:28.312 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T17:57:28.312 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T17:57:28.312 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T17:57:28.312 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.313 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.313 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.313 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T17:57:28.313 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T17:57:28.313 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T17:57:28.313 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T17:57:28.313 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T17:57:28.313 INFO:teuthology.orchestra.run.smithi170.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T17:57:28.313 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T17:57:28.313 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T17:57:28.313 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T17:57:28.313 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T17:57:28.313 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T17:57:28.313 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T17:57:28.313 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T17:57:28.313 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T17:57:28.314 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T17:57:28.314 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T17:57:28.314 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T17:57:28.314 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T17:57:28.314 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T17:57:28.314 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T17:57:28.314 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T17:57:28.314 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T17:57:28.314 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T17:57:28.314 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T17:57:28.314 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T17:57:28.314 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T17:57:28.314 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T17:57:28.314 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T17:57:28.314 INFO:teuthology.orchestra.run.smithi170.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T17:57:28.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T17:57:28.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T17:57:28.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T17:57:28.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T17:57:28.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T17:57:28.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T17:57:28.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T17:57:28.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T17:57:28.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T17:57:28.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T17:57:28.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T17:57:28.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T17:57:28.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T17:57:28.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T17:57:28.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T17:57:28.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T17:57:28.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T17:57:28.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T17:57:28.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T17:57:28.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T17:57:28.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T17:57:28.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T17:57:28.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T17:57:28.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T17:57:28.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T17:57:28.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T17:57:28.316 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T17:57:28.316 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T17:57:28.316 INFO:teuthology.orchestra.run.smithi170.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T17:57:28.317 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.317 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.317 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:28.317 INFO:teuthology.orchestra.run.smithi170.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T17:57:28.317 INFO:teuthology.orchestra.run.smithi170.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T17:57:28.317 INFO:teuthology.orchestra.run.smithi170.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T17:57:28.317 INFO:teuthology.orchestra.run.smithi170.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T17:57:28.317 INFO:teuthology.orchestra.run.smithi170.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T17:57:28.317 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T17:57:28.317 INFO:teuthology.orchestra.run.smithi170.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T17:57:28.317 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T17:57:28.317 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-28T17:57:28.768 DEBUG:teuthology.parallel:result is None 2024-03-28T17:57:29.149 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T17:57:29.190 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T17:57:29.220 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T17:57:29.249 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T17:57:29.448 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T17:57:29.567 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T17:57:29.612 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T17:57:29.651 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T17:57:29.712 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T17:57:29.721 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T17:57:29.755 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T17:57:29.788 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T17:57:29.814 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T17:57:29.814 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:29.814 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T17:57:29.815 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T17:57:29.815 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T17:57:29.815 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:57:29.819 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T17:57:29.850 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T17:57:29.914 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T17:57:29.964 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T17:57:30.006 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T17:57:30.079 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T17:57:30.150 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T17:57:30.151 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:30.151 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T17:57:30.151 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T17:57:30.151 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T17:57:30.151 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:57:30.571 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T17:57:30.604 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T17:57:30.624 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T17:57:30.680 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T17:57:30.752 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T17:57:30.753 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:30.753 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T17:57:30.753 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T17:57:30.753 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T17:57:30.753 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:57:30.882 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T17:57:31.512 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T17:57:31.590 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T17:57:31.631 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T17:57:31.651 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T17:57:31.683 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T17:57:31.714 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T17:57:31.734 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T17:57:31.743 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T17:57:31.768 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T17:57:31.794 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T17:57:31.820 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T17:57:31.820 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T17:57:31.838 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T17:57:31.870 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T17:57:31.917 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T17:57:31.946 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T17:57:31.946 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T17:57:32.014 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T17:57:32.049 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T17:57:32.082 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T17:57:32.107 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T17:57:32.141 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T17:57:32.183 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T17:57:32.213 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T17:57:32.221 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T17:57:32.253 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T17:57:32.263 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T17:57:32.287 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T17:57:32.312 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T17:57:32.338 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T17:57:32.371 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T17:57:32.378 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T17:57:32.378 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T17:57:32.582 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T17:57:32.583 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T17:57:32.648 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T17:57:32.673 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T17:57:32.706 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T17:57:32.733 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T17:57:32.758 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T17:57:32.825 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T17:57:33.418 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T17:57:33.454 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T17:57:33.605 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T17:57:33.802 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T17:57:33.831 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T17:57:33.900 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T17:57:33.900 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:33.900 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T17:57:33.901 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T17:57:33.901 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T17:57:33.901 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:57:33.957 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T17:57:33.995 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T17:57:34.093 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T17:57:35.117 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T17:57:35.151 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T17:57:35.237 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T17:57:35.237 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:35.237 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T17:57:35.237 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T17:57:35.237 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T17:57:35.237 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:57:35.285 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T17:57:35.381 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T17:57:35.381 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:35.381 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T17:57:35.381 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:57:35.452 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T17:57:35.560 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T17:57:35.560 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:35.560 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T17:57:35.561 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T17:57:35.561 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T17:57:35.561 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:57:35.944 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T17:57:36.027 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T17:57:36.027 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:57:36.027 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T17:57:36.027 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T17:57:36.027 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T17:57:36.027 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:57:37.517 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T17:57:37.517 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T17:57:37.517 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T17:57:37.517 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T17:57:37.517 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T17:57:37.517 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T17:57:37.517 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T17:57:37.517 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T17:57:37.518 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T17:57:37.518 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T17:57:37.518 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T17:57:37.518 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T17:57:37.518 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T17:57:37.518 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T17:57:37.518 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T17:57:37.518 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T17:57:37.518 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T17:57:37.518 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T17:57:37.518 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T17:57:37.518 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T17:57:37.518 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T17:57:37.518 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T17:57:37.519 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T17:57:37.519 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T17:57:37.519 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T17:57:37.519 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T17:57:37.519 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T17:57:37.519 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T17:57:37.519 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T17:57:37.534 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T17:57:37.534 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T17:57:37.534 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T17:57:37.534 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T17:57:37.534 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T17:57:37.534 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T17:57:37.534 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T17:57:37.534 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T17:57:37.534 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T17:57:37.534 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T17:57:37.535 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T17:57:37.535 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T17:57:37.535 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T17:57:37.535 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T17:57:37.535 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T17:57:37.535 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T17:57:37.535 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T17:57:37.535 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T17:57:37.535 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T17:57:37.535 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T17:57:37.535 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T17:57:37.535 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T17:57:37.536 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T17:57:37.536 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T17:57:37.536 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T17:57:37.536 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T17:57:37.536 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T17:57:37.536 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T17:57:37.536 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T17:57:37.536 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T17:57:37.536 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T17:57:37.536 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T17:57:37.536 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T17:57:37.536 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T17:57:37.537 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T17:57:37.537 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T17:57:37.537 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T17:57:37.537 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T17:57:37.537 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T17:57:37.537 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T17:57:37.537 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T17:57:37.537 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T17:57:37.537 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T17:57:37.537 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T17:57:37.537 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T17:57:37.537 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T17:57:37.537 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T17:57:37.537 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T17:57:37.538 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T17:57:37.538 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T17:57:37.538 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T17:57:37.538 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T17:57:37.538 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T17:57:37.538 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T17:57:37.538 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T17:57:37.538 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T17:57:37.538 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T17:57:37.538 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T17:57:37.538 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T17:57:37.538 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T17:57:37.539 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T17:57:37.539 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T17:57:37.539 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T17:57:37.539 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T17:57:37.539 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T17:57:37.539 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T17:57:37.539 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T17:57:37.539 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T17:57:37.539 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T17:57:37.539 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T17:57:37.539 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T17:57:37.539 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T17:57:37.539 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T17:57:37.539 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T17:57:37.540 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T17:57:37.540 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T17:57:37.540 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T17:57:37.540 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T17:57:37.540 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T17:57:37.540 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T17:57:37.540 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T17:57:37.540 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T17:57:37.540 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T17:57:37.540 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T17:57:37.540 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T17:57:37.540 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T17:57:37.540 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T17:57:37.540 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T17:57:37.540 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T17:57:37.540 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T17:57:37.541 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T17:57:37.541 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T17:57:37.541 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T17:57:37.541 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T17:57:37.541 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T17:57:37.541 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T17:57:37.541 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T17:57:37.541 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T17:57:37.541 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T17:57:37.541 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T17:57:37.541 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T17:57:37.541 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T17:57:37.541 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T17:57:37.541 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T17:57:37.541 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T17:57:37.542 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T17:57:37.542 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T17:57:37.542 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T17:57:37.542 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T17:57:37.542 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T17:57:37.542 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T17:57:37.542 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T17:57:37.542 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T17:57:37.542 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T17:57:37.542 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T17:57:37.542 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T17:57:37.542 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T17:57:37.542 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T17:57:37.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T17:57:37.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T17:57:37.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T17:57:37.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T17:57:37.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T17:57:37.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T17:57:37.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T17:57:37.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T17:57:37.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T17:57:37.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T17:57:37.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T17:57:37.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T17:57:37.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T17:57:37.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T17:57:37.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T17:57:37.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T17:57:37.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T17:57:37.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T17:57:37.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T17:57:37.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T17:57:37.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T17:57:37.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T17:57:37.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T17:57:37.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T17:57:37.545 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T17:57:37.545 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T17:57:37.545 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T17:57:37.545 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T17:57:37.545 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T17:57:37.545 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T17:57:37.545 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T17:57:37.545 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T17:57:37.545 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T17:57:37.545 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T17:57:37.545 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T17:57:37.545 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T17:57:37.545 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T17:57:37.545 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T17:57:37.546 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T17:57:37.546 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T17:57:37.546 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T17:57:37.546 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T17:57:37.546 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T17:57:37.546 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T17:57:37.546 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T17:57:37.546 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T17:57:37.546 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T17:57:37.546 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T17:57:37.546 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T17:57:37.546 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T17:57:37.546 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T17:57:40.787 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T17:57:40.787 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-28T17:57:40.787 INFO:teuthology.orchestra.run.smithi059.stdout:Upgraded: 2024-03-28T17:57:40.787 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T17:57:40.787 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T17:57:40.787 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T17:57:40.787 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T17:57:40.787 INFO:teuthology.orchestra.run.smithi059.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T17:57:40.788 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T17:57:40.788 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T17:57:40.788 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T17:57:40.788 INFO:teuthology.orchestra.run.smithi059.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T17:57:40.788 INFO:teuthology.orchestra.run.smithi059.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T17:57:40.788 INFO:teuthology.orchestra.run.smithi059.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T17:57:40.788 INFO:teuthology.orchestra.run.smithi059.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T17:57:40.788 INFO:teuthology.orchestra.run.smithi059.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T17:57:40.788 INFO:teuthology.orchestra.run.smithi059.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.788 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.788 INFO:teuthology.orchestra.run.smithi059.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T17:57:40.788 INFO:teuthology.orchestra.run.smithi059.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T17:57:40.788 INFO:teuthology.orchestra.run.smithi059.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T17:57:40.788 INFO:teuthology.orchestra.run.smithi059.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T17:57:40.788 INFO:teuthology.orchestra.run.smithi059.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T17:57:40.789 INFO:teuthology.orchestra.run.smithi059.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T17:57:40.789 INFO:teuthology.orchestra.run.smithi059.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T17:57:40.789 INFO:teuthology.orchestra.run.smithi059.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T17:57:40.789 INFO:teuthology.orchestra.run.smithi059.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T17:57:40.789 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-28T17:57:40.789 INFO:teuthology.orchestra.run.smithi059.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T17:57:40.789 INFO:teuthology.orchestra.run.smithi059.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T17:57:40.789 INFO:teuthology.orchestra.run.smithi059.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T17:57:40.789 INFO:teuthology.orchestra.run.smithi059.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T17:57:40.789 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.789 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.789 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.789 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.790 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:40.790 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.790 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.790 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.790 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:40.790 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:40.790 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:40.790 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:40.790 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:40.790 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.790 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.790 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:40.790 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.790 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.790 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.791 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:40.791 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:40.791 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.791 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.791 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:40.791 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T17:57:40.791 INFO:teuthology.orchestra.run.smithi059.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T17:57:40.791 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T17:57:40.791 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T17:57:40.791 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T17:57:40.791 INFO:teuthology.orchestra.run.smithi059.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T17:57:40.791 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T17:57:40.791 INFO:teuthology.orchestra.run.smithi059.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T17:57:40.791 INFO:teuthology.orchestra.run.smithi059.stdout: jq-1.6-16.el9.x86_64 2024-03-28T17:57:40.791 INFO:teuthology.orchestra.run.smithi059.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T17:57:40.792 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T17:57:40.792 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T17:57:40.792 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.792 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.792 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.792 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T17:57:40.792 INFO:teuthology.orchestra.run.smithi059.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T17:57:40.792 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T17:57:40.792 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T17:57:40.792 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T17:57:40.792 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.792 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.792 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T17:57:40.792 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.792 INFO:teuthology.orchestra.run.smithi059.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T17:57:40.793 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T17:57:40.793 INFO:teuthology.orchestra.run.smithi059.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T17:57:40.793 INFO:teuthology.orchestra.run.smithi059.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T17:57:40.793 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T17:57:40.793 INFO:teuthology.orchestra.run.smithi059.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T17:57:40.793 INFO:teuthology.orchestra.run.smithi059.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T17:57:40.793 INFO:teuthology.orchestra.run.smithi059.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T17:57:40.793 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T17:57:40.793 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T17:57:40.793 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T17:57:40.793 INFO:teuthology.orchestra.run.smithi059.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T17:57:40.793 INFO:teuthology.orchestra.run.smithi059.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T17:57:40.793 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T17:57:40.793 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T17:57:40.793 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T17:57:40.793 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T17:57:40.793 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T17:57:40.794 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T17:57:40.794 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T17:57:40.794 INFO:teuthology.orchestra.run.smithi059.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T17:57:40.794 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T17:57:40.794 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T17:57:40.794 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T17:57:40.794 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.794 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.794 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.794 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T17:57:40.794 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T17:57:40.794 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T17:57:40.794 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T17:57:40.794 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T17:57:40.795 INFO:teuthology.orchestra.run.smithi059.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T17:57:40.795 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T17:57:40.795 INFO:teuthology.orchestra.run.smithi059.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T17:57:40.795 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T17:57:40.795 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T17:57:40.795 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T17:57:40.795 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T17:57:40.795 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T17:57:40.795 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T17:57:40.795 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T17:57:40.795 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T17:57:40.795 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T17:57:40.795 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T17:57:40.795 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T17:57:40.795 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T17:57:40.796 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T17:57:40.796 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T17:57:40.796 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T17:57:40.796 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T17:57:40.796 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T17:57:40.796 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T17:57:40.796 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T17:57:40.796 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T17:57:40.796 INFO:teuthology.orchestra.run.smithi059.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T17:57:40.796 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T17:57:40.796 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T17:57:40.796 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T17:57:40.796 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T17:57:40.796 INFO:teuthology.orchestra.run.smithi059.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T17:57:40.796 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T17:57:40.796 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T17:57:40.797 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T17:57:40.797 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T17:57:40.797 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T17:57:40.797 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T17:57:40.797 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.797 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.797 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T17:57:40.797 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T17:57:40.797 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T17:57:40.797 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.797 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T17:57:40.797 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T17:57:40.797 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T17:57:40.797 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T17:57:40.797 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T17:57:40.797 INFO:teuthology.orchestra.run.smithi059.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T17:57:40.798 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T17:57:40.798 INFO:teuthology.orchestra.run.smithi059.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T17:57:40.798 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T17:57:40.798 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T17:57:40.798 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T17:57:40.798 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T17:57:40.798 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T17:57:40.798 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T17:57:40.798 INFO:teuthology.orchestra.run.smithi059.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T17:57:40.798 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.798 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.798 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:40.798 INFO:teuthology.orchestra.run.smithi059.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T17:57:40.799 INFO:teuthology.orchestra.run.smithi059.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T17:57:40.799 INFO:teuthology.orchestra.run.smithi059.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T17:57:40.799 INFO:teuthology.orchestra.run.smithi059.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T17:57:40.799 INFO:teuthology.orchestra.run.smithi059.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T17:57:40.799 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T17:57:40.799 INFO:teuthology.orchestra.run.smithi059.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T17:57:40.799 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-28T17:57:40.799 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-28T17:57:41.112 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T17:57:41.169 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T17:57:41.210 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T17:57:41.219 DEBUG:teuthology.parallel:result is None 2024-03-28T17:57:41.249 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T17:57:41.567 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T17:57:41.702 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T17:57:41.752 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T17:57:41.815 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T17:57:41.886 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T17:57:41.939 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T17:57:41.975 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T17:57:42.018 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T17:57:42.070 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T17:57:42.172 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T17:57:42.293 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T17:57:42.336 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T17:57:43.576 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T17:57:43.624 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T17:57:43.692 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T17:57:45.125 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T17:57:45.220 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T17:57:45.281 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T17:57:45.412 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T17:57:45.448 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T17:57:45.481 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T17:57:45.515 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T17:57:45.557 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T17:57:45.557 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T17:57:45.728 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T17:57:45.778 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T17:57:45.778 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T17:57:45.916 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T17:57:45.993 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T17:57:46.039 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T17:57:46.072 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T17:57:46.104 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T17:57:46.208 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T17:57:46.296 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T17:57:46.343 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T17:57:46.375 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T17:57:46.410 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T17:57:46.443 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T17:57:46.476 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T17:57:46.514 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T17:57:46.514 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T17:57:46.835 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T17:57:46.835 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T17:57:47.015 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T17:57:47.045 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T17:57:47.100 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T17:57:47.140 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T17:57:47.180 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T17:57:47.305 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T17:57:47.601 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T17:57:47.601 INFO:teuthology.orchestra.run.smithi037.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-28T17:57:47.601 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:57:52.437 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T17:57:52.437 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T17:57:52.438 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T17:57:52.438 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T17:57:52.438 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T17:57:52.438 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T17:57:52.438 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T17:57:52.438 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T17:57:52.438 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T17:57:52.438 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T17:57:52.438 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T17:57:52.438 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T17:57:52.438 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T17:57:52.438 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T17:57:52.439 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T17:57:52.439 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T17:57:52.439 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T17:57:52.439 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T17:57:52.439 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T17:57:52.439 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T17:57:52.439 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T17:57:52.439 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T17:57:52.439 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T17:57:52.439 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T17:57:52.439 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T17:57:52.439 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T17:57:52.439 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T17:57:52.440 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T17:57:52.440 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T17:57:52.455 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T17:57:52.456 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T17:57:52.456 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T17:57:52.456 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T17:57:52.456 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T17:57:52.456 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T17:57:52.456 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T17:57:52.456 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T17:57:52.456 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T17:57:52.456 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T17:57:52.456 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T17:57:52.456 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T17:57:52.456 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T17:57:52.456 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T17:57:52.456 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T17:57:52.457 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T17:57:52.457 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T17:57:52.457 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T17:57:52.457 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T17:57:52.457 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T17:57:52.457 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T17:57:52.457 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T17:57:52.457 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T17:57:52.457 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T17:57:52.457 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T17:57:52.457 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T17:57:52.457 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T17:57:52.457 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T17:57:52.457 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T17:57:52.458 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T17:57:52.458 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T17:57:52.458 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T17:57:52.458 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T17:57:52.458 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T17:57:52.458 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T17:57:52.458 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T17:57:52.458 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T17:57:52.458 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T17:57:52.458 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T17:57:52.458 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T17:57:52.458 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T17:57:52.458 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T17:57:52.459 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T17:57:52.459 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T17:57:52.459 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T17:57:52.459 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T17:57:52.459 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T17:57:52.459 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T17:57:52.459 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T17:57:52.459 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T17:57:52.459 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T17:57:52.459 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T17:57:52.459 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T17:57:52.459 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T17:57:52.459 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T17:57:52.460 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T17:57:52.460 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T17:57:52.460 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T17:57:52.460 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T17:57:52.460 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T17:57:52.460 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T17:57:52.460 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T17:57:52.460 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T17:57:52.460 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T17:57:52.460 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T17:57:52.460 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T17:57:52.460 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T17:57:52.460 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T17:57:52.460 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T17:57:52.461 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T17:57:52.461 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T17:57:52.461 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T17:57:52.461 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T17:57:52.461 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T17:57:52.461 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T17:57:52.461 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T17:57:52.461 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T17:57:52.461 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T17:57:52.461 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T17:57:52.461 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T17:57:52.461 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T17:57:52.461 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T17:57:52.461 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T17:57:52.461 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T17:57:52.462 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T17:57:52.462 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T17:57:52.462 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T17:57:52.462 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T17:57:52.462 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T17:57:52.462 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T17:57:52.462 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T17:57:52.462 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T17:57:52.462 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T17:57:52.462 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T17:57:52.462 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T17:57:52.462 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T17:57:52.462 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T17:57:52.462 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T17:57:52.462 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T17:57:52.463 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T17:57:52.463 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T17:57:52.463 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T17:57:52.463 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T17:57:52.463 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T17:57:52.463 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T17:57:52.463 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T17:57:52.463 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T17:57:52.463 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T17:57:52.463 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T17:57:52.463 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T17:57:52.463 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T17:57:52.463 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T17:57:52.463 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T17:57:52.463 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T17:57:52.464 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T17:57:52.464 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T17:57:52.464 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T17:57:52.464 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T17:57:52.464 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T17:57:52.464 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T17:57:52.464 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T17:57:52.464 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T17:57:52.464 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T17:57:52.464 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T17:57:52.464 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T17:57:52.464 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T17:57:52.464 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T17:57:52.464 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T17:57:52.464 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T17:57:52.465 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T17:57:52.465 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T17:57:52.465 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T17:57:52.465 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T17:57:52.465 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T17:57:52.465 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T17:57:52.465 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T17:57:52.465 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T17:57:52.465 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T17:57:52.465 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T17:57:52.465 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T17:57:52.465 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T17:57:52.465 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T17:57:52.465 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T17:57:52.465 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T17:57:52.465 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T17:57:52.466 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T17:57:52.466 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T17:57:52.466 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T17:57:52.466 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T17:57:52.466 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T17:57:52.466 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T17:57:52.466 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T17:57:52.466 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T17:57:52.466 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T17:57:52.466 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T17:57:52.466 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T17:57:52.466 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T17:57:52.466 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T17:57:52.466 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T17:57:52.466 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T17:57:52.467 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T17:57:52.467 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T17:57:52.467 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T17:57:52.467 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T17:57:52.467 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T17:57:52.467 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T17:57:52.467 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T17:57:52.467 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T17:57:54.892 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T17:57:54.892 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:57:54.893 INFO:teuthology.orchestra.run.smithi037.stdout:Upgraded: 2024-03-28T17:57:54.893 INFO:teuthology.orchestra.run.smithi037.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T17:57:54.893 INFO:teuthology.orchestra.run.smithi037.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T17:57:54.893 INFO:teuthology.orchestra.run.smithi037.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T17:57:54.893 INFO:teuthology.orchestra.run.smithi037.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T17:57:54.893 INFO:teuthology.orchestra.run.smithi037.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T17:57:54.893 INFO:teuthology.orchestra.run.smithi037.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T17:57:54.893 INFO:teuthology.orchestra.run.smithi037.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T17:57:54.893 INFO:teuthology.orchestra.run.smithi037.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T17:57:54.894 INFO:teuthology.orchestra.run.smithi037.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T17:57:54.894 INFO:teuthology.orchestra.run.smithi037.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T17:57:54.894 INFO:teuthology.orchestra.run.smithi037.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T17:57:54.894 INFO:teuthology.orchestra.run.smithi037.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T17:57:54.894 INFO:teuthology.orchestra.run.smithi037.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T17:57:54.894 INFO:teuthology.orchestra.run.smithi037.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.894 INFO:teuthology.orchestra.run.smithi037.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.894 INFO:teuthology.orchestra.run.smithi037.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T17:57:54.894 INFO:teuthology.orchestra.run.smithi037.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T17:57:54.895 INFO:teuthology.orchestra.run.smithi037.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T17:57:54.895 INFO:teuthology.orchestra.run.smithi037.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T17:57:54.895 INFO:teuthology.orchestra.run.smithi037.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T17:57:54.895 INFO:teuthology.orchestra.run.smithi037.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T17:57:54.895 INFO:teuthology.orchestra.run.smithi037.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T17:57:54.895 INFO:teuthology.orchestra.run.smithi037.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T17:57:54.895 INFO:teuthology.orchestra.run.smithi037.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T17:57:54.895 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-28T17:57:54.895 INFO:teuthology.orchestra.run.smithi037.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T17:57:54.896 INFO:teuthology.orchestra.run.smithi037.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T17:57:54.896 INFO:teuthology.orchestra.run.smithi037.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T17:57:54.896 INFO:teuthology.orchestra.run.smithi037.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T17:57:54.896 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.896 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.896 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.896 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.896 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:54.896 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.897 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.897 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.897 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:54.897 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:54.897 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:54.897 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:54.897 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:54.897 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.897 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.898 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:54.898 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.898 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.898 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.898 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:54.898 INFO:teuthology.orchestra.run.smithi037.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:54.898 INFO:teuthology.orchestra.run.smithi037.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.898 INFO:teuthology.orchestra.run.smithi037.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.898 INFO:teuthology.orchestra.run.smithi037.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:57:54.899 INFO:teuthology.orchestra.run.smithi037.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T17:57:54.899 INFO:teuthology.orchestra.run.smithi037.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T17:57:54.899 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T17:57:54.899 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T17:57:54.899 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T17:57:54.899 INFO:teuthology.orchestra.run.smithi037.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T17:57:54.899 INFO:teuthology.orchestra.run.smithi037.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T17:57:54.899 INFO:teuthology.orchestra.run.smithi037.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T17:57:54.899 INFO:teuthology.orchestra.run.smithi037.stdout: jq-1.6-16.el9.x86_64 2024-03-28T17:57:54.899 INFO:teuthology.orchestra.run.smithi037.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T17:57:54.900 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T17:57:54.900 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T17:57:54.900 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.900 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.900 INFO:teuthology.orchestra.run.smithi037.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.900 INFO:teuthology.orchestra.run.smithi037.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T17:57:54.900 INFO:teuthology.orchestra.run.smithi037.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T17:57:54.900 INFO:teuthology.orchestra.run.smithi037.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T17:57:54.900 INFO:teuthology.orchestra.run.smithi037.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T17:57:54.901 INFO:teuthology.orchestra.run.smithi037.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T17:57:54.901 INFO:teuthology.orchestra.run.smithi037.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.901 INFO:teuthology.orchestra.run.smithi037.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.901 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T17:57:54.901 INFO:teuthology.orchestra.run.smithi037.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.901 INFO:teuthology.orchestra.run.smithi037.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T17:57:54.901 INFO:teuthology.orchestra.run.smithi037.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T17:57:54.901 INFO:teuthology.orchestra.run.smithi037.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T17:57:54.901 INFO:teuthology.orchestra.run.smithi037.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T17:57:54.901 INFO:teuthology.orchestra.run.smithi037.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T17:57:54.902 INFO:teuthology.orchestra.run.smithi037.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T17:57:54.902 INFO:teuthology.orchestra.run.smithi037.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T17:57:54.902 INFO:teuthology.orchestra.run.smithi037.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T17:57:54.902 INFO:teuthology.orchestra.run.smithi037.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T17:57:54.902 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T17:57:54.902 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T17:57:54.902 INFO:teuthology.orchestra.run.smithi037.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T17:57:54.902 INFO:teuthology.orchestra.run.smithi037.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T17:57:54.902 INFO:teuthology.orchestra.run.smithi037.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T17:57:54.902 INFO:teuthology.orchestra.run.smithi037.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T17:57:54.902 INFO:teuthology.orchestra.run.smithi037.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T17:57:54.903 INFO:teuthology.orchestra.run.smithi037.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T17:57:54.903 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T17:57:54.903 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T17:57:54.903 INFO:teuthology.orchestra.run.smithi037.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T17:57:54.903 INFO:teuthology.orchestra.run.smithi037.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T17:57:54.903 INFO:teuthology.orchestra.run.smithi037.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T17:57:54.903 INFO:teuthology.orchestra.run.smithi037.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T17:57:54.903 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T17:57:54.903 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.904 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.904 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.904 INFO:teuthology.orchestra.run.smithi037.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T17:57:54.904 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T17:57:54.904 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T17:57:54.904 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T17:57:54.904 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T17:57:54.904 INFO:teuthology.orchestra.run.smithi037.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T17:57:54.904 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T17:57:54.904 INFO:teuthology.orchestra.run.smithi037.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T17:57:54.904 INFO:teuthology.orchestra.run.smithi037.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T17:57:54.905 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T17:57:54.905 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T17:57:54.905 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T17:57:54.905 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T17:57:54.905 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T17:57:54.905 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T17:57:54.905 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T17:57:54.905 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T17:57:54.905 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T17:57:54.905 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T17:57:54.906 INFO:teuthology.orchestra.run.smithi037.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T17:57:54.906 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T17:57:54.906 INFO:teuthology.orchestra.run.smithi037.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T17:57:54.906 INFO:teuthology.orchestra.run.smithi037.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T17:57:54.906 INFO:teuthology.orchestra.run.smithi037.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T17:57:54.906 INFO:teuthology.orchestra.run.smithi037.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T17:57:54.906 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T17:57:54.906 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T17:57:54.906 INFO:teuthology.orchestra.run.smithi037.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T17:57:54.906 INFO:teuthology.orchestra.run.smithi037.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T17:57:54.907 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T17:57:54.907 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T17:57:54.907 INFO:teuthology.orchestra.run.smithi037.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T17:57:54.907 INFO:teuthology.orchestra.run.smithi037.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T17:57:54.907 INFO:teuthology.orchestra.run.smithi037.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T17:57:54.907 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T17:57:54.907 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T17:57:54.907 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T17:57:54.907 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T17:57:54.907 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T17:57:54.908 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T17:57:54.908 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.908 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.908 INFO:teuthology.orchestra.run.smithi037.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T17:57:54.908 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T17:57:54.908 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T17:57:54.908 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.908 INFO:teuthology.orchestra.run.smithi037.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T17:57:54.908 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T17:57:54.908 INFO:teuthology.orchestra.run.smithi037.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T17:57:54.909 INFO:teuthology.orchestra.run.smithi037.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T17:57:54.909 INFO:teuthology.orchestra.run.smithi037.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T17:57:54.909 INFO:teuthology.orchestra.run.smithi037.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T17:57:54.909 INFO:teuthology.orchestra.run.smithi037.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T17:57:54.909 INFO:teuthology.orchestra.run.smithi037.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T17:57:54.909 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T17:57:54.909 INFO:teuthology.orchestra.run.smithi037.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T17:57:54.909 INFO:teuthology.orchestra.run.smithi037.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T17:57:54.909 INFO:teuthology.orchestra.run.smithi037.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T17:57:54.909 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T17:57:54.909 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T17:57:54.910 INFO:teuthology.orchestra.run.smithi037.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T17:57:54.910 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.910 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.910 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:57:54.910 INFO:teuthology.orchestra.run.smithi037.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T17:57:54.910 INFO:teuthology.orchestra.run.smithi037.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T17:57:54.910 INFO:teuthology.orchestra.run.smithi037.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T17:57:54.910 INFO:teuthology.orchestra.run.smithi037.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T17:57:54.910 INFO:teuthology.orchestra.run.smithi037.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T17:57:54.910 INFO:teuthology.orchestra.run.smithi037.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T17:57:54.911 INFO:teuthology.orchestra.run.smithi037.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T17:57:54.911 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:57:54.911 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-28T17:57:55.744 DEBUG:teuthology.parallel:result is None 2024-03-28T17:57:55.744 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:57:55.892 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T17:57:55.950 INFO:teuthology.orchestra.run.smithi037.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T17:57:55.951 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T17:57:55.951 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T17:57:55.953 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:57:56.096 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T17:57:56.123 INFO:teuthology.orchestra.run.smithi059.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T17:57:56.123 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T17:57:56.123 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T17:57:56.125 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:57:56.266 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T17:57:56.292 INFO:teuthology.orchestra.run.smithi170.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T17:57:56.292 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T17:57:56.292 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T17:57:56.294 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-28T17:57:56.294 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-28T17:57:56.294 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T17:57:56.375 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-28T17:57:56.375 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T17:57:56.414 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T17:57:56.414 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T17:57:56.449 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-28T17:57:56.450 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-28T17:57:56.450 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T17:57:56.524 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T17:57:56.634 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-28T17:57:56.635 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T17:57:56.672 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T17:57:56.745 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T17:57:56.745 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T17:57:56.778 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T17:57:56.846 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-28T17:57:56.847 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-28T17:57:56.847 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T17:57:56.927 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T17:57:57.039 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-28T17:57:57.039 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T17:57:57.074 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T17:57:57.146 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T17:57:57.146 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T17:57:57.174 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T17:57:57.242 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-28T17:57:57.242 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-28T17:57:57.242 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T17:57:57.330 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T17:57:57.445 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-28T17:57:57.445 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T17:57:57.479 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T17:57:57.550 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T17:57:57.550 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T17:57:57.578 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T17:57:57.645 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-28T17:57:57.742 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-28T17:57:57.742 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:57:57.743 INFO:tasks.cephadm:Cluster fsid is b5ad257c-ed2c-11ee-95d0-87774f69a715 2024-03-28T17:57:57.743 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-28T17:57:57.743 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.37', 'mon.b': '172.21.15.59', 'mon.c': '172.21.15.170'} 2024-03-28T17:57:57.743 INFO:tasks.cephadm:First mon is mon.a on smithi037 2024-03-28T17:57:57.743 INFO:tasks.cephadm:First mgr is x 2024-03-28T17:57:57.743 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-28T17:57:57.743 DEBUG:teuthology.orchestra.run.smithi037:> sudo hostname $(hostname -s) 2024-03-28T17:57:57.817 DEBUG:teuthology.orchestra.run.smithi059:> sudo hostname $(hostname -s) 2024-03-28T17:57:57.847 DEBUG:teuthology.orchestra.run.smithi170:> sudo hostname $(hostname -s) 2024-03-28T17:57:57.874 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-28T17:57:57.874 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:57:58.018 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77877/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2145-g3be9a5c9', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2145.g3be9a5c9'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-22 05:51:42.267384', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-mchangir-testing-main-20240318.032620', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-28T17:57:58.231 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-mchangir-testing-main-20240318.032620, sha1 3be9a5c9bc793e11e2800a8c0c696e8b46742033 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:57:58.232 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm 2024-03-28T17:57:58.232 DEBUG:teuthology.orchestra.run.smithi037:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T17:57:58.746 INFO:teuthology.orchestra.run.smithi037.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 17:57 /home/ubuntu/cephtest/cephadm 2024-03-28T17:57:58.747 DEBUG:teuthology.orchestra.run.smithi059:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T17:57:59.382 INFO:teuthology.orchestra.run.smithi059.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 17:57 /home/ubuntu/cephtest/cephadm 2024-03-28T17:57:59.382 DEBUG:teuthology.orchestra.run.smithi170:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T17:57:59.965 INFO:teuthology.orchestra.run.smithi170.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 17:57 /home/ubuntu/cephtest/cephadm 2024-03-28T17:57:59.965 DEBUG:teuthology.orchestra.run.smithi037:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T17:58:00.018 DEBUG:teuthology.orchestra.run.smithi059:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T17:58:00.037 DEBUG:teuthology.orchestra.run.smithi170:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T17:58:00.070 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 on all hosts... 2024-03-28T17:58:00.070 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T17:58:00.074 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T17:58:00.080 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T17:58:00.895 INFO:teuthology.orchestra.run.smithi170.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T17:58:00.926 INFO:teuthology.orchestra.run.smithi059.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T17:58:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T17:58:37.867 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T17:58:37.867 INFO:teuthology.orchestra.run.smithi037.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T17:58:37.867 INFO:teuthology.orchestra.run.smithi037.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T17:58:37.868 INFO:teuthology.orchestra.run.smithi037.stdout: "repo_digests": [ 2024-03-28T17:58:37.868 INFO:teuthology.orchestra.run.smithi037.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T17:58:37.868 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-03-28T17:58:37.868 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T17:58:47.727 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-28T17:58:47.727 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T17:58:47.727 INFO:teuthology.orchestra.run.smithi059.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T17:58:47.728 INFO:teuthology.orchestra.run.smithi059.stdout: "repo_digests": [ 2024-03-28T17:58:47.728 INFO:teuthology.orchestra.run.smithi059.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T17:58:47.728 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-03-28T17:58:47.728 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-28T17:58:52.670 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-03-28T17:58:52.670 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T17:58:52.670 INFO:teuthology.orchestra.run.smithi170.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T17:58:52.670 INFO:teuthology.orchestra.run.smithi170.stdout: "repo_digests": [ 2024-03-28T17:58:52.670 INFO:teuthology.orchestra.run.smithi170.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T17:58:52.671 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-03-28T17:58:52.671 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-03-28T17:58:52.693 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /etc/ceph 2024-03-28T17:58:52.806 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph 2024-03-28T17:58:52.846 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph 2024-03-28T17:58:52.875 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 777 /etc/ceph 2024-03-28T17:58:52.985 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 777 /etc/ceph 2024-03-28T17:58:53.020 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 777 /etc/ceph 2024-03-28T17:58:53.047 INFO:tasks.cephadm:Writing seed config... 2024-03-28T17:58:53.048 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-28T17:58:53.048 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-28T17:58:53.048 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-28T17:58:53.049 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-28T17:58:53.049 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-28T17:58:53.049 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-28T17:58:53.049 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-28T17:58:53.049 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-28T17:58:53.049 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-28T17:58:53.049 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-28T17:58:53.049 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-28T17:58:53.049 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-28T17:58:53.049 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-28T17:58:53.049 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-28T17:58:53.049 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-28T17:58:53.049 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-28T17:58:53.049 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-28T17:58:53.049 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-28T17:58:53.050 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-28T17:58:53.050 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-28T17:58:53.050 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-28T17:58:53.050 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-28T17:58:53.050 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-28T17:58:53.050 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-28T17:58:53.050 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-28T17:58:53.050 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-28T17:58:53.050 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-28T17:58:53.050 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-28T17:58:53.050 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-28T17:58:53.050 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-28T17:58:53.050 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-28T17:58:53.050 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-28T17:58:53.050 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-28T17:58:53.050 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-28T17:58:53.051 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-28T17:58:53.051 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-28T17:58:53.051 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-28T17:58:53.051 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-28T17:58:53.051 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-28T17:58:53.051 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-28T17:58:53.051 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-28T17:58:53.051 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-28T17:58:53.051 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-28T17:58:53.051 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-28T17:58:53.052 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-28T17:58:53.052 DEBUG:teuthology.orchestra.run.smithi037:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-28T17:58:53.109 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = b5ad257c-ed2c-11ee-95d0-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-28T17:58:53.109 DEBUG:teuthology.orchestra.run.smithi037:mon.a> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mon.a.service 2024-03-28T17:58:53.157 DEBUG:teuthology.orchestra.run.smithi037:mgr.x> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mgr.x.service 2024-03-28T17:58:53.203 INFO:tasks.cephadm:Bootstrapping... 2024-03-28T17:58:53.203 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 -v bootstrap --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.37 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-28T17:58:53.497 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-28T17:58:53.497 INFO:teuthology.orchestra.run.smithi037.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033', '-v', 'bootstrap', '--fsid', 'b5ad257c-ed2c-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.37', '--skip-admin-label'] 2024-03-28T17:58:53.555 INFO:teuthology.orchestra.run.smithi037.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T17:58:53.556 INFO:teuthology.orchestra.run.smithi037.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-28T17:58:53.556 INFO:teuthology.orchestra.run.smithi037.stdout:Verifying podman|docker is present... 2024-03-28T17:58:53.606 INFO:teuthology.orchestra.run.smithi037.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T17:58:53.607 INFO:teuthology.orchestra.run.smithi037.stdout:Verifying lvm2 is present... 2024-03-28T17:58:53.607 INFO:teuthology.orchestra.run.smithi037.stdout:Verifying time synchronization is in place... 2024-03-28T17:58:53.622 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T17:58:53.622 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T17:58:53.638 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T17:58:53.638 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout inactive 2024-03-28T17:58:53.664 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout enabled 2024-03-28T17:58:53.687 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout active 2024-03-28T17:58:53.688 INFO:teuthology.orchestra.run.smithi037.stdout:Unit chronyd.service is enabled and running 2024-03-28T17:58:53.688 INFO:teuthology.orchestra.run.smithi037.stdout:Repeating the final host check... 2024-03-28T17:58:53.750 INFO:teuthology.orchestra.run.smithi037.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T17:58:53.750 INFO:teuthology.orchestra.run.smithi037.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-28T17:58:53.750 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl is present 2024-03-28T17:58:53.750 INFO:teuthology.orchestra.run.smithi037.stdout:lvcreate is present 2024-03-28T17:58:53.769 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T17:58:53.770 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T17:58:53.789 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T17:58:53.789 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout inactive 2024-03-28T17:58:53.816 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout enabled 2024-03-28T17:58:53.838 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout active 2024-03-28T17:58:53.838 INFO:teuthology.orchestra.run.smithi037.stdout:Unit chronyd.service is enabled and running 2024-03-28T17:58:53.838 INFO:teuthology.orchestra.run.smithi037.stdout:Host looks OK 2024-03-28T17:58:53.838 INFO:teuthology.orchestra.run.smithi037.stdout:Cluster fsid: b5ad257c-ed2c-11ee-95d0-87774f69a715 2024-03-28T17:58:53.839 INFO:teuthology.orchestra.run.smithi037.stdout:Acquiring lock 140536870309312 on /run/cephadm/b5ad257c-ed2c-11ee-95d0-87774f69a715.lock 2024-03-28T17:58:53.839 INFO:teuthology.orchestra.run.smithi037.stdout:Lock 140536870309312 acquired on /run/cephadm/b5ad257c-ed2c-11ee-95d0-87774f69a715.lock 2024-03-28T17:58:53.839 INFO:teuthology.orchestra.run.smithi037.stdout:Verifying IP 172.21.15.37 port 3300 ... 2024-03-28T17:58:53.840 INFO:teuthology.orchestra.run.smithi037.stdout:Verifying IP 172.21.15.37 port 6789 ... 2024-03-28T17:58:53.840 INFO:teuthology.orchestra.run.smithi037.stdout:Base mon IP(s) is [172.21.15.37:3300, 172.21.15.37:6789], mon addrv is [v2:172.21.15.37:3300,v1:172.21.15.37:6789] 2024-03-28T17:58:53.851 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.37 metric 100 2024-03-28T17:58:53.851 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.37 metric 100 2024-03-28T17:58:53.861 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-28T17:58:53.861 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-28T17:58:53.862 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-28T17:58:53.871 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-28T17:58:53.871 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-28T17:58:53.871 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T17:58:53.871 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-28T17:58:53.871 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:7794/64 scope link noprefixroute 2024-03-28T17:58:53.871 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T17:58:53.874 INFO:teuthology.orchestra.run.smithi037.stdout:Mon IP `172.21.15.37` is in CIDR network `172.21.0.0/20` 2024-03-28T17:58:53.874 INFO:teuthology.orchestra.run.smithi037.stdout:Mon IP `172.21.15.37` is in CIDR network `172.21.0.0/20` 2024-03-28T17:58:53.874 INFO:teuthology.orchestra.run.smithi037.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-28T17:58:53.876 INFO:teuthology.orchestra.run.smithi037.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-28T17:58:53.878 INFO:teuthology.orchestra.run.smithi037.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T17:58:54.528 INFO:teuthology.orchestra.run.smithi037.stdout:/bin/podman: stdout c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-28T17:58:54.528 INFO:teuthology.orchestra.run.smithi037.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T17:58:54.529 INFO:teuthology.orchestra.run.smithi037.stdout:/bin/podman: stderr Getting image source signatures 2024-03-28T17:58:54.529 INFO:teuthology.orchestra.run.smithi037.stdout:/bin/podman: stderr Copying blob sha256:4dd83c6ee8921fe17396d77c31e205985ae94c87d3f752465b5f04aaf9ae340a 2024-03-28T17:58:54.529 INFO:teuthology.orchestra.run.smithi037.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-28T17:58:54.529 INFO:teuthology.orchestra.run.smithi037.stdout:/bin/podman: stderr Copying config sha256:c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-28T17:58:54.529 INFO:teuthology.orchestra.run.smithi037.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-28T17:58:56.575 INFO:teuthology.orchestra.run.smithi037.stdout:ceph: stdout ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:56.575 INFO:teuthology.orchestra.run.smithi037.stdout:Ceph version: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:56.575 INFO:teuthology.orchestra.run.smithi037.stdout:Extracting ceph user uid/gid from container image... 2024-03-28T17:58:57.720 INFO:teuthology.orchestra.run.smithi037.stdout:stat: stdout 167 167 2024-03-28T17:58:57.720 INFO:teuthology.orchestra.run.smithi037.stdout:Creating initial keys... 2024-03-28T17:58:58.876 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph-authtool: stdout AQDirwVmnbYeDhAAOVX+nBIURzxareE60kJacA== 2024-03-28T17:58:59.942 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph-authtool: stdout AQDjrwVmtYEMFBAAZBMIAHkdDmolOsT3eDZoBQ== 2024-03-28T17:59:01.004 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph-authtool: stdout AQDkrwVm1/85GRAAEUC2pAbL5SKJV22whNoWiA== 2024-03-28T17:59:01.007 INFO:teuthology.orchestra.run.smithi037.stdout:Creating initial monmap... 2024-03-28T17:59:02.145 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T17:59:02.146 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-28T17:59:02.146 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to b5ad257c-ed2c-11ee-95d0-87774f69a715 2024-03-28T17:59:02.146 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T17:59:02.147 INFO:teuthology.orchestra.run.smithi037.stdout:monmaptool for a [v2:172.21.15.37:3300,v1:172.21.15.37:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T17:59:02.147 INFO:teuthology.orchestra.run.smithi037.stdout:setting min_mon_release = quincy 2024-03-28T17:59:02.147 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/monmaptool: set fsid to b5ad257c-ed2c-11ee-95d0-87774f69a715 2024-03-28T17:59:02.147 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T17:59:02.147 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T17:59:02.148 INFO:teuthology.orchestra.run.smithi037.stdout:Creating mon... 2024-03-28T17:59:03.324 INFO:teuthology.orchestra.run.smithi037.stdout:create mon.a on 2024-03-28T17:59:03.744 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-28T17:59:03.984 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-28T17:59:04.281 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715.target. 2024-03-28T17:59:04.281 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715.target. 2024-03-28T17:59:04.691 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mon.a 2024-03-28T17:59:04.691 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stderr Failed to reset failed state of unit ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-28T17:59:04.993 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715.target.wants/ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@.service. 2024-03-28T17:59:05.860 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:05 smithi037 podman[31065]: 2024-03-28 17:59:05.404638089 +0000 UTC m=+0.144839380 container create 90b0d780c3a9ad0d50cec10a1bc3d3149390f7e18bc13cc173f37b7a27fc799a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-03-28T17:59:05.860 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:05 smithi037 podman[31065]: 2024-03-28 17:59:05.858257943 +0000 UTC m=+0.598459267 container init 90b0d780c3a9ad0d50cec10a1bc3d3149390f7e18bc13cc173f37b7a27fc799a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mon-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T17:59:05.976 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T17:59:05.977 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout disabled 2024-03-28T17:59:05.991 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T17:59:05.991 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout inactive 2024-03-28T17:59:05.991 INFO:teuthology.orchestra.run.smithi037.stdout:firewalld.service is not enabled 2024-03-28T17:59:05.991 INFO:teuthology.orchestra.run.smithi037.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T17:59:05.992 INFO:teuthology.orchestra.run.smithi037.stdout:Waiting for mon to start... 2024-03-28T17:59:05.992 INFO:teuthology.orchestra.run.smithi037.stdout:Waiting for mon... 2024-03-28T17:59:06.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:05 smithi037 podman[31065]: 2024-03-28 17:59:05.869650419 +0000 UTC m=+0.609851716 container start 90b0d780c3a9ad0d50cec10a1bc3d3149390f7e18bc13cc173f37b7a27fc799a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mon-a, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T17:59:06.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:05 smithi037 bash[31065]: 90b0d780c3a9ad0d50cec10a1bc3d3149390f7e18bc13cc173f37b7a27fc799a 2024-03-28T17:59:06.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:05 smithi037 ceph-mon[31083]: mkfs b5ad257c-ed2c-11ee-95d0-87774f69a715 2024-03-28T17:59:06.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:05 smithi037 systemd[1]: Started Ceph mon.a for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T17:59:06.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:05 smithi037 ceph-mon[31083]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T17:59:07.314 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:06 smithi037 ceph-mon[31083]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T17:59:07.314 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:06 smithi037 ceph-mon[31083]: monmap e1: 1 mons at {a=[v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T17:59:07.314 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:06 smithi037 ceph-mon[31083]: fsmap 2024-03-28T17:59:07.314 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:06 smithi037 ceph-mon[31083]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T17:59:07.314 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:06 smithi037 ceph-mon[31083]: mgrmap e1: no daemons active 2024-03-28T17:59:07.314 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:06 smithi037 ceph-mon[31083]: from='client.? 172.21.15.37:0/1502611514' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-28T17:59:07.327 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout cluster: 2024-03-28T17:59:07.329 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout id: b5ad257c-ed2c-11ee-95d0-87774f69a715 2024-03-28T17:59:07.329 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-28T17:59:07.329 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-03-28T17:59:07.329 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout services: 2024-03-28T17:59:07.329 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.727702s) 2024-03-28T17:59:07.329 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-28T17:59:07.330 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-28T17:59:07.330 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-03-28T17:59:07.330 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout data: 2024-03-28T17:59:07.330 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-28T17:59:07.335 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-28T17:59:07.335 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-28T17:59:07.335 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout pgs: 2024-03-28T17:59:07.335 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-03-28T17:59:07.335 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.581+0000 7f5e05600700 1 Processor -- start 2024-03-28T17:59:07.336 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.581+0000 7f5e05600700 1 -- start start 2024-03-28T17:59:07.336 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.581+0000 7f5e05600700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e00108540 0x7f5e0010a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:07.336 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.581+0000 7f5e05600700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0007be30 con 0x7f5e00108540 2024-03-28T17:59:07.336 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.583+0000 7f5dfea00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e00108540 0x7f5e0010a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:07.336 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.583+0000 7f5dfea00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e00108540 0x7f5e0010a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60530/0 (socket says 172.21.15.37:60530) 2024-03-28T17:59:07.336 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.583+0000 7f5dfea00700 1 -- 172.21.15.37:0/2042542081 learned_addr learned my addr 172.21.15.37:0/2042542081 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:07.336 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.583+0000 7f5dfea00700 1 -- 172.21.15.37:0/2042542081 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0010ae70 con 0x7f5e00108540 2024-03-28T17:59:07.336 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.584+0000 7f5dfea00700 1 --2- 172.21.15.37:0/2042542081 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e00108540 0x7f5e0010a930 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5df00098a0 tx=0x7f5df0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a6370a38364a3ab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:07.336 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.584+0000 7f5dfd600700 1 -- 172.21.15.37:0/2042542081 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5df0004160 con 0x7f5e00108540 2024-03-28T17:59:07.336 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.584+0000 7f5dfd600700 1 -- 172.21.15.37:0/2042542081 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df0004300 con 0x7f5e00108540 2024-03-28T17:59:07.336 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.584+0000 7f5dfd600700 1 -- 172.21.15.37:0/2042542081 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5df00036a0 con 0x7f5e00108540 2024-03-28T17:59:07.336 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.585+0000 7f5e05600700 1 -- 172.21.15.37:0/2042542081 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e00108540 msgr2=0x7f5e0010a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:07.336 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.585+0000 7f5e05600700 1 --2- 172.21.15.37:0/2042542081 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e00108540 0x7f5e0010a930 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5df00098a0 tx=0x7f5df0009bb0 comp rx=0 tx=0).stop 2024-03-28T17:59:07.336 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.585+0000 7f5e05600700 1 -- 172.21.15.37:0/2042542081 shutdown_connections 2024-03-28T17:59:07.337 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.585+0000 7f5e05600700 1 --2- 172.21.15.37:0/2042542081 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e00108540 0x7f5e0010a930 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:07.337 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.585+0000 7f5e05600700 1 -- 172.21.15.37:0/2042542081 >> 172.21.15.37:0/2042542081 conn(0x7f5e00101f70 msgr2=0x7f5e00104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:07.337 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.586+0000 7f5e05600700 1 -- 172.21.15.37:0/2042542081 shutdown_connections 2024-03-28T17:59:07.337 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.586+0000 7f5e05600700 1 -- 172.21.15.37:0/2042542081 wait complete. 2024-03-28T17:59:07.337 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.587+0000 7f5e05600700 1 Processor -- start 2024-03-28T17:59:07.337 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.587+0000 7f5e05600700 1 -- start start 2024-03-28T17:59:07.337 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.587+0000 7f5e05600700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e00108540 0x7f5e0019d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:07.337 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.587+0000 7f5e05600700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0010b2e0 con 0x7f5e00108540 2024-03-28T17:59:07.337 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.587+0000 7f5dfea00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e00108540 0x7f5e0019d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:07.337 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.587+0000 7f5dfea00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e00108540 0x7f5e0019d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60546/0 (socket says 172.21.15.37:60546) 2024-03-28T17:59:07.337 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.587+0000 7f5dfea00700 1 -- 172.21.15.37:0/1502611514 learned_addr learned my addr 172.21.15.37:0/1502611514 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:07.337 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.588+0000 7f5dfea00700 1 -- 172.21.15.37:0/1502611514 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0019d860 con 0x7f5e00108540 2024-03-28T17:59:07.337 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.588+0000 7f5dfea00700 1 --2- 172.21.15.37:0/1502611514 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e00108540 0x7f5e0019d320 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5df0000c00 tx=0x7f5df0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:07.337 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.589+0000 7f5df7400700 1 -- 172.21.15.37:0/1502611514 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5df0003db0 con 0x7f5e00108540 2024-03-28T17:59:07.338 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.589+0000 7f5df7400700 1 -- 172.21.15.37:0/1502611514 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df0011630 con 0x7f5e00108540 2024-03-28T17:59:07.338 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.589+0000 7f5df7400700 1 -- 172.21.15.37:0/1502611514 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5df0011950 con 0x7f5e00108540 2024-03-28T17:59:07.338 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.589+0000 7f5e05600700 1 -- 172.21.15.37:0/1502611514 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e0019daf0 con 0x7f5e00108540 2024-03-28T17:59:07.338 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.589+0000 7f5e05600700 1 -- 172.21.15.37:0/1502611514 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e001a07e0 con 0x7f5e00108540 2024-03-28T17:59:07.338 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.589+0000 7f5df7400700 1 -- 172.21.15.37:0/1502611514 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5df0011af0 con 0x7f5e00108540 2024-03-28T17:59:07.338 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.589+0000 7f5df7400700 1 -- 172.21.15.37:0/1502611514 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5df001c450 con 0x7f5e00108540 2024-03-28T17:59:07.338 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.590+0000 7f5e05600700 1 -- 172.21.15.37:0/1502611514 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e000637c0 con 0x7f5e00108540 2024-03-28T17:59:07.338 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.593+0000 7f5df7400700 1 -- 172.21.15.37:0/1502611514 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5df0032020 con 0x7f5e00108540 2024-03-28T17:59:07.338 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.674+0000 7f5e05600700 1 -- 172.21.15.37:0/1502611514 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f5e0002dbb0 con 0x7f5e00108540 2024-03-28T17:59:07.338 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.675+0000 7f5df7400700 1 -- 172.21.15.37:0/1502611514 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f5df0004260 con 0x7f5e00108540 2024-03-28T17:59:07.338 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.677+0000 7f5e05600700 1 -- 172.21.15.37:0/1502611514 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e00108540 msgr2=0x7f5e0019d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:07.338 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.677+0000 7f5e05600700 1 --2- 172.21.15.37:0/1502611514 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e00108540 0x7f5e0019d320 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5df0000c00 tx=0x7f5df0004750 comp rx=0 tx=0).stop 2024-03-28T17:59:07.338 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.677+0000 7f5e05600700 1 -- 172.21.15.37:0/1502611514 shutdown_connections 2024-03-28T17:59:07.338 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.677+0000 7f5e05600700 1 --2- 172.21.15.37:0/1502611514 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e00108540 0x7f5e0019d320 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:07.338 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.677+0000 7f5e05600700 1 -- 172.21.15.37:0/1502611514 >> 172.21.15.37:0/1502611514 conn(0x7f5e00101f70 msgr2=0x7f5e00102e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:07.339 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.677+0000 7f5e05600700 1 -- 172.21.15.37:0/1502611514 shutdown_connections 2024-03-28T17:59:07.339 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:06.677+0000 7f5e05600700 1 -- 172.21.15.37:0/1502611514 wait complete. 2024-03-28T17:59:07.339 INFO:teuthology.orchestra.run.smithi037.stdout:mon is available 2024-03-28T17:59:07.339 INFO:teuthology.orchestra.run.smithi037.stdout:Assimilating anything we can from ceph.conf... 2024-03-28T17:59:08.356 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:08 smithi037 ceph-mon[31083]: from='client.? 172.21.15.37:0/2652118158' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T17:59:08.356 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:08 smithi037 ceph-mon[31083]: from='client.? 172.21.15.37:0/2652118158' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-28T17:59:08.653 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-03-28T17:59:08.653 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T17:59:08.653 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout fsid = b5ad257c-ed2c-11ee-95d0-87774f69a715 2024-03-28T17:59:08.653 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.37:3300,v1:172.21.15.37:6789] 2024-03-28T17:59:08.653 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T17:59:08.654 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T17:59:08.654 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T17:59:08.654 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T17:59:08.654 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-03-28T17:59:08.654 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T17:59:08.659 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T17:59:08.659 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-03-28T17:59:08.659 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T17:59:08.660 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T17:59:08.660 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T17:59:08.660 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T17:59:08.660 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T17:59:08.660 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T17:59:08.660 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.902+0000 7f66df000700 1 Processor -- start 2024-03-28T17:59:08.660 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.903+0000 7f66df000700 1 -- start start 2024-03-28T17:59:08.660 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.903+0000 7f66df000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66d8081aa0 0x7f66d8081ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:08.660 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.903+0000 7f66df000700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d8082490 con 0x7f66d8081aa0 2024-03-28T17:59:08.660 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.904+0000 7f66d7e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66d8081aa0 0x7f66d8081ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:08.660 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.904+0000 7f66d7e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66d8081aa0 0x7f66d8081ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60550/0 (socket says 172.21.15.37:60550) 2024-03-28T17:59:08.660 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.904+0000 7f66d7e00700 1 -- 172.21.15.37:0/3184675376 learned_addr learned my addr 172.21.15.37:0/3184675376 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:08.661 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.904+0000 7f66d7e00700 1 -- 172.21.15.37:0/3184675376 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d810f280 con 0x7f66d8081aa0 2024-03-28T17:59:08.661 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.904+0000 7f66d7e00700 1 --2- 172.21.15.37:0/3184675376 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66d8081aa0 0x7f66d8081ec0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f66c80098a0 tx=0x7f66c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1529d50d061d941e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:08.661 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.905+0000 7f66d7400700 1 -- 172.21.15.37:0/3184675376 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66c8004160 con 0x7f66d8081aa0 2024-03-28T17:59:08.661 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.905+0000 7f66d7400700 1 -- 172.21.15.37:0/3184675376 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c8004300 con 0x7f66d8081aa0 2024-03-28T17:59:08.661 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.905+0000 7f66d7400700 1 -- 172.21.15.37:0/3184675376 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66c80036a0 con 0x7f66d8081aa0 2024-03-28T17:59:08.661 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.905+0000 7f66df000700 1 -- 172.21.15.37:0/3184675376 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66d8081aa0 msgr2=0x7f66d8081ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:08.661 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.905+0000 7f66df000700 1 --2- 172.21.15.37:0/3184675376 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66d8081aa0 0x7f66d8081ec0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f66c80098a0 tx=0x7f66c8009bb0 comp rx=0 tx=0).stop 2024-03-28T17:59:08.661 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.905+0000 7f66df000700 1 -- 172.21.15.37:0/3184675376 shutdown_connections 2024-03-28T17:59:08.661 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.905+0000 7f66df000700 1 --2- 172.21.15.37:0/3184675376 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66d8081aa0 0x7f66d8081ec0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:08.661 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.905+0000 7f66df000700 1 -- 172.21.15.37:0/3184675376 >> 172.21.15.37:0/3184675376 conn(0x7f66d807d040 msgr2=0x7f66d807f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:08.661 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.905+0000 7f66df000700 1 -- 172.21.15.37:0/3184675376 shutdown_connections 2024-03-28T17:59:08.661 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.906+0000 7f66df000700 1 -- 172.21.15.37:0/3184675376 wait complete. 2024-03-28T17:59:08.661 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.907+0000 7f66df000700 1 Processor -- start 2024-03-28T17:59:08.661 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.907+0000 7f66df000700 1 -- start start 2024-03-28T17:59:08.661 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.907+0000 7f66df000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66d8081aa0 0x7f66d81a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:08.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.907+0000 7f66df000700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d810f7b0 con 0x7f66d8081aa0 2024-03-28T17:59:08.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.908+0000 7f66d7e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66d8081aa0 0x7f66d81a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:08.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.908+0000 7f66d7e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66d8081aa0 0x7f66d81a1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60562/0 (socket says 172.21.15.37:60562) 2024-03-28T17:59:08.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.908+0000 7f66d7e00700 1 -- 172.21.15.37:0/2652118158 learned_addr learned my addr 172.21.15.37:0/2652118158 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:08.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.908+0000 7f66d7e00700 1 -- 172.21.15.37:0/2652118158 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d81a1c70 con 0x7f66d8081aa0 2024-03-28T17:59:08.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.908+0000 7f66d7e00700 1 --2- 172.21.15.37:0/2652118158 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66d8081aa0 0x7f66d81a1730 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f66c8000c00 tx=0x7f66c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:08.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.909+0000 7f66d5600700 1 -- 172.21.15.37:0/2652118158 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66c8003db0 con 0x7f66d8081aa0 2024-03-28T17:59:08.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.909+0000 7f66d5600700 1 -- 172.21.15.37:0/2652118158 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c8011630 con 0x7f66d8081aa0 2024-03-28T17:59:08.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.909+0000 7f66d5600700 1 -- 172.21.15.37:0/2652118158 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66c8011950 con 0x7f66d8081aa0 2024-03-28T17:59:08.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.909+0000 7f66df000700 1 -- 172.21.15.37:0/2652118158 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d81a1f00 con 0x7f66d8081aa0 2024-03-28T17:59:08.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.909+0000 7f66df000700 1 -- 172.21.15.37:0/2652118158 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d81a4bf0 con 0x7f66d8081aa0 2024-03-28T17:59:08.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.909+0000 7f66d5600700 1 -- 172.21.15.37:0/2652118158 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f66c8011af0 con 0x7f66d8081aa0 2024-03-28T17:59:08.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.909+0000 7f66d5600700 1 -- 172.21.15.37:0/2652118158 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f66c801c450 con 0x7f66d8081aa0 2024-03-28T17:59:08.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.910+0000 7f66df000700 1 -- 172.21.15.37:0/2652118158 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d80637c0 con 0x7f66d8081aa0 2024-03-28T17:59:08.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.913+0000 7f66d5600700 1 -- 172.21.15.37:0/2652118158 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f66c8032020 con 0x7f66d8081aa0 2024-03-28T17:59:08.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.989+0000 7f66df000700 1 -- 172.21.15.37:0/2652118158 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f66d802dbb0 con 0x7f66d8081aa0 2024-03-28T17:59:08.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.995+0000 7f66d5600700 1 -- 172.21.15.37:0/2652118158 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f66c8004260 con 0x7f66d8081aa0 2024-03-28T17:59:08.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.996+0000 7f66d5600700 1 -- 172.21.15.37:0/2652118158 <== mon.0 v2:172.21.15.37:3300/0 8 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f66c801cb60 con 0x7f66d8081aa0 2024-03-28T17:59:08.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.997+0000 7f66df000700 1 -- 172.21.15.37:0/2652118158 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66d8081aa0 msgr2=0x7f66d81a1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:08.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.997+0000 7f66df000700 1 --2- 172.21.15.37:0/2652118158 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66d8081aa0 0x7f66d81a1730 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f66c8000c00 tx=0x7f66c8004750 comp rx=0 tx=0).stop 2024-03-28T17:59:08.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.997+0000 7f66df000700 1 -- 172.21.15.37:0/2652118158 shutdown_connections 2024-03-28T17:59:08.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.997+0000 7f66df000700 1 --2- 172.21.15.37:0/2652118158 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66d8081aa0 0x7f66d81a1730 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:08.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.997+0000 7f66df000700 1 -- 172.21.15.37:0/2652118158 >> 172.21.15.37:0/2652118158 conn(0x7f66d807d040 msgr2=0x7f66d8198430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:08.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.997+0000 7f66df000700 1 -- 172.21.15.37:0/2652118158 shutdown_connections 2024-03-28T17:59:08.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:07.997+0000 7f66df000700 1 -- 172.21.15.37:0/2652118158 wait complete. 2024-03-28T17:59:08.663 INFO:teuthology.orchestra.run.smithi037.stdout:Generating new minimal ceph.conf... 2024-03-28T17:59:09.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:09 smithi037 ceph-mon[31083]: from='client.? 172.21.15.37:0/629660136' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:59:09.970 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.264+0000 7f9bb3400700 1 Processor -- start 2024-03-28T17:59:09.970 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.265+0000 7f9bb3400700 1 -- start start 2024-03-28T17:59:09.970 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.265+0000 7f9bb3400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bac07de60 0x7f9bac07c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:09.970 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.265+0000 7f9bb3400700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac07c890 con 0x7f9bac07de60 2024-03-28T17:59:09.971 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.266+0000 7f9bb0a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bac07de60 0x7f9bac07c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:09.971 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.266+0000 7f9bb0a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bac07de60 0x7f9bac07c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60572/0 (socket says 172.21.15.37:60572) 2024-03-28T17:59:09.971 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.266+0000 7f9bb0a00700 1 -- 172.21.15.37:0/1028942379 learned_addr learned my addr 172.21.15.37:0/1028942379 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:09.971 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.266+0000 7f9bb0a00700 1 -- 172.21.15.37:0/1028942379 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac07ca10 con 0x7f9bac07de60 2024-03-28T17:59:09.971 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.267+0000 7f9bb0a00700 1 --2- 172.21.15.37:0/1028942379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bac07de60 0x7f9bac07c2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9ba00098a0 tx=0x7f9ba0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab3c17abbf7e56e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:09.971 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.267+0000 7f9bab400700 1 -- 172.21.15.37:0/1028942379 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ba0004160 con 0x7f9bac07de60 2024-03-28T17:59:09.974 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.267+0000 7f9bab400700 1 -- 172.21.15.37:0/1028942379 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ba0004300 con 0x7f9bac07de60 2024-03-28T17:59:09.974 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.267+0000 7f9bab400700 1 -- 172.21.15.37:0/1028942379 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ba0011b50 con 0x7f9bac07de60 2024-03-28T17:59:09.974 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.268+0000 7f9bb3400700 1 -- 172.21.15.37:0/1028942379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bac07de60 msgr2=0x7f9bac07c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:09.974 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.268+0000 7f9bb3400700 1 --2- 172.21.15.37:0/1028942379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bac07de60 0x7f9bac07c2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9ba00098a0 tx=0x7f9ba0009bb0 comp rx=0 tx=0).stop 2024-03-28T17:59:09.974 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.268+0000 7f9bb3400700 1 -- 172.21.15.37:0/1028942379 shutdown_connections 2024-03-28T17:59:09.974 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.268+0000 7f9bb3400700 1 --2- 172.21.15.37:0/1028942379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bac07de60 0x7f9bac07c2c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:09.974 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.268+0000 7f9bb3400700 1 -- 172.21.15.37:0/1028942379 >> 172.21.15.37:0/1028942379 conn(0x7f9bac103080 msgr2=0x7f9bac1054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:09.975 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.268+0000 7f9bb3400700 1 -- 172.21.15.37:0/1028942379 shutdown_connections 2024-03-28T17:59:09.975 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.268+0000 7f9bb3400700 1 -- 172.21.15.37:0/1028942379 wait complete. 2024-03-28T17:59:09.975 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.269+0000 7f9bb3400700 1 Processor -- start 2024-03-28T17:59:09.975 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.269+0000 7f9bb3400700 1 -- start start 2024-03-28T17:59:09.975 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.270+0000 7f9bb3400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bac07de60 0x7f9bac19dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:09.975 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.270+0000 7f9bb3400700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac10e0b0 con 0x7f9bac07de60 2024-03-28T17:59:09.975 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.270+0000 7f9bb0a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bac07de60 0x7f9bac19dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:09.975 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.270+0000 7f9bb0a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bac07de60 0x7f9bac19dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60584/0 (socket says 172.21.15.37:60584) 2024-03-28T17:59:09.975 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.270+0000 7f9bb0a00700 1 -- 172.21.15.37:0/629660136 learned_addr learned my addr 172.21.15.37:0/629660136 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:09.975 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.270+0000 7f9bb0a00700 1 -- 172.21.15.37:0/629660136 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac19e160 con 0x7f9bac07de60 2024-03-28T17:59:09.975 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.271+0000 7f9bb0a00700 1 --2- 172.21.15.37:0/629660136 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bac07de60 0x7f9bac19dc20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0011b60 tx=0x7f9ba0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:09.975 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.271+0000 7f9ba9600700 1 -- 172.21.15.37:0/629660136 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ba0003b20 con 0x7f9bac07de60 2024-03-28T17:59:09.975 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.271+0000 7f9ba9600700 1 -- 172.21.15.37:0/629660136 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ba0003cc0 con 0x7f9bac07de60 2024-03-28T17:59:09.975 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.271+0000 7f9ba9600700 1 -- 172.21.15.37:0/629660136 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ba0024840 con 0x7f9bac07de60 2024-03-28T17:59:09.975 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.271+0000 7f9bb3400700 1 -- 172.21.15.37:0/629660136 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bac19e3f0 con 0x7f9bac07de60 2024-03-28T17:59:09.976 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.271+0000 7f9bb3400700 1 -- 172.21.15.37:0/629660136 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bac1a10e0 con 0x7f9bac07de60 2024-03-28T17:59:09.976 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.272+0000 7f9ba9600700 1 -- 172.21.15.37:0/629660136 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9ba0022020 con 0x7f9bac07de60 2024-03-28T17:59:09.976 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.272+0000 7f9ba9600700 1 -- 172.21.15.37:0/629660136 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9ba0011e00 con 0x7f9bac07de60 2024-03-28T17:59:09.976 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.273+0000 7f9bb3400700 1 -- 172.21.15.37:0/629660136 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bac0637c0 con 0x7f9bac07de60 2024-03-28T17:59:09.976 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.275+0000 7f9ba9600700 1 -- 172.21.15.37:0/629660136 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9ba0043dd0 con 0x7f9bac07de60 2024-03-28T17:59:09.976 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.351+0000 7f9bb3400700 1 -- 172.21.15.37:0/629660136 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f9bac1a17c0 con 0x7f9bac07de60 2024-03-28T17:59:09.976 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.352+0000 7f9ba9600700 1 -- 172.21.15.37:0/629660136 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f9ba0032020 con 0x7f9bac07de60 2024-03-28T17:59:09.976 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.353+0000 7f9bb3400700 1 -- 172.21.15.37:0/629660136 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bac07de60 msgr2=0x7f9bac19dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:09.976 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.353+0000 7f9bb3400700 1 --2- 172.21.15.37:0/629660136 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bac07de60 0x7f9bac19dc20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0011b60 tx=0x7f9ba0011c40 comp rx=0 tx=0).stop 2024-03-28T17:59:09.977 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.353+0000 7f9bb3400700 1 -- 172.21.15.37:0/629660136 shutdown_connections 2024-03-28T17:59:09.977 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.353+0000 7f9bb3400700 1 --2- 172.21.15.37:0/629660136 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bac07de60 0x7f9bac19dc20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:09.977 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.353+0000 7f9bb3400700 1 -- 172.21.15.37:0/629660136 >> 172.21.15.37:0/629660136 conn(0x7f9bac103080 msgr2=0x7f9bac103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:09.977 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.354+0000 7f9bb3400700 1 -- 172.21.15.37:0/629660136 shutdown_connections 2024-03-28T17:59:09.977 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:09.354+0000 7f9bb3400700 1 -- 172.21.15.37:0/629660136 wait complete. 2024-03-28T17:59:09.977 INFO:teuthology.orchestra.run.smithi037.stdout:Restarting the monitor... 2024-03-28T17:59:10.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:10 smithi037 systemd[1]: Stopping Ceph mon.a for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T17:59:10.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mon-a[31079]: 2024-03-28T17:59:10.470+0000 7fa941e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T17:59:10.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mon-a[31079]: 2024-03-28T17:59:10.470+0000 7fa941e00700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-28T17:59:10.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:10 smithi037 podman[31336]: 2024-03-28 17:59:10.595056751 +0000 UTC m=+0.260065282 container died 90b0d780c3a9ad0d50cec10a1bc3d3149390f7e18bc13cc173f37b7a27fc799a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mon-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-28T17:59:11.254 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:10 smithi037 podman[31336]: 2024-03-28 17:59:10.995583692 +0000 UTC m=+0.660592213 container cleanup 90b0d780c3a9ad0d50cec10a1bc3d3149390f7e18bc13cc173f37b7a27fc799a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-28T17:59:11.254 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:10 smithi037 bash[31336]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mon-a 2024-03-28T17:59:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:11 smithi037 podman[31349]: 2024-03-28 17:59:11.254602997 +0000 UTC m=+0.649914840 container remove 90b0d780c3a9ad0d50cec10a1bc3d3149390f7e18bc13cc173f37b7a27fc799a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T17:59:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:11 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-28T17:59:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:11 smithi037 systemd[1]: Stopped Ceph mon.a for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T17:59:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:11 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mon.a.service: Consumed 1.913s CPU time. 2024-03-28T17:59:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:11 smithi037 systemd[1]: Starting Ceph mon.a for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T17:59:12.409 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 podman[31442]: 2024-03-28 17:59:12.031437608 +0000 UTC m=+0.034610088 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:59:12.409 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 podman[31442]: 2024-03-28 17:59:12.130387486 +0000 UTC m=+0.133559986 container create 10bef11fe84fa6c8e56d2e237642e020b13f732885484cf501699d33b527bee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-28T17:59:12.409 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 podman[31442]: 2024-03-28 17:59:12.408833785 +0000 UTC m=+0.412006328 container init 10bef11fe84fa6c8e56d2e237642e020b13f732885484cf501699d33b527bee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-03-28T17:59:12.500 INFO:teuthology.orchestra.run.smithi037.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-28T17:59:12.792 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 podman[31442]: 2024-03-28 17:59:12.419406958 +0000 UTC m=+0.422579428 container start 10bef11fe84fa6c8e56d2e237642e020b13f732885484cf501699d33b527bee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-28T17:59:12.792 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: set uid:gid to 167:167 (ceph:ceph) 2024-03-28T17:59:12.792 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-28T17:59:12.792 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: pidfile_write: ignore empty --pid-file 2024-03-28T17:59:12.793 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: load: jerasure load: lrc 2024-03-28T17:59:12.793 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: RocksDB version: 7.9.2 2024-03-28T17:59:12.793 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Git sha 0 2024-03-28T17:59:12.793 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-28T17:59:12.793 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: DB SUMMARY 2024-03-28T17:59:12.793 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: DB Session ID: U2H01BI3FXZF1CBKAUTI 2024-03-28T17:59:12.793 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: CURRENT file: CURRENT 2024-03-28T17:59:12.793 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: IDENTITY file: IDENTITY 2024-03-28T17:59:12.793 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-28T17:59:12.793 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-28T17:59:12.793 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 100066 ; 2024-03-28T17:59:12.793 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.error_if_exists: 0 2024-03-28T17:59:12.793 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.create_if_missing: 0 2024-03-28T17:59:12.793 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.paranoid_checks: 1 2024-03-28T17:59:12.794 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-28T17:59:12.794 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-28T17:59:12.794 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-28T17:59:12.794 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.env: 0x560d370d4b20 2024-03-28T17:59:12.794 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.fs: PosixFileSystem 2024-03-28T17:59:12.794 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.info_log: 0x560d3987d960 2024-03-28T17:59:12.794 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_file_opening_threads: 16 2024-03-28T17:59:12.794 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.statistics: (nil) 2024-03-28T17:59:12.794 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.use_fsync: 0 2024-03-28T17:59:12.794 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_log_file_size: 0 2024-03-28T17:59:12.794 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-28T17:59:12.794 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-28T17:59:12.794 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.keep_log_file_num: 1000 2024-03-28T17:59:12.794 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.recycle_log_file_num: 0 2024-03-28T17:59:12.795 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.allow_fallocate: 1 2024-03-28T17:59:12.795 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.allow_mmap_reads: 0 2024-03-28T17:59:12.795 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.allow_mmap_writes: 0 2024-03-28T17:59:12.795 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.use_direct_reads: 0 2024-03-28T17:59:12.795 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-28T17:59:12.795 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.create_missing_column_families: 0 2024-03-28T17:59:12.796 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.db_log_dir: 2024-03-28T17:59:12.796 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.wal_dir: 2024-03-28T17:59:12.796 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-28T17:59:12.796 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-28T17:59:12.796 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-28T17:59:12.796 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-28T17:59:12.796 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-28T17:59:12.796 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-28T17:59:12.796 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.advise_random_on_open: 1 2024-03-28T17:59:12.796 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.db_write_buffer_size: 0 2024-03-28T17:59:12.796 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.write_buffer_manager: 0x560d398501e0 2024-03-28T17:59:12.796 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-28T17:59:12.796 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-28T17:59:12.797 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-28T17:59:12.797 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.rate_limiter: (nil) 2024-03-28T17:59:12.797 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-28T17:59:12.797 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.wal_recovery_mode: 2 2024-03-28T17:59:12.797 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.enable_thread_tracking: 0 2024-03-28T17:59:12.797 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.enable_pipelined_write: 0 2024-03-28T17:59:12.797 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.unordered_write: 0 2024-03-28T17:59:12.797 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-28T17:59:12.797 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-28T17:59:12.797 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-28T17:59:12.797 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-28T17:59:12.797 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.row_cache: None 2024-03-28T17:59:12.797 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.wal_filter: None 2024-03-28T17:59:12.797 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-28T17:59:12.798 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.allow_ingest_behind: 0 2024-03-28T17:59:12.798 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.two_write_queues: 0 2024-03-28T17:59:12.798 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.manual_wal_flush: 0 2024-03-28T17:59:12.798 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.wal_compression: 0 2024-03-28T17:59:12.798 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.atomic_flush: 0 2024-03-28T17:59:12.798 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-28T17:59:12.798 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-28T17:59:12.798 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-28T17:59:12.798 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.log_readahead_size: 0 2024-03-28T17:59:12.798 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-28T17:59:12.798 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.best_efforts_recovery: 0 2024-03-28T17:59:12.798 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-28T17:59:12.799 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-28T17:59:12.799 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.allow_data_in_errors: 0 2024-03-28T17:59:12.799 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.db_host_id: __hostname__ 2024-03-28T17:59:12.799 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-28T17:59:12.799 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_background_jobs: 2 2024-03-28T17:59:12.799 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_background_compactions: -1 2024-03-28T17:59:12.799 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_subcompactions: 1 2024-03-28T17:59:12.799 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-28T17:59:12.799 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-28T17:59:12.799 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-28T17:59:12.799 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_total_wal_size: 0 2024-03-28T17:59:12.799 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-28T17:59:12.799 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-28T17:59:12.799 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-28T17:59:12.800 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-28T17:59:12.800 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_open_files: -1 2024-03-28T17:59:12.800 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.bytes_per_sync: 0 2024-03-28T17:59:12.800 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-28T17:59:12.800 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-28T17:59:12.800 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compaction_readahead_size: 0 2024-03-28T17:59:12.800 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_background_flushes: -1 2024-03-28T17:59:12.800 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Compression algorithms supported: 2024-03-28T17:59:12.800 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-28T17:59:12.800 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: kZSTD supported: 0 2024-03-28T17:59:12.800 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: kXpressCompression supported: 0 2024-03-28T17:59:12.800 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: kLZ4HCCompression supported: 1 2024-03-28T17:59:12.800 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: kZlibCompression supported: 1 2024-03-28T17:59:12.800 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: kSnappyCompression supported: 1 2024-03-28T17:59:12.800 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: kLZ4Compression supported: 1 2024-03-28T17:59:12.801 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: kBZip2Compression supported: 0 2024-03-28T17:59:12.801 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-28T17:59:12.801 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-28T17:59:12.801 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-28T17:59:12.801 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-28T17:59:12.801 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-28T17:59:12.801 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.merge_operator: 2024-03-28T17:59:12.802 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compaction_filter: None 2024-03-28T17:59:12.802 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compaction_filter_factory: None 2024-03-28T17:59:12.802 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.sst_partitioner_factory: None 2024-03-28T17:59:12.802 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-28T17:59:12.802 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-28T17:59:12.803 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560d3987da60) 2024-03-28T17:59:12.803 INFO:journalctl@ceph.mon.a.smithi037.stdout: cache_index_and_filter_blocks: 1 2024-03-28T17:59:12.803 INFO:journalctl@ceph.mon.a.smithi037.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-28T17:59:12.803 INFO:journalctl@ceph.mon.a.smithi037.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-28T17:59:12.803 INFO:journalctl@ceph.mon.a.smithi037.stdout: pin_top_level_index_and_filter: 1 2024-03-28T17:59:12.803 INFO:journalctl@ceph.mon.a.smithi037.stdout: index_type: 0 2024-03-28T17:59:12.803 INFO:journalctl@ceph.mon.a.smithi037.stdout: data_block_index_type: 0 2024-03-28T17:59:12.803 INFO:journalctl@ceph.mon.a.smithi037.stdout: index_shortening: 1 2024-03-28T17:59:12.803 INFO:journalctl@ceph.mon.a.smithi037.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-28T17:59:12.803 INFO:journalctl@ceph.mon.a.smithi037.stdout: checksum: 4 2024-03-28T17:59:12.803 INFO:journalctl@ceph.mon.a.smithi037.stdout: no_block_cache: 0 2024-03-28T17:59:12.803 INFO:journalctl@ceph.mon.a.smithi037.stdout: block_cache: 0x560d39883090 2024-03-28T17:59:12.803 INFO:journalctl@ceph.mon.a.smithi037.stdout: block_cache_name: BinnedLRUCache 2024-03-28T17:59:12.803 INFO:journalctl@ceph.mon.a.smithi037.stdout: block_cache_options: 2024-03-28T17:59:12.803 INFO:journalctl@ceph.mon.a.smithi037.stdout: capacity : 536870912 2024-03-28T17:59:12.804 INFO:journalctl@ceph.mon.a.smithi037.stdout: num_shard_bits : 4 2024-03-28T17:59:12.804 INFO:journalctl@ceph.mon.a.smithi037.stdout: strict_capacity_limit : 0 2024-03-28T17:59:12.804 INFO:journalctl@ceph.mon.a.smithi037.stdout: high_pri_pool_ratio: 0.000 2024-03-28T17:59:12.804 INFO:journalctl@ceph.mon.a.smithi037.stdout: block_cache_compressed: (nil) 2024-03-28T17:59:12.804 INFO:journalctl@ceph.mon.a.smithi037.stdout: persistent_cache: (nil) 2024-03-28T17:59:12.804 INFO:journalctl@ceph.mon.a.smithi037.stdout: block_size: 4096 2024-03-28T17:59:12.804 INFO:journalctl@ceph.mon.a.smithi037.stdout: block_size_deviation: 10 2024-03-28T17:59:12.804 INFO:journalctl@ceph.mon.a.smithi037.stdout: block_restart_interval: 16 2024-03-28T17:59:12.804 INFO:journalctl@ceph.mon.a.smithi037.stdout: index_block_restart_interval: 1 2024-03-28T17:59:12.804 INFO:journalctl@ceph.mon.a.smithi037.stdout: metadata_block_size: 4096 2024-03-28T17:59:12.804 INFO:journalctl@ceph.mon.a.smithi037.stdout: partition_filters: 0 2024-03-28T17:59:12.804 INFO:journalctl@ceph.mon.a.smithi037.stdout: use_delta_encoding: 1 2024-03-28T17:59:12.804 INFO:journalctl@ceph.mon.a.smithi037.stdout: filter_policy: bloomfilter 2024-03-28T17:59:12.804 INFO:journalctl@ceph.mon.a.smithi037.stdout: whole_key_filtering: 1 2024-03-28T17:59:12.805 INFO:journalctl@ceph.mon.a.smithi037.stdout: verify_compression: 0 2024-03-28T17:59:12.805 INFO:journalctl@ceph.mon.a.smithi037.stdout: read_amp_bytes_per_bit: 0 2024-03-28T17:59:12.805 INFO:journalctl@ceph.mon.a.smithi037.stdout: format_version: 5 2024-03-28T17:59:12.805 INFO:journalctl@ceph.mon.a.smithi037.stdout: enable_index_compression: 1 2024-03-28T17:59:12.805 INFO:journalctl@ceph.mon.a.smithi037.stdout: block_align: 0 2024-03-28T17:59:12.805 INFO:journalctl@ceph.mon.a.smithi037.stdout: max_auto_readahead_size: 262144 2024-03-28T17:59:12.805 INFO:journalctl@ceph.mon.a.smithi037.stdout: prepopulate_block_cache: 0 2024-03-28T17:59:12.805 INFO:journalctl@ceph.mon.a.smithi037.stdout: initial_auto_readahead_size: 8192 2024-03-28T17:59:12.805 INFO:journalctl@ceph.mon.a.smithi037.stdout: num_file_reads_for_auto_readahead: 2 2024-03-28T17:59:12.805 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.write_buffer_size: 33554432 2024-03-28T17:59:12.805 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_write_buffer_number: 2 2024-03-28T17:59:12.805 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compression: NoCompression 2024-03-28T17:59:12.805 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.bottommost_compression: Disabled 2024-03-28T17:59:12.805 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.prefix_extractor: nullptr 2024-03-28T17:59:12.805 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-28T17:59:12.806 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.num_levels: 7 2024-03-28T17:59:12.806 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-28T17:59:12.806 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-28T17:59:12.806 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-28T17:59:12.806 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-28T17:59:12.806 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-28T17:59:12.806 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-28T17:59:12.806 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-28T17:59:12.806 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-28T17:59:12.806 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-28T17:59:12.806 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-28T17:59:12.806 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-28T17:59:12.806 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-28T17:59:12.806 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-28T17:59:12.807 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compression_opts.level: 32767 2024-03-28T17:59:12.807 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compression_opts.strategy: 0 2024-03-28T17:59:12.807 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-28T17:59:12.807 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-28T17:59:12.807 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-28T17:59:12.807 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-28T17:59:12.807 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compression_opts.enabled: false 2024-03-28T17:59:12.807 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-28T17:59:12.807 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-28T17:59:12.807 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-28T17:59:12.807 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-28T17:59:12.807 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.target_file_size_base: 67108864 2024-03-28T17:59:12.807 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-28T17:59:12.807 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-28T17:59:12.807 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-28T17:59:12.808 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-28T17:59:12.808 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-28T17:59:12.808 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-28T17:59:12.808 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-28T17:59:12.808 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-28T17:59:12.808 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-28T17:59:12.808 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-28T17:59:12.808 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-28T17:59:12.808 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-28T17:59:12.808 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-28T17:59:12.808 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-28T17:59:12.808 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.arena_block_size: 1048576 2024-03-28T17:59:12.808 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-28T17:59:12.808 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-28T17:59:12.809 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.disable_auto_compactions: 0 2024-03-28T17:59:12.809 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-28T17:59:12.809 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-28T17:59:12.809 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-28T17:59:12.809 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-28T17:59:12.809 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-28T17:59:12.809 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-28T17:59:12.809 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-28T17:59:12.809 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-28T17:59:12.809 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-28T17:59:12.809 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-28T17:59:12.809 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.table_properties_collectors: 2024-03-28T17:59:12.809 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.inplace_update_support: 0 2024-03-28T17:59:12.809 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-28T17:59:12.810 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-28T17:59:12.810 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-28T17:59:12.810 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-28T17:59:12.810 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.bloom_locality: 0 2024-03-28T17:59:12.810 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.max_successive_merges: 0 2024-03-28T17:59:12.810 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-28T17:59:12.810 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.paranoid_file_checks: 0 2024-03-28T17:59:12.810 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.force_consistency_checks: 1 2024-03-28T17:59:12.810 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.report_bg_io_stats: 0 2024-03-28T17:59:12.810 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.ttl: 2592000 2024-03-28T17:59:12.810 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-28T17:59:12.810 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-28T17:59:12.810 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-28T17:59:12.810 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.enable_blob_files: false 2024-03-28T17:59:12.811 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.min_blob_size: 0 2024-03-28T17:59:12.811 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.blob_file_size: 268435456 2024-03-28T17:59:12.811 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-28T17:59:12.811 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-28T17:59:12.811 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-28T17:59:12.811 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-28T17:59:12.811 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-28T17:59:12.811 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.blob_file_starting_level: 0 2024-03-28T17:59:12.811 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-28T17:59:12.811 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-28T17:59:12.811 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-28T17:59:12.811 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e5f0fad8-3439-43de-a3ca-ff3b74dcfb30 2024-03-28T17:59:12.811 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711648752481410, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-28T17:59:12.811 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-28T17:59:12.812 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711648752483242, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 95284, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 316, "table_properties": {"data_size": 93283, "index_size": 283, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13771, "raw_average_key_size": 50, "raw_value_size": 85756, "raw_average_value_size": 311, "num_data_blocks": 11, "num_entries": 275, "num_filter_entries": 275, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711648752, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e5f0fad8-3439-43de-a3ca-ff3b74dcfb30", "db_session_id": "U2H01BI3FXZF1CBKAUTI", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-28T17:59:12.812 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711648752483398, "job": 1, "event": "recovery_finished"} 2024-03-28T17:59:12.812 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-28T17:59:12.812 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-28T17:59:12.812 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560d39976000 2024-03-28T17:59:12.812 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: DB pointer 0x560d39960000 2024-03-28T17:59:12.812 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-28T17:59:12.812 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-28T17:59:12.812 INFO:journalctl@ceph.mon.a.smithi037.stdout: ** DB Stats ** 2024-03-28T17:59:12.812 INFO:journalctl@ceph.mon.a.smithi037.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T17:59:12.812 INFO:journalctl@ceph.mon.a.smithi037.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-28T17:59:12.812 INFO:journalctl@ceph.mon.a.smithi037.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T17:59:12.812 INFO:journalctl@ceph.mon.a.smithi037.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T17:59:12.812 INFO:journalctl@ceph.mon.a.smithi037.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-28T17:59:12.813 INFO:journalctl@ceph.mon.a.smithi037.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T17:59:12.813 INFO:journalctl@ceph.mon.a.smithi037.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T17:59:12.813 INFO:journalctl@ceph.mon.a.smithi037.stdout: 2024-03-28T17:59:12.813 INFO:journalctl@ceph.mon.a.smithi037.stdout: ** Compaction Stats [default] ** 2024-03-28T17:59:12.813 INFO:journalctl@ceph.mon.a.smithi037.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T17:59:12.813 INFO:journalctl@ceph.mon.a.smithi037.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-28T17:59:12.813 INFO:journalctl@ceph.mon.a.smithi037.stdout: L0 2/0 94.89 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 75.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T17:59:12.813 INFO:journalctl@ceph.mon.a.smithi037.stdout: Sum 2/0 94.89 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 75.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T17:59:12.813 INFO:journalctl@ceph.mon.a.smithi037.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 75.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T17:59:12.813 INFO:journalctl@ceph.mon.a.smithi037.stdout: 2024-03-28T17:59:12.813 INFO:journalctl@ceph.mon.a.smithi037.stdout: ** Compaction Stats [default] ** 2024-03-28T17:59:12.813 INFO:journalctl@ceph.mon.a.smithi037.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T17:59:12.813 INFO:journalctl@ceph.mon.a.smithi037.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-28T17:59:12.813 INFO:journalctl@ceph.mon.a.smithi037.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 75.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T17:59:12.813 INFO:journalctl@ceph.mon.a.smithi037.stdout: 2024-03-28T17:59:12.814 INFO:journalctl@ceph.mon.a.smithi037.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-28T17:59:12.814 INFO:journalctl@ceph.mon.a.smithi037.stdout: 2024-03-28T17:59:12.814 INFO:journalctl@ceph.mon.a.smithi037.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T17:59:12.814 INFO:journalctl@ceph.mon.a.smithi037.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-28T17:59:12.814 INFO:journalctl@ceph.mon.a.smithi037.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-28T17:59:12.814 INFO:journalctl@ceph.mon.a.smithi037.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-28T17:59:12.814 INFO:journalctl@ceph.mon.a.smithi037.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-28T17:59:12.814 INFO:journalctl@ceph.mon.a.smithi037.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-28T17:59:12.814 INFO:journalctl@ceph.mon.a.smithi037.stdout: Cumulative compaction: 0.00 GB write, 13.18 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T17:59:12.814 INFO:journalctl@ceph.mon.a.smithi037.stdout: Interval compaction: 0.00 GB write, 13.18 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T17:59:12.814 INFO:journalctl@ceph.mon.a.smithi037.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-28T17:59:12.814 INFO:journalctl@ceph.mon.a.smithi037.stdout: Block cache BinnedLRUCache@0x560d39883090#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-03-28T17:59:12.814 INFO:journalctl@ceph.mon.a.smithi037.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-28T17:59:12.814 INFO:journalctl@ceph.mon.a.smithi037.stdout: 2024-03-28T17:59:12.815 INFO:journalctl@ceph.mon.a.smithi037.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-28T17:59:12.815 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: starting mon.a rank 0 at public addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] at bind addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 2024-03-28T17:59:12.815 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: mon.a@-1(???) e1 preinit fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 2024-03-28T17:59:12.815 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: mon.a@-1(???).mds e1 new map 2024-03-28T17:59:12.815 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: mon.a@-1(???).mds e1 print_map 2024-03-28T17:59:12.815 INFO:journalctl@ceph.mon.a.smithi037.stdout: e1 2024-03-28T17:59:12.815 INFO:journalctl@ceph.mon.a.smithi037.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T17:59:12.815 INFO:journalctl@ceph.mon.a.smithi037.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T17:59:12.815 INFO:journalctl@ceph.mon.a.smithi037.stdout: legacy client fscid: -1 2024-03-28T17:59:12.815 INFO:journalctl@ceph.mon.a.smithi037.stdout: 2024-03-28T17:59:12.815 INFO:journalctl@ceph.mon.a.smithi037.stdout: No filesystems configured 2024-03-28T17:59:12.815 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T17:59:12.815 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T17:59:12.815 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T17:59:12.816 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T17:59:12.816 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-28T17:59:12.816 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 bash[31442]: 10bef11fe84fa6c8e56d2e237642e020b13f732885484cf501699d33b527bee4 2024-03-28T17:59:12.816 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 systemd[1]: Started Ceph mon.a for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T17:59:12.816 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T17:59:12.816 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: monmap e1: 1 mons at {a=[v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T17:59:12.816 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: fsmap 2024-03-28T17:59:12.816 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T17:59:12.816 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:12 smithi037 ceph-mon[31460]: mgrmap e1: no daemons active 2024-03-28T17:59:13.881 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.102+0000 7f0eaf200700 1 Processor -- start 2024-03-28T17:59:13.881 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.102+0000 7f0eaf200700 1 -- start start 2024-03-28T17:59:13.882 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.103+0000 7f0eaf200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0ea8109c80 0x7f0ea810a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:13.882 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.103+0000 7f0eaf200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea810a670 con 0x7f0ea8109c80 2024-03-28T17:59:13.882 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.104+0000 7f0ea7e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0ea8109c80 0x7f0ea810a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:13.882 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.104+0000 7f0ea7e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0ea8109c80 0x7f0ea810a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60598/0 (socket says 172.21.15.37:60598) 2024-03-28T17:59:13.882 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.104+0000 7f0ea7e00700 1 -- 172.21.15.37:0/2986424293 learned_addr learned my addr 172.21.15.37:0/2986424293 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:13.882 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.105+0000 7f0ea7e00700 1 -- 172.21.15.37:0/2986424293 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea810ae80 con 0x7f0ea8109c80 2024-03-28T17:59:13.882 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.105+0000 7f0ea7e00700 1 --2- 172.21.15.37:0/2986424293 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0ea8109c80 0x7f0ea810a0a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0e980098a0 tx=0x7f0e98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=669b382016984abe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:13.882 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.106+0000 7f0ea6a00700 1 -- 172.21.15.37:0/2986424293 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0e98004160 con 0x7f0ea8109c80 2024-03-28T17:59:13.882 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.106+0000 7f0ea6a00700 1 -- 172.21.15.37:0/2986424293 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e98004300 con 0x7f0ea8109c80 2024-03-28T17:59:13.882 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.106+0000 7f0ea6a00700 1 -- 172.21.15.37:0/2986424293 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0e98011b50 con 0x7f0ea8109c80 2024-03-28T17:59:13.882 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.106+0000 7f0eaf200700 1 -- 172.21.15.37:0/2986424293 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0ea8109c80 msgr2=0x7f0ea810a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:13.883 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.106+0000 7f0eaf200700 1 --2- 172.21.15.37:0/2986424293 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0ea8109c80 0x7f0ea810a0a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0e980098a0 tx=0x7f0e98009bb0 comp rx=0 tx=0).stop 2024-03-28T17:59:13.883 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.106+0000 7f0eaf200700 1 -- 172.21.15.37:0/2986424293 shutdown_connections 2024-03-28T17:59:13.883 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.106+0000 7f0eaf200700 1 --2- 172.21.15.37:0/2986424293 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0ea8109c80 0x7f0ea810a0a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:13.883 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.106+0000 7f0eaf200700 1 -- 172.21.15.37:0/2986424293 >> 172.21.15.37:0/2986424293 conn(0x7f0ea807d040 msgr2=0x7f0ea807d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:13.883 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.107+0000 7f0eaf200700 1 -- 172.21.15.37:0/2986424293 shutdown_connections 2024-03-28T17:59:13.883 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.107+0000 7f0eaf200700 1 -- 172.21.15.37:0/2986424293 wait complete. 2024-03-28T17:59:13.883 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.108+0000 7f0eaf200700 1 Processor -- start 2024-03-28T17:59:13.883 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.108+0000 7f0eaf200700 1 -- start start 2024-03-28T17:59:13.883 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.108+0000 7f0eaf200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0ea8109c80 0x7f0ea819dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:13.883 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.108+0000 7f0eaf200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea810e230 con 0x7f0ea8109c80 2024-03-28T17:59:13.883 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.108+0000 7f0ea7e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0ea8109c80 0x7f0ea819dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:13.883 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.108+0000 7f0ea7e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0ea8109c80 0x7f0ea819dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60608/0 (socket says 172.21.15.37:60608) 2024-03-28T17:59:13.884 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.108+0000 7f0ea7e00700 1 -- 172.21.15.37:0/3890554830 learned_addr learned my addr 172.21.15.37:0/3890554830 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:13.884 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.109+0000 7f0ea7e00700 1 -- 172.21.15.37:0/3890554830 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea819e150 con 0x7f0ea8109c80 2024-03-28T17:59:13.886 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.109+0000 7f0ea7e00700 1 --2- 172.21.15.37:0/3890554830 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0ea8109c80 0x7f0ea819dc10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0e98011670 tx=0x7f0e98011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:13.886 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.109+0000 7f0ea4c00700 1 -- 172.21.15.37:0/3890554830 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0e980039c0 con 0x7f0ea8109c80 2024-03-28T17:59:13.886 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.110+0000 7f0ea4c00700 1 -- 172.21.15.37:0/3890554830 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e98011e50 con 0x7f0ea8109c80 2024-03-28T17:59:13.886 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.110+0000 7f0ea4c00700 1 -- 172.21.15.37:0/3890554830 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0e9801a5c0 con 0x7f0ea8109c80 2024-03-28T17:59:13.886 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.110+0000 7f0eaf200700 1 -- 172.21.15.37:0/3890554830 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea819e3e0 con 0x7f0ea8109c80 2024-03-28T17:59:13.887 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.110+0000 7f0eaf200700 1 -- 172.21.15.37:0/3890554830 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea81a10d0 con 0x7f0ea8109c80 2024-03-28T17:59:13.887 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.110+0000 7f0ea4c00700 1 -- 172.21.15.37:0/3890554830 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0e9801ac00 con 0x7f0ea8109c80 2024-03-28T17:59:13.887 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.110+0000 7f0ea4c00700 1 -- 172.21.15.37:0/3890554830 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0e98019700 con 0x7f0ea8109c80 2024-03-28T17:59:13.887 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.111+0000 7f0eaf200700 1 -- 172.21.15.37:0/3890554830 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea80637c0 con 0x7f0ea8109c80 2024-03-28T17:59:13.887 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.114+0000 7f0ea4c00700 1 -- 172.21.15.37:0/3890554830 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f0e98003b60 con 0x7f0ea8109c80 2024-03-28T17:59:13.887 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.191+0000 7f0eaf200700 1 -- 172.21.15.37:0/3890554830 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f0ea81a12a0 con 0x7f0ea8109c80 2024-03-28T17:59:13.887 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.196+0000 7f0ea4c00700 1 -- 172.21.15.37:0/3890554830 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f0e98019900 con 0x7f0ea8109c80 2024-03-28T17:59:13.887 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.197+0000 7f0eaf200700 1 -- 172.21.15.37:0/3890554830 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0ea8109c80 msgr2=0x7f0ea819dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:13.887 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.197+0000 7f0eaf200700 1 --2- 172.21.15.37:0/3890554830 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0ea8109c80 0x7f0ea819dc10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0e98011670 tx=0x7f0e98011b60 comp rx=0 tx=0).stop 2024-03-28T17:59:13.887 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.197+0000 7f0eaf200700 1 -- 172.21.15.37:0/3890554830 shutdown_connections 2024-03-28T17:59:13.887 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.197+0000 7f0eaf200700 1 --2- 172.21.15.37:0/3890554830 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0ea8109c80 0x7f0ea819dc10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:13.887 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.197+0000 7f0eaf200700 1 -- 172.21.15.37:0/3890554830 >> 172.21.15.37:0/3890554830 conn(0x7f0ea807d040 msgr2=0x7f0ea8107460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:13.888 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.198+0000 7f0eaf200700 1 -- 172.21.15.37:0/3890554830 shutdown_connections 2024-03-28T17:59:13.888 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:13.198+0000 7f0eaf200700 1 -- 172.21.15.37:0/3890554830 wait complete. 2024-03-28T17:59:13.888 INFO:teuthology.orchestra.run.smithi037.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-28T17:59:13.912 INFO:teuthology.orchestra.run.smithi037.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-28T17:59:13.913 INFO:teuthology.orchestra.run.smithi037.stdout:Creating mgr... 2024-03-28T17:59:13.914 INFO:teuthology.orchestra.run.smithi037.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-28T17:59:13.915 INFO:teuthology.orchestra.run.smithi037.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-28T17:59:14.348 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mgr.x 2024-03-28T17:59:14.348 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stderr Failed to reset failed state of unit ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mgr.x.service: Unit ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mgr.x.service not loaded. 2024-03-28T17:59:14.605 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715.target.wants/ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@.service. 2024-03-28T17:59:14.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:14 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3890554830' entity='client.admin' 2024-03-28T17:59:15.013 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:14 smithi037 systemd[1]: Starting Ceph mgr.x for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T17:59:15.267 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:15 smithi037 podman[31675]: 2024-03-28 17:59:14.918725773 +0000 UTC m=+0.034214948 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:59:15.267 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:15 smithi037 podman[31675]: 2024-03-28 17:59:15.024283209 +0000 UTC m=+0.139772412 container create 203afa0bd6e2e0b6afa8ca8f85a5bb2612dfa8221ba551095433010763ae3183 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-28T17:59:15.268 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:15 smithi037 podman[31675]: 2024-03-28 17:59:15.254776629 +0000 UTC m=+0.370265865 container init 203afa0bd6e2e0b6afa8ca8f85a5bb2612dfa8221ba551095433010763ae3183 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-28T17:59:15.268 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:15 smithi037 podman[31675]: 2024-03-28 17:59:15.265285415 +0000 UTC m=+0.380774595 container start 203afa0bd6e2e0b6afa8ca8f85a5bb2612dfa8221ba551095433010763ae3183 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-28T17:59:15.355 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T17:59:15.355 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout disabled 2024-03-28T17:59:15.370 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T17:59:15.370 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout inactive 2024-03-28T17:59:15.370 INFO:teuthology.orchestra.run.smithi037.stdout:firewalld.service is not enabled 2024-03-28T17:59:15.370 INFO:teuthology.orchestra.run.smithi037.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T17:59:15.390 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T17:59:15.390 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout disabled 2024-03-28T17:59:15.407 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T17:59:15.407 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout inactive 2024-03-28T17:59:15.407 INFO:teuthology.orchestra.run.smithi037.stdout:firewalld.service is not enabled 2024-03-28T17:59:15.407 INFO:teuthology.orchestra.run.smithi037.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-28T17:59:15.407 INFO:teuthology.orchestra.run.smithi037.stdout:Waiting for mgr to start... 2024-03-28T17:59:15.408 INFO:teuthology.orchestra.run.smithi037.stdout:Waiting for mgr... 2024-03-28T17:59:15.566 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:15 smithi037 bash[31675]: 203afa0bd6e2e0b6afa8ca8f85a5bb2612dfa8221ba551095433010763ae3183 2024-03-28T17:59:15.566 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:15 smithi037 systemd[1]: Started Ceph mgr.x for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T17:59:16.214 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:16 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:16.071+0000 7f207bc3b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T17:59:16.215 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:16 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:16.198+0000 7f207bc3b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T17:59:16.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:16 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3406798564' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T17:59:16.979 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-03-28T17:59:16.979 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout { 2024-03-28T17:59:16.980 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "fsid": "b5ad257c-ed2c-11ee-95d0-87774f69a715", 2024-03-28T17:59:16.980 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T17:59:16.980 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T17:59:16.980 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T17:59:16.980 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T17:59:16.980 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:16.980 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T17:59:16.980 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T17:59:16.980 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 0 2024-03-28T17:59:16.980 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:59:16.980 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T17:59:16.980 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T17:59:16.980 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:59:16.981 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-28T17:59:16.981 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T17:59:16.981 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:59:16.981 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T17:59:16.981 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T17:59:16.981 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:16.981 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T17:59:16.981 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:59:16.981 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T17:59:16.981 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T17:59:16.981 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T17:59:16.989 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T17:59:16.989 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T17:59:16.989 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T17:59:16.989 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:16.989 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T17:59:16.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T17:59:16.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T17:59:16.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T17:59:16.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T17:59:16.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T17:59:16.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T17:59:16.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T17:59:16.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T17:59:16.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:16.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T17:59:16.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:59:16.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T17:59:16.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T17:59:16.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:16.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T17:59:16.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T17:59:16.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T17:59:16.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T17:59:16.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T17:59:16.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T17:59:16.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T17:59:16.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:59:16.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T17:59:16.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:16.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T17:59:16.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:59:16.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T17:59:05.950436+0000", 2024-03-28T17:59:16.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T17:59:16.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:16.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T17:59:16.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout } 2024-03-28T17:59:16.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.120+0000 7fbaca800700 1 Processor -- start 2024-03-28T17:59:16.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.120+0000 7fbaca800700 1 -- start start 2024-03-28T17:59:16.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.121+0000 7fbaca800700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbac4106970 0x7fbac4106d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:16.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.121+0000 7fbaca800700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac4107360 con 0x7fbac4106970 2024-03-28T17:59:16.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.121+0000 7fbac3400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbac4106970 0x7fbac4106d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:16.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.121+0000 7fbac3400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbac4106970 0x7fbac4106d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50190/0 (socket says 172.21.15.37:50190) 2024-03-28T17:59:16.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.121+0000 7fbac3400700 1 -- 172.21.15.37:0/3058758713 learned_addr learned my addr 172.21.15.37:0/3058758713 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:16.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.122+0000 7fbac3400700 1 -- 172.21.15.37:0/3058758713 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac4107b70 con 0x7fbac4106970 2024-03-28T17:59:16.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.122+0000 7fbac3400700 1 --2- 172.21.15.37:0/3058758713 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbac4106970 0x7fbac4106d90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbab40098a0 tx=0x7fbab4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f156a67f1a45b99 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:16.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.122+0000 7fbac2000700 1 -- 172.21.15.37:0/3058758713 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbab4004160 con 0x7fbac4106970 2024-03-28T17:59:16.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.122+0000 7fbac2000700 1 -- 172.21.15.37:0/3058758713 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbab4004300 con 0x7fbac4106970 2024-03-28T17:59:16.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.122+0000 7fbac2000700 1 -- 172.21.15.37:0/3058758713 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbab4011b50 con 0x7fbac4106970 2024-03-28T17:59:16.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.123+0000 7fbaca800700 1 -- 172.21.15.37:0/3058758713 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbac4106970 msgr2=0x7fbac4106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:16.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.123+0000 7fbaca800700 1 --2- 172.21.15.37:0/3058758713 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbac4106970 0x7fbac4106d90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbab40098a0 tx=0x7fbab4009bb0 comp rx=0 tx=0).stop 2024-03-28T17:59:16.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.123+0000 7fbaca800700 1 -- 172.21.15.37:0/3058758713 shutdown_connections 2024-03-28T17:59:16.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.123+0000 7fbaca800700 1 --2- 172.21.15.37:0/3058758713 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbac4106970 0x7fbac4106d90 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:16.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.123+0000 7fbaca800700 1 -- 172.21.15.37:0/3058758713 >> 172.21.15.37:0/3058758713 conn(0x7fbac4101f70 msgr2=0x7fbac4104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:16.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.123+0000 7fbaca800700 1 -- 172.21.15.37:0/3058758713 shutdown_connections 2024-03-28T17:59:16.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.123+0000 7fbaca800700 1 -- 172.21.15.37:0/3058758713 wait complete. 2024-03-28T17:59:16.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.124+0000 7fbaca800700 1 Processor -- start 2024-03-28T17:59:16.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.124+0000 7fbaca800700 1 -- start start 2024-03-28T17:59:16.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.125+0000 7fbaca800700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbac4106970 0x7fbac4191b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:16.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.125+0000 7fbaca800700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac410af20 con 0x7fbac4106970 2024-03-28T17:59:16.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.125+0000 7fbac3400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbac4106970 0x7fbac4191b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:16.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.125+0000 7fbac3400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbac4106970 0x7fbac4191b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50204/0 (socket says 172.21.15.37:50204) 2024-03-28T17:59:16.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.125+0000 7fbac3400700 1 -- 172.21.15.37:0/3406798564 learned_addr learned my addr 172.21.15.37:0/3406798564 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:16.994 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.125+0000 7fbac3400700 1 -- 172.21.15.37:0/3406798564 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac4192060 con 0x7fbac4106970 2024-03-28T17:59:16.994 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.126+0000 7fbac3400700 1 --2- 172.21.15.37:0/3406798564 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbac4106970 0x7fbac4191b20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbab4000c00 tx=0x7fbab4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:16.994 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.126+0000 7fbabbe00700 1 -- 172.21.15.37:0/3406798564 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbab4003b20 con 0x7fbac4106970 2024-03-28T17:59:16.994 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.126+0000 7fbabbe00700 1 -- 172.21.15.37:0/3406798564 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbab4003cc0 con 0x7fbac4106970 2024-03-28T17:59:16.994 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.126+0000 7fbabbe00700 1 -- 172.21.15.37:0/3406798564 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbab4024910 con 0x7fbac4106970 2024-03-28T17:59:16.994 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.126+0000 7fbaca800700 1 -- 172.21.15.37:0/3406798564 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac4190250 con 0x7fbac4106970 2024-03-28T17:59:16.994 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.126+0000 7fbaca800700 1 -- 172.21.15.37:0/3406798564 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac4190790 con 0x7fbac4106970 2024-03-28T17:59:16.994 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.126+0000 7fbabbe00700 1 -- 172.21.15.37:0/3406798564 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fbab4022020 con 0x7fbac4106970 2024-03-28T17:59:16.994 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.127+0000 7fbabbe00700 1 -- 172.21.15.37:0/3406798564 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbab4011e00 con 0x7fbac4106970 2024-03-28T17:59:16.994 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.127+0000 7fbaca800700 1 -- 172.21.15.37:0/3406798564 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac40637c0 con 0x7fbac4106970 2024-03-28T17:59:16.994 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.130+0000 7fbabbe00700 1 -- 172.21.15.37:0/3406798564 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fbab402ddf0 con 0x7fbac4106970 2024-03-28T17:59:16.994 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.210+0000 7fbaca800700 1 -- 172.21.15.37:0/3406798564 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbac4195430 con 0x7fbac4106970 2024-03-28T17:59:16.994 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.211+0000 7fbabbe00700 1 -- 172.21.15.37:0/3406798564 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fbab4032030 con 0x7fbac4106970 2024-03-28T17:59:16.994 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.213+0000 7fbaca800700 1 -- 172.21.15.37:0/3406798564 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbac4106970 msgr2=0x7fbac4191b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:16.995 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.213+0000 7fbaca800700 1 --2- 172.21.15.37:0/3406798564 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbac4106970 0x7fbac4191b20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbab4000c00 tx=0x7fbab4011c10 comp rx=0 tx=0).stop 2024-03-28T17:59:16.995 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.213+0000 7fbaca800700 1 -- 172.21.15.37:0/3406798564 shutdown_connections 2024-03-28T17:59:16.995 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.213+0000 7fbaca800700 1 --2- 172.21.15.37:0/3406798564 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbac4106970 0x7fbac4191b20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:16.995 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.213+0000 7fbaca800700 1 -- 172.21.15.37:0/3406798564 >> 172.21.15.37:0/3406798564 conn(0x7fbac4101f70 msgr2=0x7fbac4102e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:16.995 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.213+0000 7fbaca800700 1 -- 172.21.15.37:0/3406798564 shutdown_connections 2024-03-28T17:59:16.995 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:16.213+0000 7fbaca800700 1 -- 172.21.15.37:0/3406798564 wait complete. 2024-03-28T17:59:16.995 INFO:teuthology.orchestra.run.smithi037.stdout:mgr not available, waiting (1/15)... 2024-03-28T17:59:17.354 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:17.047+0000 7f207bc3b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T17:59:17.354 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:17.173+0000 7f207bc3b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T17:59:17.354 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:17.353+0000 7f207bc3b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T17:59:17.624 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:17.482+0000 7f207bc3b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T17:59:17.625 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:17.596+0000 7f207bc3b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T17:59:18.205 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:17.936+0000 7f207bc3b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T17:59:18.206 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:18 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:18.084+0000 7f207bc3b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T17:59:18.578 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:18 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:18.205+0000 7f207bc3b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T17:59:18.579 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:18 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:18.334+0000 7f207bc3b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T17:59:18.579 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:18 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:18.577+0000 7f207bc3b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T17:59:19.958 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:19 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2652462132' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T17:59:20.332 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:19 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:19.957+0000 7f207bc3b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T17:59:20.332 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:20 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:20.135+0000 7f207bc3b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T17:59:20.369 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-03-28T17:59:20.369 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout { 2024-03-28T17:59:20.370 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "fsid": "b5ad257c-ed2c-11ee-95d0-87774f69a715", 2024-03-28T17:59:20.370 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T17:59:20.370 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T17:59:20.370 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T17:59:20.370 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T17:59:20.371 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:20.371 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T17:59:20.371 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T17:59:20.371 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 0 2024-03-28T17:59:20.371 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:59:20.372 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T17:59:20.372 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T17:59:20.372 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:59:20.372 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-03-28T17:59:20.373 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T17:59:20.373 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:59:20.373 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T17:59:20.373 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T17:59:20.373 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:20.374 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T17:59:20.374 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:59:20.374 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T17:59:20.374 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T17:59:20.375 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T17:59:20.375 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T17:59:20.375 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T17:59:20.375 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T17:59:20.375 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:20.376 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T17:59:20.376 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T17:59:20.376 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T17:59:20.376 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T17:59:20.376 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T17:59:20.377 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T17:59:20.377 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T17:59:20.377 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T17:59:20.377 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T17:59:20.377 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:20.378 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T17:59:20.378 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:59:20.378 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T17:59:20.378 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T17:59:20.379 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:20.379 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T17:59:20.379 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T17:59:20.379 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T17:59:20.379 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T17:59:20.380 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T17:59:20.380 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T17:59:20.380 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T17:59:20.380 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:59:20.380 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T17:59:20.381 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:20.381 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T17:59:20.381 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:59:20.381 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T17:59:05.950436+0000", 2024-03-28T17:59:20.382 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T17:59:20.382 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:20.382 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T17:59:20.382 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout } 2024-03-28T17:59:20.382 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.557+0000 7fef76e00700 1 Processor -- start 2024-03-28T17:59:20.383 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.558+0000 7fef76e00700 1 -- start start 2024-03-28T17:59:20.383 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.558+0000 7fef76e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fef7007de60 0x7fef7007c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:20.383 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.558+0000 7fef76e00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7007c890 con 0x7fef7007de60 2024-03-28T17:59:20.383 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.558+0000 7fef6fe00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fef7007de60 0x7fef7007c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:20.383 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.558+0000 7fef6fe00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fef7007de60 0x7fef7007c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50216/0 (socket says 172.21.15.37:50216) 2024-03-28T17:59:20.384 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.558+0000 7fef6fe00700 1 -- 172.21.15.37:0/1054336626 learned_addr learned my addr 172.21.15.37:0/1054336626 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:20.384 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.559+0000 7fef6fe00700 1 -- 172.21.15.37:0/1054336626 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7007ca10 con 0x7fef7007de60 2024-03-28T17:59:20.384 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.559+0000 7fef6fe00700 1 --2- 172.21.15.37:0/1054336626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fef7007de60 0x7fef7007c2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fef600098a0 tx=0x7fef60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9634e274fb2dba70 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:20.384 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.559+0000 7fef6f400700 1 -- 172.21.15.37:0/1054336626 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef60004160 con 0x7fef7007de60 2024-03-28T17:59:20.385 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.559+0000 7fef6f400700 1 -- 172.21.15.37:0/1054336626 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef60004300 con 0x7fef7007de60 2024-03-28T17:59:20.385 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.559+0000 7fef6f400700 1 -- 172.21.15.37:0/1054336626 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef60011b50 con 0x7fef7007de60 2024-03-28T17:59:20.385 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.560+0000 7fef76e00700 1 -- 172.21.15.37:0/1054336626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fef7007de60 msgr2=0x7fef7007c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:20.385 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.560+0000 7fef76e00700 1 --2- 172.21.15.37:0/1054336626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fef7007de60 0x7fef7007c2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fef600098a0 tx=0x7fef60009bb0 comp rx=0 tx=0).stop 2024-03-28T17:59:20.385 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.560+0000 7fef76e00700 1 -- 172.21.15.37:0/1054336626 shutdown_connections 2024-03-28T17:59:20.386 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.560+0000 7fef76e00700 1 --2- 172.21.15.37:0/1054336626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fef7007de60 0x7fef7007c2c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:20.386 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.560+0000 7fef76e00700 1 -- 172.21.15.37:0/1054336626 >> 172.21.15.37:0/1054336626 conn(0x7fef70103080 msgr2=0x7fef701054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:20.386 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.560+0000 7fef76e00700 1 -- 172.21.15.37:0/1054336626 shutdown_connections 2024-03-28T17:59:20.386 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.560+0000 7fef76e00700 1 -- 172.21.15.37:0/1054336626 wait complete. 2024-03-28T17:59:20.386 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.562+0000 7fef76e00700 1 Processor -- start 2024-03-28T17:59:20.387 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.562+0000 7fef76e00700 1 -- start start 2024-03-28T17:59:20.387 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.562+0000 7fef76e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fef7007de60 0x7fef7019dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:20.387 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.562+0000 7fef76e00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7010e0b0 con 0x7fef7007de60 2024-03-28T17:59:20.387 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.563+0000 7fef6fe00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fef7007de60 0x7fef7019dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:20.387 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.563+0000 7fef6fe00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fef7007de60 0x7fef7019dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50222/0 (socket says 172.21.15.37:50222) 2024-03-28T17:59:20.388 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.563+0000 7fef6fe00700 1 -- 172.21.15.37:0/2652462132 learned_addr learned my addr 172.21.15.37:0/2652462132 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:20.388 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.563+0000 7fef6fe00700 1 -- 172.21.15.37:0/2652462132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7019e160 con 0x7fef7007de60 2024-03-28T17:59:20.388 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.563+0000 7fef6fe00700 1 --2- 172.21.15.37:0/2652462132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fef7007de60 0x7fef7019dc20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fef60011b60 tx=0x7fef60011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:20.388 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.564+0000 7fef6d600700 1 -- 172.21.15.37:0/2652462132 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef60003b20 con 0x7fef7007de60 2024-03-28T17:59:20.389 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.564+0000 7fef6d600700 1 -- 172.21.15.37:0/2652462132 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef60003cc0 con 0x7fef7007de60 2024-03-28T17:59:20.389 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.564+0000 7fef6d600700 1 -- 172.21.15.37:0/2652462132 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef60024840 con 0x7fef7007de60 2024-03-28T17:59:20.389 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.564+0000 7fef76e00700 1 -- 172.21.15.37:0/2652462132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef7019e3f0 con 0x7fef7007de60 2024-03-28T17:59:20.389 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.564+0000 7fef76e00700 1 -- 172.21.15.37:0/2652462132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef701a10e0 con 0x7fef7007de60 2024-03-28T17:59:20.389 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.565+0000 7fef6d600700 1 -- 172.21.15.37:0/2652462132 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fef60022020 con 0x7fef7007de60 2024-03-28T17:59:20.389 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.566+0000 7fef6d600700 1 -- 172.21.15.37:0/2652462132 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fef60011e00 con 0x7fef7007de60 2024-03-28T17:59:20.390 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.566+0000 7fef66000700 1 -- 172.21.15.37:0/2652462132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef48005330 con 0x7fef7007de60 2024-03-28T17:59:20.390 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.569+0000 7fef6d600700 1 -- 172.21.15.37:0/2652462132 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fef60043dd0 con 0x7fef7007de60 2024-03-28T17:59:20.390 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.648+0000 7fef66000700 1 -- 172.21.15.37:0/2652462132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fef480058f0 con 0x7fef7007de60 2024-03-28T17:59:20.390 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.649+0000 7fef6d600700 1 -- 172.21.15.37:0/2652462132 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fef60032020 con 0x7fef7007de60 2024-03-28T17:59:20.390 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.650+0000 7fef66000700 1 -- 172.21.15.37:0/2652462132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fef7007de60 msgr2=0x7fef7019dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:20.391 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.650+0000 7fef66000700 1 --2- 172.21.15.37:0/2652462132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fef7007de60 0x7fef7019dc20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fef60011b60 tx=0x7fef60011c40 comp rx=0 tx=0).stop 2024-03-28T17:59:20.391 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.650+0000 7fef66000700 1 -- 172.21.15.37:0/2652462132 shutdown_connections 2024-03-28T17:59:20.391 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.650+0000 7fef66000700 1 --2- 172.21.15.37:0/2652462132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fef7007de60 0x7fef7019dc20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:20.391 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.650+0000 7fef66000700 1 -- 172.21.15.37:0/2652462132 >> 172.21.15.37:0/2652462132 conn(0x7fef70103080 msgr2=0x7fef70103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:20.391 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.651+0000 7fef66000700 1 -- 172.21.15.37:0/2652462132 shutdown_connections 2024-03-28T17:59:20.392 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:19.651+0000 7fef66000700 1 -- 172.21.15.37:0/2652462132 wait complete. 2024-03-28T17:59:20.392 INFO:teuthology.orchestra.run.smithi037.stdout:mgr not available, waiting (2/15)... 2024-03-28T17:59:20.624 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:20 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:20.567+0000 7f207bc3b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T17:59:20.958 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:20 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:20.689+0000 7f207bc3b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T17:59:21.595 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:20 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:20.957+0000 7f207bc3b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T17:59:21.595 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:21 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:21.209+0000 7f207bc3b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T17:59:21.595 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:21 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:21.594+0000 7f207bc3b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T17:59:22.233 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:21 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:21.919+0000 7f207bc3b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T17:59:22.518 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:22 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:22.232+0000 7f207bc3b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T17:59:23.334 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:23 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/763349757' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T17:59:23.624 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:23.333+0000 7f207bc3b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T17:59:23.753 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-03-28T17:59:23.754 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout { 2024-03-28T17:59:23.754 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "fsid": "b5ad257c-ed2c-11ee-95d0-87774f69a715", 2024-03-28T17:59:23.754 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T17:59:23.754 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T17:59:23.754 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T17:59:23.754 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T17:59:23.757 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:23.757 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T17:59:23.757 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T17:59:23.757 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 0 2024-03-28T17:59:23.757 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:59:23.757 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T17:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T17:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-28T17:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T17:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T17:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T17:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T17:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T17:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T17:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T17:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T17:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T17:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T17:59:23.759 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:23.759 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T17:59:23.763 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T17:59:23.763 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T17:59:23.763 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T17:59:23.763 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T17:59:23.763 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T17:59:23.763 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T17:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T17:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T17:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T17:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T17:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T17:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T17:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T17:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T17:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T17:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T17:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T17:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T17:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T17:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T17:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T17:59:05.950436+0000", 2024-03-28T17:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T17:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T17:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout } 2024-03-28T17:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.946+0000 7f17e5800700 1 Processor -- start 2024-03-28T17:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.947+0000 7f17e5800700 1 -- start start 2024-03-28T17:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.947+0000 7f17e5800700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17e0108540 0x7f17e010a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.947+0000 7f17e5800700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e007be30 con 0x7f17e0108540 2024-03-28T17:59:23.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.948+0000 7f17dea00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17e0108540 0x7f17e010a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:23.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.948+0000 7f17dea00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17e0108540 0x7f17e010a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50232/0 (socket says 172.21.15.37:50232) 2024-03-28T17:59:23.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.948+0000 7f17dea00700 1 -- 172.21.15.37:0/1554949440 learned_addr learned my addr 172.21.15.37:0/1554949440 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:23.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.948+0000 7f17dea00700 1 -- 172.21.15.37:0/1554949440 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e010ae70 con 0x7f17e0108540 2024-03-28T17:59:23.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.949+0000 7f17dea00700 1 --2- 172.21.15.37:0/1554949440 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17e0108540 0x7f17e010a930 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f17d00098a0 tx=0x7f17d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb1dd8da5cd374d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:23.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.949+0000 7f17de000700 1 -- 172.21.15.37:0/1554949440 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f17d0004160 con 0x7f17e0108540 2024-03-28T17:59:23.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.949+0000 7f17de000700 1 -- 172.21.15.37:0/1554949440 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f17d0004300 con 0x7f17e0108540 2024-03-28T17:59:23.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.949+0000 7f17de000700 1 -- 172.21.15.37:0/1554949440 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f17d0011b50 con 0x7f17e0108540 2024-03-28T17:59:23.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.950+0000 7f17e5800700 1 -- 172.21.15.37:0/1554949440 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17e0108540 msgr2=0x7f17e010a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:23.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.950+0000 7f17e5800700 1 --2- 172.21.15.37:0/1554949440 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17e0108540 0x7f17e010a930 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f17d00098a0 tx=0x7f17d0009bb0 comp rx=0 tx=0).stop 2024-03-28T17:59:23.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.950+0000 7f17e5800700 1 -- 172.21.15.37:0/1554949440 shutdown_connections 2024-03-28T17:59:23.769 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.950+0000 7f17e5800700 1 --2- 172.21.15.37:0/1554949440 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17e0108540 0x7f17e010a930 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:23.770 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.950+0000 7f17e5800700 1 -- 172.21.15.37:0/1554949440 >> 172.21.15.37:0/1554949440 conn(0x7f17e0101f70 msgr2=0x7f17e0104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:23.770 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.950+0000 7f17e5800700 1 -- 172.21.15.37:0/1554949440 shutdown_connections 2024-03-28T17:59:23.770 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.950+0000 7f17e5800700 1 -- 172.21.15.37:0/1554949440 wait complete. 2024-03-28T17:59:23.770 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.951+0000 7f17e5800700 1 Processor -- start 2024-03-28T17:59:23.770 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.951+0000 7f17e5800700 1 -- start start 2024-03-28T17:59:23.770 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.951+0000 7f17e5800700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17e0108540 0x7f17e019dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:23.770 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.952+0000 7f17e5800700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e010be40 con 0x7f17e0108540 2024-03-28T17:59:23.770 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.952+0000 7f17dea00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17e0108540 0x7f17e019dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:23.770 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.952+0000 7f17dea00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17e0108540 0x7f17e019dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50244/0 (socket says 172.21.15.37:50244) 2024-03-28T17:59:23.770 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.952+0000 7f17dea00700 1 -- 172.21.15.37:0/763349757 learned_addr learned my addr 172.21.15.37:0/763349757 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:23.770 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.952+0000 7f17dea00700 1 -- 172.21.15.37:0/763349757 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e019e140 con 0x7f17e0108540 2024-03-28T17:59:23.770 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.952+0000 7f17dea00700 1 --2- 172.21.15.37:0/763349757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17e0108540 0x7f17e019dc00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f17d0011670 tx=0x7f17d0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:23.770 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.953+0000 7f17d7400700 1 -- 172.21.15.37:0/763349757 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f17d00039c0 con 0x7f17e0108540 2024-03-28T17:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.953+0000 7f17d7400700 1 -- 172.21.15.37:0/763349757 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f17d0011e50 con 0x7f17e0108540 2024-03-28T17:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.953+0000 7f17e5800700 1 -- 172.21.15.37:0/763349757 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e019e3d0 con 0x7f17e0108540 2024-03-28T17:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.953+0000 7f17e5800700 1 -- 172.21.15.37:0/763349757 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e01a10c0 con 0x7f17e0108540 2024-03-28T17:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.953+0000 7f17d7400700 1 -- 172.21.15.37:0/763349757 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f17d001a5c0 con 0x7f17e0108540 2024-03-28T17:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.953+0000 7f17d7400700 1 -- 172.21.15.37:0/763349757 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f17d001ac70 con 0x7f17e0108540 2024-03-28T17:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.953+0000 7f17d7400700 1 -- 172.21.15.37:0/763349757 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f17d0019700 con 0x7f17e0108540 2024-03-28T17:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.954+0000 7f17e5800700 1 -- 172.21.15.37:0/763349757 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e00637c0 con 0x7f17e0108540 2024-03-28T17:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:22.957+0000 7f17d7400700 1 -- 172.21.15.37:0/763349757 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f17d0003b60 con 0x7f17e0108540 2024-03-28T17:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:23.036+0000 7f17e5800700 1 -- 172.21.15.37:0/763349757 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f17e01a1290 con 0x7f17e0108540 2024-03-28T17:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:23.036+0000 7f17d7400700 1 -- 172.21.15.37:0/763349757 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f17d0019900 con 0x7f17e0108540 2024-03-28T17:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:23.038+0000 7f17e5800700 1 -- 172.21.15.37:0/763349757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17e0108540 msgr2=0x7f17e019dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:23.038+0000 7f17e5800700 1 --2- 172.21.15.37:0/763349757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17e0108540 0x7f17e019dc00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f17d0011670 tx=0x7f17d0011b60 comp rx=0 tx=0).stop 2024-03-28T17:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:23.038+0000 7f17e5800700 1 -- 172.21.15.37:0/763349757 shutdown_connections 2024-03-28T17:59:23.772 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:23.038+0000 7f17e5800700 1 --2- 172.21.15.37:0/763349757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17e0108540 0x7f17e019dc00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:23.772 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:23.038+0000 7f17e5800700 1 -- 172.21.15.37:0/763349757 >> 172.21.15.37:0/763349757 conn(0x7f17e0101f70 msgr2=0x7f17e0102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:23.772 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:23.038+0000 7f17e5800700 1 -- 172.21.15.37:0/763349757 shutdown_connections 2024-03-28T17:59:23.772 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:23.038+0000 7f17e5800700 1 -- 172.21.15.37:0/763349757 wait complete. 2024-03-28T17:59:23.772 INFO:teuthology.orchestra.run.smithi037.stdout:mgr not available, waiting (3/15)... 2024-03-28T17:59:24.374 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:24 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:24.053+0000 7f207bc3b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T17:59:24.375 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:24 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:24.186+0000 7f207bc3b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T17:59:24.874 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:24 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:24.476+0000 7f207bc3b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T17:59:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:24 smithi037 ceph-mon[31460]: Activating manager daemon x 2024-03-28T17:59:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:24 smithi037 ceph-mon[31460]: mgrmap e2: x(active, starting, since 0.0110961s) 2024-03-28T17:59:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:24 smithi037 ceph-mon[31460]: from='mgr.14100 172.21.15.37:0/1155252435' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T17:59:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:24 smithi037 ceph-mon[31460]: from='mgr.14100 172.21.15.37:0/1155252435' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T17:59:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:24 smithi037 ceph-mon[31460]: from='mgr.14100 172.21.15.37:0/1155252435' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T17:59:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:24 smithi037 ceph-mon[31460]: from='mgr.14100 172.21.15.37:0/1155252435' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:59:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:24 smithi037 ceph-mon[31460]: from='mgr.14100 172.21.15.37:0/1155252435' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T17:59:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:24 smithi037 ceph-mon[31460]: Manager daemon x is now available 2024-03-28T17:59:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:24 smithi037 ceph-mon[31460]: from='mgr.14100 172.21.15.37:0/1155252435' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T17:59:24.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:24 smithi037 ceph-mon[31460]: from='mgr.14100 172.21.15.37:0/1155252435' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T17:59:24.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:24 smithi037 ceph-mon[31460]: from='mgr.14100 172.21.15.37:0/1155252435' entity='mgr.x' 2024-03-28T17:59:24.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:24 smithi037 ceph-mon[31460]: from='mgr.14100 172.21.15.37:0/1155252435' entity='mgr.x' 2024-03-28T17:59:25.878 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:25 smithi037 ceph-mon[31460]: from='mgr.14100 172.21.15.37:0/1155252435' entity='mgr.x' 2024-03-28T17:59:25.878 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:25 smithi037 ceph-mon[31460]: mgrmap e3: x(active, since 1.02084s) 2024-03-28T17:59:27.466 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-03-28T17:59:27.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout { 2024-03-28T17:59:27.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "fsid": "b5ad257c-ed2c-11ee-95d0-87774f69a715", 2024-03-28T17:59:27.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T17:59:27.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T17:59:27.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T17:59:27.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T17:59:27.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:27.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T17:59:27.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T17:59:27.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 0 2024-03-28T17:59:27.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:59:27.468 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T17:59:27.468 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T17:59:27.468 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:59:27.469 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-28T17:59:27.469 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T17:59:27.469 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:59:27.469 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T17:59:27.469 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T17:59:27.469 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:27.469 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T17:59:27.469 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:59:27.470 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T17:59:27.470 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T17:59:27.470 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T17:59:27.476 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T17:59:27.476 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T17:59:27.476 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T17:59:27.476 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:27.476 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T17:59:27.477 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T17:59:27.477 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T17:59:27.477 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T17:59:27.477 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T17:59:27.477 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T17:59:27.477 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T17:59:27.477 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T17:59:27.477 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T17:59:27.477 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:27.477 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T17:59:27.477 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:59:27.477 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T17:59:27.477 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T17:59:27.477 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:27.478 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T17:59:27.478 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T17:59:27.478 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T17:59:27.478 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T17:59:27.478 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T17:59:27.478 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T17:59:27.478 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T17:59:27.478 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:59:27.478 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T17:59:27.478 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:27.478 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T17:59:27.478 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:59:27.478 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T17:59:05.950436+0000", 2024-03-28T17:59:27.478 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T17:59:27.479 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:59:27.479 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T17:59:27.479 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout } 2024-03-28T17:59:27.479 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.353+0000 7f7f2f200700 1 Processor -- start 2024-03-28T17:59:27.479 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.353+0000 7f7f2f200700 1 -- start start 2024-03-28T17:59:27.479 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.354+0000 7f7f2f200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f28081870 0x7f7f28081c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:27.479 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.354+0000 7f7f2f200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f280821d0 con 0x7f7f28081870 2024-03-28T17:59:27.479 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.354+0000 7f7f2de00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f28081870 0x7f7f28081c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:27.479 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.354+0000 7f7f2de00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f28081870 0x7f7f28081c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50314/0 (socket says 172.21.15.37:50314) 2024-03-28T17:59:27.479 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.354+0000 7f7f2de00700 1 -- 172.21.15.37:0/3197112980 learned_addr learned my addr 172.21.15.37:0/3197112980 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:27.479 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.355+0000 7f7f2de00700 1 -- 172.21.15.37:0/3197112980 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2810f040 con 0x7f7f28081870 2024-03-28T17:59:27.479 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.355+0000 7f7f2de00700 1 --2- 172.21.15.37:0/3197112980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f28081870 0x7f7f28081c90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c0098a0 tx=0x7f7f1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b60516ac38159b43 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:27.479 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.355+0000 7f7f2ca00700 1 -- 172.21.15.37:0/3197112980 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7f1c004160 con 0x7f7f28081870 2024-03-28T17:59:27.480 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.355+0000 7f7f2ca00700 1 -- 172.21.15.37:0/3197112980 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f1c004300 con 0x7f7f28081870 2024-03-28T17:59:27.480 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.355+0000 7f7f2ca00700 1 -- 172.21.15.37:0/3197112980 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7f1c011b50 con 0x7f7f28081870 2024-03-28T17:59:27.480 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.356+0000 7f7f2f200700 1 -- 172.21.15.37:0/3197112980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f28081870 msgr2=0x7f7f28081c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:27.480 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.356+0000 7f7f2f200700 1 --2- 172.21.15.37:0/3197112980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f28081870 0x7f7f28081c90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c0098a0 tx=0x7f7f1c009bb0 comp rx=0 tx=0).stop 2024-03-28T17:59:27.480 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.356+0000 7f7f2f200700 1 -- 172.21.15.37:0/3197112980 shutdown_connections 2024-03-28T17:59:27.480 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.356+0000 7f7f2f200700 1 --2- 172.21.15.37:0/3197112980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f28081870 0x7f7f28081c90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:27.480 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.356+0000 7f7f2f200700 1 -- 172.21.15.37:0/3197112980 >> 172.21.15.37:0/3197112980 conn(0x7f7f2807cee0 msgr2=0x7f7f2807f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:27.480 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.356+0000 7f7f2f200700 1 -- 172.21.15.37:0/3197112980 shutdown_connections 2024-03-28T17:59:27.480 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.356+0000 7f7f2f200700 1 -- 172.21.15.37:0/3197112980 wait complete. 2024-03-28T17:59:27.480 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.357+0000 7f7f2f200700 1 Processor -- start 2024-03-28T17:59:27.480 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.358+0000 7f7f2f200700 1 -- start start 2024-03-28T17:59:27.480 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.358+0000 7f7f2f200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f28081870 0x7f7f281a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:27.480 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.358+0000 7f7f2f200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f28112300 con 0x7f7f28081870 2024-03-28T17:59:27.480 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.358+0000 7f7f2de00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f28081870 0x7f7f281a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:27.481 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.358+0000 7f7f2de00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f28081870 0x7f7f281a1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50322/0 (socket says 172.21.15.37:50322) 2024-03-28T17:59:27.481 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.358+0000 7f7f2de00700 1 -- 172.21.15.37:0/2174849022 learned_addr learned my addr 172.21.15.37:0/2174849022 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:27.481 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.359+0000 7f7f2de00700 1 -- 172.21.15.37:0/2174849022 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f281a2390 con 0x7f7f28081870 2024-03-28T17:59:27.481 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.359+0000 7f7f2de00700 1 --2- 172.21.15.37:0/2174849022 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f28081870 0x7f7f281a1e50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c009fb0 tx=0x7f7f1c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:27.484 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.359+0000 7f7f22a00700 1 -- 172.21.15.37:0/2174849022 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7f1c0038b0 con 0x7f7f28081870 2024-03-28T17:59:27.484 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.359+0000 7f7f2f200700 1 -- 172.21.15.37:0/2174849022 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f281a2620 con 0x7f7f28081870 2024-03-28T17:59:27.484 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.359+0000 7f7f22a00700 1 -- 172.21.15.37:0/2174849022 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f1c011e50 con 0x7f7f28081870 2024-03-28T17:59:27.484 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.359+0000 7f7f22a00700 1 -- 172.21.15.37:0/2174849022 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7f1c019420 con 0x7f7f28081870 2024-03-28T17:59:27.484 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.359+0000 7f7f2f200700 1 -- 172.21.15.37:0/2174849022 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f281a5310 con 0x7f7f28081870 2024-03-28T17:59:27.484 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.360+0000 7f7f22a00700 1 -- 172.21.15.37:0/2174849022 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 3) v1 ==== 45989+0+0 (secure 0 0 0) 0x7f7f1c01a430 con 0x7f7f28081870 2024-03-28T17:59:27.484 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.360+0000 7f7f2f200700 1 -- 172.21.15.37:0/2174849022 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f28112480 con 0x7f7f28081870 2024-03-28T17:59:27.484 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.361+0000 7f7f22a00700 1 --2- 172.21.15.37:0/2174849022 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f7f10038dc0 0x7f7f1003b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:27.484 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.361+0000 7f7f22a00700 1 -- 172.21.15.37:0/2174849022 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7f1c04ce40 con 0x7f7f28081870 2024-03-28T17:59:27.484 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.361+0000 7f7f2d400700 1 --2- 172.21.15.37:0/2174849022 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f7f10038dc0 0x7f7f1003b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:27.484 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.362+0000 7f7f2d400700 1 --2- 172.21.15.37:0/2174849022 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f7f10038dc0 0x7f7f1003b280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7f18006fd0 tx=0x7f7f18006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:27.485 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.367+0000 7f7f22a00700 1 -- 172.21.15.37:0/2174849022 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7f1c050030 con 0x7f7f28081870 2024-03-28T17:59:27.485 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.544+0000 7f7f22a00700 1 -- 172.21.15.37:0/2174849022 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f7f1c0196f0 con 0x7f7f28081870 2024-03-28T17:59:27.485 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.681+0000 7f7f2f200700 1 -- 172.21.15.37:0/2174849022 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7f281a58c0 con 0x7f7f28081870 2024-03-28T17:59:27.485 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.682+0000 7f7f22a00700 1 -- 172.21.15.37:0/2174849022 <== mon.0 v2:172.21.15.37:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f7f1c022340 con 0x7f7f28081870 2024-03-28T17:59:27.485 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.685+0000 7f7f2f200700 1 -- 172.21.15.37:0/2174849022 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f7f10038dc0 msgr2=0x7f7f1003b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:27.485 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.685+0000 7f7f2f200700 1 --2- 172.21.15.37:0/2174849022 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f7f10038dc0 0x7f7f1003b280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7f18006fd0 tx=0x7f7f18006e40 comp rx=0 tx=0).stop 2024-03-28T17:59:27.485 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.685+0000 7f7f2f200700 1 -- 172.21.15.37:0/2174849022 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f28081870 msgr2=0x7f7f281a1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:27.485 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.685+0000 7f7f2f200700 1 --2- 172.21.15.37:0/2174849022 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f28081870 0x7f7f281a1e50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c009fb0 tx=0x7f7f1c011bf0 comp rx=0 tx=0).stop 2024-03-28T17:59:27.485 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.685+0000 7f7f2f200700 1 -- 172.21.15.37:0/2174849022 shutdown_connections 2024-03-28T17:59:27.485 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.685+0000 7f7f2f200700 1 --2- 172.21.15.37:0/2174849022 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f7f10038dc0 0x7f7f1003b280 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:27.485 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.685+0000 7f7f2f200700 1 --2- 172.21.15.37:0/2174849022 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f28081870 0x7f7f281a1e50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:27.485 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.685+0000 7f7f2f200700 1 -- 172.21.15.37:0/2174849022 >> 172.21.15.37:0/2174849022 conn(0x7f7f2807cee0 msgr2=0x7f7f2807d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:27.485 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.686+0000 7f7f2f200700 1 -- 172.21.15.37:0/2174849022 shutdown_connections 2024-03-28T17:59:27.485 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:26.686+0000 7f7f2f200700 1 -- 172.21.15.37:0/2174849022 wait complete. 2024-03-28T17:59:27.485 INFO:teuthology.orchestra.run.smithi037.stdout:mgr is available 2024-03-28T17:59:27.762 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:27 smithi037 ceph-mon[31460]: mgrmap e4: x(active, since 2s) 2024-03-28T17:59:27.762 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:27 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2174849022' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T17:59:28.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:28 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1848216291' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T17:59:29.028 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-03-28T17:59:29.036 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T17:59:29.036 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout fsid = b5ad257c-ed2c-11ee-95d0-87774f69a715 2024-03-28T17:59:29.036 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T17:59:29.036 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T17:59:29.036 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T17:59:29.036 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T17:59:29.036 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-03-28T17:59:29.036 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T17:59:29.036 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T17:59:29.036 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-03-28T17:59:29.036 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T17:59:29.037 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T17:59:29.037 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T17:59:29.037 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T17:59:29.037 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T17:59:29.037 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T17:59:29.037 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.064+0000 7f3dfba00700 1 Processor -- start 2024-03-28T17:59:29.037 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.064+0000 7f3dfba00700 1 -- start start 2024-03-28T17:59:29.037 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.065+0000 7f3dfba00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df4108540 0x7f3df410a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:29.037 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.065+0000 7f3dfba00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df407be30 con 0x7f3df4108540 2024-03-28T17:59:29.037 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.065+0000 7f3df9000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df4108540 0x7f3df410a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:29.037 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.065+0000 7f3df9000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df4108540 0x7f3df410a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50336/0 (socket says 172.21.15.37:50336) 2024-03-28T17:59:29.037 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.065+0000 7f3df9000700 1 -- 172.21.15.37:0/1758184490 learned_addr learned my addr 172.21.15.37:0/1758184490 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:29.037 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.066+0000 7f3df9000700 1 -- 172.21.15.37:0/1758184490 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df410ae70 con 0x7f3df4108540 2024-03-28T17:59:29.037 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.066+0000 7f3df9000700 1 --2- 172.21.15.37:0/1758184490 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df4108540 0x7f3df410a930 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3de80098a0 tx=0x7f3de8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a02cc4d835ae1fed server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:29.037 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.066+0000 7f3df3400700 1 -- 172.21.15.37:0/1758184490 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3de8004160 con 0x7f3df4108540 2024-03-28T17:59:29.038 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.066+0000 7f3df3400700 1 -- 172.21.15.37:0/1758184490 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3de8004300 con 0x7f3df4108540 2024-03-28T17:59:29.038 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.066+0000 7f3df3400700 1 -- 172.21.15.37:0/1758184490 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3de8011b50 con 0x7f3df4108540 2024-03-28T17:59:29.038 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.067+0000 7f3dfba00700 1 -- 172.21.15.37:0/1758184490 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df4108540 msgr2=0x7f3df410a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:29.038 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.067+0000 7f3dfba00700 1 --2- 172.21.15.37:0/1758184490 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df4108540 0x7f3df410a930 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3de80098a0 tx=0x7f3de8009bb0 comp rx=0 tx=0).stop 2024-03-28T17:59:29.038 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.067+0000 7f3dfba00700 1 -- 172.21.15.37:0/1758184490 shutdown_connections 2024-03-28T17:59:29.038 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.067+0000 7f3dfba00700 1 --2- 172.21.15.37:0/1758184490 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df4108540 0x7f3df410a930 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:29.038 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.067+0000 7f3dfba00700 1 -- 172.21.15.37:0/1758184490 >> 172.21.15.37:0/1758184490 conn(0x7f3df4101f10 msgr2=0x7f3df4104350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:29.038 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.068+0000 7f3dfba00700 1 -- 172.21.15.37:0/1758184490 shutdown_connections 2024-03-28T17:59:29.038 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.068+0000 7f3dfba00700 1 -- 172.21.15.37:0/1758184490 wait complete. 2024-03-28T17:59:29.038 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.069+0000 7f3dfba00700 1 Processor -- start 2024-03-28T17:59:29.038 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.069+0000 7f3dfba00700 1 -- start start 2024-03-28T17:59:29.038 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.069+0000 7f3dfba00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df4108540 0x7f3df4107dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:29.038 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.069+0000 7f3dfba00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df410be40 con 0x7f3df4108540 2024-03-28T17:59:29.039 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.069+0000 7f3df9000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df4108540 0x7f3df4107dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:29.039 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.070+0000 7f3df9000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df4108540 0x7f3df4107dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50348/0 (socket says 172.21.15.37:50348) 2024-03-28T17:59:29.039 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.070+0000 7f3df9000700 1 -- 172.21.15.37:0/1848216291 learned_addr learned my addr 172.21.15.37:0/1848216291 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:29.039 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.070+0000 7f3df9000700 1 -- 172.21.15.37:0/1848216291 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df4108300 con 0x7f3df4108540 2024-03-28T17:59:29.039 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.070+0000 7f3df9000700 1 --2- 172.21.15.37:0/1848216291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df4108540 0x7f3df4107dc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3de8011b60 tx=0x7f3de8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:29.039 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.070+0000 7f3df1600700 1 -- 172.21.15.37:0/1848216291 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3de8003b20 con 0x7f3df4108540 2024-03-28T17:59:29.039 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.071+0000 7f3df1600700 1 -- 172.21.15.37:0/1848216291 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3de8003cc0 con 0x7f3df4108540 2024-03-28T17:59:29.039 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.071+0000 7f3dfba00700 1 -- 172.21.15.37:0/1848216291 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df41064f0 con 0x7f3df4108540 2024-03-28T17:59:29.039 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.071+0000 7f3df1600700 1 -- 172.21.15.37:0/1848216291 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3de8024840 con 0x7f3df4108540 2024-03-28T17:59:29.039 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.071+0000 7f3dfba00700 1 -- 172.21.15.37:0/1848216291 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df4106a30 con 0x7f3df4108540 2024-03-28T17:59:29.039 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.072+0000 7f3dfba00700 1 -- 172.21.15.37:0/1848216291 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df40637c0 con 0x7f3df4108540 2024-03-28T17:59:29.039 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.072+0000 7f3df1600700 1 -- 172.21.15.37:0/1848216291 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f3de8022020 con 0x7f3df4108540 2024-03-28T17:59:29.039 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.072+0000 7f3df1600700 1 --2- 172.21.15.37:0/1848216291 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f3ddc038f70 0x7f3ddc03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:29.040 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.072+0000 7f3df1600700 1 -- 172.21.15.37:0/1848216291 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3de8011e00 con 0x7f3df4108540 2024-03-28T17:59:29.040 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.073+0000 7f3df3e00700 1 --2- 172.21.15.37:0/1848216291 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f3ddc038f70 0x7f3ddc03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:29.040 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.073+0000 7f3df3e00700 1 --2- 172.21.15.37:0/1848216291 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f3ddc038f70 0x7f3ddc03b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3de0006fd0 tx=0x7f3de0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:29.040 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.078+0000 7f3df1600700 1 -- 172.21.15.37:0/1848216291 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3de800b3e0 con 0x7f3df4108540 2024-03-28T17:59:29.040 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.305+0000 7f3dfba00700 1 -- 172.21.15.37:0/1848216291 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f3df4106cb0 con 0x7f3df4108540 2024-03-28T17:59:29.040 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.307+0000 7f3df1600700 1 -- 172.21.15.37:0/1848216291 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f3de800b5c0 con 0x7f3df4108540 2024-03-28T17:59:29.040 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.310+0000 7f3dfba00700 1 -- 172.21.15.37:0/1848216291 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f3ddc038f70 msgr2=0x7f3ddc03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:29.040 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.310+0000 7f3dfba00700 1 --2- 172.21.15.37:0/1848216291 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f3ddc038f70 0x7f3ddc03b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3de0006fd0 tx=0x7f3de0006e40 comp rx=0 tx=0).stop 2024-03-28T17:59:29.040 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.310+0000 7f3dfba00700 1 -- 172.21.15.37:0/1848216291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df4108540 msgr2=0x7f3df4107dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:29.040 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.310+0000 7f3dfba00700 1 --2- 172.21.15.37:0/1848216291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df4108540 0x7f3df4107dc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3de8011b60 tx=0x7f3de8011c40 comp rx=0 tx=0).stop 2024-03-28T17:59:29.040 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.310+0000 7f3dfba00700 1 -- 172.21.15.37:0/1848216291 shutdown_connections 2024-03-28T17:59:29.040 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.310+0000 7f3dfba00700 1 --2- 172.21.15.37:0/1848216291 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f3ddc038f70 0x7f3ddc03b430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:29.040 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.310+0000 7f3dfba00700 1 --2- 172.21.15.37:0/1848216291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df4108540 0x7f3df4107dc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:29.040 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.310+0000 7f3dfba00700 1 -- 172.21.15.37:0/1848216291 >> 172.21.15.37:0/1848216291 conn(0x7f3df4101f10 msgr2=0x7f3df41029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:29.040 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.310+0000 7f3dfba00700 1 -- 172.21.15.37:0/1848216291 shutdown_connections 2024-03-28T17:59:29.041 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:28.310+0000 7f3dfba00700 1 -- 172.21.15.37:0/1848216291 wait complete. 2024-03-28T17:59:29.041 INFO:teuthology.orchestra.run.smithi037.stdout:Enabling cephadm module... 2024-03-28T17:59:30.299 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:30 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1158018660' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-28T17:59:30.299 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:30 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: ignoring --setuser ceph since I am not root 2024-03-28T17:59:30.300 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:30 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: ignoring --setgroup ceph since I am not root 2024-03-28T17:59:30.624 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.653+0000 7f1d9b400700 1 Processor -- start 2024-03-28T17:59:30.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.654+0000 7f1d9b400700 1 -- start start 2024-03-28T17:59:30.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.654+0000 7f1d9b400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d94106110 0x7f1d94106530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:30.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.654+0000 7f1d9b400700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d94106b00 con 0x7f1d94106110 2024-03-28T17:59:30.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.654+0000 7f1d98a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d94106110 0x7f1d94106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:30.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.655+0000 7f1d98a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d94106110 0x7f1d94106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50350/0 (socket says 172.21.15.37:50350) 2024-03-28T17:59:30.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.655+0000 7f1d98a00700 1 -- 172.21.15.37:0/3774588038 learned_addr learned my addr 172.21.15.37:0/3774588038 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:30.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.655+0000 7f1d98a00700 1 -- 172.21.15.37:0/3774588038 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d94107310 con 0x7f1d94106110 2024-03-28T17:59:30.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.656+0000 7f1d98a00700 1 --2- 172.21.15.37:0/3774588038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d94106110 0x7f1d94106530 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1d8800bb40 tx=0x7f1d8800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7004e30be19a809e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:30.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.656+0000 7f1d93400700 1 -- 172.21.15.37:0/3774588038 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d8800dad0 con 0x7f1d94106110 2024-03-28T17:59:30.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.656+0000 7f1d93400700 1 -- 172.21.15.37:0/3774588038 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d880044f0 con 0x7f1d94106110 2024-03-28T17:59:30.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.656+0000 7f1d93400700 1 -- 172.21.15.37:0/3774588038 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d88017410 con 0x7f1d94106110 2024-03-28T17:59:30.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.657+0000 7f1d9b400700 1 -- 172.21.15.37:0/3774588038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d94106110 msgr2=0x7f1d94106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:30.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.657+0000 7f1d9b400700 1 --2- 172.21.15.37:0/3774588038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d94106110 0x7f1d94106530 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1d8800bb40 tx=0x7f1d8800be50 comp rx=0 tx=0).stop 2024-03-28T17:59:30.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.657+0000 7f1d9b400700 1 -- 172.21.15.37:0/3774588038 shutdown_connections 2024-03-28T17:59:30.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.657+0000 7f1d9b400700 1 --2- 172.21.15.37:0/3774588038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d94106110 0x7f1d94106530 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:30.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.657+0000 7f1d9b400700 1 -- 172.21.15.37:0/3774588038 >> 172.21.15.37:0/3774588038 conn(0x7f1d941016f0 msgr2=0x7f1d94103b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:30.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.657+0000 7f1d9b400700 1 -- 172.21.15.37:0/3774588038 shutdown_connections 2024-03-28T17:59:30.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.657+0000 7f1d9b400700 1 -- 172.21.15.37:0/3774588038 wait complete. 2024-03-28T17:59:30.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.658+0000 7f1d9b400700 1 Processor -- start 2024-03-28T17:59:30.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.658+0000 7f1d9b400700 1 -- start start 2024-03-28T17:59:30.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.659+0000 7f1d9b400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d94106110 0x7f1d9419dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:30.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.659+0000 7f1d9b400700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d94107a90 con 0x7f1d94106110 2024-03-28T17:59:30.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.659+0000 7f1d98a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d94106110 0x7f1d9419dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:30.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.659+0000 7f1d98a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d94106110 0x7f1d9419dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50360/0 (socket says 172.21.15.37:50360) 2024-03-28T17:59:30.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.659+0000 7f1d98a00700 1 -- 172.21.15.37:0/1158018660 learned_addr learned my addr 172.21.15.37:0/1158018660 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:30.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.660+0000 7f1d98a00700 1 -- 172.21.15.37:0/1158018660 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9419e180 con 0x7f1d94106110 2024-03-28T17:59:30.627 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.660+0000 7f1d98a00700 1 --2- 172.21.15.37:0/1158018660 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d94106110 0x7f1d9419dc40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1d880046e0 tx=0x7f1d8800deb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:30.627 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.660+0000 7f1d91600700 1 -- 172.21.15.37:0/1158018660 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d88004150 con 0x7f1d94106110 2024-03-28T17:59:30.627 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.660+0000 7f1d91600700 1 -- 172.21.15.37:0/1158018660 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d880042f0 con 0x7f1d94106110 2024-03-28T17:59:30.627 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.660+0000 7f1d91600700 1 -- 172.21.15.37:0/1158018660 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d8801fbc0 con 0x7f1d94106110 2024-03-28T17:59:30.627 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.660+0000 7f1d9b400700 1 -- 172.21.15.37:0/1158018660 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d9419e410 con 0x7f1d94106110 2024-03-28T17:59:30.627 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.660+0000 7f1d9b400700 1 -- 172.21.15.37:0/1158018660 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d941a1100 con 0x7f1d94106110 2024-03-28T17:59:30.627 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.661+0000 7f1d9b400700 1 -- 172.21.15.37:0/1158018660 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d940637c0 con 0x7f1d94106110 2024-03-28T17:59:30.627 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.662+0000 7f1d91600700 1 -- 172.21.15.37:0/1158018660 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f1d88026030 con 0x7f1d94106110 2024-03-28T17:59:30.627 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.662+0000 7f1d91600700 1 --2- 172.21.15.37:0/1158018660 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f1d7c038f20 0x7f1d7c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:30.627 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.662+0000 7f1d91600700 1 -- 172.21.15.37:0/1158018660 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1d88024070 con 0x7f1d94106110 2024-03-28T17:59:30.627 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.663+0000 7f1d93e00700 1 --2- 172.21.15.37:0/1158018660 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f1d7c038f20 0x7f1d7c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:30.627 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.663+0000 7f1d93e00700 1 --2- 172.21.15.37:0/1158018660 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f1d7c038f20 0x7f1d7c03b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1d80006fd0 tx=0x7f1d80006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:30.627 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.667+0000 7f1d91600700 1 -- 172.21.15.37:0/1158018660 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1d8802d060 con 0x7f1d94106110 2024-03-28T17:59:30.627 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:29.960+0000 7f1d9b400700 1 -- 172.21.15.37:0/1158018660 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f1d9402ddd0 con 0x7f1d94106110 2024-03-28T17:59:30.627 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:30.014+0000 7f1d91600700 1 -- 172.21.15.37:0/1158018660 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f1d88012070 con 0x7f1d94106110 2024-03-28T17:59:30.628 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:30.017+0000 7f1d91600700 1 -- 172.21.15.37:0/1158018660 <== mon.0 v2:172.21.15.37:3300/0 8 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f1d88017940 con 0x7f1d94106110 2024-03-28T17:59:30.628 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:30.018+0000 7f1d9b400700 1 -- 172.21.15.37:0/1158018660 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f1d7c038f20 msgr2=0x7f1d7c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:30.628 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:30.018+0000 7f1d9b400700 1 --2- 172.21.15.37:0/1158018660 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f1d7c038f20 0x7f1d7c03b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1d80006fd0 tx=0x7f1d80006e40 comp rx=0 tx=0).stop 2024-03-28T17:59:30.628 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:30.018+0000 7f1d9b400700 1 -- 172.21.15.37:0/1158018660 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d94106110 msgr2=0x7f1d9419dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:30.628 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:30.018+0000 7f1d9b400700 1 --2- 172.21.15.37:0/1158018660 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d94106110 0x7f1d9419dc40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1d880046e0 tx=0x7f1d8800deb0 comp rx=0 tx=0).stop 2024-03-28T17:59:30.628 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:30.019+0000 7f1d9b400700 1 -- 172.21.15.37:0/1158018660 shutdown_connections 2024-03-28T17:59:30.628 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:30.019+0000 7f1d9b400700 1 --2- 172.21.15.37:0/1158018660 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f1d7c038f20 0x7f1d7c03b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:30.628 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:30.019+0000 7f1d9b400700 1 --2- 172.21.15.37:0/1158018660 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d94106110 0x7f1d9419dc40 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1d880046e0 tx=0x7f1d8800deb0 comp rx=0 tx=0).stop 2024-03-28T17:59:30.628 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:30.019+0000 7f1d9b400700 1 -- 172.21.15.37:0/1158018660 >> 172.21.15.37:0/1158018660 conn(0x7f1d941016f0 msgr2=0x7f1d94190410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:30.628 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:30.019+0000 7f1d9b400700 1 -- 172.21.15.37:0/1158018660 shutdown_connections 2024-03-28T17:59:30.628 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:30.019+0000 7f1d9b400700 1 -- 172.21.15.37:0/1158018660 wait complete. 2024-03-28T17:59:30.874 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:30 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:30.828+0000 7fef92c77200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T17:59:31.259 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:31 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1158018660' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-28T17:59:31.259 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:31 smithi037 ceph-mon[31460]: mgrmap e5: x(active, since 5s) 2024-03-28T17:59:31.259 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:30 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:30.954+0000 7fef92c77200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T17:59:31.874 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:31 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:31.791+0000 7fef92c77200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T17:59:32.239 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:32 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/4254071931' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T17:59:32.240 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:31 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:31.921+0000 7fef92c77200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T17:59:32.240 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:32 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:32.107+0000 7fef92c77200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T17:59:32.496 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout { 2024-03-28T17:59:32.496 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-28T17:59:32.496 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T17:59:32.496 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T17:59:32.496 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T17:59:32.496 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout } 2024-03-28T17:59:32.496 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.258+0000 7f3929600700 1 Processor -- start 2024-03-28T17:59:32.496 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.259+0000 7f3929600700 1 -- start start 2024-03-28T17:59:32.496 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.259+0000 7f3929600700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3924107a50 0x7f3924107e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:32.496 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.259+0000 7f3929600700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3924108440 con 0x7f3924107a50 2024-03-28T17:59:32.497 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.259+0000 7f3922a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3924107a50 0x7f3924107e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:32.497 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.260+0000 7f3922a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3924107a50 0x7f3924107e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50392/0 (socket says 172.21.15.37:50392) 2024-03-28T17:59:32.497 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.260+0000 7f3922a00700 1 -- 172.21.15.37:0/2109709190 learned_addr learned my addr 172.21.15.37:0/2109709190 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:32.497 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.260+0000 7f3922a00700 1 -- 172.21.15.37:0/2109709190 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3924108c50 con 0x7f3924107a50 2024-03-28T17:59:32.497 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.260+0000 7f3922a00700 1 --2- 172.21.15.37:0/2109709190 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3924107a50 0x7f3924107e70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3914009e80 tx=0x7f391400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fefed25f1b5385ce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:32.497 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.261+0000 7f3922000700 1 -- 172.21.15.37:0/2109709190 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3914017070 con 0x7f3924107a50 2024-03-28T17:59:32.497 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.261+0000 7f3922000700 1 -- 172.21.15.37:0/2109709190 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f391400b8c0 con 0x7f3924107a50 2024-03-28T17:59:32.497 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.261+0000 7f3922000700 1 -- 172.21.15.37:0/2109709190 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39140045c0 con 0x7f3924107a50 2024-03-28T17:59:32.497 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.262+0000 7f3929600700 1 -- 172.21.15.37:0/2109709190 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3924107a50 msgr2=0x7f3924107e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:32.497 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.262+0000 7f3929600700 1 --2- 172.21.15.37:0/2109709190 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3924107a50 0x7f3924107e70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3914009e80 tx=0x7f391400b330 comp rx=0 tx=0).stop 2024-03-28T17:59:32.497 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.262+0000 7f3929600700 1 -- 172.21.15.37:0/2109709190 shutdown_connections 2024-03-28T17:59:32.497 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.262+0000 7f3929600700 1 --2- 172.21.15.37:0/2109709190 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3924107a50 0x7f3924107e70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:32.497 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.262+0000 7f3929600700 1 -- 172.21.15.37:0/2109709190 >> 172.21.15.37:0/2109709190 conn(0x7f3924103070 msgr2=0x7f3924105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:32.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.262+0000 7f3929600700 1 -- 172.21.15.37:0/2109709190 shutdown_connections 2024-03-28T17:59:32.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.262+0000 7f3929600700 1 -- 172.21.15.37:0/2109709190 wait complete. 2024-03-28T17:59:32.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.263+0000 7f3929600700 1 Processor -- start 2024-03-28T17:59:32.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.263+0000 7f3929600700 1 -- start start 2024-03-28T17:59:32.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.264+0000 7f3929600700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3924107a50 0x7f392419dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:32.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.264+0000 7f3929600700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392410c000 con 0x7f3924107a50 2024-03-28T17:59:32.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.264+0000 7f3922a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3924107a50 0x7f392419dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:32.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.264+0000 7f3922a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3924107a50 0x7f392419dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50406/0 (socket says 172.21.15.37:50406) 2024-03-28T17:59:32.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.264+0000 7f3922a00700 1 -- 172.21.15.37:0/4254071931 learned_addr learned my addr 172.21.15.37:0/4254071931 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:32.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.264+0000 7f3922a00700 1 -- 172.21.15.37:0/4254071931 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392419e120 con 0x7f3924107a50 2024-03-28T17:59:32.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.264+0000 7f3922a00700 1 --2- 172.21.15.37:0/4254071931 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3924107a50 0x7f392419dbe0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3914009530 tx=0x7f391400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:32.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.265+0000 7f391be00700 1 -- 172.21.15.37:0/4254071931 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3914020020 con 0x7f3924107a50 2024-03-28T17:59:32.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.265+0000 7f391be00700 1 -- 172.21.15.37:0/4254071931 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3914003b50 con 0x7f3924107a50 2024-03-28T17:59:32.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.265+0000 7f391be00700 1 -- 172.21.15.37:0/4254071931 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3914017040 con 0x7f3924107a50 2024-03-28T17:59:32.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.265+0000 7f3929600700 1 -- 172.21.15.37:0/4254071931 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f392419e3b0 con 0x7f3924107a50 2024-03-28T17:59:32.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.265+0000 7f3929600700 1 -- 172.21.15.37:0/4254071931 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39241a10a0 con 0x7f3924107a50 2024-03-28T17:59:32.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.266+0000 7f3929600700 1 -- 172.21.15.37:0/4254071931 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39240637c0 con 0x7f3924107a50 2024-03-28T17:59:32.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.266+0000 7f391be00700 1 -- 172.21.15.37:0/4254071931 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f3914023020 con 0x7f3924107a50 2024-03-28T17:59:32.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.267+0000 7f391be00700 1 --2- 172.21.15.37:0/4254071931 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f3908038f70 0x7f390803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:32.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.267+0000 7f391be00700 1 -- 172.21.15.37:0/4254071931 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f391400f050 con 0x7f3924107a50 2024-03-28T17:59:32.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.267+0000 7f391a000700 1 -- 172.21.15.37:0/4254071931 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f3908038f70 msgr2=0x7f390803b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/1052839239 2024-03-28T17:59:32.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.267+0000 7f391a000700 1 --2- 172.21.15.37:0/4254071931 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f3908038f70 0x7f390803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T17:59:32.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.272+0000 7f391be00700 1 -- 172.21.15.37:0/4254071931 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f391402a020 con 0x7f3924107a50 2024-03-28T17:59:32.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.468+0000 7f391a000700 1 -- 172.21.15.37:0/4254071931 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f3908038f70 msgr2=0x7f390803b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/1052839239 2024-03-28T17:59:32.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.468+0000 7f391a000700 1 --2- 172.21.15.37:0/4254071931 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f3908038f70 0x7f390803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T17:59:32.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.557+0000 7f3929600700 1 -- 172.21.15.37:0/4254071931 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f39241a1270 con 0x7f3924107a50 2024-03-28T17:59:32.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.557+0000 7f391be00700 1 -- 172.21.15.37:0/4254071931 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f39241a1270 con 0x7f3924107a50 2024-03-28T17:59:32.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.560+0000 7f3929600700 1 -- 172.21.15.37:0/4254071931 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f3908038f70 msgr2=0x7f390803b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T17:59:32.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.560+0000 7f3929600700 1 --2- 172.21.15.37:0/4254071931 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f3908038f70 0x7f390803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:32.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.560+0000 7f3929600700 1 -- 172.21.15.37:0/4254071931 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3924107a50 msgr2=0x7f392419dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:32.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.560+0000 7f3929600700 1 --2- 172.21.15.37:0/4254071931 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3924107a50 0x7f392419dbe0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3914009530 tx=0x7f391400bd50 comp rx=0 tx=0).stop 2024-03-28T17:59:32.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.560+0000 7f3929600700 1 -- 172.21.15.37:0/4254071931 shutdown_connections 2024-03-28T17:59:32.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.560+0000 7f3929600700 1 --2- 172.21.15.37:0/4254071931 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7f3908038f70 0x7f390803b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:32.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.560+0000 7f3929600700 1 --2- 172.21.15.37:0/4254071931 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3924107a50 0x7f392419dbe0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:32.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.560+0000 7f3929600700 1 -- 172.21.15.37:0/4254071931 >> 172.21.15.37:0/4254071931 conn(0x7f3924103070 msgr2=0x7f3924103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:32.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.561+0000 7f3929600700 1 -- 172.21.15.37:0/4254071931 shutdown_connections 2024-03-28T17:59:32.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:31.561+0000 7f3929600700 1 -- 172.21.15.37:0/4254071931 wait complete. 2024-03-28T17:59:32.500 INFO:teuthology.orchestra.run.smithi037.stdout:Waiting for the mgr to restart... 2024-03-28T17:59:32.500 INFO:teuthology.orchestra.run.smithi037.stdout:Waiting for mgr epoch 5... 2024-03-28T17:59:32.624 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:32 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:32.239+0000 7fef92c77200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T17:59:32.624 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:32 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:32.355+0000 7fef92c77200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T17:59:32.973 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:32 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:32.709+0000 7fef92c77200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T17:59:32.973 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:32 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:32.865+0000 7fef92c77200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T17:59:33.252 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:32 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:32.987+0000 7fef92c77200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T17:59:33.252 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:33 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:33.118+0000 7fef92c77200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T17:59:33.624 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:33 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:33.341+0000 7fef92c77200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T17:59:34.829 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:34.652+0000 7fef92c77200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T17:59:35.124 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:34.829+0000 7fef92c77200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T17:59:35.624 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:35.253+0000 7fef92c77200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T17:59:35.624 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:35.376+0000 7fef92c77200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T17:59:35.893 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:35.640+0000 7fef92c77200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T17:59:36.279 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:35.892+0000 7fef92c77200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T17:59:36.279 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:36 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:36.279+0000 7fef92c77200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T17:59:36.875 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:36 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:36.610+0000 7fef92c77200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T17:59:37.374 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:36 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:36.931+0000 7fef92c77200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T17:59:38.124 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:38 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:38.012+0000 7fef92c77200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T17:59:39.124 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:38 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:38.734+0000 7fef92c77200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T17:59:39.124 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:38 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:38.867+0000 7fef92c77200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T17:59:39.625 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:39 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T17:59:39.157+0000 7fef92c77200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T17:59:39.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:39 smithi037 ceph-mon[31460]: Active manager daemon x restarted 2024-03-28T17:59:39.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:39 smithi037 ceph-mon[31460]: Activating manager daemon x 2024-03-28T17:59:39.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:39 smithi037 ceph-mon[31460]: osdmap e2: 0 total, 0 up, 0 in 2024-03-28T17:59:39.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:39 smithi037 ceph-mon[31460]: mgrmap e6: x(active, starting, since 0.0067768s) 2024-03-28T17:59:39.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:39 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:59:39.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:39 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T17:59:39.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:39 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T17:59:39.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:39 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T17:59:39.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:39 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T17:59:39.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:39 smithi037 ceph-mon[31460]: Manager daemon x is now available 2024-03-28T17:59:39.627 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:39 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' 2024-03-28T17:59:40.498 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:40 smithi037 ceph-mon[31460]: Found migration_current of "None". Setting to last migration. 2024-03-28T17:59:40.498 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:40 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' 2024-03-28T17:59:40.498 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:40 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:59:40.498 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:40 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:59:40.498 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:40 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:59:40.498 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:40 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T17:59:40.498 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:40 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T17:59:40.498 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:40 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' 2024-03-28T17:59:40.498 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:40 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' 2024-03-28T17:59:40.498 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:40 smithi037 ceph-mon[31460]: [28/Mar/2024:17:59:40] ENGINE Bus STARTING 2024-03-28T17:59:40.499 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:40 smithi037 ceph-mon[31460]: [28/Mar/2024:17:59:40] ENGINE Serving on http://172.21.15.37:8765 2024-03-28T17:59:40.499 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:40 smithi037 ceph-mon[31460]: mgrmap e7: x(active, since 1.01272s) 2024-03-28T17:59:40.836 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout { 2024-03-28T17:59:40.837 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-28T17:59:40.837 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T17:59:40.837 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout } 2024-03-28T17:59:40.837 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.042+0000 7fbb53c00700 1 Processor -- start 2024-03-28T17:59:40.837 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.042+0000 7fbb53c00700 1 -- start start 2024-03-28T17:59:40.838 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.043+0000 7fbb53c00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb4c108520 0x7fbb4c10a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:40.839 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.043+0000 7fbb53c00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c07be50 con 0x7fbb4c108520 2024-03-28T17:59:40.839 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.043+0000 7fbb51200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb4c108520 0x7fbb4c10a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:40.839 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.043+0000 7fbb51200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb4c108520 0x7fbb4c10a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50422/0 (socket says 172.21.15.37:50422) 2024-03-28T17:59:40.840 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.043+0000 7fbb51200700 1 -- 172.21.15.37:0/597161406 learned_addr learned my addr 172.21.15.37:0/597161406 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:40.840 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.043+0000 7fbb51200700 1 -- 172.21.15.37:0/597161406 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4c10ae50 con 0x7fbb4c108520 2024-03-28T17:59:40.840 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.044+0000 7fbb51200700 1 --2- 172.21.15.37:0/597161406 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb4c108520 0x7fbb4c10a910 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbb400098a0 tx=0x7fbb40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e216e0120f49b540 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:40.840 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.044+0000 7fbb4b400700 1 -- 172.21.15.37:0/597161406 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb40004160 con 0x7fbb4c108520 2024-03-28T17:59:40.840 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.044+0000 7fbb4b400700 1 -- 172.21.15.37:0/597161406 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb40004300 con 0x7fbb4c108520 2024-03-28T17:59:40.840 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.044+0000 7fbb4b400700 1 -- 172.21.15.37:0/597161406 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb40011b50 con 0x7fbb4c108520 2024-03-28T17:59:40.840 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.045+0000 7fbb53c00700 1 -- 172.21.15.37:0/597161406 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb4c108520 msgr2=0x7fbb4c10a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:40.840 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.045+0000 7fbb53c00700 1 --2- 172.21.15.37:0/597161406 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb4c108520 0x7fbb4c10a910 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbb400098a0 tx=0x7fbb40009bb0 comp rx=0 tx=0).stop 2024-03-28T17:59:40.840 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.045+0000 7fbb53c00700 1 -- 172.21.15.37:0/597161406 shutdown_connections 2024-03-28T17:59:40.840 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.045+0000 7fbb53c00700 1 --2- 172.21.15.37:0/597161406 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb4c108520 0x7fbb4c10a910 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:40.840 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.045+0000 7fbb53c00700 1 -- 172.21.15.37:0/597161406 >> 172.21.15.37:0/597161406 conn(0x7fbb4c101f70 msgr2=0x7fbb4c104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:40.842 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.045+0000 7fbb53c00700 1 -- 172.21.15.37:0/597161406 shutdown_connections 2024-03-28T17:59:40.842 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.045+0000 7fbb53c00700 1 -- 172.21.15.37:0/597161406 wait complete. 2024-03-28T17:59:40.842 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.046+0000 7fbb53c00700 1 Processor -- start 2024-03-28T17:59:40.842 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.046+0000 7fbb53c00700 1 -- start start 2024-03-28T17:59:40.842 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.047+0000 7fbb53c00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb4c108520 0x7fbb4c107da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:40.842 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.047+0000 7fbb53c00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c10be70 con 0x7fbb4c108520 2024-03-28T17:59:40.842 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.047+0000 7fbb51200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb4c108520 0x7fbb4c107da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:40.842 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.047+0000 7fbb51200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb4c108520 0x7fbb4c107da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50432/0 (socket says 172.21.15.37:50432) 2024-03-28T17:59:40.842 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.047+0000 7fbb51200700 1 -- 172.21.15.37:0/58016059 learned_addr learned my addr 172.21.15.37:0/58016059 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:40.842 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.047+0000 7fbb51200700 1 -- 172.21.15.37:0/58016059 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4c1082e0 con 0x7fbb4c108520 2024-03-28T17:59:40.844 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.048+0000 7fbb51200700 1 --2- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb4c108520 0x7fbb4c107da0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbb40011670 tx=0x7fbb40011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:40.844 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.048+0000 7fbb49600700 1 -- 172.21.15.37:0/58016059 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb400039c0 con 0x7fbb4c108520 2024-03-28T17:59:40.844 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.048+0000 7fbb53c00700 1 -- 172.21.15.37:0/58016059 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb4c1064d0 con 0x7fbb4c108520 2024-03-28T17:59:40.844 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.048+0000 7fbb49600700 1 -- 172.21.15.37:0/58016059 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb40011e50 con 0x7fbb4c108520 2024-03-28T17:59:40.844 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.048+0000 7fbb49600700 1 -- 172.21.15.37:0/58016059 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb4001a5c0 con 0x7fbb4c108520 2024-03-28T17:59:40.844 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.048+0000 7fbb53c00700 1 -- 172.21.15.37:0/58016059 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb4c106a10 con 0x7fbb4c108520 2024-03-28T17:59:40.844 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.050+0000 7fbb49600700 1 -- 172.21.15.37:0/58016059 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fbb40024430 con 0x7fbb4c108520 2024-03-28T17:59:40.844 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.050+0000 7fbb49600700 1 --2- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7fbb34038ed0 0x7fbb3403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:40.844 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.050+0000 7fbb49600700 1 -- 172.21.15.37:0/58016059 --> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb3403baa0 con 0x7fbb34038ed0 2024-03-28T17:59:40.845 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.050+0000 7fbb49600700 1 -- 172.21.15.37:0/58016059 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbb4004d210 con 0x7fbb4c108520 2024-03-28T17:59:40.845 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.050+0000 7fbb4be00700 1 -- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7fbb34038ed0 msgr2=0x7fbb3403b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/1052839239 2024-03-28T17:59:40.845 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.050+0000 7fbb4be00700 1 --2- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7fbb34038ed0 0x7fbb3403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T17:59:40.846 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.251+0000 7fbb4be00700 1 -- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7fbb34038ed0 msgr2=0x7fbb3403b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/1052839239 2024-03-28T17:59:40.846 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.251+0000 7fbb4be00700 1 --2- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7fbb34038ed0 0x7fbb3403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T17:59:40.846 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.652+0000 7fbb4be00700 1 -- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7fbb34038ed0 msgr2=0x7fbb3403b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/1052839239 2024-03-28T17:59:40.846 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:33.652+0000 7fbb4be00700 1 --2- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7fbb34038ed0 0x7fbb3403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T17:59:40.847 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:34.453+0000 7fbb4be00700 1 -- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7fbb34038ed0 msgr2=0x7fbb3403b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/1052839239 2024-03-28T17:59:40.847 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:34.453+0000 7fbb4be00700 1 --2- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7fbb34038ed0 0x7fbb3403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T17:59:40.847 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:36.055+0000 7fbb4be00700 1 -- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7fbb34038ed0 msgr2=0x7fbb3403b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/1052839239 2024-03-28T17:59:40.847 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:36.055+0000 7fbb4be00700 1 --2- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7fbb34038ed0 0x7fbb3403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T17:59:40.847 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:39.166+0000 7fbb49600700 1 -- 172.21.15.37:0/58016059 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mgrmap(e 6) v1 ==== 45873+0+0 (secure 0 0 0) 0x7fbb4002d3f0 con 0x7fbb4c108520 2024-03-28T17:59:40.847 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:39.166+0000 7fbb49600700 1 -- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7fbb34038ed0 msgr2=0x7fbb3403b390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T17:59:40.847 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:39.166+0000 7fbb49600700 1 --2- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7fbb34038ed0 0x7fbb3403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:40.848 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:40.168+0000 7fbb49600700 1 -- 172.21.15.37:0/58016059 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7fbb4004eca0 con 0x7fbb4c108520 2024-03-28T17:59:40.848 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:40.169+0000 7fbb49600700 1 --2- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7fbb3403c7c0 0x7fbb3403ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:40.848 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:40.169+0000 7fbb49600700 1 -- 172.21.15.37:0/58016059 --> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb3403baa0 con 0x7fbb3403c7c0 2024-03-28T17:59:40.848 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:40.170+0000 7fbb4be00700 1 --2- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7fbb3403c7c0 0x7fbb3403ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:40.848 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:40.172+0000 7fbb4be00700 1 --2- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7fbb3403c7c0 0x7fbb3403ebb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbb38003a10 tx=0x7fbb38009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:40.849 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:40.173+0000 7fbb49600700 1 -- 172.21.15.37:0/58016059 <== mgr.14120 v2:172.21.15.37:6800/3459498509 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7fbb3403baa0 con 0x7fbb3403c7c0 2024-03-28T17:59:40.849 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:40.181+0000 7fbb53c00700 1 -- 172.21.15.37:0/58016059 --> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fbb4c0637c0 con 0x7fbb3403c7c0 2024-03-28T17:59:40.849 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:40.182+0000 7fbb49600700 1 -- 172.21.15.37:0/58016059 <== mgr.14120 v2:172.21.15.37:6800/3459498509 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fbb4c0637c0 con 0x7fbb3403c7c0 2024-03-28T17:59:40.849 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:40.182+0000 7fbb53c00700 1 -- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7fbb3403c7c0 msgr2=0x7fbb3403ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:40.850 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:40.182+0000 7fbb53c00700 1 --2- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7fbb3403c7c0 0x7fbb3403ebb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbb38003a10 tx=0x7fbb38009290 comp rx=0 tx=0).stop 2024-03-28T17:59:40.850 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:40.183+0000 7fbb53c00700 1 -- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb4c108520 msgr2=0x7fbb4c107da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:40.850 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:40.183+0000 7fbb53c00700 1 --2- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb4c108520 0x7fbb4c107da0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbb40011670 tx=0x7fbb40011b60 comp rx=0 tx=0).stop 2024-03-28T17:59:40.850 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:40.183+0000 7fbb53c00700 1 -- 172.21.15.37:0/58016059 shutdown_connections 2024-03-28T17:59:40.851 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:40.183+0000 7fbb53c00700 1 --2- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7fbb3403c7c0 0x7fbb3403ebb0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:40.851 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:40.183+0000 7fbb53c00700 1 --2- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb4c108520 0x7fbb4c107da0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:40.851 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:40.183+0000 7fbb53c00700 1 --2- 172.21.15.37:0/58016059 >> [v2:172.21.15.37:6800/1052839239,v1:172.21.15.37:6801/1052839239] conn(0x7fbb34038ed0 0x7fbb3403b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:40.851 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:40.183+0000 7fbb53c00700 1 -- 172.21.15.37:0/58016059 >> 172.21.15.37:0/58016059 conn(0x7fbb4c101f70 msgr2=0x7fbb4c102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:40.852 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:40.183+0000 7fbb53c00700 1 -- 172.21.15.37:0/58016059 shutdown_connections 2024-03-28T17:59:40.852 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:40.183+0000 7fbb53c00700 1 -- 172.21.15.37:0/58016059 wait complete. 2024-03-28T17:59:40.852 INFO:teuthology.orchestra.run.smithi037.stdout:mgr epoch 5 is available 2024-03-28T17:59:40.852 INFO:teuthology.orchestra.run.smithi037.stdout:Setting orchestrator backend to cephadm... 2024-03-28T17:59:41.405 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:41 smithi037 ceph-mon[31460]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-28T17:59:41.405 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:41 smithi037 ceph-mon[31460]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-28T17:59:41.405 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:41 smithi037 ceph-mon[31460]: [28/Mar/2024:17:59:40] ENGINE Serving on https://172.21.15.37:7150 2024-03-28T17:59:41.405 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:41 smithi037 ceph-mon[31460]: [28/Mar/2024:17:59:40] ENGINE Bus STARTED 2024-03-28T17:59:41.405 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:41 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:59:42.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:42 smithi037 ceph-mon[31460]: mgrmap e8: x(active, since 2s) 2024-03-28T17:59:42.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:42 smithi037 ceph-mon[31460]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:59:42.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:42 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' 2024-03-28T17:59:42.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:42 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:59:42.456 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.479+0000 7f9096c00700 1 Processor -- start 2024-03-28T17:59:42.456 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.479+0000 7f9096c00700 1 -- start start 2024-03-28T17:59:42.456 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.479+0000 7f9096c00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9090107a50 0x7f9090107e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:42.456 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.479+0000 7f9096c00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9090108440 con 0x7f9090107a50 2024-03-28T17:59:42.456 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.480+0000 7f908fe00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9090107a50 0x7f9090107e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:42.456 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.480+0000 7f908fe00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9090107a50 0x7f9090107e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47396/0 (socket says 172.21.15.37:47396) 2024-03-28T17:59:42.456 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.480+0000 7f908fe00700 1 -- 172.21.15.37:0/3117681105 learned_addr learned my addr 172.21.15.37:0/3117681105 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:42.456 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.480+0000 7f908fe00700 1 -- 172.21.15.37:0/3117681105 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9090108c50 con 0x7f9090107a50 2024-03-28T17:59:42.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.481+0000 7f908fe00700 1 --2- 172.21.15.37:0/3117681105 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9090107a50 0x7f9090107e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f90800098a0 tx=0x7f9080009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c71ccd6d4b89029 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:42.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.481+0000 7f908ea00700 1 -- 172.21.15.37:0/3117681105 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9080004160 con 0x7f9090107a50 2024-03-28T17:59:42.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.481+0000 7f908ea00700 1 -- 172.21.15.37:0/3117681105 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9080004300 con 0x7f9090107a50 2024-03-28T17:59:42.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.481+0000 7f908ea00700 1 -- 172.21.15.37:0/3117681105 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9080011b50 con 0x7f9090107a50 2024-03-28T17:59:42.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.482+0000 7f9096c00700 1 -- 172.21.15.37:0/3117681105 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9090107a50 msgr2=0x7f9090107e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:42.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.482+0000 7f9096c00700 1 --2- 172.21.15.37:0/3117681105 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9090107a50 0x7f9090107e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f90800098a0 tx=0x7f9080009bb0 comp rx=0 tx=0).stop 2024-03-28T17:59:42.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.482+0000 7f9096c00700 1 -- 172.21.15.37:0/3117681105 shutdown_connections 2024-03-28T17:59:42.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.482+0000 7f9096c00700 1 --2- 172.21.15.37:0/3117681105 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9090107a50 0x7f9090107e70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:42.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.482+0000 7f9096c00700 1 -- 172.21.15.37:0/3117681105 >> 172.21.15.37:0/3117681105 conn(0x7f9090103070 msgr2=0x7f9090105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:42.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.482+0000 7f9096c00700 1 -- 172.21.15.37:0/3117681105 shutdown_connections 2024-03-28T17:59:42.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.482+0000 7f9096c00700 1 -- 172.21.15.37:0/3117681105 wait complete. 2024-03-28T17:59:42.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.483+0000 7f9096c00700 1 Processor -- start 2024-03-28T17:59:42.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.483+0000 7f9096c00700 1 -- start start 2024-03-28T17:59:42.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.484+0000 7f9096c00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9090107a50 0x7f909019dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:42.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.484+0000 7f9096c00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909010c000 con 0x7f9090107a50 2024-03-28T17:59:42.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.484+0000 7f908fe00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9090107a50 0x7f909019dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:42.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.484+0000 7f908fe00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9090107a50 0x7f909019dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47406/0 (socket says 172.21.15.37:47406) 2024-03-28T17:59:42.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.484+0000 7f908fe00700 1 -- 172.21.15.37:0/2767760674 learned_addr learned my addr 172.21.15.37:0/2767760674 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:42.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.485+0000 7f908fe00700 1 -- 172.21.15.37:0/2767760674 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909019e180 con 0x7f9090107a50 2024-03-28T17:59:42.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.485+0000 7f908fe00700 1 --2- 172.21.15.37:0/2767760674 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9090107a50 0x7f909019dc40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9080000c00 tx=0x7f9080011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:42.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.485+0000 7f908cc00700 1 -- 172.21.15.37:0/2767760674 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9080003b20 con 0x7f9090107a50 2024-03-28T17:59:42.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.485+0000 7f908cc00700 1 -- 172.21.15.37:0/2767760674 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9080003cc0 con 0x7f9090107a50 2024-03-28T17:59:42.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.485+0000 7f9096c00700 1 -- 172.21.15.37:0/2767760674 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f909019e410 con 0x7f9090107a50 2024-03-28T17:59:42.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.485+0000 7f908cc00700 1 -- 172.21.15.37:0/2767760674 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9080024910 con 0x7f9090107a50 2024-03-28T17:59:42.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.486+0000 7f9096c00700 1 -- 172.21.15.37:0/2767760674 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90901a1100 con 0x7f9090107a50 2024-03-28T17:59:42.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.487+0000 7f9096c00700 1 -- 172.21.15.37:0/2767760674 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90900637c0 con 0x7f9090107a50 2024-03-28T17:59:42.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.487+0000 7f908cc00700 1 -- 172.21.15.37:0/2767760674 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f9080022020 con 0x7f9090107a50 2024-03-28T17:59:42.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.487+0000 7f908cc00700 1 --2- 172.21.15.37:0/2767760674 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f9074038f70 0x7f907403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:42.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.487+0000 7f908cc00700 1 -- 172.21.15.37:0/2767760674 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9080011e00 con 0x7f9090107a50 2024-03-28T17:59:42.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.488+0000 7f908f400700 1 --2- 172.21.15.37:0/2767760674 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f9074038f70 0x7f907403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:42.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.488+0000 7f908f400700 1 --2- 172.21.15.37:0/2767760674 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f9074038f70 0x7f907403b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f907c006fd0 tx=0x7f907c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:42.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.494+0000 7f908cc00700 1 -- 172.21.15.37:0/2767760674 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9080016260 con 0x7f9090107a50 2024-03-28T17:59:42.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.738+0000 7f9096c00700 1 -- 172.21.15.37:0/2767760674 --> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f9090104200 con 0x7f9074038f70 2024-03-28T17:59:42.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.751+0000 7f908cc00700 1 -- 172.21.15.37:0/2767760674 <== mgr.14120 v2:172.21.15.37:6800/3459498509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9090104200 con 0x7f9074038f70 2024-03-28T17:59:42.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.755+0000 7f9096c00700 1 -- 172.21.15.37:0/2767760674 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f9074038f70 msgr2=0x7f907403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:42.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.755+0000 7f9096c00700 1 --2- 172.21.15.37:0/2767760674 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f9074038f70 0x7f907403b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f907c006fd0 tx=0x7f907c006e40 comp rx=0 tx=0).stop 2024-03-28T17:59:42.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.755+0000 7f9096c00700 1 -- 172.21.15.37:0/2767760674 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9090107a50 msgr2=0x7f909019dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:42.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.755+0000 7f9096c00700 1 --2- 172.21.15.37:0/2767760674 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9090107a50 0x7f909019dc40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9080000c00 tx=0x7f9080011c10 comp rx=0 tx=0).stop 2024-03-28T17:59:42.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.755+0000 7f9096c00700 1 -- 172.21.15.37:0/2767760674 shutdown_connections 2024-03-28T17:59:42.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.755+0000 7f9096c00700 1 --2- 172.21.15.37:0/2767760674 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f9074038f70 0x7f907403b430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:42.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.755+0000 7f9096c00700 1 --2- 172.21.15.37:0/2767760674 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9090107a50 0x7f909019dc40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:42.460 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.755+0000 7f9096c00700 1 -- 172.21.15.37:0/2767760674 >> 172.21.15.37:0/2767760674 conn(0x7f9090103070 msgr2=0x7f9090103af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:42.460 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.756+0000 7f9096c00700 1 -- 172.21.15.37:0/2767760674 shutdown_connections 2024-03-28T17:59:42.460 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:41.756+0000 7f9096c00700 1 -- 172.21.15.37:0/2767760674 wait complete. 2024-03-28T17:59:44.009 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-28T17:59:44.009 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.071+0000 7fd547000700 1 Processor -- start 2024-03-28T17:59:44.009 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.072+0000 7fd547000700 1 -- start start 2024-03-28T17:59:44.009 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.072+0000 7fd547000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd540081aa0 0x7fd540081ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:44.009 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.072+0000 7fd547000700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd540082490 con 0x7fd540081aa0 2024-03-28T17:59:44.009 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.072+0000 7fd53fe00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd540081aa0 0x7fd540081ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:44.009 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.072+0000 7fd53fe00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd540081aa0 0x7fd540081ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47412/0 (socket says 172.21.15.37:47412) 2024-03-28T17:59:44.010 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.072+0000 7fd53fe00700 1 -- 172.21.15.37:0/1284553033 learned_addr learned my addr 172.21.15.37:0/1284553033 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:44.010 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.073+0000 7fd53fe00700 1 -- 172.21.15.37:0/1284553033 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54010f280 con 0x7fd540081aa0 2024-03-28T17:59:44.010 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.073+0000 7fd53fe00700 1 --2- 172.21.15.37:0/1284553033 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd540081aa0 0x7fd540081ec0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd5300098a0 tx=0x7fd530009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec23f3e22834e9af server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:44.010 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.074+0000 7fd53ea00700 1 -- 172.21.15.37:0/1284553033 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd530004160 con 0x7fd540081aa0 2024-03-28T17:59:44.010 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.074+0000 7fd53ea00700 1 -- 172.21.15.37:0/1284553033 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd530004300 con 0x7fd540081aa0 2024-03-28T17:59:44.010 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.074+0000 7fd53ea00700 1 -- 172.21.15.37:0/1284553033 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd530011b50 con 0x7fd540081aa0 2024-03-28T17:59:44.010 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.074+0000 7fd547000700 1 -- 172.21.15.37:0/1284553033 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd540081aa0 msgr2=0x7fd540081ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:44.010 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.074+0000 7fd547000700 1 --2- 172.21.15.37:0/1284553033 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd540081aa0 0x7fd540081ec0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd5300098a0 tx=0x7fd530009bb0 comp rx=0 tx=0).stop 2024-03-28T17:59:44.010 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.074+0000 7fd547000700 1 -- 172.21.15.37:0/1284553033 shutdown_connections 2024-03-28T17:59:44.010 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.074+0000 7fd547000700 1 --2- 172.21.15.37:0/1284553033 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd540081aa0 0x7fd540081ec0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:44.010 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.074+0000 7fd547000700 1 -- 172.21.15.37:0/1284553033 >> 172.21.15.37:0/1284553033 conn(0x7fd54007d040 msgr2=0x7fd54007f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:44.010 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.074+0000 7fd547000700 1 -- 172.21.15.37:0/1284553033 shutdown_connections 2024-03-28T17:59:44.010 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.075+0000 7fd547000700 1 -- 172.21.15.37:0/1284553033 wait complete. 2024-03-28T17:59:44.011 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.075+0000 7fd547000700 1 Processor -- start 2024-03-28T17:59:44.011 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.076+0000 7fd547000700 1 -- start start 2024-03-28T17:59:44.011 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.076+0000 7fd547000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd540081aa0 0x7fd5401a2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:44.011 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.076+0000 7fd547000700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd540112630 con 0x7fd540081aa0 2024-03-28T17:59:44.011 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.076+0000 7fd53fe00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd540081aa0 0x7fd5401a2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:44.014 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.076+0000 7fd53fe00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd540081aa0 0x7fd5401a2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47416/0 (socket says 172.21.15.37:47416) 2024-03-28T17:59:44.014 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.076+0000 7fd53fe00700 1 -- 172.21.15.37:0/3861385287 learned_addr learned my addr 172.21.15.37:0/3861385287 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:44.014 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.077+0000 7fd53fe00700 1 -- 172.21.15.37:0/3861385287 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5401a2570 con 0x7fd540081aa0 2024-03-28T17:59:44.014 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.077+0000 7fd53fe00700 1 --2- 172.21.15.37:0/3861385287 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd540081aa0 0x7fd5401a2030 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd530000c00 tx=0x7fd530011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:44.014 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.077+0000 7fd53cc00700 1 -- 172.21.15.37:0/3861385287 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd530003b20 con 0x7fd540081aa0 2024-03-28T17:59:44.014 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.077+0000 7fd53cc00700 1 -- 172.21.15.37:0/3861385287 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd530003cc0 con 0x7fd540081aa0 2024-03-28T17:59:44.014 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.077+0000 7fd53cc00700 1 -- 172.21.15.37:0/3861385287 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd530024910 con 0x7fd540081aa0 2024-03-28T17:59:44.014 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.077+0000 7fd547000700 1 -- 172.21.15.37:0/3861385287 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5401a2800 con 0x7fd540081aa0 2024-03-28T17:59:44.014 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.077+0000 7fd547000700 1 -- 172.21.15.37:0/3861385287 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5401a54f0 con 0x7fd540081aa0 2024-03-28T17:59:44.014 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.078+0000 7fd547000700 1 -- 172.21.15.37:0/3861385287 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5400637c0 con 0x7fd540081aa0 2024-03-28T17:59:44.014 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.078+0000 7fd53cc00700 1 -- 172.21.15.37:0/3861385287 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fd530022020 con 0x7fd540081aa0 2024-03-28T17:59:44.015 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.079+0000 7fd53cc00700 1 --2- 172.21.15.37:0/3861385287 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7fd524038f70 0x7fd52403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:44.015 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.079+0000 7fd53cc00700 1 -- 172.21.15.37:0/3861385287 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd530011e00 con 0x7fd540081aa0 2024-03-28T17:59:44.015 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.079+0000 7fd53f400700 1 --2- 172.21.15.37:0/3861385287 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7fd524038f70 0x7fd52403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:44.015 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.080+0000 7fd53f400700 1 --2- 172.21.15.37:0/3861385287 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7fd524038f70 0x7fd52403b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd52c006fd0 tx=0x7fd52c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:44.015 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.085+0000 7fd53cc00700 1 -- 172.21.15.37:0/3861385287 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd530016260 con 0x7fd540081aa0 2024-03-28T17:59:44.015 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.312+0000 7fd547000700 1 -- 172.21.15.37:0/3861385287 --> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fd54007e240 con 0x7fd524038f70 2024-03-28T17:59:44.015 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.313+0000 7fd53cc00700 1 -- 172.21.15.37:0/3861385287 <== mgr.14120 v2:172.21.15.37:6800/3459498509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fd54007e240 con 0x7fd524038f70 2024-03-28T17:59:44.015 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.316+0000 7fd547000700 1 -- 172.21.15.37:0/3861385287 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7fd524038f70 msgr2=0x7fd52403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:44.015 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.316+0000 7fd547000700 1 --2- 172.21.15.37:0/3861385287 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7fd524038f70 0x7fd52403b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd52c006fd0 tx=0x7fd52c006e40 comp rx=0 tx=0).stop 2024-03-28T17:59:44.015 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.316+0000 7fd547000700 1 -- 172.21.15.37:0/3861385287 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd540081aa0 msgr2=0x7fd5401a2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:44.015 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.316+0000 7fd547000700 1 --2- 172.21.15.37:0/3861385287 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd540081aa0 0x7fd5401a2030 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd530000c00 tx=0x7fd530011c10 comp rx=0 tx=0).stop 2024-03-28T17:59:44.015 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.317+0000 7fd547000700 1 -- 172.21.15.37:0/3861385287 shutdown_connections 2024-03-28T17:59:44.015 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.317+0000 7fd547000700 1 --2- 172.21.15.37:0/3861385287 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7fd524038f70 0x7fd52403b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:44.015 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.317+0000 7fd547000700 1 --2- 172.21.15.37:0/3861385287 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd540081aa0 0x7fd5401a2030 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:44.016 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.317+0000 7fd547000700 1 -- 172.21.15.37:0/3861385287 >> 172.21.15.37:0/3861385287 conn(0x7fd54007d040 msgr2=0x7fd54007db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:44.016 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.317+0000 7fd547000700 1 -- 172.21.15.37:0/3861385287 shutdown_connections 2024-03-28T17:59:44.016 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:43.317+0000 7fd547000700 1 -- 172.21.15.37:0/3861385287 wait complete. 2024-03-28T17:59:44.016 INFO:teuthology.orchestra.run.smithi037.stdout:Generating ssh key... 2024-03-28T17:59:44.279 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:44 smithi037 ceph-mon[31460]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:59:45.325 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: Generating public/private rsa key pair. 2024-03-28T17:59:45.326 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: Your identification has been saved in /tmp/tmpbmk0ra5u/key. 2024-03-28T17:59:45.326 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: Your public key has been saved in /tmp/tmpbmk0ra5u/key.pub. 2024-03-28T17:59:45.326 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: The key fingerprint is: 2024-03-28T17:59:45.326 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: SHA256:UBMkZ05WhJQ5I/OZteJ90OFVlYtJMcF/dBsENd6ggf8 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715 2024-03-28T17:59:45.326 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: The key's randomart image is: 2024-03-28T17:59:45.326 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: +---[RSA 3072]----+ 2024-03-28T17:59:45.327 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: | .o@Bo.o=B==| 2024-03-28T17:59:45.327 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: | oO*.o .=++=| 2024-03-28T17:59:45.327 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: | .+.* =oo+o*| 2024-03-28T17:59:45.327 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: | .= o +o +.| 2024-03-28T17:59:45.327 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: | .So . . .| 2024-03-28T17:59:45.327 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: | . . . E | 2024-03-28T17:59:45.327 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: | . | 2024-03-28T17:59:45.327 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: | | 2024-03-28T17:59:45.327 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: | | 2024-03-28T17:59:45.327 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 17:59:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: +----[SHA256]-----+ 2024-03-28T17:59:45.656 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.573+0000 7ff423600700 1 Processor -- start 2024-03-28T17:59:45.656 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.573+0000 7ff423600700 1 -- start start 2024-03-28T17:59:45.656 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.574+0000 7ff423600700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff41c106980 0x7ff41c106da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:45.656 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.574+0000 7ff423600700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c107370 con 0x7ff41c106980 2024-03-28T17:59:45.656 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.574+0000 7ff420c00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff41c106980 0x7ff41c106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:45.661 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.574+0000 7ff420c00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff41c106980 0x7ff41c106da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51320/0 (socket says 172.21.15.37:51320) 2024-03-28T17:59:45.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.574+0000 7ff420c00700 1 -- 172.21.15.37:0/1077518314 learned_addr learned my addr 172.21.15.37:0/1077518314 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:45.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.575+0000 7ff420c00700 1 -- 172.21.15.37:0/1077518314 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41c107b80 con 0x7ff41c106980 2024-03-28T17:59:45.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.575+0000 7ff420c00700 1 --2- 172.21.15.37:0/1077518314 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff41c106980 0x7ff41c106da0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff4100098a0 tx=0x7ff410009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbd8d44d4153d4c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:45.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.576+0000 7ff41b400700 1 -- 172.21.15.37:0/1077518314 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff410004160 con 0x7ff41c106980 2024-03-28T17:59:45.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.576+0000 7ff41b400700 1 -- 172.21.15.37:0/1077518314 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff410004300 con 0x7ff41c106980 2024-03-28T17:59:45.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.576+0000 7ff41b400700 1 -- 172.21.15.37:0/1077518314 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff410011b50 con 0x7ff41c106980 2024-03-28T17:59:45.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.576+0000 7ff423600700 1 -- 172.21.15.37:0/1077518314 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff41c106980 msgr2=0x7ff41c106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:45.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.576+0000 7ff423600700 1 --2- 172.21.15.37:0/1077518314 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff41c106980 0x7ff41c106da0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff4100098a0 tx=0x7ff410009bb0 comp rx=0 tx=0).stop 2024-03-28T17:59:45.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.577+0000 7ff423600700 1 -- 172.21.15.37:0/1077518314 shutdown_connections 2024-03-28T17:59:45.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.577+0000 7ff423600700 1 --2- 172.21.15.37:0/1077518314 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff41c106980 0x7ff41c106da0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:45.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.577+0000 7ff423600700 1 -- 172.21.15.37:0/1077518314 >> 172.21.15.37:0/1077518314 conn(0x7ff41c101f20 msgr2=0x7ff41c104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:45.662 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.577+0000 7ff423600700 1 -- 172.21.15.37:0/1077518314 shutdown_connections 2024-03-28T17:59:45.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.577+0000 7ff423600700 1 -- 172.21.15.37:0/1077518314 wait complete. 2024-03-28T17:59:45.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.578+0000 7ff423600700 1 Processor -- start 2024-03-28T17:59:45.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.578+0000 7ff423600700 1 -- start start 2024-03-28T17:59:45.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.578+0000 7ff423600700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff41c106980 0x7ff41c19dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:45.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.578+0000 7ff423600700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c10af30 con 0x7ff41c106980 2024-03-28T17:59:45.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.579+0000 7ff420c00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff41c106980 0x7ff41c19dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:45.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.579+0000 7ff420c00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff41c106980 0x7ff41c19dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51334/0 (socket says 172.21.15.37:51334) 2024-03-28T17:59:45.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.579+0000 7ff420c00700 1 -- 172.21.15.37:0/1045693865 learned_addr learned my addr 172.21.15.37:0/1045693865 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:45.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.579+0000 7ff420c00700 1 -- 172.21.15.37:0/1045693865 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41c19e1b0 con 0x7ff41c106980 2024-03-28T17:59:45.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.579+0000 7ff420c00700 1 --2- 172.21.15.37:0/1045693865 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff41c106980 0x7ff41c19dc70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff410000c00 tx=0x7ff410011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:45.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.580+0000 7ff419600700 1 -- 172.21.15.37:0/1045693865 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff410003b20 con 0x7ff41c106980 2024-03-28T17:59:45.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.580+0000 7ff419600700 1 -- 172.21.15.37:0/1045693865 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff410003cc0 con 0x7ff41c106980 2024-03-28T17:59:45.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.580+0000 7ff423600700 1 -- 172.21.15.37:0/1045693865 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff41c19e440 con 0x7ff41c106980 2024-03-28T17:59:45.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.580+0000 7ff419600700 1 -- 172.21.15.37:0/1045693865 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff410024910 con 0x7ff41c106980 2024-03-28T17:59:45.663 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.580+0000 7ff423600700 1 -- 172.21.15.37:0/1045693865 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff41c1a1130 con 0x7ff41c106980 2024-03-28T17:59:45.664 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.581+0000 7ff419600700 1 -- 172.21.15.37:0/1045693865 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7ff410022020 con 0x7ff41c106980 2024-03-28T17:59:45.664 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.581+0000 7ff423600700 1 -- 172.21.15.37:0/1045693865 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff41c0637c0 con 0x7ff41c106980 2024-03-28T17:59:45.664 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.582+0000 7ff419600700 1 --2- 172.21.15.37:0/1045693865 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7ff404038f70 0x7ff40403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:45.664 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.582+0000 7ff419600700 1 -- 172.21.15.37:0/1045693865 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff410011e00 con 0x7ff41c106980 2024-03-28T17:59:45.664 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.582+0000 7ff41be00700 1 --2- 172.21.15.37:0/1045693865 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7ff404038f70 0x7ff40403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:45.664 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.583+0000 7ff41be00700 1 --2- 172.21.15.37:0/1045693865 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7ff404038f70 0x7ff40403b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff408006fd0 tx=0x7ff408006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:45.664 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.588+0000 7ff419600700 1 -- 172.21.15.37:0/1045693865 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff410016260 con 0x7ff41c106980 2024-03-28T17:59:45.664 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.814+0000 7ff423600700 1 -- 172.21.15.37:0/1045693865 --> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff41c0008d0 con 0x7ff404038f70 2024-03-28T17:59:45.664 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.936+0000 7ff419600700 1 -- 172.21.15.37:0/1045693865 <== mgr.14120 v2:172.21.15.37:6800/3459498509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff41c0008d0 con 0x7ff404038f70 2024-03-28T17:59:45.664 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.939+0000 7ff423600700 1 -- 172.21.15.37:0/1045693865 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7ff404038f70 msgr2=0x7ff40403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:45.664 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.939+0000 7ff423600700 1 --2- 172.21.15.37:0/1045693865 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7ff404038f70 0x7ff40403b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff408006fd0 tx=0x7ff408006e40 comp rx=0 tx=0).stop 2024-03-28T17:59:45.664 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.939+0000 7ff423600700 1 -- 172.21.15.37:0/1045693865 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff41c106980 msgr2=0x7ff41c19dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:45.664 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.939+0000 7ff423600700 1 --2- 172.21.15.37:0/1045693865 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff41c106980 0x7ff41c19dc70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff410000c00 tx=0x7ff410011c10 comp rx=0 tx=0).stop 2024-03-28T17:59:45.665 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.939+0000 7ff423600700 1 -- 172.21.15.37:0/1045693865 shutdown_connections 2024-03-28T17:59:45.665 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.939+0000 7ff423600700 1 --2- 172.21.15.37:0/1045693865 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7ff404038f70 0x7ff40403b430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:45.665 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.939+0000 7ff423600700 1 --2- 172.21.15.37:0/1045693865 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff41c106980 0x7ff41c19dc70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:45.665 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.939+0000 7ff423600700 1 -- 172.21.15.37:0/1045693865 >> 172.21.15.37:0/1045693865 conn(0x7ff41c101f20 msgr2=0x7ff41c102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:45.665 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.940+0000 7ff423600700 1 -- 172.21.15.37:0/1045693865 shutdown_connections 2024-03-28T17:59:45.665 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:44.940+0000 7ff423600700 1 -- 172.21.15.37:0/1045693865 wait complete. 2024-03-28T17:59:46.174 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:45 smithi037 ceph-mon[31460]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:59:46.174 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:45 smithi037 ceph-mon[31460]: Generating ssh key... 2024-03-28T17:59:46.175 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:45 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' 2024-03-28T17:59:46.175 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:45 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' 2024-03-28T17:59:47.282 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD1F/cnl50HxYSSFY9/w1WPzQL0j1VQwl2pJqTdZ+7bihhWxC6YMhT5fityfUZ4Ac2HZOm+RDIBFCLo78r5NmSWvWSMz9cLoCLKiyPS7OH2GVX00RPFgoAbeIwm6OfIF/aae1Gxsc1TsjsNSoCdWWJcbj2Kxg5jqsbV2j028FIIwePvymequbOrHMQgpNNeHFWdeqvAZ36DwAZyvYwxPV1ntCOi+MGPEDMHynYMp8jFiOb9HwRvPICkTu1+7uVDHtA4B68WKg5QMvL4UY/3oM38MzOsuM+BtwcyG5PCRQQ6y3JUf0KPvyOeTBczlEzlEDPkJs7XqtT0AfHNiOQa6Ufrpq59MimbREGm4N0JScbzbkQvsydKEvp02WKC1iKWVnYSJeVzgEIOYZY9F9lH2V2VxF/7NWPx+8P8/cBx+pOnRJpMlpufe5Ov75PetgLNQzpUTvt9jSRQWOtJ2j4sThLNJ1GfgcSct8mLAkgH36swHSYPCTFE1dlN4WxhdvxBhgk= ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715 2024-03-28T17:59:47.282 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.343+0000 7f7d6fc00700 1 Processor -- start 2024-03-28T17:59:47.282 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.343+0000 7f7d6fc00700 1 -- start start 2024-03-28T17:59:47.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.343+0000 7f7d6fc00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d68109c80 0x7f7d6810a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:47.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.343+0000 7f7d6fc00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6810a670 con 0x7f7d68109c80 2024-03-28T17:59:47.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.344+0000 7f7d6d200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d68109c80 0x7f7d6810a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:47.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.344+0000 7f7d6d200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d68109c80 0x7f7d6810a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51350/0 (socket says 172.21.15.37:51350) 2024-03-28T17:59:47.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.344+0000 7f7d6d200700 1 -- 172.21.15.37:0/3826098648 learned_addr learned my addr 172.21.15.37:0/3826098648 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:47.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.344+0000 7f7d6d200700 1 -- 172.21.15.37:0/3826098648 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6810ae80 con 0x7f7d68109c80 2024-03-28T17:59:47.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.345+0000 7f7d6d200700 1 --2- 172.21.15.37:0/3826098648 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d68109c80 0x7f7d6810a0a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7d580098a0 tx=0x7f7d58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6511b4a4468fd352 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:47.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.345+0000 7f7d67e00700 1 -- 172.21.15.37:0/3826098648 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d58004160 con 0x7f7d68109c80 2024-03-28T17:59:47.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.345+0000 7f7d67e00700 1 -- 172.21.15.37:0/3826098648 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d58004300 con 0x7f7d68109c80 2024-03-28T17:59:47.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.345+0000 7f7d67e00700 1 -- 172.21.15.37:0/3826098648 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d58011b50 con 0x7f7d68109c80 2024-03-28T17:59:47.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.345+0000 7f7d6fc00700 1 -- 172.21.15.37:0/3826098648 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d68109c80 msgr2=0x7f7d6810a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:47.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.345+0000 7f7d6fc00700 1 --2- 172.21.15.37:0/3826098648 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d68109c80 0x7f7d6810a0a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7d580098a0 tx=0x7f7d58009bb0 comp rx=0 tx=0).stop 2024-03-28T17:59:47.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.346+0000 7f7d6fc00700 1 -- 172.21.15.37:0/3826098648 shutdown_connections 2024-03-28T17:59:47.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.346+0000 7f7d6fc00700 1 --2- 172.21.15.37:0/3826098648 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d68109c80 0x7f7d6810a0a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:47.284 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.346+0000 7f7d6fc00700 1 -- 172.21.15.37:0/3826098648 >> 172.21.15.37:0/3826098648 conn(0x7f7d6807d040 msgr2=0x7f7d6807d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:47.284 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.346+0000 7f7d6fc00700 1 -- 172.21.15.37:0/3826098648 shutdown_connections 2024-03-28T17:59:47.284 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.346+0000 7f7d6fc00700 1 -- 172.21.15.37:0/3826098648 wait complete. 2024-03-28T17:59:47.284 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.347+0000 7f7d6fc00700 1 Processor -- start 2024-03-28T17:59:47.284 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.347+0000 7f7d6fc00700 1 -- start start 2024-03-28T17:59:47.284 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.347+0000 7f7d6fc00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d68109c80 0x7f7d6819dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:47.284 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.347+0000 7f7d6fc00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6810e230 con 0x7f7d68109c80 2024-03-28T17:59:47.284 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.348+0000 7f7d6d200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d68109c80 0x7f7d6819dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:47.284 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.348+0000 7f7d6d200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d68109c80 0x7f7d6819dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51362/0 (socket says 172.21.15.37:51362) 2024-03-28T17:59:47.284 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.348+0000 7f7d6d200700 1 -- 172.21.15.37:0/3446189766 learned_addr learned my addr 172.21.15.37:0/3446189766 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:47.284 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.348+0000 7f7d6d200700 1 -- 172.21.15.37:0/3446189766 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6819e170 con 0x7f7d68109c80 2024-03-28T17:59:47.284 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.348+0000 7f7d6d200700 1 --2- 172.21.15.37:0/3446189766 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d68109c80 0x7f7d6819dc30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7d58000c00 tx=0x7f7d58011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:47.284 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.349+0000 7f7d66000700 1 -- 172.21.15.37:0/3446189766 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d58003b20 con 0x7f7d68109c80 2024-03-28T17:59:47.284 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.349+0000 7f7d66000700 1 -- 172.21.15.37:0/3446189766 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d58003cc0 con 0x7f7d68109c80 2024-03-28T17:59:47.284 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.349+0000 7f7d6fc00700 1 -- 172.21.15.37:0/3446189766 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d6819e400 con 0x7f7d68109c80 2024-03-28T17:59:47.285 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.349+0000 7f7d66000700 1 -- 172.21.15.37:0/3446189766 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d58024910 con 0x7f7d68109c80 2024-03-28T17:59:47.285 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.349+0000 7f7d6fc00700 1 -- 172.21.15.37:0/3446189766 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d681a10f0 con 0x7f7d68109c80 2024-03-28T17:59:47.285 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.350+0000 7f7d66000700 1 -- 172.21.15.37:0/3446189766 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f7d58022020 con 0x7f7d68109c80 2024-03-28T17:59:47.285 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.350+0000 7f7d6fc00700 1 -- 172.21.15.37:0/3446189766 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d680637c0 con 0x7f7d68109c80 2024-03-28T17:59:47.285 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.351+0000 7f7d66000700 1 --2- 172.21.15.37:0/3446189766 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f7d48038f70 0x7f7d4803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:47.285 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.351+0000 7f7d66000700 1 -- 172.21.15.37:0/3446189766 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7d58011e00 con 0x7f7d68109c80 2024-03-28T17:59:47.285 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.351+0000 7f7d5fe00700 1 --2- 172.21.15.37:0/3446189766 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f7d48038f70 0x7f7d4803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:47.285 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.352+0000 7f7d5fe00700 1 --2- 172.21.15.37:0/3446189766 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f7d48038f70 0x7f7d4803b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7d50006fd0 tx=0x7f7d50006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:47.285 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.356+0000 7f7d66000700 1 -- 172.21.15.37:0/3446189766 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7d58016260 con 0x7f7d68109c80 2024-03-28T17:59:47.285 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.583+0000 7f7d6fc00700 1 -- 172.21.15.37:0/3446189766 --> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d68107b90 con 0x7f7d48038f70 2024-03-28T17:59:47.285 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.585+0000 7f7d66000700 1 -- 172.21.15.37:0/3446189766 <== mgr.14120 v2:172.21.15.37:6800/3459498509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f7d68107b90 con 0x7f7d48038f70 2024-03-28T17:59:47.285 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.589+0000 7f7d6fc00700 1 -- 172.21.15.37:0/3446189766 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f7d48038f70 msgr2=0x7f7d4803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:47.285 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.589+0000 7f7d6fc00700 1 --2- 172.21.15.37:0/3446189766 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f7d48038f70 0x7f7d4803b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7d50006fd0 tx=0x7f7d50006e40 comp rx=0 tx=0).stop 2024-03-28T17:59:47.285 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.589+0000 7f7d6fc00700 1 -- 172.21.15.37:0/3446189766 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d68109c80 msgr2=0x7f7d6819dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:47.285 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.589+0000 7f7d6fc00700 1 --2- 172.21.15.37:0/3446189766 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d68109c80 0x7f7d6819dc30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7d58000c00 tx=0x7f7d58011c10 comp rx=0 tx=0).stop 2024-03-28T17:59:47.286 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.589+0000 7f7d6fc00700 1 -- 172.21.15.37:0/3446189766 shutdown_connections 2024-03-28T17:59:47.286 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.589+0000 7f7d6fc00700 1 --2- 172.21.15.37:0/3446189766 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f7d48038f70 0x7f7d4803b430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:47.286 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.589+0000 7f7d6fc00700 1 --2- 172.21.15.37:0/3446189766 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d68109c80 0x7f7d6819dc30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:47.286 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.589+0000 7f7d6fc00700 1 -- 172.21.15.37:0/3446189766 >> 172.21.15.37:0/3446189766 conn(0x7f7d6807d040 msgr2=0x7f7d68107480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:47.286 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.590+0000 7f7d6fc00700 1 -- 172.21.15.37:0/3446189766 shutdown_connections 2024-03-28T17:59:47.286 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:46.590+0000 7f7d6fc00700 1 -- 172.21.15.37:0/3446189766 wait complete. 2024-03-28T17:59:47.286 INFO:teuthology.orchestra.run.smithi037.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-28T17:59:47.287 INFO:teuthology.orchestra.run.smithi037.stdout:Adding key to root@localhost authorized_keys... 2024-03-28T17:59:47.287 INFO:teuthology.orchestra.run.smithi037.stdout:Adding host smithi037... 2024-03-28T17:59:47.556 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:47 smithi037 ceph-mon[31460]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:59:48.435 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:48 smithi037 ceph-mon[31460]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi037", "addr": "172.21.15.37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:59:50.350 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:50 smithi037 ceph-mon[31460]: Deploying cephadm binary to smithi037 2024-03-28T17:59:55.464 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout Added host 'smithi037' with addr '172.21.15.37' 2024-03-28T17:59:55.465 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.861+0000 7f138b200700 1 Processor -- start 2024-03-28T17:59:55.465 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.861+0000 7f138b200700 1 -- start start 2024-03-28T17:59:55.465 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.862+0000 7f138b200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f138c106130 0x7f138c108560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:55.465 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.862+0000 7f138b200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c07be70 con 0x7f138c106130 2024-03-28T17:59:55.465 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.862+0000 7f1389e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f138c106130 0x7f138c108560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:55.465 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.862+0000 7f1389e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f138c106130 0x7f138c108560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51374/0 (socket says 172.21.15.37:51374) 2024-03-28T17:59:55.465 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.862+0000 7f1389e00700 1 -- 172.21.15.37:0/3778073206 learned_addr learned my addr 172.21.15.37:0/3778073206 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:55.465 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.863+0000 7f1389e00700 1 -- 172.21.15.37:0/3778073206 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c108aa0 con 0x7f138c106130 2024-03-28T17:59:55.465 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.863+0000 7f1389e00700 1 --2- 172.21.15.37:0/3778073206 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f138c106130 0x7f138c108560 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f137c0098a0 tx=0x7f137c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85ec8d3dc2a9a131 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:55.465 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.864+0000 7f1388a00700 1 -- 172.21.15.37:0/3778073206 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f137c004160 con 0x7f138c106130 2024-03-28T17:59:55.465 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.864+0000 7f1388a00700 1 -- 172.21.15.37:0/3778073206 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f137c004300 con 0x7f138c106130 2024-03-28T17:59:55.465 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.864+0000 7f1388a00700 1 -- 172.21.15.37:0/3778073206 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f137c011b50 con 0x7f138c106130 2024-03-28T17:59:55.465 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.864+0000 7f138b200700 1 -- 172.21.15.37:0/3778073206 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f138c106130 msgr2=0x7f138c108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:55.466 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.864+0000 7f138b200700 1 --2- 172.21.15.37:0/3778073206 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f138c106130 0x7f138c108560 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f137c0098a0 tx=0x7f137c009bb0 comp rx=0 tx=0).stop 2024-03-28T17:59:55.466 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.865+0000 7f138b200700 1 -- 172.21.15.37:0/3778073206 shutdown_connections 2024-03-28T17:59:55.466 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.865+0000 7f138b200700 1 --2- 172.21.15.37:0/3778073206 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f138c106130 0x7f138c108560 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:55.466 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.865+0000 7f138b200700 1 -- 172.21.15.37:0/3778073206 >> 172.21.15.37:0/3778073206 conn(0x7f138c101dd0 msgr2=0x7f138c1041f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:55.466 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.865+0000 7f138b200700 1 -- 172.21.15.37:0/3778073206 shutdown_connections 2024-03-28T17:59:55.466 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.865+0000 7f138b200700 1 -- 172.21.15.37:0/3778073206 wait complete. 2024-03-28T17:59:55.466 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.866+0000 7f138b200700 1 Processor -- start 2024-03-28T17:59:55.466 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.866+0000 7f138b200700 1 -- start start 2024-03-28T17:59:55.466 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.866+0000 7f138b200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f138c106130 0x7f138c1a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:55.466 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.866+0000 7f138b200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c109b30 con 0x7f138c106130 2024-03-28T17:59:55.466 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.867+0000 7f1389e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f138c106130 0x7f138c1a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:55.466 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.867+0000 7f1389e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f138c106130 0x7f138c1a1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51376/0 (socket says 172.21.15.37:51376) 2024-03-28T17:59:55.466 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.867+0000 7f1389e00700 1 -- 172.21.15.37:0/2809661232 learned_addr learned my addr 172.21.15.37:0/2809661232 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:55.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.867+0000 7f1389e00700 1 -- 172.21.15.37:0/2809661232 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c1a2370 con 0x7f138c106130 2024-03-28T17:59:55.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.867+0000 7f1389e00700 1 --2- 172.21.15.37:0/2809661232 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f138c106130 0x7f138c1a1e30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f137c011b60 tx=0x7f137c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:55.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.868+0000 7f1382a00700 1 -- 172.21.15.37:0/2809661232 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f137c003970 con 0x7f138c106130 2024-03-28T17:59:55.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.868+0000 7f1382a00700 1 -- 172.21.15.37:0/2809661232 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f137c003b10 con 0x7f138c106130 2024-03-28T17:59:55.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.868+0000 7f138b200700 1 -- 172.21.15.37:0/2809661232 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f138c1a2600 con 0x7f138c106130 2024-03-28T17:59:55.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.868+0000 7f1382a00700 1 -- 172.21.15.37:0/2809661232 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f137c024610 con 0x7f138c106130 2024-03-28T17:59:55.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.868+0000 7f138b200700 1 -- 172.21.15.37:0/2809661232 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f138c1a52f0 con 0x7f138c106130 2024-03-28T17:59:55.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.869+0000 7f1382a00700 1 -- 172.21.15.37:0/2809661232 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f137c01a9f0 con 0x7f138c106130 2024-03-28T17:59:55.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.869+0000 7f138b200700 1 -- 172.21.15.37:0/2809661232 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f138c0637c0 con 0x7f138c106130 2024-03-28T17:59:55.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.870+0000 7f1382a00700 1 --2- 172.21.15.37:0/2809661232 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f1370038f70 0x7f137003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:55.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.870+0000 7f1382a00700 1 -- 172.21.15.37:0/2809661232 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f137c011e00 con 0x7f138c106130 2024-03-28T17:59:55.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.870+0000 7f1389400700 1 --2- 172.21.15.37:0/2809661232 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f1370038f70 0x7f137003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:55.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.871+0000 7f1389400700 1 --2- 172.21.15.37:0/2809661232 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f1370038f70 0x7f137003b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1378006fd0 tx=0x7f1378006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:55.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:47.876+0000 7f1382a00700 1 -- 172.21.15.37:0/2809661232 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f137c019e10 con 0x7f138c106130 2024-03-28T17:59:55.467 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:48.107+0000 7f138b200700 1 -- 172.21.15.37:0/2809661232 --> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi037", "addr": "172.21.15.37", "target": ["mon-mgr", ""]}) v1 -- 0x7f138c0008d0 con 0x7f1370038f70 2024-03-28T17:59:55.468 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:54.809+0000 7f1382a00700 1 -- 172.21.15.37:0/2809661232 <== mgr.14120 v2:172.21.15.37:6800/3459498509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f138c0008d0 con 0x7f1370038f70 2024-03-28T17:59:55.468 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:54.814+0000 7f138b200700 1 -- 172.21.15.37:0/2809661232 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f1370038f70 msgr2=0x7f137003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:55.468 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:54.814+0000 7f138b200700 1 --2- 172.21.15.37:0/2809661232 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f1370038f70 0x7f137003b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1378006fd0 tx=0x7f1378006e40 comp rx=0 tx=0).stop 2024-03-28T17:59:55.468 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:54.814+0000 7f138b200700 1 -- 172.21.15.37:0/2809661232 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f138c106130 msgr2=0x7f138c1a1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:55.468 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:54.814+0000 7f138b200700 1 --2- 172.21.15.37:0/2809661232 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f138c106130 0x7f138c1a1e30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f137c011b60 tx=0x7f137c011c40 comp rx=0 tx=0).stop 2024-03-28T17:59:55.468 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:54.814+0000 7f138b200700 1 -- 172.21.15.37:0/2809661232 shutdown_connections 2024-03-28T17:59:55.468 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:54.814+0000 7f138b200700 1 --2- 172.21.15.37:0/2809661232 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f1370038f70 0x7f137003b430 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:55.468 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:54.814+0000 7f138b200700 1 --2- 172.21.15.37:0/2809661232 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f138c106130 0x7f138c1a1e30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:55.468 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:54.814+0000 7f138b200700 1 -- 172.21.15.37:0/2809661232 >> 172.21.15.37:0/2809661232 conn(0x7f138c101dd0 msgr2=0x7f138c1041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:55.468 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:54.814+0000 7f138b200700 1 -- 172.21.15.37:0/2809661232 shutdown_connections 2024-03-28T17:59:55.468 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:54.815+0000 7f138b200700 1 -- 172.21.15.37:0/2809661232 wait complete. 2024-03-28T17:59:55.468 INFO:teuthology.orchestra.run.smithi037.stdout:Deploying unmanaged mon service... 2024-03-28T17:59:56.087 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:55 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' 2024-03-28T17:59:56.087 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:55 smithi037 ceph-mon[31460]: Added host smithi037 2024-03-28T17:59:56.087 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:55 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:59:57.264 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-28T17:59:57.265 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.122+0000 7fd49c400700 1 Processor -- start 2024-03-28T17:59:57.265 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.122+0000 7fd49c400700 1 -- start start 2024-03-28T17:59:57.265 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.122+0000 7fd49c400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd494081ab0 0x7fd494081ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:57.265 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.122+0000 7fd49c400700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4940824a0 con 0x7fd494081ab0 2024-03-28T17:59:57.265 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.123+0000 7fd499a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd494081ab0 0x7fd494081ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:57.265 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.123+0000 7fd499a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd494081ab0 0x7fd494081ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51104/0 (socket says 172.21.15.37:51104) 2024-03-28T17:59:57.265 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.123+0000 7fd499a00700 1 -- 172.21.15.37:0/1569021773 learned_addr learned my addr 172.21.15.37:0/1569021773 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:57.265 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.123+0000 7fd499a00700 1 -- 172.21.15.37:0/1569021773 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49410f280 con 0x7fd494081ab0 2024-03-28T17:59:57.265 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.124+0000 7fd499a00700 1 --2- 172.21.15.37:0/1569021773 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd494081ab0 0x7fd494081ed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd488009e80 tx=0x7fd48800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a92561e905eb430b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:57.265 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.124+0000 7fd493e00700 1 -- 172.21.15.37:0/1569021773 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd488017070 con 0x7fd494081ab0 2024-03-28T17:59:57.265 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.124+0000 7fd493e00700 1 -- 172.21.15.37:0/1569021773 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd48800b8c0 con 0x7fd494081ab0 2024-03-28T17:59:57.265 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.124+0000 7fd493e00700 1 -- 172.21.15.37:0/1569021773 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd4880045c0 con 0x7fd494081ab0 2024-03-28T17:59:57.266 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.125+0000 7fd49c400700 1 -- 172.21.15.37:0/1569021773 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd494081ab0 msgr2=0x7fd494081ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:57.266 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.125+0000 7fd49c400700 1 --2- 172.21.15.37:0/1569021773 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd494081ab0 0x7fd494081ed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd488009e80 tx=0x7fd48800b330 comp rx=0 tx=0).stop 2024-03-28T17:59:57.267 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.125+0000 7fd49c400700 1 -- 172.21.15.37:0/1569021773 shutdown_connections 2024-03-28T17:59:57.267 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.125+0000 7fd49c400700 1 --2- 172.21.15.37:0/1569021773 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd494081ab0 0x7fd494081ed0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:57.267 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.125+0000 7fd49c400700 1 -- 172.21.15.37:0/1569021773 >> 172.21.15.37:0/1569021773 conn(0x7fd49407d090 msgr2=0x7fd49407f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:57.267 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.125+0000 7fd49c400700 1 -- 172.21.15.37:0/1569021773 shutdown_connections 2024-03-28T17:59:57.267 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.125+0000 7fd49c400700 1 -- 172.21.15.37:0/1569021773 wait complete. 2024-03-28T17:59:57.267 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.126+0000 7fd49c400700 1 Processor -- start 2024-03-28T17:59:57.267 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.126+0000 7fd49c400700 1 -- start start 2024-03-28T17:59:57.267 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.127+0000 7fd49c400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd494081ab0 0x7fd4941a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:57.268 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.127+0000 7fd49c400700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd494112630 con 0x7fd494081ab0 2024-03-28T17:59:57.268 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.127+0000 7fd499a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd494081ab0 0x7fd4941a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:57.268 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.127+0000 7fd499a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd494081ab0 0x7fd4941a1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51112/0 (socket says 172.21.15.37:51112) 2024-03-28T17:59:57.269 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.127+0000 7fd499a00700 1 -- 172.21.15.37:0/3785227844 learned_addr learned my addr 172.21.15.37:0/3785227844 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:57.269 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.128+0000 7fd499a00700 1 -- 172.21.15.37:0/3785227844 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4941a24d0 con 0x7fd494081ab0 2024-03-28T17:59:57.269 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.128+0000 7fd499a00700 1 --2- 172.21.15.37:0/3785227844 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd494081ab0 0x7fd4941a1f90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd488009530 tx=0x7fd48800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:57.269 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.128+0000 7fd492000700 1 -- 172.21.15.37:0/3785227844 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd488020020 con 0x7fd494081ab0 2024-03-28T17:59:57.269 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.128+0000 7fd492000700 1 -- 172.21.15.37:0/3785227844 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd488003b50 con 0x7fd494081ab0 2024-03-28T17:59:57.269 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.128+0000 7fd49c400700 1 -- 172.21.15.37:0/3785227844 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4941a2760 con 0x7fd494081ab0 2024-03-28T17:59:57.269 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.128+0000 7fd492000700 1 -- 172.21.15.37:0/3785227844 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd488017040 con 0x7fd494081ab0 2024-03-28T17:59:57.270 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.128+0000 7fd49c400700 1 -- 172.21.15.37:0/3785227844 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4941a5450 con 0x7fd494081ab0 2024-03-28T17:59:57.270 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.130+0000 7fd492000700 1 -- 172.21.15.37:0/3785227844 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fd488023020 con 0x7fd494081ab0 2024-03-28T17:59:57.270 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.130+0000 7fd49c400700 1 -- 172.21.15.37:0/3785227844 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4940637c0 con 0x7fd494081ab0 2024-03-28T17:59:57.270 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.130+0000 7fd492000700 1 --2- 172.21.15.37:0/3785227844 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7fd47c038f20 0x7fd47c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:57.270 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.130+0000 7fd492000700 1 -- 172.21.15.37:0/3785227844 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd48800f050 con 0x7fd494081ab0 2024-03-28T17:59:57.271 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.130+0000 7fd499000700 1 --2- 172.21.15.37:0/3785227844 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7fd47c038f20 0x7fd47c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:57.271 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.131+0000 7fd499000700 1 --2- 172.21.15.37:0/3785227844 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7fd47c038f20 0x7fd47c03b3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd480006fd0 tx=0x7fd480006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:57.271 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.136+0000 7fd492000700 1 -- 172.21.15.37:0/3785227844 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd48802a020 con 0x7fd494081ab0 2024-03-28T17:59:57.271 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.410+0000 7fd49c400700 1 -- 172.21.15.37:0/3785227844 --> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fd49407e200 con 0x7fd47c038f20 2024-03-28T17:59:57.271 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.418+0000 7fd492000700 1 -- 172.21.15.37:0/3785227844 <== mgr.14120 v2:172.21.15.37:6800/3459498509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd49407e200 con 0x7fd47c038f20 2024-03-28T17:59:57.272 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.421+0000 7fd49c400700 1 -- 172.21.15.37:0/3785227844 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7fd47c038f20 msgr2=0x7fd47c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:57.272 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.421+0000 7fd49c400700 1 --2- 172.21.15.37:0/3785227844 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7fd47c038f20 0x7fd47c03b3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd480006fd0 tx=0x7fd480006e40 comp rx=0 tx=0).stop 2024-03-28T17:59:57.272 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.421+0000 7fd49c400700 1 -- 172.21.15.37:0/3785227844 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd494081ab0 msgr2=0x7fd4941a1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:57.272 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.421+0000 7fd49c400700 1 --2- 172.21.15.37:0/3785227844 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd494081ab0 0x7fd4941a1f90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd488009530 tx=0x7fd48800bd50 comp rx=0 tx=0).stop 2024-03-28T17:59:57.273 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.421+0000 7fd49c400700 1 -- 172.21.15.37:0/3785227844 shutdown_connections 2024-03-28T17:59:57.273 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.421+0000 7fd49c400700 1 --2- 172.21.15.37:0/3785227844 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7fd47c038f20 0x7fd47c03b3e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:57.273 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.421+0000 7fd49c400700 1 --2- 172.21.15.37:0/3785227844 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd494081ab0 0x7fd4941a1f90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:57.273 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.421+0000 7fd49c400700 1 -- 172.21.15.37:0/3785227844 >> 172.21.15.37:0/3785227844 conn(0x7fd49407d090 msgr2=0x7fd49407daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:57.274 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.421+0000 7fd49c400700 1 -- 172.21.15.37:0/3785227844 shutdown_connections 2024-03-28T17:59:57.274 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:56.421+0000 7fd49c400700 1 -- 172.21.15.37:0/3785227844 wait complete. 2024-03-28T17:59:57.274 INFO:teuthology.orchestra.run.smithi037.stdout:Deploying unmanaged mgr service... 2024-03-28T17:59:57.512 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:57 smithi037 ceph-mon[31460]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:59:57.512 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:57 smithi037 ceph-mon[31460]: Saving service mon spec with placement count:5 2024-03-28T17:59:57.512 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:57 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' 2024-03-28T17:59:59.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-28T17:59:59.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.415+0000 7f8f5f800700 1 Processor -- start 2024-03-28T17:59:59.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.416+0000 7f8f5f800700 1 -- start start 2024-03-28T17:59:59.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.416+0000 7f8f5f800700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f58107a80 0x7f8f58107ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:59.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.416+0000 7f8f5f800700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f58108470 con 0x7f8f58107a80 2024-03-28T17:59:59.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.416+0000 7f8f5ce00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f58107a80 0x7f8f58107ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:59.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.416+0000 7f8f5ce00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f58107a80 0x7f8f58107ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51120/0 (socket says 172.21.15.37:51120) 2024-03-28T17:59:59.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.416+0000 7f8f5ce00700 1 -- 172.21.15.37:0/3832917286 learned_addr learned my addr 172.21.15.37:0/3832917286 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:59.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.417+0000 7f8f5ce00700 1 -- 172.21.15.37:0/3832917286 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f58108c80 con 0x7f8f58107a80 2024-03-28T17:59:59.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.417+0000 7f8f5ce00700 1 --2- 172.21.15.37:0/3832917286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f58107a80 0x7f8f58107ea0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8f480098a0 tx=0x7f8f48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f516b1cf75906204 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:59.457 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.417+0000 7f8f57e00700 1 -- 172.21.15.37:0/3832917286 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f48004160 con 0x7f8f58107a80 2024-03-28T17:59:59.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.418+0000 7f8f57e00700 1 -- 172.21.15.37:0/3832917286 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f48004300 con 0x7f8f58107a80 2024-03-28T17:59:59.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.418+0000 7f8f57e00700 1 -- 172.21.15.37:0/3832917286 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f48011b50 con 0x7f8f58107a80 2024-03-28T17:59:59.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.418+0000 7f8f5f800700 1 -- 172.21.15.37:0/3832917286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f58107a80 msgr2=0x7f8f58107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:59.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.418+0000 7f8f5f800700 1 --2- 172.21.15.37:0/3832917286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f58107a80 0x7f8f58107ea0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8f480098a0 tx=0x7f8f48009bb0 comp rx=0 tx=0).stop 2024-03-28T17:59:59.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.418+0000 7f8f5f800700 1 -- 172.21.15.37:0/3832917286 shutdown_connections 2024-03-28T17:59:59.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.418+0000 7f8f5f800700 1 --2- 172.21.15.37:0/3832917286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f58107a80 0x7f8f58107ea0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:59.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.418+0000 7f8f5f800700 1 -- 172.21.15.37:0/3832917286 >> 172.21.15.37:0/3832917286 conn(0x7f8f58103080 msgr2=0x7f8f581054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:59.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.418+0000 7f8f5f800700 1 -- 172.21.15.37:0/3832917286 shutdown_connections 2024-03-28T17:59:59.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.418+0000 7f8f5f800700 1 -- 172.21.15.37:0/3832917286 wait complete. 2024-03-28T17:59:59.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.420+0000 7f8f5f800700 1 Processor -- start 2024-03-28T17:59:59.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.420+0000 7f8f5f800700 1 -- start start 2024-03-28T17:59:59.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.420+0000 7f8f5f800700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f58107a80 0x7f8f5807db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:59.458 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.420+0000 7f8f5f800700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5810c030 con 0x7f8f58107a80 2024-03-28T17:59:59.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.420+0000 7f8f5ce00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f58107a80 0x7f8f5807db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:59.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.420+0000 7f8f5ce00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f58107a80 0x7f8f5807db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51126/0 (socket says 172.21.15.37:51126) 2024-03-28T17:59:59.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.420+0000 7f8f5ce00700 1 -- 172.21.15.37:0/2698244085 learned_addr learned my addr 172.21.15.37:0/2698244085 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T17:59:59.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.421+0000 7f8f5ce00700 1 -- 172.21.15.37:0/2698244085 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5807e090 con 0x7f8f58107a80 2024-03-28T17:59:59.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.421+0000 7f8f5ce00700 1 --2- 172.21.15.37:0/2698244085 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f58107a80 0x7f8f5807db50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8f48000c00 tx=0x7f8f48011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:59.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.421+0000 7f8f56000700 1 -- 172.21.15.37:0/2698244085 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f48003b20 con 0x7f8f58107a80 2024-03-28T17:59:59.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.421+0000 7f8f56000700 1 -- 172.21.15.37:0/2698244085 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f48003cc0 con 0x7f8f58107a80 2024-03-28T17:59:59.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.421+0000 7f8f56000700 1 -- 172.21.15.37:0/2698244085 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f48024910 con 0x7f8f58107a80 2024-03-28T17:59:59.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.421+0000 7f8f5f800700 1 -- 172.21.15.37:0/2698244085 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f5807c280 con 0x7f8f58107a80 2024-03-28T17:59:59.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.421+0000 7f8f5f800700 1 -- 172.21.15.37:0/2698244085 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f5807c7c0 con 0x7f8f58107a80 2024-03-28T17:59:59.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.423+0000 7f8f56000700 1 -- 172.21.15.37:0/2698244085 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f8f48022020 con 0x7f8f58107a80 2024-03-28T17:59:59.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.423+0000 7f8f5f800700 1 -- 172.21.15.37:0/2698244085 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f580637c0 con 0x7f8f58107a80 2024-03-28T17:59:59.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.423+0000 7f8f56000700 1 --2- 172.21.15.37:0/2698244085 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f8f38038fd0 0x7f8f3803b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:59.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.423+0000 7f8f56000700 1 -- 172.21.15.37:0/2698244085 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8f48011e00 con 0x7f8f58107a80 2024-03-28T17:59:59.459 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.423+0000 7f8f4fe00700 1 --2- 172.21.15.37:0/2698244085 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f8f38038fd0 0x7f8f3803b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:59.460 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.424+0000 7f8f4fe00700 1 --2- 172.21.15.37:0/2698244085 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f8f38038fd0 0x7f8f3803b490 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8f40006fd0 tx=0x7f8f40006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:59.460 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.429+0000 7f8f56000700 1 -- 172.21.15.37:0/2698244085 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8f48016260 con 0x7f8f58107a80 2024-03-28T17:59:59.460 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.643+0000 7f8f56000700 1 -- 172.21.15.37:0/2698244085 <== mon.0 v2:172.21.15.37:3300/0 7 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f48018530 con 0x7f8f58107a80 2024-03-28T17:59:59.460 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.678+0000 7f8f5f800700 1 -- 172.21.15.37:0/2698244085 --> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f8f581042f0 con 0x7f8f38038fd0 2024-03-28T17:59:59.460 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.688+0000 7f8f56000700 1 -- 172.21.15.37:0/2698244085 <== mgr.14120 v2:172.21.15.37:6800/3459498509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f8f581042f0 con 0x7f8f38038fd0 2024-03-28T17:59:59.460 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.691+0000 7f8f5f800700 1 -- 172.21.15.37:0/2698244085 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f8f38038fd0 msgr2=0x7f8f3803b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:59.460 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.691+0000 7f8f5f800700 1 --2- 172.21.15.37:0/2698244085 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f8f38038fd0 0x7f8f3803b490 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8f40006fd0 tx=0x7f8f40006e40 comp rx=0 tx=0).stop 2024-03-28T17:59:59.460 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.691+0000 7f8f5f800700 1 -- 172.21.15.37:0/2698244085 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f58107a80 msgr2=0x7f8f5807db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:59.460 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.691+0000 7f8f5f800700 1 --2- 172.21.15.37:0/2698244085 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f58107a80 0x7f8f5807db50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8f48000c00 tx=0x7f8f48011c10 comp rx=0 tx=0).stop 2024-03-28T17:59:59.461 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.691+0000 7f8f5f800700 1 -- 172.21.15.37:0/2698244085 shutdown_connections 2024-03-28T17:59:59.461 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.691+0000 7f8f5f800700 1 --2- 172.21.15.37:0/2698244085 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f8f38038fd0 0x7f8f3803b490 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:59.461 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.691+0000 7f8f5f800700 1 --2- 172.21.15.37:0/2698244085 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f58107a80 0x7f8f5807db50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:59.461 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.691+0000 7f8f5f800700 1 -- 172.21.15.37:0/2698244085 >> 172.21.15.37:0/2698244085 conn(0x7f8f58103080 msgr2=0x7f8f58103be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:59.461 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.692+0000 7f8f5f800700 1 -- 172.21.15.37:0/2698244085 shutdown_connections 2024-03-28T17:59:59.461 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T17:59:58.692+0000 7f8f5f800700 1 -- 172.21.15.37:0/2698244085 wait complete. 2024-03-28T17:59:59.754 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:59 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' 2024-03-28T17:59:59.754 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:59 smithi037 ceph-mon[31460]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:59:59.754 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:59 smithi037 ceph-mon[31460]: Saving service mgr spec with placement count:2 2024-03-28T17:59:59.754 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:59 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' 2024-03-28T17:59:59.754 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 17:59:59 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' 2024-03-28T18:00:00.988 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:00 smithi037 ceph-mon[31460]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:00:00.988 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:00 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3473265414' entity='client.admin' 2024-03-28T18:00:01.000 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.052+0000 7f27f3400700 1 Processor -- start 2024-03-28T18:00:01.000 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.052+0000 7f27f3400700 1 -- start start 2024-03-28T18:00:01.000 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.052+0000 7f27f3400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27ec107a80 0x7f27ec107ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:01.000 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.052+0000 7f27f3400700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec108470 con 0x7f27ec107a80 2024-03-28T18:00:01.000 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.053+0000 7f27f0a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27ec107a80 0x7f27ec107ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:01.001 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.053+0000 7f27f0a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27ec107a80 0x7f27ec107ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51134/0 (socket says 172.21.15.37:51134) 2024-03-28T18:00:01.001 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.053+0000 7f27f0a00700 1 -- 172.21.15.37:0/505524641 learned_addr learned my addr 172.21.15.37:0/505524641 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:01.001 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.053+0000 7f27f0a00700 1 -- 172.21.15.37:0/505524641 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ec108c80 con 0x7f27ec107a80 2024-03-28T18:00:01.001 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.053+0000 7f27f0a00700 1 --2- 172.21.15.37:0/505524641 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27ec107a80 0x7f27ec107ea0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f27e00098a0 tx=0x7f27e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a2d83aabb89a178 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:01.001 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.054+0000 7f27eb400700 1 -- 172.21.15.37:0/505524641 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27e0004160 con 0x7f27ec107a80 2024-03-28T18:00:01.001 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.054+0000 7f27eb400700 1 -- 172.21.15.37:0/505524641 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f27e0004300 con 0x7f27ec107a80 2024-03-28T18:00:01.001 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.054+0000 7f27eb400700 1 -- 172.21.15.37:0/505524641 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27e0011b20 con 0x7f27ec107a80 2024-03-28T18:00:01.001 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.055+0000 7f27f3400700 1 -- 172.21.15.37:0/505524641 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27ec107a80 msgr2=0x7f27ec107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:01.001 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.055+0000 7f27f3400700 1 --2- 172.21.15.37:0/505524641 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27ec107a80 0x7f27ec107ea0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f27e00098a0 tx=0x7f27e0009bb0 comp rx=0 tx=0).stop 2024-03-28T18:00:01.001 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.055+0000 7f27f3400700 1 -- 172.21.15.37:0/505524641 shutdown_connections 2024-03-28T18:00:01.001 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.055+0000 7f27f3400700 1 --2- 172.21.15.37:0/505524641 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27ec107a80 0x7f27ec107ea0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:01.001 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.055+0000 7f27f3400700 1 -- 172.21.15.37:0/505524641 >> 172.21.15.37:0/505524641 conn(0x7f27ec103080 msgr2=0x7f27ec1054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:01.001 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.055+0000 7f27f3400700 1 -- 172.21.15.37:0/505524641 shutdown_connections 2024-03-28T18:00:01.002 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.055+0000 7f27f3400700 1 -- 172.21.15.37:0/505524641 wait complete. 2024-03-28T18:00:01.002 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.056+0000 7f27f3400700 1 Processor -- start 2024-03-28T18:00:01.002 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.056+0000 7f27f3400700 1 -- start start 2024-03-28T18:00:01.002 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.057+0000 7f27f3400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27ec107a80 0x7f27ec19b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:01.002 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.057+0000 7f27f3400700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec10c030 con 0x7f27ec107a80 2024-03-28T18:00:01.002 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.057+0000 7f27f0a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27ec107a80 0x7f27ec19b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:01.002 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.057+0000 7f27f0a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27ec107a80 0x7f27ec19b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51148/0 (socket says 172.21.15.37:51148) 2024-03-28T18:00:01.002 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.057+0000 7f27f0a00700 1 -- 172.21.15.37:0/3473265414 learned_addr learned my addr 172.21.15.37:0/3473265414 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:01.002 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.057+0000 7f27f0a00700 1 -- 172.21.15.37:0/3473265414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ec19bef0 con 0x7f27ec107a80 2024-03-28T18:00:01.002 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.058+0000 7f27f0a00700 1 --2- 172.21.15.37:0/3473265414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27ec107a80 0x7f27ec19b9b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f27e00117e0 tx=0x7f27e0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:01.002 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.058+0000 7f27e9600700 1 -- 172.21.15.37:0/3473265414 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27e0003bf0 con 0x7f27ec107a80 2024-03-28T18:00:01.002 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.058+0000 7f27e9600700 1 -- 172.21.15.37:0/3473265414 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f27e001a910 con 0x7f27ec107a80 2024-03-28T18:00:01.002 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.058+0000 7f27e9600700 1 -- 172.21.15.37:0/3473265414 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27e0024970 con 0x7f27ec107a80 2024-03-28T18:00:01.002 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.058+0000 7f27f3400700 1 -- 172.21.15.37:0/3473265414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27ec19c180 con 0x7f27ec107a80 2024-03-28T18:00:01.003 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.058+0000 7f27f3400700 1 -- 172.21.15.37:0/3473265414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27ec19c5e0 con 0x7f27ec107a80 2024-03-28T18:00:01.003 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.059+0000 7f27f3400700 1 -- 172.21.15.37:0/3473265414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27ec0637c0 con 0x7f27ec107a80 2024-03-28T18:00:01.003 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.059+0000 7f27e9600700 1 -- 172.21.15.37:0/3473265414 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f27e0022020 con 0x7f27ec107a80 2024-03-28T18:00:01.003 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.060+0000 7f27e9600700 1 --2- 172.21.15.37:0/3473265414 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f27d4038f70 0x7f27d403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:01.003 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.060+0000 7f27e9600700 1 -- 172.21.15.37:0/3473265414 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f27e004d2b0 con 0x7f27ec107a80 2024-03-28T18:00:01.003 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.060+0000 7f27ebe00700 1 --2- 172.21.15.37:0/3473265414 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f27d4038f70 0x7f27d403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:01.003 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.061+0000 7f27ebe00700 1 --2- 172.21.15.37:0/3473265414 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f27d4038f70 0x7f27d403b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f27d8006fd0 tx=0x7f27d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:01.003 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.069+0000 7f27e9600700 1 -- 172.21.15.37:0/3473265414 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f27e004cc00 con 0x7f27ec107a80 2024-03-28T18:00:01.003 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.294+0000 7f27f3400700 1 -- 172.21.15.37:0/3473265414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f27ec19ee70 con 0x7f27ec107a80 2024-03-28T18:00:01.003 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.298+0000 7f27e9600700 1 -- 172.21.15.37:0/3473265414 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f27e001ab70 con 0x7f27ec107a80 2024-03-28T18:00:01.003 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.301+0000 7f27f3400700 1 -- 172.21.15.37:0/3473265414 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f27d4038f70 msgr2=0x7f27d403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:01.003 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.301+0000 7f27f3400700 1 --2- 172.21.15.37:0/3473265414 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f27d4038f70 0x7f27d403b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f27d8006fd0 tx=0x7f27d8006e40 comp rx=0 tx=0).stop 2024-03-28T18:00:01.003 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.301+0000 7f27f3400700 1 -- 172.21.15.37:0/3473265414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27ec107a80 msgr2=0x7f27ec19b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:01.003 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.301+0000 7f27f3400700 1 --2- 172.21.15.37:0/3473265414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27ec107a80 0x7f27ec19b9b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f27e00117e0 tx=0x7f27e0011ba0 comp rx=0 tx=0).stop 2024-03-28T18:00:01.004 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.301+0000 7f27f3400700 1 -- 172.21.15.37:0/3473265414 shutdown_connections 2024-03-28T18:00:01.004 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.301+0000 7f27f3400700 1 --2- 172.21.15.37:0/3473265414 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f27d4038f70 0x7f27d403b430 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:01.004 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.301+0000 7f27f3400700 1 --2- 172.21.15.37:0/3473265414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27ec107a80 0x7f27ec19b9b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:01.004 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.301+0000 7f27f3400700 1 -- 172.21.15.37:0/3473265414 >> 172.21.15.37:0/3473265414 conn(0x7f27ec103080 msgr2=0x7f27ec103af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:01.004 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.302+0000 7f27f3400700 1 -- 172.21.15.37:0/3473265414 shutdown_connections 2024-03-28T18:00:01.004 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:00.302+0000 7f27f3400700 1 -- 172.21.15.37:0/3473265414 wait complete. 2024-03-28T18:00:02.836 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:02 smithi037 ceph-mon[31460]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:00:02.836 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:02 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' 2024-03-28T18:00:02.836 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:02 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1097624118' entity='client.admin' 2024-03-28T18:00:03.154 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.229+0000 7feefd200700 1 Processor -- start 2024-03-28T18:00:03.154 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.229+0000 7feefd200700 1 -- start start 2024-03-28T18:00:03.154 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.229+0000 7feefd200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feef807de40 0x7feef807c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:03.154 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.229+0000 7feefd200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef807c870 con 0x7feef807de40 2024-03-28T18:00:03.155 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.230+0000 7feef5e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feef807de40 0x7feef807c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:03.156 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.230+0000 7feef5e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feef807de40 0x7feef807c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51160/0 (socket says 172.21.15.37:51160) 2024-03-28T18:00:03.156 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.230+0000 7feef5e00700 1 -- 172.21.15.37:0/2438500557 learned_addr learned my addr 172.21.15.37:0/2438500557 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:03.156 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.231+0000 7feef5e00700 1 -- 172.21.15.37:0/2438500557 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef807c9f0 con 0x7feef807de40 2024-03-28T18:00:03.156 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.231+0000 7feef5e00700 1 --2- 172.21.15.37:0/2438500557 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feef807de40 0x7feef807c2a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7feee80098a0 tx=0x7feee8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db2e36dc8172228d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:03.156 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.231+0000 7feef4a00700 1 -- 172.21.15.37:0/2438500557 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feee8004160 con 0x7feef807de40 2024-03-28T18:00:03.156 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.231+0000 7feef4a00700 1 -- 172.21.15.37:0/2438500557 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feee8004300 con 0x7feef807de40 2024-03-28T18:00:03.156 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.232+0000 7feef4a00700 1 -- 172.21.15.37:0/2438500557 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feee8011b20 con 0x7feef807de40 2024-03-28T18:00:03.157 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.232+0000 7feefd200700 1 -- 172.21.15.37:0/2438500557 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feef807de40 msgr2=0x7feef807c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:03.157 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.232+0000 7feefd200700 1 --2- 172.21.15.37:0/2438500557 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feef807de40 0x7feef807c2a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7feee80098a0 tx=0x7feee8009bb0 comp rx=0 tx=0).stop 2024-03-28T18:00:03.157 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.232+0000 7feefd200700 1 -- 172.21.15.37:0/2438500557 shutdown_connections 2024-03-28T18:00:03.161 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.232+0000 7feefd200700 1 --2- 172.21.15.37:0/2438500557 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feef807de40 0x7feef807c2a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:03.161 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.232+0000 7feefd200700 1 -- 172.21.15.37:0/2438500557 >> 172.21.15.37:0/2438500557 conn(0x7feef8103080 msgr2=0x7feef81054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:03.161 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.232+0000 7feefd200700 1 -- 172.21.15.37:0/2438500557 shutdown_connections 2024-03-28T18:00:03.161 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.232+0000 7feefd200700 1 -- 172.21.15.37:0/2438500557 wait complete. 2024-03-28T18:00:03.161 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.233+0000 7feefd200700 1 Processor -- start 2024-03-28T18:00:03.161 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.234+0000 7feefd200700 1 -- start start 2024-03-28T18:00:03.162 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.234+0000 7feefd200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feef807de40 0x7feef819dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:03.162 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.234+0000 7feefd200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef810e0c0 con 0x7feef807de40 2024-03-28T18:00:03.162 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.234+0000 7feef5e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feef807de40 0x7feef819dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:03.162 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.234+0000 7feef5e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feef807de40 0x7feef819dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51162/0 (socket says 172.21.15.37:51162) 2024-03-28T18:00:03.162 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.234+0000 7feef5e00700 1 -- 172.21.15.37:0/1097624118 learned_addr learned my addr 172.21.15.37:0/1097624118 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:03.162 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.235+0000 7feef5e00700 1 -- 172.21.15.37:0/1097624118 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef819e0f0 con 0x7feef807de40 2024-03-28T18:00:03.162 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.235+0000 7feef5e00700 1 --2- 172.21.15.37:0/1097624118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feef807de40 0x7feef819dbb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feee80117e0 tx=0x7feee8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:03.162 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.235+0000 7feeeea00700 1 -- 172.21.15.37:0/1097624118 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feee8003ba0 con 0x7feef807de40 2024-03-28T18:00:03.162 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.235+0000 7feeeea00700 1 -- 172.21.15.37:0/1097624118 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feee801a910 con 0x7feef807de40 2024-03-28T18:00:03.162 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.235+0000 7feeeea00700 1 -- 172.21.15.37:0/1097624118 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feee80248f0 con 0x7feef807de40 2024-03-28T18:00:03.162 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.236+0000 7feefd200700 1 -- 172.21.15.37:0/1097624118 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef819e380 con 0x7feef807de40 2024-03-28T18:00:03.162 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.236+0000 7feefd200700 1 -- 172.21.15.37:0/1097624118 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef819e7e0 con 0x7feef807de40 2024-03-28T18:00:03.163 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.237+0000 7feefd200700 1 -- 172.21.15.37:0/1097624118 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef80637c0 con 0x7feef807de40 2024-03-28T18:00:03.163 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.237+0000 7feeeea00700 1 -- 172.21.15.37:0/1097624118 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7feee8022020 con 0x7feef807de40 2024-03-28T18:00:03.163 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.237+0000 7feeeea00700 1 --2- 172.21.15.37:0/1097624118 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7feedc038f70 0x7feedc03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:03.163 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.237+0000 7feeeea00700 1 -- 172.21.15.37:0/1097624118 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7feee804d2f0 con 0x7feef807de40 2024-03-28T18:00:03.163 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.238+0000 7feef5400700 1 --2- 172.21.15.37:0/1097624118 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7feedc038f70 0x7feedc03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:03.163 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.238+0000 7feef5400700 1 --2- 172.21.15.37:0/1097624118 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7feedc038f70 0x7feedc03b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7feee4006fd0 tx=0x7feee4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:03.163 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.243+0000 7feeeea00700 1 -- 172.21.15.37:0/1097624118 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feee801ab70 con 0x7feef807de40 2024-03-28T18:00:03.163 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.472+0000 7feefd200700 1 -- 172.21.15.37:0/1097624118 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7feef81a1270 con 0x7feef807de40 2024-03-28T18:00:03.163 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.478+0000 7feeeea00700 1 -- 172.21.15.37:0/1097624118 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7feee804cc80 con 0x7feef807de40 2024-03-28T18:00:03.163 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.481+0000 7feefd200700 1 -- 172.21.15.37:0/1097624118 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7feedc038f70 msgr2=0x7feedc03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:03.163 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.481+0000 7feefd200700 1 --2- 172.21.15.37:0/1097624118 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7feedc038f70 0x7feedc03b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7feee4006fd0 tx=0x7feee4006e40 comp rx=0 tx=0).stop 2024-03-28T18:00:03.163 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.481+0000 7feefd200700 1 -- 172.21.15.37:0/1097624118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feef807de40 msgr2=0x7feef819dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:03.163 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.481+0000 7feefd200700 1 --2- 172.21.15.37:0/1097624118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feef807de40 0x7feef819dbb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feee80117e0 tx=0x7feee8011ba0 comp rx=0 tx=0).stop 2024-03-28T18:00:03.163 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.482+0000 7feefd200700 1 -- 172.21.15.37:0/1097624118 shutdown_connections 2024-03-28T18:00:03.164 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.482+0000 7feefd200700 1 --2- 172.21.15.37:0/1097624118 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7feedc038f70 0x7feedc03b430 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:03.164 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.482+0000 7feefd200700 1 --2- 172.21.15.37:0/1097624118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feef807de40 0x7feef819dbb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:03.164 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.482+0000 7feefd200700 1 -- 172.21.15.37:0/1097624118 >> 172.21.15.37:0/1097624118 conn(0x7feef8103080 msgr2=0x7feef8103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:03.164 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.482+0000 7feefd200700 1 -- 172.21.15.37:0/1097624118 shutdown_connections 2024-03-28T18:00:03.164 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:02.482+0000 7feefd200700 1 -- 172.21.15.37:0/1097624118 wait complete. 2024-03-28T18:00:03.165 INFO:teuthology.orchestra.run.smithi037.stdout:Enabling the dashboard module... 2024-03-28T18:00:05.065 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:04 smithi037 ceph-mon[31460]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:00:05.065 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:04 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3031410681' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-28T18:00:05.065 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:04 smithi037 ceph-mon[31460]: from='mgr.14120 172.21.15.37:0/1746232664' entity='mgr.x' 2024-03-28T18:00:05.065 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:04 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: ignoring --setuser ceph since I am not root 2024-03-28T18:00:05.065 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:04 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: ignoring --setgroup ceph since I am not root 2024-03-28T18:00:05.409 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.806+0000 7f6466a00700 1 Processor -- start 2024-03-28T18:00:05.409 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.807+0000 7f6466a00700 1 -- start start 2024-03-28T18:00:05.409 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.807+0000 7f6466a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f646007de60 0x7f646007c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:05.409 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.807+0000 7f6466a00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646007c890 con 0x7f646007de60 2024-03-28T18:00:05.409 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.808+0000 7f645fe00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f646007de60 0x7f646007c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:05.409 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.808+0000 7f645fe00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f646007de60 0x7f646007c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:42832/0 (socket says 172.21.15.37:42832) 2024-03-28T18:00:05.409 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.808+0000 7f645fe00700 1 -- 172.21.15.37:0/2587786879 learned_addr learned my addr 172.21.15.37:0/2587786879 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:05.409 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.808+0000 7f645fe00700 1 -- 172.21.15.37:0/2587786879 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646007ca10 con 0x7f646007de60 2024-03-28T18:00:05.409 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.808+0000 7f645fe00700 1 --2- 172.21.15.37:0/2587786879 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f646007de60 0x7f646007c2c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f64500098a0 tx=0x7f6450009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c7623faee919f77 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:05.409 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.809+0000 7f645ea00700 1 -- 172.21.15.37:0/2587786879 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6450004160 con 0x7f646007de60 2024-03-28T18:00:05.409 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.809+0000 7f645ea00700 1 -- 172.21.15.37:0/2587786879 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6450004300 con 0x7f646007de60 2024-03-28T18:00:05.409 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.809+0000 7f645ea00700 1 -- 172.21.15.37:0/2587786879 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6450011b20 con 0x7f646007de60 2024-03-28T18:00:05.410 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.809+0000 7f6466a00700 1 -- 172.21.15.37:0/2587786879 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f646007de60 msgr2=0x7f646007c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:05.410 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.809+0000 7f6466a00700 1 --2- 172.21.15.37:0/2587786879 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f646007de60 0x7f646007c2c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f64500098a0 tx=0x7f6450009bb0 comp rx=0 tx=0).stop 2024-03-28T18:00:05.410 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.810+0000 7f6466a00700 1 -- 172.21.15.37:0/2587786879 shutdown_connections 2024-03-28T18:00:05.410 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.810+0000 7f6466a00700 1 --2- 172.21.15.37:0/2587786879 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f646007de60 0x7f646007c2c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:05.410 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.810+0000 7f6466a00700 1 -- 172.21.15.37:0/2587786879 >> 172.21.15.37:0/2587786879 conn(0x7f6460103080 msgr2=0x7f64601054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:05.410 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.810+0000 7f6466a00700 1 -- 172.21.15.37:0/2587786879 shutdown_connections 2024-03-28T18:00:05.410 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.810+0000 7f6466a00700 1 -- 172.21.15.37:0/2587786879 wait complete. 2024-03-28T18:00:05.410 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.811+0000 7f6466a00700 1 Processor -- start 2024-03-28T18:00:05.410 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.812+0000 7f6466a00700 1 -- start start 2024-03-28T18:00:05.410 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.812+0000 7f6466a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f646007de60 0x7f646019dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:05.410 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.812+0000 7f6466a00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646010e100 con 0x7f646007de60 2024-03-28T18:00:05.410 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.812+0000 7f645fe00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f646007de60 0x7f646019dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:05.410 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.812+0000 7f645fe00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f646007de60 0x7f646019dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:42842/0 (socket says 172.21.15.37:42842) 2024-03-28T18:00:05.411 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.812+0000 7f645fe00700 1 -- 172.21.15.37:0/3031410681 learned_addr learned my addr 172.21.15.37:0/3031410681 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:05.411 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.813+0000 7f645fe00700 1 -- 172.21.15.37:0/3031410681 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646019e0e0 con 0x7f646007de60 2024-03-28T18:00:05.411 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.813+0000 7f645fe00700 1 --2- 172.21.15.37:0/3031410681 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f646007de60 0x7f646019dba0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f64500117e0 tx=0x7f6450011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:05.411 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.813+0000 7f645cc00700 1 -- 172.21.15.37:0/3031410681 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6450003bf0 con 0x7f646007de60 2024-03-28T18:00:05.411 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.813+0000 7f645cc00700 1 -- 172.21.15.37:0/3031410681 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f645001a910 con 0x7f646007de60 2024-03-28T18:00:05.411 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.813+0000 7f645cc00700 1 -- 172.21.15.37:0/3031410681 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6450024970 con 0x7f646007de60 2024-03-28T18:00:05.411 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.813+0000 7f6466a00700 1 -- 172.21.15.37:0/3031410681 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f646019e370 con 0x7f646007de60 2024-03-28T18:00:05.411 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.814+0000 7f6466a00700 1 -- 172.21.15.37:0/3031410681 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f646019e7d0 con 0x7f646007de60 2024-03-28T18:00:05.411 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.815+0000 7f645cc00700 1 -- 172.21.15.37:0/3031410681 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f6450022020 con 0x7f646007de60 2024-03-28T18:00:05.411 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.815+0000 7f6466a00700 1 -- 172.21.15.37:0/3031410681 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64600637c0 con 0x7f646007de60 2024-03-28T18:00:05.411 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.815+0000 7f645cc00700 1 --2- 172.21.15.37:0/3031410681 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f6444038f20 0x7f644403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:05.411 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.815+0000 7f645cc00700 1 -- 172.21.15.37:0/3031410681 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f645004d220 con 0x7f646007de60 2024-03-28T18:00:05.411 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.815+0000 7f645f400700 1 --2- 172.21.15.37:0/3031410681 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f6444038f20 0x7f644403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:05.411 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.816+0000 7f645f400700 1 --2- 172.21.15.37:0/3031410681 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f6444038f20 0x7f644403b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f644c006fd0 tx=0x7f644c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:05.412 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:03.821+0000 7f645cc00700 1 -- 172.21.15.37:0/3031410681 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6450011e20 con 0x7f646007de60 2024-03-28T18:00:05.412 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:04.113+0000 7f6466a00700 1 -- 172.21.15.37:0/3031410681 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f64601a11a0 con 0x7f646007de60 2024-03-28T18:00:05.412 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:04.653+0000 7f645cc00700 1 -- 172.21.15.37:0/3031410681 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f645000b3e0 con 0x7f646007de60 2024-03-28T18:00:05.412 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:04.655+0000 7f645cc00700 1 -- 172.21.15.37:0/3031410681 <== mon.0 v2:172.21.15.37:3300/0 8 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f645001a430 con 0x7f646007de60 2024-03-28T18:00:05.412 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:04.658+0000 7f6466a00700 1 -- 172.21.15.37:0/3031410681 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f6444038f20 msgr2=0x7f644403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:05.412 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:04.658+0000 7f6466a00700 1 --2- 172.21.15.37:0/3031410681 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f6444038f20 0x7f644403b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f644c006fd0 tx=0x7f644c006e40 comp rx=0 tx=0).stop 2024-03-28T18:00:05.412 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:04.658+0000 7f6466a00700 1 -- 172.21.15.37:0/3031410681 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f646007de60 msgr2=0x7f646019dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:05.412 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:04.658+0000 7f6466a00700 1 --2- 172.21.15.37:0/3031410681 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f646007de60 0x7f646019dba0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f64500117e0 tx=0x7f6450011c10 comp rx=0 tx=0).stop 2024-03-28T18:00:05.412 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:04.658+0000 7f6466a00700 1 -- 172.21.15.37:0/3031410681 shutdown_connections 2024-03-28T18:00:05.412 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:04.658+0000 7f6466a00700 1 --2- 172.21.15.37:0/3031410681 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f6444038f20 0x7f644403b3e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:05.412 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:04.658+0000 7f6466a00700 1 --2- 172.21.15.37:0/3031410681 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f646007de60 0x7f646019dba0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:05.412 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:04.658+0000 7f6466a00700 1 -- 172.21.15.37:0/3031410681 >> 172.21.15.37:0/3031410681 conn(0x7f6460103080 msgr2=0x7f6460103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:05.412 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:04.659+0000 7f6466a00700 1 -- 172.21.15.37:0/3031410681 shutdown_connections 2024-03-28T18:00:05.412 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:04.659+0000 7f6466a00700 1 -- 172.21.15.37:0/3031410681 wait complete. 2024-03-28T18:00:05.656 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:05 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:05.487+0000 7f90621c0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:00:05.656 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:05 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:05.611+0000 7f90621c0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:00:05.939 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:05 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3031410681' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-28T18:00:05.939 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:05 smithi037 ceph-mon[31460]: mgrmap e9: x(active, since 25s) 2024-03-28T18:00:06.577 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:06 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:06.446+0000 7f90621c0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:00:06.577 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:06 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:06.575+0000 7f90621c0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:00:06.874 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:06 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:06.779+0000 7f90621c0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:00:06.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:06 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/770453921' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T18:00:07.024 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout { 2024-03-28T18:00:07.031 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-28T18:00:07.031 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T18:00:07.031 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T18:00:07.031 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T18:00:07.031 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout } 2024-03-28T18:00:07.031 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.004+0000 7f5bece00700 1 Processor -- start 2024-03-28T18:00:07.031 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.004+0000 7f5bece00700 1 -- start start 2024-03-28T18:00:07.031 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.005+0000 7f5bece00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5be81084e0 0x7f5be810a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.031 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.005+0000 7f5bece00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be807be90 con 0x7f5be81084e0 2024-03-28T18:00:07.031 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.005+0000 7f5be5e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5be81084e0 0x7f5be810a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:07.031 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.005+0000 7f5be5e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5be81084e0 0x7f5be810a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:42868/0 (socket says 172.21.15.37:42868) 2024-03-28T18:00:07.032 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.005+0000 7f5be5e00700 1 -- 172.21.15.37:0/2853463572 learned_addr learned my addr 172.21.15.37:0/2853463572 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:07.032 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.006+0000 7f5be5e00700 1 -- 172.21.15.37:0/2853463572 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be810ae10 con 0x7f5be81084e0 2024-03-28T18:00:07.032 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.006+0000 7f5be5e00700 1 --2- 172.21.15.37:0/2853463572 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5be81084e0 0x7f5be810a8d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80098a0 tx=0x7f5bd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb65a7cf5044cff5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:07.032 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.006+0000 7f5be4a00700 1 -- 172.21.15.37:0/2853463572 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bd8004160 con 0x7f5be81084e0 2024-03-28T18:00:07.032 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.006+0000 7f5be4a00700 1 -- 172.21.15.37:0/2853463572 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5bd8004300 con 0x7f5be81084e0 2024-03-28T18:00:07.032 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.006+0000 7f5be4a00700 1 -- 172.21.15.37:0/2853463572 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bd8011b20 con 0x7f5be81084e0 2024-03-28T18:00:07.032 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.007+0000 7f5bece00700 1 -- 172.21.15.37:0/2853463572 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5be81084e0 msgr2=0x7f5be810a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:07.032 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.007+0000 7f5bece00700 1 --2- 172.21.15.37:0/2853463572 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5be81084e0 0x7f5be810a8d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80098a0 tx=0x7f5bd8009bb0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.032 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.007+0000 7f5bece00700 1 -- 172.21.15.37:0/2853463572 shutdown_connections 2024-03-28T18:00:07.032 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.007+0000 7f5bece00700 1 --2- 172.21.15.37:0/2853463572 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5be81084e0 0x7f5be810a8d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.032 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.007+0000 7f5bece00700 1 -- 172.21.15.37:0/2853463572 >> 172.21.15.37:0/2853463572 conn(0x7f5be8101f70 msgr2=0x7f5be8104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:07.032 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.007+0000 7f5bece00700 1 -- 172.21.15.37:0/2853463572 shutdown_connections 2024-03-28T18:00:07.032 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.007+0000 7f5bece00700 1 -- 172.21.15.37:0/2853463572 wait complete. 2024-03-28T18:00:07.032 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.008+0000 7f5bece00700 1 Processor -- start 2024-03-28T18:00:07.032 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.008+0000 7f5bece00700 1 -- start start 2024-03-28T18:00:07.033 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.009+0000 7f5bece00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5be81084e0 0x7f5be819db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.033 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.009+0000 7f5bece00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be810be80 con 0x7f5be81084e0 2024-03-28T18:00:07.033 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.009+0000 7f5be5e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5be81084e0 0x7f5be819db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:07.033 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.009+0000 7f5be5e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5be81084e0 0x7f5be819db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:42870/0 (socket says 172.21.15.37:42870) 2024-03-28T18:00:07.033 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.009+0000 7f5be5e00700 1 -- 172.21.15.37:0/770453921 learned_addr learned my addr 172.21.15.37:0/770453921 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:07.033 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.009+0000 7f5be5e00700 1 -- 172.21.15.37:0/770453921 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be819e0b0 con 0x7f5be81084e0 2024-03-28T18:00:07.033 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.010+0000 7f5be5e00700 1 --2- 172.21.15.37:0/770453921 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5be81084e0 0x7f5be819db70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80117e0 tx=0x7f5bd8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:07.033 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.010+0000 7f5bdea00700 1 -- 172.21.15.37:0/770453921 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bd8003bf0 con 0x7f5be81084e0 2024-03-28T18:00:07.033 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.010+0000 7f5bdea00700 1 -- 172.21.15.37:0/770453921 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5bd801a910 con 0x7f5be81084e0 2024-03-28T18:00:07.033 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.010+0000 7f5bece00700 1 -- 172.21.15.37:0/770453921 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be819e340 con 0x7f5be81084e0 2024-03-28T18:00:07.033 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.010+0000 7f5bdea00700 1 -- 172.21.15.37:0/770453921 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bd8024970 con 0x7f5be81084e0 2024-03-28T18:00:07.033 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.010+0000 7f5bece00700 1 -- 172.21.15.37:0/770453921 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be819e7a0 con 0x7f5be81084e0 2024-03-28T18:00:07.033 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.011+0000 7f5bdea00700 1 -- 172.21.15.37:0/770453921 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f5bd8022020 con 0x7f5be81084e0 2024-03-28T18:00:07.033 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.011+0000 7f5bece00700 1 -- 172.21.15.37:0/770453921 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be80637c0 con 0x7f5be81084e0 2024-03-28T18:00:07.033 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.012+0000 7f5bdea00700 1 --2- 172.21.15.37:0/770453921 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5bcc038f80 0x7f5bcc03b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.034 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.012+0000 7f5bdea00700 1 -- 172.21.15.37:0/770453921 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5bd804d280 con 0x7f5be81084e0 2024-03-28T18:00:07.034 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.012+0000 7f5be5400700 1 -- 172.21.15.37:0/770453921 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5bcc038f80 msgr2=0x7f5bcc03b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/3459498509 2024-03-28T18:00:07.034 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.012+0000 7f5be5400700 1 --2- 172.21.15.37:0/770453921 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5bcc038f80 0x7f5bcc03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:00:07.034 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.021+0000 7f5bdea00700 1 -- 172.21.15.37:0/770453921 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5bd8011e20 con 0x7f5be81084e0 2024-03-28T18:00:07.034 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.213+0000 7f5be5400700 1 -- 172.21.15.37:0/770453921 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5bcc038f80 msgr2=0x7f5bcc03b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/3459498509 2024-03-28T18:00:07.034 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.213+0000 7f5be5400700 1 --2- 172.21.15.37:0/770453921 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5bcc038f80 0x7f5bcc03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T18:00:07.034 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.304+0000 7f5bece00700 1 -- 172.21.15.37:0/770453921 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f5be81a1150 con 0x7f5be81084e0 2024-03-28T18:00:07.034 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.305+0000 7f5bdea00700 1 -- 172.21.15.37:0/770453921 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f5bd800b3e0 con 0x7f5be81084e0 2024-03-28T18:00:07.034 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.308+0000 7f5bece00700 1 -- 172.21.15.37:0/770453921 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5bcc038f80 msgr2=0x7f5bcc03b440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:00:07.034 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.308+0000 7f5bece00700 1 --2- 172.21.15.37:0/770453921 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5bcc038f80 0x7f5bcc03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.034 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.308+0000 7f5bece00700 1 -- 172.21.15.37:0/770453921 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5be81084e0 msgr2=0x7f5be819db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:07.034 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.308+0000 7f5bece00700 1 --2- 172.21.15.37:0/770453921 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5be81084e0 0x7f5be819db70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80117e0 tx=0x7f5bd8011c10 comp rx=0 tx=0).stop 2024-03-28T18:00:07.034 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.308+0000 7f5bece00700 1 -- 172.21.15.37:0/770453921 shutdown_connections 2024-03-28T18:00:07.035 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.308+0000 7f5bece00700 1 --2- 172.21.15.37:0/770453921 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5bcc038f80 0x7f5bcc03b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.035 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.308+0000 7f5bece00700 1 --2- 172.21.15.37:0/770453921 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5be81084e0 0x7f5be819db70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.035 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.308+0000 7f5bece00700 1 -- 172.21.15.37:0/770453921 >> 172.21.15.37:0/770453921 conn(0x7f5be8101f70 msgr2=0x7f5be8102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:07.035 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.308+0000 7f5bece00700 1 -- 172.21.15.37:0/770453921 shutdown_connections 2024-03-28T18:00:07.035 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:06.308+0000 7f5bece00700 1 -- 172.21.15.37:0/770453921 wait complete. 2024-03-28T18:00:07.035 INFO:teuthology.orchestra.run.smithi037.stdout:Waiting for the mgr to restart... 2024-03-28T18:00:07.035 INFO:teuthology.orchestra.run.smithi037.stdout:Waiting for mgr epoch 9... 2024-03-28T18:00:07.182 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:06 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:06.911+0000 7f90621c0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:00:07.183 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:07 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:07.027+0000 7f90621c0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:00:07.454 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:07 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:07.394+0000 7f90621c0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:00:07.793 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:07 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:07.544+0000 7f90621c0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:00:07.793 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:07 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:07.663+0000 7f90621c0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:00:07.793 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:07 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:07.791+0000 7f90621c0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:00:08.124 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:08.018+0000 7f90621c0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:00:09.624 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:09 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:09.328+0000 7f90621c0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:00:09.624 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:09 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:09.505+0000 7f90621c0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:00:10.324 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:09 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:09.931+0000 7f90621c0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:00:10.325 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:10.054+0000 7f90621c0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:00:10.325 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:10.323+0000 7f90621c0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:00:10.874 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:10.575+0000 7f90621c0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:00:11.291 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:10.965+0000 7f90621c0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:00:11.291 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:11 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:11.289+0000 7f90621c0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:00:11.874 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:11 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:11.605+0000 7f90621c0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:00:13.124 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:12 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:12.666+0000 7f90621c0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:00:13.806 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:13.381+0000 7f90621c0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:00:13.807 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:13.515+0000 7f90621c0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:00:14.124 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:00:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:00:13.805+0000 7f90621c0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:00:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:13 smithi037 ceph-mon[31460]: Active manager daemon x restarted 2024-03-28T18:00:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:13 smithi037 ceph-mon[31460]: Activating manager daemon x 2024-03-28T18:00:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:13 smithi037 ceph-mon[31460]: osdmap e3: 0 total, 0 up, 0 in 2024-03-28T18:00:14.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:13 smithi037 ceph-mon[31460]: mgrmap e10: x(active, starting, since 0.00925472s) 2024-03-28T18:00:14.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:13 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:00:14.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:13 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T18:00:14.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:13 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T18:00:14.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:13 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T18:00:14.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:13 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T18:00:14.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:13 smithi037 ceph-mon[31460]: Manager daemon x is now available 2024-03-28T18:00:15.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:00:15.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:00:15.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T18:00:15.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T18:00:15.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:14 smithi037 ceph-mon[31460]: mgrmap e11: x(active, since 1.00886s) 2024-03-28T18:00:15.530 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout { 2024-03-28T18:00:15.530 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-28T18:00:15.530 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T18:00:15.530 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout } 2024-03-28T18:00:15.530 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.612+0000 7f5b69200700 1 Processor -- start 2024-03-28T18:00:15.530 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.613+0000 7f5b69200700 1 -- start start 2024-03-28T18:00:15.530 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.613+0000 7f5b69200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b64107a80 0x7f5b64107ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:15.531 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.613+0000 7f5b69200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b64108470 con 0x7f5b64107a80 2024-03-28T18:00:15.531 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.613+0000 7f5b61e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b64107a80 0x7f5b64107ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:15.531 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.614+0000 7f5b61e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b64107a80 0x7f5b64107ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:42886/0 (socket says 172.21.15.37:42886) 2024-03-28T18:00:15.531 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.614+0000 7f5b61e00700 1 -- 172.21.15.37:0/2038567221 learned_addr learned my addr 172.21.15.37:0/2038567221 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:15.531 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.614+0000 7f5b61e00700 1 -- 172.21.15.37:0/2038567221 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b64108c80 con 0x7f5b64107a80 2024-03-28T18:00:15.531 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.614+0000 7f5b61e00700 1 --2- 172.21.15.37:0/2038567221 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b64107a80 0x7f5b64107ea0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5b540098a0 tx=0x7f5b54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c31deea2b7edb0c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:15.531 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.615+0000 7f5b60a00700 1 -- 172.21.15.37:0/2038567221 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b54004160 con 0x7f5b64107a80 2024-03-28T18:00:15.531 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.615+0000 7f5b60a00700 1 -- 172.21.15.37:0/2038567221 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b54004300 con 0x7f5b64107a80 2024-03-28T18:00:15.531 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.615+0000 7f5b60a00700 1 -- 172.21.15.37:0/2038567221 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b54011b20 con 0x7f5b64107a80 2024-03-28T18:00:15.531 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.616+0000 7f5b69200700 1 -- 172.21.15.37:0/2038567221 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b64107a80 msgr2=0x7f5b64107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:15.531 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.616+0000 7f5b69200700 1 --2- 172.21.15.37:0/2038567221 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b64107a80 0x7f5b64107ea0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5b540098a0 tx=0x7f5b54009bb0 comp rx=0 tx=0).stop 2024-03-28T18:00:15.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.616+0000 7f5b69200700 1 -- 172.21.15.37:0/2038567221 shutdown_connections 2024-03-28T18:00:15.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.616+0000 7f5b69200700 1 --2- 172.21.15.37:0/2038567221 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b64107a80 0x7f5b64107ea0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:15.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.616+0000 7f5b69200700 1 -- 172.21.15.37:0/2038567221 >> 172.21.15.37:0/2038567221 conn(0x7f5b64103020 msgr2=0x7f5b64105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:15.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.616+0000 7f5b69200700 1 -- 172.21.15.37:0/2038567221 shutdown_connections 2024-03-28T18:00:15.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.616+0000 7f5b69200700 1 -- 172.21.15.37:0/2038567221 wait complete. 2024-03-28T18:00:15.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.617+0000 7f5b69200700 1 Processor -- start 2024-03-28T18:00:15.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.618+0000 7f5b69200700 1 -- start start 2024-03-28T18:00:15.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.618+0000 7f5b69200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b64107a80 0x7f5b6419b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:15.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.618+0000 7f5b69200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6410c030 con 0x7f5b64107a80 2024-03-28T18:00:15.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.618+0000 7f5b61e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b64107a80 0x7f5b6419b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:15.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.618+0000 7f5b61e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b64107a80 0x7f5b6419b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:42900/0 (socket says 172.21.15.37:42900) 2024-03-28T18:00:15.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.618+0000 7f5b61e00700 1 -- 172.21.15.37:0/1134472585 learned_addr learned my addr 172.21.15.37:0/1134472585 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:15.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.619+0000 7f5b61e00700 1 -- 172.21.15.37:0/1134472585 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6419bec0 con 0x7f5b64107a80 2024-03-28T18:00:15.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.619+0000 7f5b61e00700 1 --2- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b64107a80 0x7f5b6419b980 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5b540117e0 tx=0x7f5b54011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:15.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.620+0000 7f5b5aa00700 1 -- 172.21.15.37:0/1134472585 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b54003bf0 con 0x7f5b64107a80 2024-03-28T18:00:15.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.620+0000 7f5b5aa00700 1 -- 172.21.15.37:0/1134472585 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b5401a910 con 0x7f5b64107a80 2024-03-28T18:00:15.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.620+0000 7f5b69200700 1 -- 172.21.15.37:0/1134472585 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6419c150 con 0x7f5b64107a80 2024-03-28T18:00:15.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.620+0000 7f5b5aa00700 1 -- 172.21.15.37:0/1134472585 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b54024970 con 0x7f5b64107a80 2024-03-28T18:00:15.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.620+0000 7f5b69200700 1 -- 172.21.15.37:0/1134472585 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b6419c5b0 con 0x7f5b64107a80 2024-03-28T18:00:15.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.621+0000 7f5b5aa00700 1 -- 172.21.15.37:0/1134472585 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f5b54022020 con 0x7f5b64107a80 2024-03-28T18:00:15.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.622+0000 7f5b5aa00700 1 --2- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5b48038fd0 0x7f5b4803b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:15.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.622+0000 7f5b5aa00700 1 -- 172.21.15.37:0/1134472585 --> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b4803bba0 con 0x7f5b48038fd0 2024-03-28T18:00:15.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.622+0000 7f5b5aa00700 1 -- 172.21.15.37:0/1134472585 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5b5404d3e0 con 0x7f5b64107a80 2024-03-28T18:00:15.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.622+0000 7f5b61400700 1 -- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5b48038fd0 msgr2=0x7f5b4803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/3459498509 2024-03-28T18:00:15.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.622+0000 7f5b61400700 1 --2- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5b48038fd0 0x7f5b4803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:00:15.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.823+0000 7f5b61400700 1 -- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5b48038fd0 msgr2=0x7f5b4803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/3459498509 2024-03-28T18:00:15.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:07.823+0000 7f5b61400700 1 --2- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5b48038fd0 0x7f5b4803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T18:00:15.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:08.224+0000 7f5b61400700 1 -- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5b48038fd0 msgr2=0x7f5b4803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/3459498509 2024-03-28T18:00:15.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:08.224+0000 7f5b61400700 1 --2- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5b48038fd0 0x7f5b4803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T18:00:15.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:09.025+0000 7f5b61400700 1 -- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5b48038fd0 msgr2=0x7f5b4803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/3459498509 2024-03-28T18:00:15.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:09.025+0000 7f5b61400700 1 --2- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5b48038fd0 0x7f5b4803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T18:00:15.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:10.627+0000 7f5b61400700 1 -- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5b48038fd0 msgr2=0x7f5b4803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/3459498509 2024-03-28T18:00:15.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:10.627+0000 7f5b61400700 1 --2- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5b48038fd0 0x7f5b4803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T18:00:15.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:13.814+0000 7f5b5aa00700 1 -- 172.21.15.37:0/1134472585 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mgrmap(e 10) v1 ==== 45886+0+0 (secure 0 0 0) 0x7f5b5401a430 con 0x7f5b64107a80 2024-03-28T18:00:15.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:13.814+0000 7f5b5aa00700 1 -- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5b48038fd0 msgr2=0x7f5b4803b490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:00:15.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:13.814+0000 7f5b5aa00700 1 --2- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5b48038fd0 0x7f5b4803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:15.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:14.815+0000 7f5b5aa00700 1 -- 172.21.15.37:0/1134472585 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7f5b540188a0 con 0x7f5b64107a80 2024-03-28T18:00:15.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:14.815+0000 7f5b5aa00700 1 --2- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5b4803ca80 0x7f5b4803ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:15.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:14.815+0000 7f5b5aa00700 1 -- 172.21.15.37:0/1134472585 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b54001d30 con 0x7f5b4803ca80 2024-03-28T18:00:15.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:14.816+0000 7f5b61400700 1 --2- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5b4803ca80 0x7f5b4803ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:15.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:14.818+0000 7f5b61400700 1 --2- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5b4803ca80 0x7f5b4803ee70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5b50003a10 tx=0x7f5b50009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:15.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:14.819+0000 7f5b5aa00700 1 -- 172.21.15.37:0/1134472585 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f5b54001d30 con 0x7f5b4803ca80 2024-03-28T18:00:15.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:14.826+0000 7f5b69200700 1 -- 172.21.15.37:0/1134472585 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f5b640637c0 con 0x7f5b4803ca80 2024-03-28T18:00:15.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:14.826+0000 7f5b5aa00700 1 -- 172.21.15.37:0/1134472585 <== mgr.14152 v2:172.21.15.37:6800/521857560 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f5b640637c0 con 0x7f5b4803ca80 2024-03-28T18:00:15.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:14.827+0000 7f5b69200700 1 -- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5b4803ca80 msgr2=0x7f5b4803ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:15.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:14.827+0000 7f5b69200700 1 --2- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5b4803ca80 0x7f5b4803ee70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5b50003a10 tx=0x7f5b50009270 comp rx=0 tx=0).stop 2024-03-28T18:00:15.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:14.827+0000 7f5b69200700 1 -- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b64107a80 msgr2=0x7f5b6419b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:15.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:14.827+0000 7f5b69200700 1 --2- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b64107a80 0x7f5b6419b980 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5b540117e0 tx=0x7f5b54011ba0 comp rx=0 tx=0).stop 2024-03-28T18:00:15.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:14.827+0000 7f5b69200700 1 -- 172.21.15.37:0/1134472585 shutdown_connections 2024-03-28T18:00:15.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:14.827+0000 7f5b69200700 1 --2- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5b4803ca80 0x7f5b4803ee70 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:15.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:14.827+0000 7f5b69200700 1 --2- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b64107a80 0x7f5b6419b980 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:15.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:14.827+0000 7f5b69200700 1 --2- 172.21.15.37:0/1134472585 >> [v2:172.21.15.37:6800/3459498509,v1:172.21.15.37:6801/3459498509] conn(0x7f5b48038fd0 0x7f5b4803b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:15.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:14.827+0000 7f5b69200700 1 -- 172.21.15.37:0/1134472585 >> 172.21.15.37:0/1134472585 conn(0x7f5b64103020 msgr2=0x7f5b64103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:15.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:14.828+0000 7f5b69200700 1 -- 172.21.15.37:0/1134472585 shutdown_connections 2024-03-28T18:00:15.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:14.828+0000 7f5b69200700 1 -- 172.21.15.37:0/1134472585 wait complete. 2024-03-28T18:00:15.536 INFO:teuthology.orchestra.run.smithi037.stdout:mgr epoch 9 is available 2024-03-28T18:00:15.536 INFO:teuthology.orchestra.run.smithi037.stdout:Generating a dashboard self-signed certificate... 2024-03-28T18:00:16.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:15 smithi037 ceph-mon[31460]: [28/Mar/2024:18:00:14] ENGINE Bus STARTING 2024-03-28T18:00:16.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:15 smithi037 ceph-mon[31460]: [28/Mar/2024:18:00:14] ENGINE Serving on https://172.21.15.37:7150 2024-03-28T18:00:16.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:15 smithi037 ceph-mon[31460]: [28/Mar/2024:18:00:15] ENGINE Serving on http://172.21.15.37:8765 2024-03-28T18:00:16.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:15 smithi037 ceph-mon[31460]: [28/Mar/2024:18:00:15] ENGINE Bus STARTED 2024-03-28T18:00:17.260 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:16 smithi037 ceph-mon[31460]: mgrmap e12: x(active, since 2s) 2024-03-28T18:00:17.261 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:16 smithi037 ceph-mon[31460]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:00:17.261 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:16 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:17.261 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:16 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:17.617 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-28T18:00:17.617 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.166+0000 7fe490200700 1 Processor -- start 2024-03-28T18:00:17.617 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.166+0000 7fe490200700 1 -- start start 2024-03-28T18:00:17.617 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.166+0000 7fe490200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe488109c80 0x7fe48810a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:17.618 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.166+0000 7fe490200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48810a670 con 0x7fe488109c80 2024-03-28T18:00:17.618 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.167+0000 7fe48d800700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe488109c80 0x7fe48810a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:17.618 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.167+0000 7fe48d800700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe488109c80 0x7fe48810a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58606/0 (socket says 172.21.15.37:58606) 2024-03-28T18:00:17.618 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.167+0000 7fe48d800700 1 -- 172.21.15.37:0/1747048538 learned_addr learned my addr 172.21.15.37:0/1747048538 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:17.618 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.167+0000 7fe48d800700 1 -- 172.21.15.37:0/1747048538 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48810ae80 con 0x7fe488109c80 2024-03-28T18:00:17.623 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.167+0000 7fe48d800700 1 --2- 172.21.15.37:0/1747048538 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe488109c80 0x7fe48810a0a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0098a0 tx=0x7fe47c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b88f9003d03477a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:17.623 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.168+0000 7fe487e00700 1 -- 172.21.15.37:0/1747048538 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe47c004160 con 0x7fe488109c80 2024-03-28T18:00:17.623 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.168+0000 7fe487e00700 1 -- 172.21.15.37:0/1747048538 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe47c004300 con 0x7fe488109c80 2024-03-28T18:00:17.623 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.168+0000 7fe487e00700 1 -- 172.21.15.37:0/1747048538 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe47c011b20 con 0x7fe488109c80 2024-03-28T18:00:17.623 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.168+0000 7fe490200700 1 -- 172.21.15.37:0/1747048538 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe488109c80 msgr2=0x7fe48810a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:17.623 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.168+0000 7fe490200700 1 --2- 172.21.15.37:0/1747048538 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe488109c80 0x7fe48810a0a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0098a0 tx=0x7fe47c009bb0 comp rx=0 tx=0).stop 2024-03-28T18:00:17.623 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.169+0000 7fe490200700 1 -- 172.21.15.37:0/1747048538 shutdown_connections 2024-03-28T18:00:17.623 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.169+0000 7fe490200700 1 --2- 172.21.15.37:0/1747048538 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe488109c80 0x7fe48810a0a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:17.623 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.169+0000 7fe490200700 1 -- 172.21.15.37:0/1747048538 >> 172.21.15.37:0/1747048538 conn(0x7fe48807d040 msgr2=0x7fe48807d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:17.623 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.169+0000 7fe490200700 1 -- 172.21.15.37:0/1747048538 shutdown_connections 2024-03-28T18:00:17.624 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.169+0000 7fe490200700 1 -- 172.21.15.37:0/1747048538 wait complete. 2024-03-28T18:00:17.624 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.170+0000 7fe490200700 1 Processor -- start 2024-03-28T18:00:17.624 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.170+0000 7fe490200700 1 -- start start 2024-03-28T18:00:17.624 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.170+0000 7fe490200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe488109c80 0x7fe48819dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:17.624 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.170+0000 7fe490200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48810e230 con 0x7fe488109c80 2024-03-28T18:00:17.624 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.171+0000 7fe48d800700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe488109c80 0x7fe48819dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:17.624 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.171+0000 7fe48d800700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe488109c80 0x7fe48819dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58622/0 (socket says 172.21.15.37:58622) 2024-03-28T18:00:17.624 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.171+0000 7fe48d800700 1 -- 172.21.15.37:0/2909388361 learned_addr learned my addr 172.21.15.37:0/2909388361 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:17.624 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.171+0000 7fe48d800700 1 -- 172.21.15.37:0/2909388361 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48819e0f0 con 0x7fe488109c80 2024-03-28T18:00:17.624 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.171+0000 7fe48d800700 1 --2- 172.21.15.37:0/2909388361 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe488109c80 0x7fe48819dbb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0117e0 tx=0x7fe47c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:17.624 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.172+0000 7fe486000700 1 -- 172.21.15.37:0/2909388361 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe47c003bf0 con 0x7fe488109c80 2024-03-28T18:00:17.624 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.172+0000 7fe486000700 1 -- 172.21.15.37:0/2909388361 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe47c01a910 con 0x7fe488109c80 2024-03-28T18:00:17.624 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.172+0000 7fe490200700 1 -- 172.21.15.37:0/2909388361 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48819e380 con 0x7fe488109c80 2024-03-28T18:00:17.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.172+0000 7fe486000700 1 -- 172.21.15.37:0/2909388361 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe47c0249a0 con 0x7fe488109c80 2024-03-28T18:00:17.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.172+0000 7fe490200700 1 -- 172.21.15.37:0/2909388361 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48819e7e0 con 0x7fe488109c80 2024-03-28T18:00:17.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.173+0000 7fe486000700 1 -- 172.21.15.37:0/2909388361 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fe47c022020 con 0x7fe488109c80 2024-03-28T18:00:17.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.173+0000 7fe490200700 1 -- 172.21.15.37:0/2909388361 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4880637c0 con 0x7fe488109c80 2024-03-28T18:00:17.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.174+0000 7fe486000700 1 --2- 172.21.15.37:0/2909388361 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe470038f80 0x7fe47003b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:17.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.174+0000 7fe486000700 1 -- 172.21.15.37:0/2909388361 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe47c04d380 con 0x7fe488109c80 2024-03-28T18:00:17.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.174+0000 7fe48ce00700 1 --2- 172.21.15.37:0/2909388361 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe470038f80 0x7fe47003b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:17.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.175+0000 7fe48ce00700 1 --2- 172.21.15.37:0/2909388361 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe470038f80 0x7fe47003b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe474006fd0 tx=0x7fe474006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:17.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.179+0000 7fe486000700 1 -- 172.21.15.37:0/2909388361 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe47c011e20 con 0x7fe488109c80 2024-03-28T18:00:17.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.415+0000 7fe490200700 1 -- 172.21.15.37:0/2909388361 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fe488107b20 con 0x7fe470038f80 2024-03-28T18:00:17.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.565+0000 7fe486000700 1 -- 172.21.15.37:0/2909388361 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fe488107b20 con 0x7fe470038f80 2024-03-28T18:00:17.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.567+0000 7fe490200700 1 -- 172.21.15.37:0/2909388361 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe470038f80 msgr2=0x7fe47003b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:17.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.567+0000 7fe490200700 1 --2- 172.21.15.37:0/2909388361 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe470038f80 0x7fe47003b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe474006fd0 tx=0x7fe474006e40 comp rx=0 tx=0).stop 2024-03-28T18:00:17.625 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.567+0000 7fe490200700 1 -- 172.21.15.37:0/2909388361 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe488109c80 msgr2=0x7fe48819dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:17.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.567+0000 7fe490200700 1 --2- 172.21.15.37:0/2909388361 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe488109c80 0x7fe48819dbb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0117e0 tx=0x7fe47c011c10 comp rx=0 tx=0).stop 2024-03-28T18:00:17.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.568+0000 7fe490200700 1 -- 172.21.15.37:0/2909388361 shutdown_connections 2024-03-28T18:00:17.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.568+0000 7fe490200700 1 --2- 172.21.15.37:0/2909388361 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe470038f80 0x7fe47003b440 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:17.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.568+0000 7fe490200700 1 --2- 172.21.15.37:0/2909388361 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe488109c80 0x7fe48819dbb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:17.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.568+0000 7fe490200700 1 -- 172.21.15.37:0/2909388361 >> 172.21.15.37:0/2909388361 conn(0x7fe48807d040 msgr2=0x7fe488107410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:17.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.568+0000 7fe490200700 1 -- 172.21.15.37:0/2909388361 shutdown_connections 2024-03-28T18:00:17.626 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:16.568+0000 7fe490200700 1 -- 172.21.15.37:0/2909388361 wait complete. 2024-03-28T18:00:17.626 INFO:teuthology.orchestra.run.smithi037.stdout:Creating initial admin user... 2024-03-28T18:00:18.219 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:18 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:19.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$f3aomiCD6Mal8/iBo.i6g.BPLEx3WWDlUy7cQSLxL88z5B/p0W3Va", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711648818, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-28T18:00:19.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.219+0000 7fc5f2200700 1 Processor -- start 2024-03-28T18:00:19.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.219+0000 7fc5f2200700 1 -- start start 2024-03-28T18:00:19.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.219+0000 7fc5f2200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5ec07dec0 0x7fc5ec07c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:19.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.219+0000 7fc5f2200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec07c8f0 con 0x7fc5ec07dec0 2024-03-28T18:00:19.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.220+0000 7fc5eb400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5ec07dec0 0x7fc5ec07c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:19.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.220+0000 7fc5eb400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5ec07dec0 0x7fc5ec07c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58632/0 (socket says 172.21.15.37:58632) 2024-03-28T18:00:19.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.220+0000 7fc5eb400700 1 -- 172.21.15.37:0/1906126721 learned_addr learned my addr 172.21.15.37:0/1906126721 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:19.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.221+0000 7fc5eb400700 1 -- 172.21.15.37:0/1906126721 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec07ca70 con 0x7fc5ec07dec0 2024-03-28T18:00:19.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.221+0000 7fc5eb400700 1 --2- 172.21.15.37:0/1906126721 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5ec07dec0 0x7fc5ec07c320 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0098a0 tx=0x7fc5dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=888bf93798ffc826 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:19.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.221+0000 7fc5ea000700 1 -- 172.21.15.37:0/1906126721 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5dc004160 con 0x7fc5ec07dec0 2024-03-28T18:00:19.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.221+0000 7fc5ea000700 1 -- 172.21.15.37:0/1906126721 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc5dc004300 con 0x7fc5ec07dec0 2024-03-28T18:00:19.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.221+0000 7fc5ea000700 1 -- 172.21.15.37:0/1906126721 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5dc011b20 con 0x7fc5ec07dec0 2024-03-28T18:00:19.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.222+0000 7fc5f2200700 1 -- 172.21.15.37:0/1906126721 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5ec07dec0 msgr2=0x7fc5ec07c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:19.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.222+0000 7fc5f2200700 1 --2- 172.21.15.37:0/1906126721 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5ec07dec0 0x7fc5ec07c320 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0098a0 tx=0x7fc5dc009bb0 comp rx=0 tx=0).stop 2024-03-28T18:00:19.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.222+0000 7fc5f2200700 1 -- 172.21.15.37:0/1906126721 shutdown_connections 2024-03-28T18:00:19.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.222+0000 7fc5f2200700 1 --2- 172.21.15.37:0/1906126721 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5ec07dec0 0x7fc5ec07c320 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:19.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.222+0000 7fc5f2200700 1 -- 172.21.15.37:0/1906126721 >> 172.21.15.37:0/1906126721 conn(0x7fc5ec103080 msgr2=0x7fc5ec1054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:19.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.222+0000 7fc5f2200700 1 -- 172.21.15.37:0/1906126721 shutdown_connections 2024-03-28T18:00:19.501 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.222+0000 7fc5f2200700 1 -- 172.21.15.37:0/1906126721 wait complete. 2024-03-28T18:00:19.501 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.223+0000 7fc5f2200700 1 Processor -- start 2024-03-28T18:00:19.501 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.224+0000 7fc5f2200700 1 -- start start 2024-03-28T18:00:19.501 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.224+0000 7fc5f2200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5ec07dec0 0x7fc5ec1a2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:19.501 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.224+0000 7fc5f2200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec109ce0 con 0x7fc5ec07dec0 2024-03-28T18:00:19.501 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.224+0000 7fc5eb400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5ec07dec0 0x7fc5ec1a2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:19.501 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.224+0000 7fc5eb400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5ec07dec0 0x7fc5ec1a2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58634/0 (socket says 172.21.15.37:58634) 2024-03-28T18:00:19.501 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.224+0000 7fc5eb400700 1 -- 172.21.15.37:0/1930483105 learned_addr learned my addr 172.21.15.37:0/1930483105 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:19.501 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.225+0000 7fc5eb400700 1 -- 172.21.15.37:0/1930483105 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec1a2540 con 0x7fc5ec07dec0 2024-03-28T18:00:19.501 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.225+0000 7fc5eb400700 1 --2- 172.21.15.37:0/1930483105 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5ec07dec0 0x7fc5ec1a2000 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0117e0 tx=0x7fc5dc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:19.501 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.225+0000 7fc5e3e00700 1 -- 172.21.15.37:0/1930483105 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5dc003bf0 con 0x7fc5ec07dec0 2024-03-28T18:00:19.501 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.226+0000 7fc5e3e00700 1 -- 172.21.15.37:0/1930483105 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc5dc003d90 con 0x7fc5ec07dec0 2024-03-28T18:00:19.501 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.226+0000 7fc5e3e00700 1 -- 172.21.15.37:0/1930483105 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5dc024800 con 0x7fc5ec07dec0 2024-03-28T18:00:19.501 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.226+0000 7fc5f2200700 1 -- 172.21.15.37:0/1930483105 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5ec1a27d0 con 0x7fc5ec07dec0 2024-03-28T18:00:19.502 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.226+0000 7fc5f2200700 1 -- 172.21.15.37:0/1930483105 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5ec1a2c30 con 0x7fc5ec07dec0 2024-03-28T18:00:19.502 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.227+0000 7fc5e3e00700 1 -- 172.21.15.37:0/1930483105 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fc5dc022020 con 0x7fc5ec07dec0 2024-03-28T18:00:19.502 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.227+0000 7fc5f2200700 1 -- 172.21.15.37:0/1930483105 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5ec0637c0 con 0x7fc5ec07dec0 2024-03-28T18:00:19.502 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.227+0000 7fc5e3e00700 1 --2- 172.21.15.37:0/1930483105 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc5d0038fd0 0x7fc5d003b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:19.502 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.228+0000 7fc5e3e00700 1 -- 172.21.15.37:0/1930483105 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc5dc04d1e0 con 0x7fc5ec07dec0 2024-03-28T18:00:19.502 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.228+0000 7fc5eaa00700 1 --2- 172.21.15.37:0/1930483105 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc5d0038fd0 0x7fc5d003b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:19.502 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.229+0000 7fc5eaa00700 1 --2- 172.21.15.37:0/1930483105 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc5d0038fd0 0x7fc5d003b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8006fd0 tx=0x7fc5d8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:19.502 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.234+0000 7fc5e3e00700 1 -- 172.21.15.37:0/1930483105 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc5dc016260 con 0x7fc5ec07dec0 2024-03-28T18:00:19.502 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.484+0000 7fc5f2200700 1 -- 172.21.15.37:0/1930483105 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fc5ec1a5640 con 0x7fc5d0038fd0 2024-03-28T18:00:19.502 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.733+0000 7fc5e3e00700 1 -- 172.21.15.37:0/1930483105 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fc5ec1a5640 con 0x7fc5d0038fd0 2024-03-28T18:00:19.502 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.736+0000 7fc5f2200700 1 -- 172.21.15.37:0/1930483105 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc5d0038fd0 msgr2=0x7fc5d003b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:19.502 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.736+0000 7fc5f2200700 1 --2- 172.21.15.37:0/1930483105 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc5d0038fd0 0x7fc5d003b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8006fd0 tx=0x7fc5d8006e40 comp rx=0 tx=0).stop 2024-03-28T18:00:19.502 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.736+0000 7fc5f2200700 1 -- 172.21.15.37:0/1930483105 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5ec07dec0 msgr2=0x7fc5ec1a2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:19.502 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.736+0000 7fc5f2200700 1 --2- 172.21.15.37:0/1930483105 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5ec07dec0 0x7fc5ec1a2000 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0117e0 tx=0x7fc5dc011ba0 comp rx=0 tx=0).stop 2024-03-28T18:00:19.503 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.737+0000 7fc5f2200700 1 -- 172.21.15.37:0/1930483105 shutdown_connections 2024-03-28T18:00:19.503 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.737+0000 7fc5f2200700 1 --2- 172.21.15.37:0/1930483105 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc5d0038fd0 0x7fc5d003b490 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:19.503 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.737+0000 7fc5f2200700 1 --2- 172.21.15.37:0/1930483105 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5ec07dec0 0x7fc5ec1a2000 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:19.503 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.737+0000 7fc5f2200700 1 -- 172.21.15.37:0/1930483105 >> 172.21.15.37:0/1930483105 conn(0x7fc5ec103080 msgr2=0x7fc5ec105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:19.503 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.737+0000 7fc5f2200700 1 -- 172.21.15.37:0/1930483105 shutdown_connections 2024-03-28T18:00:19.503 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:18.737+0000 7fc5f2200700 1 -- 172.21.15.37:0/1930483105 wait complete. 2024-03-28T18:00:19.503 INFO:teuthology.orchestra.run.smithi037.stdout:Fetching dashboard port number... 2024-03-28T18:00:19.778 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:19 smithi037 ceph-mon[31460]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:00:19.778 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:19 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:19.778 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:19 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:21.058 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:20 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2705337395' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-28T18:00:21.874 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 8443 2024-03-28T18:00:21.874 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.087+0000 7f002d200700 1 Processor -- start 2024-03-28T18:00:21.874 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.087+0000 7f002d200700 1 -- start start 2024-03-28T18:00:21.874 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.087+0000 7f002d200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f002807de80 0x7f002807c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:21.874 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.088+0000 7f002d200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002807c8b0 con 0x7f002807de80 2024-03-28T18:00:21.874 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.088+0000 7f001de00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f002807de80 0x7f002807c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:21.874 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.088+0000 7f001de00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f002807de80 0x7f002807c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58638/0 (socket says 172.21.15.37:58638) 2024-03-28T18:00:21.875 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.088+0000 7f001de00700 1 -- 172.21.15.37:0/865333236 learned_addr learned my addr 172.21.15.37:0/865333236 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:21.875 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.088+0000 7f001de00700 1 -- 172.21.15.37:0/865333236 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002807ca30 con 0x7f002807de80 2024-03-28T18:00:21.875 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.089+0000 7f001de00700 1 --2- 172.21.15.37:0/865333236 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f002807de80 0x7f002807c2e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f00180098a0 tx=0x7f0018009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cfc2b698b1c3a70 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:21.875 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.089+0000 7f0025400700 1 -- 172.21.15.37:0/865333236 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0018004160 con 0x7f002807de80 2024-03-28T18:00:21.875 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.089+0000 7f0025400700 1 -- 172.21.15.37:0/865333236 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0018004300 con 0x7f002807de80 2024-03-28T18:00:21.875 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.089+0000 7f0025400700 1 -- 172.21.15.37:0/865333236 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0018011b20 con 0x7f002807de80 2024-03-28T18:00:21.875 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.090+0000 7f002d200700 1 -- 172.21.15.37:0/865333236 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f002807de80 msgr2=0x7f002807c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:21.875 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.090+0000 7f002d200700 1 --2- 172.21.15.37:0/865333236 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f002807de80 0x7f002807c2e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f00180098a0 tx=0x7f0018009bb0 comp rx=0 tx=0).stop 2024-03-28T18:00:21.875 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.090+0000 7f002d200700 1 -- 172.21.15.37:0/865333236 shutdown_connections 2024-03-28T18:00:21.875 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.090+0000 7f002d200700 1 --2- 172.21.15.37:0/865333236 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f002807de80 0x7f002807c2e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:21.875 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.090+0000 7f002d200700 1 -- 172.21.15.37:0/865333236 >> 172.21.15.37:0/865333236 conn(0x7f0028103080 msgr2=0x7f00281054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:21.876 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.090+0000 7f002d200700 1 -- 172.21.15.37:0/865333236 shutdown_connections 2024-03-28T18:00:21.876 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.090+0000 7f002d200700 1 -- 172.21.15.37:0/865333236 wait complete. 2024-03-28T18:00:21.876 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.091+0000 7f002d200700 1 Processor -- start 2024-03-28T18:00:21.876 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.092+0000 7f002d200700 1 -- start start 2024-03-28T18:00:21.876 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.092+0000 7f002d200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f002807de80 0x7f002819dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:21.876 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.092+0000 7f002d200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002810e0e0 con 0x7f002807de80 2024-03-28T18:00:21.876 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.092+0000 7f001de00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f002807de80 0x7f002819dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:21.876 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.092+0000 7f001de00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f002807de80 0x7f002819dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58640/0 (socket says 172.21.15.37:58640) 2024-03-28T18:00:21.876 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.092+0000 7f001de00700 1 -- 172.21.15.37:0/2705337395 learned_addr learned my addr 172.21.15.37:0/2705337395 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:21.876 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.093+0000 7f001de00700 1 -- 172.21.15.37:0/2705337395 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002819e120 con 0x7f002807de80 2024-03-28T18:00:21.876 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.093+0000 7f001de00700 1 --2- 172.21.15.37:0/2705337395 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f002807de80 0x7f002819dbe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f00180117e0 tx=0x7f0018011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:21.876 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.093+0000 7f001f400700 1 -- 172.21.15.37:0/2705337395 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0018003bf0 con 0x7f002807de80 2024-03-28T18:00:21.876 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.094+0000 7f001f400700 1 -- 172.21.15.37:0/2705337395 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f001801a910 con 0x7f002807de80 2024-03-28T18:00:21.877 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.094+0000 7f001f400700 1 -- 172.21.15.37:0/2705337395 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0018024970 con 0x7f002807de80 2024-03-28T18:00:21.877 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.094+0000 7f002d200700 1 -- 172.21.15.37:0/2705337395 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f002819e3b0 con 0x7f002807de80 2024-03-28T18:00:21.877 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.094+0000 7f002d200700 1 -- 172.21.15.37:0/2705337395 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f002819e810 con 0x7f002807de80 2024-03-28T18:00:21.877 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.095+0000 7f002d200700 1 -- 172.21.15.37:0/2705337395 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00280637c0 con 0x7f002807de80 2024-03-28T18:00:21.877 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.095+0000 7f001f400700 1 -- 172.21.15.37:0/2705337395 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f0018022020 con 0x7f002807de80 2024-03-28T18:00:21.877 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.096+0000 7f001f400700 1 --2- 172.21.15.37:0/2705337395 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f000c038f70 0x7f000c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:21.877 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.096+0000 7f001f400700 1 -- 172.21.15.37:0/2705337395 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f001804d3d0 con 0x7f002807de80 2024-03-28T18:00:21.877 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.096+0000 7f0025e00700 1 --2- 172.21.15.37:0/2705337395 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f000c038f70 0x7f000c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:21.877 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.097+0000 7f0025e00700 1 --2- 172.21.15.37:0/2705337395 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f000c038f70 0x7f000c03b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0010006fd0 tx=0x7f0010006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:21.877 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.102+0000 7f001f400700 1 -- 172.21.15.37:0/2705337395 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0018017ca0 con 0x7f002807de80 2024-03-28T18:00:21.877 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.330+0000 7f002d200700 1 -- 172.21.15.37:0/2705337395 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f00281a12e0 con 0x7f002807de80 2024-03-28T18:00:21.877 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.331+0000 7f001f400700 1 -- 172.21.15.37:0/2705337395 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f001801ab70 con 0x7f002807de80 2024-03-28T18:00:21.877 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.334+0000 7f002d200700 1 -- 172.21.15.37:0/2705337395 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f000c038f70 msgr2=0x7f000c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:21.878 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.334+0000 7f002d200700 1 --2- 172.21.15.37:0/2705337395 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f000c038f70 0x7f000c03b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0010006fd0 tx=0x7f0010006e40 comp rx=0 tx=0).stop 2024-03-28T18:00:21.878 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.334+0000 7f002d200700 1 -- 172.21.15.37:0/2705337395 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f002807de80 msgr2=0x7f002819dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:21.878 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.334+0000 7f002d200700 1 --2- 172.21.15.37:0/2705337395 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f002807de80 0x7f002819dbe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f00180117e0 tx=0x7f0018011ba0 comp rx=0 tx=0).stop 2024-03-28T18:00:21.878 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.335+0000 7f002d200700 1 -- 172.21.15.37:0/2705337395 shutdown_connections 2024-03-28T18:00:21.878 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.335+0000 7f002d200700 1 --2- 172.21.15.37:0/2705337395 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f000c038f70 0x7f000c03b430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:21.878 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.335+0000 7f002d200700 1 --2- 172.21.15.37:0/2705337395 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f002807de80 0x7f002819dbe0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:21.878 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.335+0000 7f002d200700 1 -- 172.21.15.37:0/2705337395 >> 172.21.15.37:0/2705337395 conn(0x7f0028103080 msgr2=0x7f0028103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:21.878 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.335+0000 7f002d200700 1 -- 172.21.15.37:0/2705337395 shutdown_connections 2024-03-28T18:00:21.878 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:20.335+0000 7f002d200700 1 -- 172.21.15.37:0/2705337395 wait complete. 2024-03-28T18:00:21.901 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T18:00:21.901 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout disabled 2024-03-28T18:00:21.924 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T18:00:21.924 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout inactive 2024-03-28T18:00:21.925 INFO:teuthology.orchestra.run.smithi037.stdout:firewalld.service is not enabled 2024-03-28T18:00:21.925 INFO:teuthology.orchestra.run.smithi037.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-28T18:00:21.932 INFO:teuthology.orchestra.run.smithi037.stdout:Ceph Dashboard is now available at: 2024-03-28T18:00:21.932 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:00:21.932 INFO:teuthology.orchestra.run.smithi037.stdout: URL: https://smithi037.front.sepia.ceph.com:8443/ 2024-03-28T18:00:21.932 INFO:teuthology.orchestra.run.smithi037.stdout: User: admin 2024-03-28T18:00:21.932 INFO:teuthology.orchestra.run.smithi037.stdout: Password: oeghbewtjz 2024-03-28T18:00:21.932 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:00:21.933 INFO:teuthology.orchestra.run.smithi037.stdout:Saving cluster configuration to /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config directory 2024-03-28T18:00:21.935 INFO:teuthology.orchestra.run.smithi037.stdout:Enabling autotune for osd_memory_target 2024-03-28T18:00:22.094 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:21 smithi037 ceph-mon[31460]: mgrmap e13: x(active, since 6s) 2024-03-28T18:00:23.481 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.546+0000 7f975da00700 1 Processor -- start 2024-03-28T18:00:23.482 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.546+0000 7f975da00700 1 -- start start 2024-03-28T18:00:23.482 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.547+0000 7f975da00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9758109c50 0x7f975810a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:23.483 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.547+0000 7f975da00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975810a640 con 0x7f9758109c50 2024-03-28T18:00:23.483 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.547+0000 7f9756a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9758109c50 0x7f975810a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:23.483 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.547+0000 7f9756a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9758109c50 0x7f975810a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58642/0 (socket says 172.21.15.37:58642) 2024-03-28T18:00:23.483 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.547+0000 7f9756a00700 1 -- 172.21.15.37:0/936916835 learned_addr learned my addr 172.21.15.37:0/936916835 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:23.483 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.548+0000 7f9756a00700 1 -- 172.21.15.37:0/936916835 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975810ae50 con 0x7f9758109c50 2024-03-28T18:00:23.484 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.548+0000 7f9756a00700 1 --2- 172.21.15.37:0/936916835 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9758109c50 0x7f975810a070 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f97480098a0 tx=0x7f9748009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b94496a5a71236a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:23.484 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.548+0000 7f9755600700 1 -- 172.21.15.37:0/936916835 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9748004160 con 0x7f9758109c50 2024-03-28T18:00:23.484 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.548+0000 7f9755600700 1 -- 172.21.15.37:0/936916835 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9748004300 con 0x7f9758109c50 2024-03-28T18:00:23.485 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.548+0000 7f9755600700 1 -- 172.21.15.37:0/936916835 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9748011b20 con 0x7f9758109c50 2024-03-28T18:00:23.485 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.549+0000 7f975da00700 1 -- 172.21.15.37:0/936916835 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9758109c50 msgr2=0x7f975810a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:23.485 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.549+0000 7f975da00700 1 --2- 172.21.15.37:0/936916835 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9758109c50 0x7f975810a070 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f97480098a0 tx=0x7f9748009bb0 comp rx=0 tx=0).stop 2024-03-28T18:00:23.485 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.549+0000 7f975da00700 1 -- 172.21.15.37:0/936916835 shutdown_connections 2024-03-28T18:00:23.486 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.549+0000 7f975da00700 1 --2- 172.21.15.37:0/936916835 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9758109c50 0x7f975810a070 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:23.486 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.549+0000 7f975da00700 1 -- 172.21.15.37:0/936916835 >> 172.21.15.37:0/936916835 conn(0x7f975807d040 msgr2=0x7f975807d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:23.486 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.549+0000 7f975da00700 1 -- 172.21.15.37:0/936916835 shutdown_connections 2024-03-28T18:00:23.487 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.549+0000 7f975da00700 1 -- 172.21.15.37:0/936916835 wait complete. 2024-03-28T18:00:23.487 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.550+0000 7f975da00700 1 Processor -- start 2024-03-28T18:00:23.487 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.551+0000 7f975da00700 1 -- start start 2024-03-28T18:00:23.487 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.551+0000 7f975da00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9758109c50 0x7f975819db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:23.489 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.551+0000 7f975da00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975810e200 con 0x7f9758109c50 2024-03-28T18:00:23.489 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.551+0000 7f9756a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9758109c50 0x7f975819db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:23.489 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.551+0000 7f9756a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9758109c50 0x7f975819db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58644/0 (socket says 172.21.15.37:58644) 2024-03-28T18:00:23.489 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.551+0000 7f9756a00700 1 -- 172.21.15.37:0/3010023194 learned_addr learned my addr 172.21.15.37:0/3010023194 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:23.489 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.552+0000 7f9756a00700 1 -- 172.21.15.37:0/3010023194 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975819e0c0 con 0x7f9758109c50 2024-03-28T18:00:23.489 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.552+0000 7f9756a00700 1 --2- 172.21.15.37:0/3010023194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9758109c50 0x7f975819db80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f97480117e0 tx=0x7f9748011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:23.490 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.552+0000 7f974f400700 1 -- 172.21.15.37:0/3010023194 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9748003bf0 con 0x7f9758109c50 2024-03-28T18:00:23.490 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.552+0000 7f974f400700 1 -- 172.21.15.37:0/3010023194 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f974801a910 con 0x7f9758109c50 2024-03-28T18:00:23.490 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.552+0000 7f975da00700 1 -- 172.21.15.37:0/3010023194 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f975819e350 con 0x7f9758109c50 2024-03-28T18:00:23.490 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.552+0000 7f974f400700 1 -- 172.21.15.37:0/3010023194 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9748024970 con 0x7f9758109c50 2024-03-28T18:00:23.490 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.553+0000 7f975da00700 1 -- 172.21.15.37:0/3010023194 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f975819e7b0 con 0x7f9758109c50 2024-03-28T18:00:23.490 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.554+0000 7f974f400700 1 -- 172.21.15.37:0/3010023194 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f9748022020 con 0x7f9758109c50 2024-03-28T18:00:23.490 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.554+0000 7f975da00700 1 -- 172.21.15.37:0/3010023194 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97580637c0 con 0x7f9758109c50 2024-03-28T18:00:23.490 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.554+0000 7f974f400700 1 --2- 172.21.15.37:0/3010023194 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f973c039070 0x7f973c03b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:23.490 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.554+0000 7f974f400700 1 -- 172.21.15.37:0/3010023194 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f974804d4f0 con 0x7f9758109c50 2024-03-28T18:00:23.492 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.555+0000 7f9756000700 1 --2- 172.21.15.37:0/3010023194 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f973c039070 0x7f973c03b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:23.492 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.555+0000 7f9756000700 1 --2- 172.21.15.37:0/3010023194 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f973c039070 0x7f973c03b530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9744006fd0 tx=0x7f9744006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:23.492 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.560+0000 7f974f400700 1 -- 172.21.15.37:0/3010023194 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f974802d910 con 0x7f9758109c50 2024-03-28T18:00:23.492 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.784+0000 7f975da00700 1 -- 172.21.15.37:0/3010023194 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f97581a1180 con 0x7f9758109c50 2024-03-28T18:00:23.492 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.785+0000 7f974f400700 1 -- 172.21.15.37:0/3010023194 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f974802d3f0 con 0x7f9758109c50 2024-03-28T18:00:23.492 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.787+0000 7f975da00700 1 -- 172.21.15.37:0/3010023194 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f973c039070 msgr2=0x7f973c03b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:23.493 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.787+0000 7f975da00700 1 --2- 172.21.15.37:0/3010023194 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f973c039070 0x7f973c03b530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9744006fd0 tx=0x7f9744006e40 comp rx=0 tx=0).stop 2024-03-28T18:00:23.493 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.787+0000 7f975da00700 1 -- 172.21.15.37:0/3010023194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9758109c50 msgr2=0x7f975819db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:23.493 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.787+0000 7f975da00700 1 --2- 172.21.15.37:0/3010023194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9758109c50 0x7f975819db80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f97480117e0 tx=0x7f9748011ba0 comp rx=0 tx=0).stop 2024-03-28T18:00:23.493 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.787+0000 7f975da00700 1 -- 172.21.15.37:0/3010023194 shutdown_connections 2024-03-28T18:00:23.493 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.787+0000 7f975da00700 1 --2- 172.21.15.37:0/3010023194 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f973c039070 0x7f973c03b530 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:23.493 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.787+0000 7f975da00700 1 --2- 172.21.15.37:0/3010023194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9758109c50 0x7f975819db80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:23.493 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.787+0000 7f975da00700 1 -- 172.21.15.37:0/3010023194 >> 172.21.15.37:0/3010023194 conn(0x7f975807d040 msgr2=0x7f97581074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:23.493 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.788+0000 7f975da00700 1 -- 172.21.15.37:0/3010023194 shutdown_connections 2024-03-28T18:00:23.493 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:22.788+0000 7f975da00700 1 -- 172.21.15.37:0/3010023194 wait complete. 2024-03-28T18:00:25.132 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.139+0000 7f8954400700 1 Processor -- start 2024-03-28T18:00:25.132 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.139+0000 7f8954400700 1 -- start start 2024-03-28T18:00:25.132 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.140+0000 7f8954400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f894c1085a0 0x7f894c10a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:25.132 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.140+0000 7f8954400700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894c07bdd0 con 0x7f894c1085a0 2024-03-28T18:00:25.132 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.140+0000 7f8951a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f894c1085a0 0x7f894c10a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:25.132 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.140+0000 7f8951a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f894c1085a0 0x7f894c10a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:59002/0 (socket says 172.21.15.37:59002) 2024-03-28T18:00:25.132 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.140+0000 7f8951a00700 1 -- 172.21.15.37:0/1140032799 learned_addr learned my addr 172.21.15.37:0/1140032799 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:25.133 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.141+0000 7f8951a00700 1 -- 172.21.15.37:0/1140032799 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894c10aed0 con 0x7f894c1085a0 2024-03-28T18:00:25.133 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.142+0000 7f8951a00700 1 --2- 172.21.15.37:0/1140032799 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f894c1085a0 0x7f894c10a990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f89400098a0 tx=0x7f8940009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=828dee04ddd56c42 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:25.133 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.142+0000 7f894be00700 1 -- 172.21.15.37:0/1140032799 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8940004160 con 0x7f894c1085a0 2024-03-28T18:00:25.133 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.142+0000 7f894be00700 1 -- 172.21.15.37:0/1140032799 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8940004300 con 0x7f894c1085a0 2024-03-28T18:00:25.133 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.142+0000 7f894be00700 1 -- 172.21.15.37:0/1140032799 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8940011b20 con 0x7f894c1085a0 2024-03-28T18:00:25.133 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.143+0000 7f8954400700 1 -- 172.21.15.37:0/1140032799 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f894c1085a0 msgr2=0x7f894c10a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:25.133 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.143+0000 7f8954400700 1 --2- 172.21.15.37:0/1140032799 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f894c1085a0 0x7f894c10a990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f89400098a0 tx=0x7f8940009bb0 comp rx=0 tx=0).stop 2024-03-28T18:00:25.133 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.143+0000 7f8954400700 1 -- 172.21.15.37:0/1140032799 shutdown_connections 2024-03-28T18:00:25.133 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.143+0000 7f8954400700 1 --2- 172.21.15.37:0/1140032799 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f894c1085a0 0x7f894c10a990 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:25.133 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.143+0000 7f8954400700 1 -- 172.21.15.37:0/1140032799 >> 172.21.15.37:0/1140032799 conn(0x7f894c101f70 msgr2=0x7f894c104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:25.133 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.143+0000 7f8954400700 1 -- 172.21.15.37:0/1140032799 shutdown_connections 2024-03-28T18:00:25.133 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.143+0000 7f8954400700 1 -- 172.21.15.37:0/1140032799 wait complete. 2024-03-28T18:00:25.133 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.144+0000 7f8954400700 1 Processor -- start 2024-03-28T18:00:25.133 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.144+0000 7f8954400700 1 -- start start 2024-03-28T18:00:25.134 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.145+0000 7f8954400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f894c1085a0 0x7f894c19dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:25.134 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.145+0000 7f8954400700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894c10be80 con 0x7f894c1085a0 2024-03-28T18:00:25.134 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.145+0000 7f8951a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f894c1085a0 0x7f894c19dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:25.134 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.145+0000 7f8951a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f894c1085a0 0x7f894c19dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:59004/0 (socket says 172.21.15.37:59004) 2024-03-28T18:00:25.134 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.145+0000 7f8951a00700 1 -- 172.21.15.37:0/1863304188 learned_addr learned my addr 172.21.15.37:0/1863304188 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:25.134 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.145+0000 7f8951a00700 1 -- 172.21.15.37:0/1863304188 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894c19e0e0 con 0x7f894c1085a0 2024-03-28T18:00:25.134 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.146+0000 7f8951a00700 1 --2- 172.21.15.37:0/1863304188 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f894c1085a0 0x7f894c19dba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f89400117e0 tx=0x7f8940011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:25.134 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.146+0000 7f894a000700 1 -- 172.21.15.37:0/1863304188 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8940003ba0 con 0x7f894c1085a0 2024-03-28T18:00:25.134 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.146+0000 7f894a000700 1 -- 172.21.15.37:0/1863304188 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f894001a910 con 0x7f894c1085a0 2024-03-28T18:00:25.134 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.146+0000 7f8954400700 1 -- 172.21.15.37:0/1863304188 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f894c19e370 con 0x7f894c1085a0 2024-03-28T18:00:25.134 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.146+0000 7f894a000700 1 -- 172.21.15.37:0/1863304188 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f89400248f0 con 0x7f894c1085a0 2024-03-28T18:00:25.134 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.146+0000 7f8954400700 1 -- 172.21.15.37:0/1863304188 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f894c19e7d0 con 0x7f894c1085a0 2024-03-28T18:00:25.134 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.147+0000 7f8954400700 1 -- 172.21.15.37:0/1863304188 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f894c0637c0 con 0x7f894c1085a0 2024-03-28T18:00:25.135 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.148+0000 7f894a000700 1 -- 172.21.15.37:0/1863304188 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f8940022020 con 0x7f894c1085a0 2024-03-28T18:00:25.135 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.149+0000 7f894a000700 1 --2- 172.21.15.37:0/1863304188 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8934039010 0x7f893403b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:25.135 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.149+0000 7f894a000700 1 -- 172.21.15.37:0/1863304188 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f894004d9f0 con 0x7f894c1085a0 2024-03-28T18:00:25.135 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.149+0000 7f8951000700 1 --2- 172.21.15.37:0/1863304188 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8934039010 0x7f893403b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:25.135 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.150+0000 7f8951000700 1 --2- 172.21.15.37:0/1863304188 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8934039010 0x7f893403b4d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8938006fd0 tx=0x7f8938006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:25.135 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.154+0000 7f894a000700 1 -- 172.21.15.37:0/1863304188 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f894001ab70 con 0x7f894c1085a0 2024-03-28T18:00:25.135 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.498+0000 7f8954400700 1 -- 172.21.15.37:0/1863304188 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f894c1a1260 con 0x7f894c1085a0 2024-03-28T18:00:25.135 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.501+0000 7f894a000700 1 -- 172.21.15.37:0/1863304188 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f894002d3f0 con 0x7f894c1085a0 2024-03-28T18:00:25.135 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.504+0000 7f8954400700 1 -- 172.21.15.37:0/1863304188 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8934039010 msgr2=0x7f893403b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:25.135 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.504+0000 7f8954400700 1 --2- 172.21.15.37:0/1863304188 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8934039010 0x7f893403b4d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8938006fd0 tx=0x7f8938006e40 comp rx=0 tx=0).stop 2024-03-28T18:00:25.135 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.504+0000 7f8954400700 1 -- 172.21.15.37:0/1863304188 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f894c1085a0 msgr2=0x7f894c19dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:25.135 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.504+0000 7f8954400700 1 --2- 172.21.15.37:0/1863304188 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f894c1085a0 0x7f894c19dba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f89400117e0 tx=0x7f8940011ba0 comp rx=0 tx=0).stop 2024-03-28T18:00:25.135 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.504+0000 7f8954400700 1 -- 172.21.15.37:0/1863304188 shutdown_connections 2024-03-28T18:00:25.135 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.504+0000 7f8954400700 1 --2- 172.21.15.37:0/1863304188 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8934039010 0x7f893403b4d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:25.136 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.504+0000 7f8954400700 1 --2- 172.21.15.37:0/1863304188 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f894c1085a0 0x7f894c19dba0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:25.136 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.505+0000 7f8954400700 1 -- 172.21.15.37:0/1863304188 >> 172.21.15.37:0/1863304188 conn(0x7f894c101f70 msgr2=0x7f894c102970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:25.136 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.505+0000 7f8954400700 1 -- 172.21.15.37:0/1863304188 shutdown_connections 2024-03-28T18:00:25.136 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-03-28T18:00:24.505+0000 7f8954400700 1 -- 172.21.15.37:0/1863304188 wait complete. 2024-03-28T18:00:25.136 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-28T18:00:25.136 INFO:teuthology.orchestra.run.smithi037.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-28T18:00:25.136 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:00:25.136 INFO:teuthology.orchestra.run.smithi037.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-28T18:00:25.136 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:00:25.136 INFO:teuthology.orchestra.run.smithi037.stdout:Or, if you are only running a single cluster on this host: 2024-03-28T18:00:25.136 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:00:25.136 INFO:teuthology.orchestra.run.smithi037.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-28T18:00:25.136 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:00:25.136 INFO:teuthology.orchestra.run.smithi037.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-28T18:00:25.136 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:00:25.137 INFO:teuthology.orchestra.run.smithi037.stdout: ceph telemetry on 2024-03-28T18:00:25.137 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:00:25.137 INFO:teuthology.orchestra.run.smithi037.stdout:For more information see: 2024-03-28T18:00:25.137 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:00:25.137 INFO:teuthology.orchestra.run.smithi037.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-28T18:00:25.137 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:00:25.137 INFO:teuthology.orchestra.run.smithi037.stdout:Bootstrap complete. 2024-03-28T18:00:25.245 INFO:tasks.cephadm:Fetching config... 2024-03-28T18:00:25.245 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-28T18:00:25.245 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-28T18:00:25.296 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-28T18:00:25.296 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-28T18:00:25.296 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-28T18:00:25.341 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-28T18:00:25.341 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-28T18:00:25.341 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-28T18:00:25.474 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-28T18:00:25.474 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-28T18:00:25.474 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-28T18:00:25.530 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-28T18:00:25.531 DEBUG:teuthology.orchestra.run.smithi037:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD1F/cnl50HxYSSFY9/w1WPzQL0j1VQwl2pJqTdZ+7bihhWxC6YMhT5fityfUZ4Ac2HZOm+RDIBFCLo78r5NmSWvWSMz9cLoCLKiyPS7OH2GVX00RPFgoAbeIwm6OfIF/aae1Gxsc1TsjsNSoCdWWJcbj2Kxg5jqsbV2j028FIIwePvymequbOrHMQgpNNeHFWdeqvAZ36DwAZyvYwxPV1ntCOi+MGPEDMHynYMp8jFiOb9HwRvPICkTu1+7uVDHtA4B68WKg5QMvL4UY/3oM38MzOsuM+BtwcyG5PCRQQ6y3JUf0KPvyOeTBczlEzlEDPkJs7XqtT0AfHNiOQa6Ufrpq59MimbREGm4N0JScbzbkQvsydKEvp02WKC1iKWVnYSJeVzgEIOYZY9F9lH2V2VxF/7NWPx+8P8/cBx+pOnRJpMlpufe5Ov75PetgLNQzpUTvt9jSRQWOtJ2j4sThLNJ1GfgcSct8mLAkgH36swHSYPCTFE1dlN4WxhdvxBhgk= ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T18:00:25.677 INFO:teuthology.orchestra.run.smithi037.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD1F/cnl50HxYSSFY9/w1WPzQL0j1VQwl2pJqTdZ+7bihhWxC6YMhT5fityfUZ4Ac2HZOm+RDIBFCLo78r5NmSWvWSMz9cLoCLKiyPS7OH2GVX00RPFgoAbeIwm6OfIF/aae1Gxsc1TsjsNSoCdWWJcbj2Kxg5jqsbV2j028FIIwePvymequbOrHMQgpNNeHFWdeqvAZ36DwAZyvYwxPV1ntCOi+MGPEDMHynYMp8jFiOb9HwRvPICkTu1+7uVDHtA4B68WKg5QMvL4UY/3oM38MzOsuM+BtwcyG5PCRQQ6y3JUf0KPvyOeTBczlEzlEDPkJs7XqtT0AfHNiOQa6Ufrpq59MimbREGm4N0JScbzbkQvsydKEvp02WKC1iKWVnYSJeVzgEIOYZY9F9lH2V2VxF/7NWPx+8P8/cBx+pOnRJpMlpufe5Ov75PetgLNQzpUTvt9jSRQWOtJ2j4sThLNJ1GfgcSct8mLAkgH36swHSYPCTFE1dlN4WxhdvxBhgk= ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715 2024-03-28T18:00:25.727 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:25 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1863304188' entity='client.admin' 2024-03-28T18:00:25.745 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD1F/cnl50HxYSSFY9/w1WPzQL0j1VQwl2pJqTdZ+7bihhWxC6YMhT5fityfUZ4Ac2HZOm+RDIBFCLo78r5NmSWvWSMz9cLoCLKiyPS7OH2GVX00RPFgoAbeIwm6OfIF/aae1Gxsc1TsjsNSoCdWWJcbj2Kxg5jqsbV2j028FIIwePvymequbOrHMQgpNNeHFWdeqvAZ36DwAZyvYwxPV1ntCOi+MGPEDMHynYMp8jFiOb9HwRvPICkTu1+7uVDHtA4B68WKg5QMvL4UY/3oM38MzOsuM+BtwcyG5PCRQQ6y3JUf0KPvyOeTBczlEzlEDPkJs7XqtT0AfHNiOQa6Ufrpq59MimbREGm4N0JScbzbkQvsydKEvp02WKC1iKWVnYSJeVzgEIOYZY9F9lH2V2VxF/7NWPx+8P8/cBx+pOnRJpMlpufe5Ov75PetgLNQzpUTvt9jSRQWOtJ2j4sThLNJ1GfgcSct8mLAkgH36swHSYPCTFE1dlN4WxhdvxBhgk= ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T18:00:25.802 INFO:teuthology.orchestra.run.smithi059.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD1F/cnl50HxYSSFY9/w1WPzQL0j1VQwl2pJqTdZ+7bihhWxC6YMhT5fityfUZ4Ac2HZOm+RDIBFCLo78r5NmSWvWSMz9cLoCLKiyPS7OH2GVX00RPFgoAbeIwm6OfIF/aae1Gxsc1TsjsNSoCdWWJcbj2Kxg5jqsbV2j028FIIwePvymequbOrHMQgpNNeHFWdeqvAZ36DwAZyvYwxPV1ntCOi+MGPEDMHynYMp8jFiOb9HwRvPICkTu1+7uVDHtA4B68WKg5QMvL4UY/3oM38MzOsuM+BtwcyG5PCRQQ6y3JUf0KPvyOeTBczlEzlEDPkJs7XqtT0AfHNiOQa6Ufrpq59MimbREGm4N0JScbzbkQvsydKEvp02WKC1iKWVnYSJeVzgEIOYZY9F9lH2V2VxF/7NWPx+8P8/cBx+pOnRJpMlpufe5Ov75PetgLNQzpUTvt9jSRQWOtJ2j4sThLNJ1GfgcSct8mLAkgH36swHSYPCTFE1dlN4WxhdvxBhgk= ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715 2024-03-28T18:00:25.821 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD1F/cnl50HxYSSFY9/w1WPzQL0j1VQwl2pJqTdZ+7bihhWxC6YMhT5fityfUZ4Ac2HZOm+RDIBFCLo78r5NmSWvWSMz9cLoCLKiyPS7OH2GVX00RPFgoAbeIwm6OfIF/aae1Gxsc1TsjsNSoCdWWJcbj2Kxg5jqsbV2j028FIIwePvymequbOrHMQgpNNeHFWdeqvAZ36DwAZyvYwxPV1ntCOi+MGPEDMHynYMp8jFiOb9HwRvPICkTu1+7uVDHtA4B68WKg5QMvL4UY/3oM38MzOsuM+BtwcyG5PCRQQ6y3JUf0KPvyOeTBczlEzlEDPkJs7XqtT0AfHNiOQa6Ufrpq59MimbREGm4N0JScbzbkQvsydKEvp02WKC1iKWVnYSJeVzgEIOYZY9F9lH2V2VxF/7NWPx+8P8/cBx+pOnRJpMlpufe5Ov75PetgLNQzpUTvt9jSRQWOtJ2j4sThLNJ1GfgcSct8mLAkgH36swHSYPCTFE1dlN4WxhdvxBhgk= ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T18:00:25.869 INFO:teuthology.orchestra.run.smithi170.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD1F/cnl50HxYSSFY9/w1WPzQL0j1VQwl2pJqTdZ+7bihhWxC6YMhT5fityfUZ4Ac2HZOm+RDIBFCLo78r5NmSWvWSMz9cLoCLKiyPS7OH2GVX00RPFgoAbeIwm6OfIF/aae1Gxsc1TsjsNSoCdWWJcbj2Kxg5jqsbV2j028FIIwePvymequbOrHMQgpNNeHFWdeqvAZ36DwAZyvYwxPV1ntCOi+MGPEDMHynYMp8jFiOb9HwRvPICkTu1+7uVDHtA4B68WKg5QMvL4UY/3oM38MzOsuM+BtwcyG5PCRQQ6y3JUf0KPvyOeTBczlEzlEDPkJs7XqtT0AfHNiOQa6Ufrpq59MimbREGm4N0JScbzbkQvsydKEvp02WKC1iKWVnYSJeVzgEIOYZY9F9lH2V2VxF/7NWPx+8P8/cBx+pOnRJpMlpufe5Ov75PetgLNQzpUTvt9jSRQWOtJ2j4sThLNJ1GfgcSct8mLAkgH36swHSYPCTFE1dlN4WxhdvxBhgk= ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715 2024-03-28T18:00:25.884 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-28T18:00:26.317 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:00:28.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.845+0000 7ff04da00700 1 -- 172.21.15.37:0/2105116012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff048102670 msgr2=0x7ff048104a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:28.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.845+0000 7ff04da00700 1 --2- 172.21.15.37:0/2105116012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff048102670 0x7ff048104a60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff038009910 tx=0x7ff038009c20 comp rx=0 tx=0).stop 2024-03-28T18:00:28.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.846+0000 7ff04da00700 1 -- 172.21.15.37:0/2105116012 shutdown_connections 2024-03-28T18:00:28.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.846+0000 7ff04da00700 1 --2- 172.21.15.37:0/2105116012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff048102670 0x7ff048104a60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:28.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.846+0000 7ff04da00700 1 -- 172.21.15.37:0/2105116012 >> 172.21.15.37:0/2105116012 conn(0x7ff0480fc080 msgr2=0x7ff0480fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:28.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.846+0000 7ff04da00700 1 -- 172.21.15.37:0/2105116012 shutdown_connections 2024-03-28T18:00:28.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.846+0000 7ff04da00700 1 -- 172.21.15.37:0/2105116012 wait complete. 2024-03-28T18:00:28.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.847+0000 7ff04da00700 1 Processor -- start 2024-03-28T18:00:28.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.847+0000 7ff04da00700 1 -- start start 2024-03-28T18:00:28.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.847+0000 7ff04da00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff048102670 0x7ff048199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:28.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.847+0000 7ff04da00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff048107c10 con 0x7ff048102670 2024-03-28T18:00:28.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.848+0000 7ff046a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff048102670 0x7ff048199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:28.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.848+0000 7ff046a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff048102670 0x7ff048199470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:59018/0 (socket says 172.21.15.37:59018) 2024-03-28T18:00:28.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.848+0000 7ff046a00700 1 -- 172.21.15.37:0/2503623043 learned_addr learned my addr 172.21.15.37:0/2503623043 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:28.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.848+0000 7ff046a00700 1 -- 172.21.15.37:0/2503623043 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0481999b0 con 0x7ff048102670 2024-03-28T18:00:28.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.848+0000 7ff046a00700 1 --2- 172.21.15.37:0/2503623043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff048102670 0x7ff048199470 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff038005950 tx=0x7ff038005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:28.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.849+0000 7ff03f400700 1 -- 172.21.15.37:0/2503623043 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff038026070 con 0x7ff048102670 2024-03-28T18:00:28.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.849+0000 7ff03f400700 1 -- 172.21.15.37:0/2503623043 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff03800bbd0 con 0x7ff048102670 2024-03-28T18:00:28.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.849+0000 7ff04da00700 1 -- 172.21.15.37:0/2503623043 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff048199ca0 con 0x7ff048102670 2024-03-28T18:00:28.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.849+0000 7ff03f400700 1 -- 172.21.15.37:0/2503623043 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff038003ad0 con 0x7ff048102670 2024-03-28T18:00:28.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.849+0000 7ff04da00700 1 -- 172.21.15.37:0/2503623043 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff04819c930 con 0x7ff048102670 2024-03-28T18:00:28.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.850+0000 7ff03f400700 1 -- 172.21.15.37:0/2503623043 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7ff038029070 con 0x7ff048102670 2024-03-28T18:00:28.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.850+0000 7ff04da00700 1 -- 172.21.15.37:0/2503623043 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff048108390 con 0x7ff048102670 2024-03-28T18:00:28.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.851+0000 7ff03f400700 1 --2- 172.21.15.37:0/2503623043 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff02c039020 0x7ff02c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:28.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.851+0000 7ff03f400700 1 -- 172.21.15.37:0/2503623043 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff03801e070 con 0x7ff048102670 2024-03-28T18:00:28.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.851+0000 7ff046000700 1 --2- 172.21.15.37:0/2503623043 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff02c039020 0x7ff02c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:28.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.852+0000 7ff046000700 1 --2- 172.21.15.37:0/2503623043 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff02c039020 0x7ff02c03b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff034006fd0 tx=0x7ff034006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:28.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:28.856+0000 7ff03f400700 1 -- 172.21.15.37:0/2503623043 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff038013070 con 0x7ff048102670 2024-03-28T18:00:29.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:29.082+0000 7ff04da00700 1 -- 172.21.15.37:0/2503623043 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7ff04802dc20 con 0x7ff048102670 2024-03-28T18:00:29.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:29.086+0000 7ff03f400700 1 -- 172.21.15.37:0/2503623043 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7ff0380300c0 con 0x7ff048102670 2024-03-28T18:00:29.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:29.089+0000 7ff04da00700 1 -- 172.21.15.37:0/2503623043 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff02c039020 msgr2=0x7ff02c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:29.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:29.089+0000 7ff04da00700 1 --2- 172.21.15.37:0/2503623043 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff02c039020 0x7ff02c03b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff034006fd0 tx=0x7ff034006e40 comp rx=0 tx=0).stop 2024-03-28T18:00:29.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:29.089+0000 7ff04da00700 1 -- 172.21.15.37:0/2503623043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff048102670 msgr2=0x7ff048199470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:29.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:29.089+0000 7ff04da00700 1 --2- 172.21.15.37:0/2503623043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff048102670 0x7ff048199470 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff038005950 tx=0x7ff038005030 comp rx=0 tx=0).stop 2024-03-28T18:00:29.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:29.089+0000 7ff04da00700 1 -- 172.21.15.37:0/2503623043 shutdown_connections 2024-03-28T18:00:29.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:29.089+0000 7ff04da00700 1 --2- 172.21.15.37:0/2503623043 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff02c039020 0x7ff02c03b4e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:29.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:29.089+0000 7ff04da00700 1 --2- 172.21.15.37:0/2503623043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff048102670 0x7ff048199470 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:29.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:29.089+0000 7ff04da00700 1 -- 172.21.15.37:0/2503623043 >> 172.21.15.37:0/2503623043 conn(0x7ff0480fc080 msgr2=0x7ff0480fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:29.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:29.090+0000 7ff04da00700 1 -- 172.21.15.37:0/2503623043 shutdown_connections 2024-03-28T18:00:29.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:29.090+0000 7ff04da00700 1 -- 172.21.15.37:0/2503623043 wait complete. 2024-03-28T18:00:29.847 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:29.847 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:29.847 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi037", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:00:29.847 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:29.847 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:00:29.847 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:29.847 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:29 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2503623043' entity='client.admin' 2024-03-28T18:00:29.847 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:00:29.847 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:29.918 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-28T18:00:29.918 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-28T18:00:30.387 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:00:32.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.359+0000 7fab6be00700 1 -- 172.21.15.37:0/3978695767 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab6c103c20 msgr2=0x7fab6c104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:32.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.359+0000 7fab6be00700 1 --2- 172.21.15.37:0/3978695767 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab6c103c20 0x7fab6c104040 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fab5c009910 tx=0x7fab5c009c20 comp rx=0 tx=0).stop 2024-03-28T18:00:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.359+0000 7fab6be00700 1 -- 172.21.15.37:0/3978695767 shutdown_connections 2024-03-28T18:00:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.359+0000 7fab6be00700 1 --2- 172.21.15.37:0/3978695767 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab6c103c20 0x7fab6c104040 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.359+0000 7fab6be00700 1 -- 172.21.15.37:0/3978695767 >> 172.21.15.37:0/3978695767 conn(0x7fab6c0ff1c0 msgr2=0x7fab6c101600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.359+0000 7fab6be00700 1 -- 172.21.15.37:0/3978695767 shutdown_connections 2024-03-28T18:00:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.359+0000 7fab6be00700 1 -- 172.21.15.37:0/3978695767 wait complete. 2024-03-28T18:00:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.360+0000 7fab6be00700 1 Processor -- start 2024-03-28T18:00:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.361+0000 7fab6be00700 1 -- start start 2024-03-28T18:00:32.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.361+0000 7fab6be00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab6c103c20 0x7fab6c199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:32.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.361+0000 7fab6be00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6c109cb0 con 0x7fab6c103c20 2024-03-28T18:00:32.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.361+0000 7fab6aa00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab6c103c20 0x7fab6c199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:32.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.361+0000 7fab6aa00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab6c103c20 0x7fab6c199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:59028/0 (socket says 172.21.15.37:59028) 2024-03-28T18:00:32.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.361+0000 7fab6aa00700 1 -- 172.21.15.37:0/2784369655 learned_addr learned my addr 172.21.15.37:0/2784369655 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:32.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.362+0000 7fab6aa00700 1 -- 172.21.15.37:0/2784369655 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6c199870 con 0x7fab6c103c20 2024-03-28T18:00:32.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.362+0000 7fab6aa00700 1 --2- 172.21.15.37:0/2784369655 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab6c103c20 0x7fab6c199330 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fab5c006010 tx=0x7fab5c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:32.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.362+0000 7fab70a00700 1 -- 172.21.15.37:0/2784369655 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab5c027070 con 0x7fab6c103c20 2024-03-28T18:00:32.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.362+0000 7fab6be00700 1 -- 172.21.15.37:0/2784369655 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab6c199b00 con 0x7fab6c103c20 2024-03-28T18:00:32.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.362+0000 7fab70a00700 1 -- 172.21.15.37:0/2784369655 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab5c00be40 con 0x7fab6c103c20 2024-03-28T18:00:32.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.363+0000 7fab70a00700 1 -- 172.21.15.37:0/2784369655 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab5c00f460 con 0x7fab6c103c20 2024-03-28T18:00:32.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.363+0000 7fab6be00700 1 -- 172.21.15.37:0/2784369655 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab6c19c7f0 con 0x7fab6c103c20 2024-03-28T18:00:32.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.364+0000 7fab70a00700 1 -- 172.21.15.37:0/2784369655 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fab5c02a070 con 0x7fab6c103c20 2024-03-28T18:00:32.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.364+0000 7fab6be00700 1 -- 172.21.15.37:0/2784369655 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab6c10a430 con 0x7fab6c103c20 2024-03-28T18:00:32.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.364+0000 7fab70a00700 1 --2- 172.21.15.37:0/2784369655 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fab50038fc0 0x7fab5003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:32.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.364+0000 7fab70a00700 1 -- 172.21.15.37:0/2784369655 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fab5c01e030 con 0x7fab6c103c20 2024-03-28T18:00:32.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.365+0000 7fab63e00700 1 --2- 172.21.15.37:0/2784369655 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fab50038fc0 0x7fab5003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:32.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.365+0000 7fab63e00700 1 --2- 172.21.15.37:0/2784369655 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fab50038fc0 0x7fab5003b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fab54006fd0 tx=0x7fab54006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:32.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.369+0000 7fab70a00700 1 -- 172.21.15.37:0/2784369655 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fab5c013070 con 0x7fab6c103c20 2024-03-28T18:00:32.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.600+0000 7fab6be00700 1 -- 172.21.15.37:0/2784369655 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fab6c02db40 con 0x7fab50038fc0 2024-03-28T18:00:32.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.608+0000 7fab70a00700 1 -- 172.21.15.37:0/2784369655 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fab6c02db40 con 0x7fab50038fc0 2024-03-28T18:00:32.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.610+0000 7fab6be00700 1 -- 172.21.15.37:0/2784369655 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fab50038fc0 msgr2=0x7fab5003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:32.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.610+0000 7fab6be00700 1 --2- 172.21.15.37:0/2784369655 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fab50038fc0 0x7fab5003b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fab54006fd0 tx=0x7fab54006e40 comp rx=0 tx=0).stop 2024-03-28T18:00:32.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.610+0000 7fab6be00700 1 -- 172.21.15.37:0/2784369655 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab6c103c20 msgr2=0x7fab6c199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:32.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.610+0000 7fab6be00700 1 --2- 172.21.15.37:0/2784369655 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab6c103c20 0x7fab6c199330 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fab5c006010 tx=0x7fab5c005050 comp rx=0 tx=0).stop 2024-03-28T18:00:32.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.610+0000 7fab6be00700 1 -- 172.21.15.37:0/2784369655 shutdown_connections 2024-03-28T18:00:32.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.610+0000 7fab6be00700 1 --2- 172.21.15.37:0/2784369655 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fab50038fc0 0x7fab5003b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:32.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.610+0000 7fab6be00700 1 --2- 172.21.15.37:0/2784369655 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab6c103c20 0x7fab6c199330 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:32.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.610+0000 7fab6be00700 1 -- 172.21.15.37:0/2784369655 >> 172.21.15.37:0/2784369655 conn(0x7fab6c0ff1c0 msgr2=0x7fab6c0ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:32.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.611+0000 7fab6be00700 1 -- 172.21.15.37:0/2784369655 shutdown_connections 2024-03-28T18:00:32.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:32.611+0000 7fab6be00700 1 -- 172.21.15.37:0/2784369655 wait complete. 2024-03-28T18:00:33.471 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi059 2024-03-28T18:00:33.471 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-28T18:00:33.472 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/etc/ceph/ceph.conf 2024-03-28T18:00:33.490 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-28T18:00:33.490 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:00:33.547 INFO:tasks.cephadm:Adding host smithi059 to orchestrator... 2024-03-28T18:00:33.548 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph orch host add smithi059 2024-03-28T18:00:33.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:33 smithi037 ceph-mon[31460]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:00:33.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:33.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:00:33.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:00:33.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:00:33.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:33 smithi037 ceph-mon[31460]: Updating smithi037:/etc/ceph/ceph.conf 2024-03-28T18:00:33.974 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:00:34.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:34 smithi037 ceph-mon[31460]: Updating smithi037:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:00:35.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:35 smithi037 ceph-mon[31460]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:00:35.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:35 smithi037 ceph-mon[31460]: Updating smithi037:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:00:35.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:35 smithi037 ceph-mon[31460]: Updating smithi037:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:00:35.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:35 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:35.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:35 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:35.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:35 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:36.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.098+0000 7f0720e00700 1 -- 172.21.15.37:0/2700586115 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071c103fe0 msgr2=0x7f071c1043c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:36.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.098+0000 7f0720e00700 1 --2- 172.21.15.37:0/2700586115 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071c103fe0 0x7f071c1043c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f070c009910 tx=0x7f070c009c20 comp rx=0 tx=0).stop 2024-03-28T18:00:36.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.098+0000 7f0720e00700 1 -- 172.21.15.37:0/2700586115 shutdown_connections 2024-03-28T18:00:36.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.098+0000 7f0720e00700 1 --2- 172.21.15.37:0/2700586115 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071c103fe0 0x7f071c1043c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:36.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.098+0000 7f0720e00700 1 -- 172.21.15.37:0/2700586115 >> 172.21.15.37:0/2700586115 conn(0x7f071c0ff870 msgr2=0x7f071c101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:36.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.098+0000 7f0720e00700 1 -- 172.21.15.37:0/2700586115 shutdown_connections 2024-03-28T18:00:36.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.098+0000 7f0720e00700 1 -- 172.21.15.37:0/2700586115 wait complete. 2024-03-28T18:00:36.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.100+0000 7f0720e00700 1 Processor -- start 2024-03-28T18:00:36.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.100+0000 7f0720e00700 1 -- start start 2024-03-28T18:00:36.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.101+0000 7f0720e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071c103fe0 0x7f071c19f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:36.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.101+0000 7f0720e00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071c10e3b0 con 0x7f071c103fe0 2024-03-28T18:00:36.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.101+0000 7f0719e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071c103fe0 0x7f071c19f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:36.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.101+0000 7f0719e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071c103fe0 0x7f071c19f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44504/0 (socket says 172.21.15.37:44504) 2024-03-28T18:00:36.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.101+0000 7f0719e00700 1 -- 172.21.15.37:0/4212361226 learned_addr learned my addr 172.21.15.37:0/4212361226 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:36.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.101+0000 7f0719e00700 1 -- 172.21.15.37:0/4212361226 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f071c19fb60 con 0x7f071c103fe0 2024-03-28T18:00:36.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.102+0000 7f0719e00700 1 --2- 172.21.15.37:0/4212361226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071c103fe0 0x7f071c19f620 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f070c006010 tx=0x7f070c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:36.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.102+0000 7f0712a00700 1 -- 172.21.15.37:0/4212361226 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f070c026070 con 0x7f071c103fe0 2024-03-28T18:00:36.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.102+0000 7f0712a00700 1 -- 172.21.15.37:0/4212361226 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f070c00bbd0 con 0x7f071c103fe0 2024-03-28T18:00:36.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.102+0000 7f0712a00700 1 -- 172.21.15.37:0/4212361226 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f070c003910 con 0x7f071c103fe0 2024-03-28T18:00:36.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.102+0000 7f0720e00700 1 -- 172.21.15.37:0/4212361226 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f071c19fe50 con 0x7f071c103fe0 2024-03-28T18:00:36.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.102+0000 7f0720e00700 1 -- 172.21.15.37:0/4212361226 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f071c1998c0 con 0x7f071c103fe0 2024-03-28T18:00:36.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.103+0000 7f0720e00700 1 -- 172.21.15.37:0/4212361226 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f071c1054c0 con 0x7f071c103fe0 2024-03-28T18:00:36.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.104+0000 7f0712a00700 1 -- 172.21.15.37:0/4212361226 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f070c029070 con 0x7f071c103fe0 2024-03-28T18:00:36.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.104+0000 7f0712a00700 1 --2- 172.21.15.37:0/4212361226 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f0700038fc0 0x7f070003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:36.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.104+0000 7f0712a00700 1 -- 172.21.15.37:0/4212361226 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f070c01e070 con 0x7f071c103fe0 2024-03-28T18:00:36.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.104+0000 7f0719400700 1 --2- 172.21.15.37:0/4212361226 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f0700038fc0 0x7f070003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:36.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.105+0000 7f0719400700 1 --2- 172.21.15.37:0/4212361226 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f0700038fc0 0x7f070003b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0708006fd0 tx=0x7f0708006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:36.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.110+0000 7f0712a00700 1 -- 172.21.15.37:0/4212361226 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f070c013070 con 0x7f071c103fe0 2024-03-28T18:00:36.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:36.343+0000 7f0720e00700 1 -- 172.21.15.37:0/4212361226 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi059", "target": ["mon-mgr", ""]}) v1 -- 0x7f071c19ac20 con 0x7f0700038fc0 2024-03-28T18:00:37.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:37 smithi037 ceph-mon[31460]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:00:37.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:37 smithi037 ceph-mon[31460]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi059", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:00:38.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:38 smithi037 ceph-mon[31460]: Deploying cephadm binary to smithi059 2024-03-28T18:00:39.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:39 smithi037 ceph-mon[31460]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:00:41.318 INFO:teuthology.orchestra.run.smithi037.stdout:Added host 'smithi059' with addr '172.21.15.59' 2024-03-28T18:00:41.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:41.315+0000 7f0712a00700 1 -- 172.21.15.37:0/4212361226 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f071c19ac20 con 0x7f0700038fc0 2024-03-28T18:00:41.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:41.320+0000 7f0720e00700 1 -- 172.21.15.37:0/4212361226 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f0700038fc0 msgr2=0x7f070003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:41.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:41.320+0000 7f0720e00700 1 --2- 172.21.15.37:0/4212361226 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f0700038fc0 0x7f070003b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0708006fd0 tx=0x7f0708006e40 comp rx=0 tx=0).stop 2024-03-28T18:00:41.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:41.320+0000 7f0720e00700 1 -- 172.21.15.37:0/4212361226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071c103fe0 msgr2=0x7f071c19f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:41.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:41.320+0000 7f0720e00700 1 --2- 172.21.15.37:0/4212361226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071c103fe0 0x7f071c19f620 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f070c006010 tx=0x7f070c005030 comp rx=0 tx=0).stop 2024-03-28T18:00:41.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:41.320+0000 7f0720e00700 1 -- 172.21.15.37:0/4212361226 shutdown_connections 2024-03-28T18:00:41.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:41.320+0000 7f0720e00700 1 --2- 172.21.15.37:0/4212361226 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f0700038fc0 0x7f070003b480 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:41.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:41.320+0000 7f0720e00700 1 --2- 172.21.15.37:0/4212361226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071c103fe0 0x7f071c19f620 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:41.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:41.320+0000 7f0720e00700 1 -- 172.21.15.37:0/4212361226 >> 172.21.15.37:0/4212361226 conn(0x7f071c0ff870 msgr2=0x7f071c1059b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:41.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:41.320+0000 7f0720e00700 1 -- 172.21.15.37:0/4212361226 shutdown_connections 2024-03-28T18:00:41.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:41.320+0000 7f0720e00700 1 -- 172.21.15.37:0/4212361226 wait complete. 2024-03-28T18:00:42.042 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:41 smithi037 ceph-mon[31460]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:00:42.042 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:41 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:42.042 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:41 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:00:42.098 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T18:00:42.546 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:00:42.885 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:42 smithi037 ceph-mon[31460]: Added host smithi059 2024-03-28T18:00:42.886 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:42 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:42.886 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:42 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:43.748 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:43 smithi037 ceph-mon[31460]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:00:44.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.544+0000 7f69f4400700 1 -- 172.21.15.37:0/4259529243 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69ec101dc0 msgr2=0x7f69ec1021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:44.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.544+0000 7f69f4400700 1 --2- 172.21.15.37:0/4259529243 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69ec101dc0 0x7f69ec1021a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f69e0009910 tx=0x7f69e0009c20 comp rx=0 tx=0).stop 2024-03-28T18:00:44.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.545+0000 7f69f4400700 1 -- 172.21.15.37:0/4259529243 shutdown_connections 2024-03-28T18:00:44.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.545+0000 7f69f4400700 1 --2- 172.21.15.37:0/4259529243 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69ec101dc0 0x7f69ec1021a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:44.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.545+0000 7f69f4400700 1 -- 172.21.15.37:0/4259529243 >> 172.21.15.37:0/4259529243 conn(0x7f69ec0fd670 msgr2=0x7f69ec0ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:44.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.545+0000 7f69f4400700 1 -- 172.21.15.37:0/4259529243 shutdown_connections 2024-03-28T18:00:44.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.545+0000 7f69f4400700 1 -- 172.21.15.37:0/4259529243 wait complete. 2024-03-28T18:00:44.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.546+0000 7f69f4400700 1 Processor -- start 2024-03-28T18:00:44.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.546+0000 7f69f4400700 1 -- start start 2024-03-28T18:00:44.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.546+0000 7f69f4400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69ec101dc0 0x7f69ec19e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:44.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.546+0000 7f69f4400700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec10c190 con 0x7f69ec101dc0 2024-03-28T18:00:44.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.547+0000 7f69f1a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69ec101dc0 0x7f69ec19e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:44.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.547+0000 7f69f1a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69ec101dc0 0x7f69ec19e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52008/0 (socket says 172.21.15.37:52008) 2024-03-28T18:00:44.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.547+0000 7f69f1a00700 1 -- 172.21.15.37:0/1646999123 learned_addr learned my addr 172.21.15.37:0/1646999123 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:44.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.547+0000 7f69f1a00700 1 -- 172.21.15.37:0/1646999123 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ec19e950 con 0x7f69ec101dc0 2024-03-28T18:00:44.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.547+0000 7f69f1a00700 1 --2- 172.21.15.37:0/1646999123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69ec101dc0 0x7f69ec19e410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f69e0006010 tx=0x7f69e0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:44.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.548+0000 7f69ea000700 1 -- 172.21.15.37:0/1646999123 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69e0026070 con 0x7f69ec101dc0 2024-03-28T18:00:44.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.548+0000 7f69ea000700 1 -- 172.21.15.37:0/1646999123 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69e000bbd0 con 0x7f69ec101dc0 2024-03-28T18:00:44.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.548+0000 7f69ea000700 1 -- 172.21.15.37:0/1646999123 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69e0003910 con 0x7f69ec101dc0 2024-03-28T18:00:44.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.548+0000 7f69f4400700 1 -- 172.21.15.37:0/1646999123 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69ec19ec40 con 0x7f69ec101dc0 2024-03-28T18:00:44.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.548+0000 7f69f4400700 1 -- 172.21.15.37:0/1646999123 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69ec1a25b0 con 0x7f69ec101dc0 2024-03-28T18:00:44.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.549+0000 7f69f4400700 1 -- 172.21.15.37:0/1646999123 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69ec1032a0 con 0x7f69ec101dc0 2024-03-28T18:00:44.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.549+0000 7f69ea000700 1 -- 172.21.15.37:0/1646999123 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f69e0029070 con 0x7f69ec101dc0 2024-03-28T18:00:44.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.550+0000 7f69ea000700 1 --2- 172.21.15.37:0/1646999123 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f69d4039020 0x7f69d403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:44.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.550+0000 7f69ea000700 1 -- 172.21.15.37:0/1646999123 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f69e001e070 con 0x7f69ec101dc0 2024-03-28T18:00:44.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.550+0000 7f69f1000700 1 --2- 172.21.15.37:0/1646999123 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f69d4039020 0x7f69d403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:44.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.551+0000 7f69f1000700 1 --2- 172.21.15.37:0/1646999123 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f69d4039020 0x7f69d403b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f69d8006fd0 tx=0x7f69d8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:44.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.555+0000 7f69ea000700 1 -- 172.21.15.37:0/1646999123 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f69e0013070 con 0x7f69ec101dc0 2024-03-28T18:00:44.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.795+0000 7f69f4400700 1 -- 172.21.15.37:0/1646999123 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f69ec19f100 con 0x7f69d4039020 2024-03-28T18:00:44.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.798+0000 7f69ea000700 1 -- 172.21.15.37:0/1646999123 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f69ec19f100 con 0x7f69d4039020 2024-03-28T18:00:44.802 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:00:44.802 INFO:teuthology.orchestra.run.smithi037.stdout:[{"addr": "172.21.15.37", "hostname": "smithi037", "labels": [], "status": ""}, {"addr": "172.21.15.59", "hostname": "smithi059", "labels": [], "status": ""}] 2024-03-28T18:00:44.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.801+0000 7f69f4400700 1 -- 172.21.15.37:0/1646999123 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f69d4039020 msgr2=0x7f69d403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:44.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.801+0000 7f69f4400700 1 --2- 172.21.15.37:0/1646999123 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f69d4039020 0x7f69d403b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f69d8006fd0 tx=0x7f69d8006e40 comp rx=0 tx=0).stop 2024-03-28T18:00:44.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.801+0000 7f69f4400700 1 -- 172.21.15.37:0/1646999123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69ec101dc0 msgr2=0x7f69ec19e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.801+0000 7f69f4400700 1 --2- 172.21.15.37:0/1646999123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69ec101dc0 0x7f69ec19e410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f69e0006010 tx=0x7f69e0005030 comp rx=0 tx=0).stop 2024-03-28T18:00:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.801+0000 7f69f4400700 1 -- 172.21.15.37:0/1646999123 shutdown_connections 2024-03-28T18:00:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.801+0000 7f69f4400700 1 --2- 172.21.15.37:0/1646999123 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f69d4039020 0x7f69d403b4e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.801+0000 7f69f4400700 1 --2- 172.21.15.37:0/1646999123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69ec101dc0 0x7f69ec19e410 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.801+0000 7f69f4400700 1 -- 172.21.15.37:0/1646999123 >> 172.21.15.37:0/1646999123 conn(0x7f69ec0fd670 msgr2=0x7f69ec0743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.802+0000 7f69f4400700 1 -- 172.21.15.37:0/1646999123 shutdown_connections 2024-03-28T18:00:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:44.802+0000 7f69f4400700 1 -- 172.21.15.37:0/1646999123 wait complete. 2024-03-28T18:00:45.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:45 smithi037 ceph-mon[31460]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:00:45.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:45 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:45.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:45 smithi037 ceph-mon[31460]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:00:45.540 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi170 2024-03-28T18:00:45.540 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T18:00:45.540 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/etc/ceph/ceph.conf 2024-03-28T18:00:45.558 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T18:00:45.558 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:00:45.616 INFO:tasks.cephadm:Adding host smithi170 to orchestrator... 2024-03-28T18:00:45.616 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph orch host add smithi170 2024-03-28T18:00:46.064 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:00:47.257 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:47 smithi037 ceph-mon[31460]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:00:48.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.175+0000 7fedd8600700 1 -- 172.21.15.37:0/1208143306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fedd0101df0 msgr2=0x7fedd01021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:48.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.175+0000 7fedd8600700 1 --2- 172.21.15.37:0/1208143306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fedd0101df0 0x7fedd01021d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fedc400b0a0 tx=0x7fedc400b3b0 comp rx=0 tx=0).stop 2024-03-28T18:00:48.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.176+0000 7fedd8600700 1 -- 172.21.15.37:0/1208143306 shutdown_connections 2024-03-28T18:00:48.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.176+0000 7fedd8600700 1 --2- 172.21.15.37:0/1208143306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fedd0101df0 0x7fedd01021d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:48.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.176+0000 7fedd8600700 1 -- 172.21.15.37:0/1208143306 >> 172.21.15.37:0/1208143306 conn(0x7fedd00fd6a0 msgr2=0x7fedd00ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:48.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.176+0000 7fedd8600700 1 -- 172.21.15.37:0/1208143306 shutdown_connections 2024-03-28T18:00:48.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.176+0000 7fedd8600700 1 -- 172.21.15.37:0/1208143306 wait complete. 2024-03-28T18:00:48.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.177+0000 7fedd8600700 1 Processor -- start 2024-03-28T18:00:48.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.177+0000 7fedd8600700 1 -- start start 2024-03-28T18:00:48.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.178+0000 7fedd8600700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fedd0101df0 0x7fedd019f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:48.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.178+0000 7fedd8600700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd010c1c0 con 0x7fedd0101df0 2024-03-28T18:00:48.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.178+0000 7fedd5c00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fedd0101df0 0x7fedd019f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:48.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.178+0000 7fedd5c00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fedd0101df0 0x7fedd019f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52014/0 (socket says 172.21.15.37:52014) 2024-03-28T18:00:48.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.178+0000 7fedd5c00700 1 -- 172.21.15.37:0/402555955 learned_addr learned my addr 172.21.15.37:0/402555955 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:48.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.178+0000 7fedd5c00700 1 -- 172.21.15.37:0/402555955 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd019f980 con 0x7fedd0101df0 2024-03-28T18:00:48.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.179+0000 7fedd5c00700 1 --2- 172.21.15.37:0/402555955 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fedd0101df0 0x7fedd019f440 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fedc40093e0 tx=0x7fedc4003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:48.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.179+0000 7fedce000700 1 -- 172.21.15.37:0/402555955 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fedc4017040 con 0x7fedd0101df0 2024-03-28T18:00:48.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.179+0000 7fedce000700 1 -- 172.21.15.37:0/402555955 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedc4022070 con 0x7fedd0101df0 2024-03-28T18:00:48.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.179+0000 7fedce000700 1 -- 172.21.15.37:0/402555955 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fedc401d440 con 0x7fedd0101df0 2024-03-28T18:00:48.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.179+0000 7fedd8600700 1 -- 172.21.15.37:0/402555955 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd019fc10 con 0x7fedd0101df0 2024-03-28T18:00:48.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.179+0000 7fedd8600700 1 -- 172.21.15.37:0/402555955 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd0199600 con 0x7fedd0101df0 2024-03-28T18:00:48.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.180+0000 7fedce000700 1 -- 172.21.15.37:0/402555955 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fedc4004000 con 0x7fedd0101df0 2024-03-28T18:00:48.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.181+0000 7fedd8600700 1 -- 172.21.15.37:0/402555955 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd01032d0 con 0x7fedd0101df0 2024-03-28T18:00:48.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.181+0000 7fedce000700 1 --2- 172.21.15.37:0/402555955 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fedb8039070 0x7fedb803b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:48.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.181+0000 7fedce000700 1 -- 172.21.15.37:0/402555955 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fedc400f050 con 0x7fedd0101df0 2024-03-28T18:00:48.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.181+0000 7fedd5200700 1 --2- 172.21.15.37:0/402555955 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fedb8039070 0x7fedb803b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:48.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.182+0000 7fedd5200700 1 --2- 172.21.15.37:0/402555955 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fedb8039070 0x7fedb803b530 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fedbc006fd0 tx=0x7fedbc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:48.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.187+0000 7fedce000700 1 -- 172.21.15.37:0/402555955 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fedc4007e20 con 0x7fedd0101df0 2024-03-28T18:00:48.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:48 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:48.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:48 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:48.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:48 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:48.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:48 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:48.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:48 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi059", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:00:48.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:48 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:00:48.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:48 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:00:48.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:48 smithi037 ceph-mon[31460]: Updating smithi059:/etc/ceph/ceph.conf 2024-03-28T18:00:48.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:48 smithi037 ceph-mon[31460]: Updating smithi059:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:00:48.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:48.426+0000 7fedd8600700 1 -- 172.21.15.37:0/402555955 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi170", "target": ["mon-mgr", ""]}) v1 -- 0x7fedd019ab30 con 0x7fedb8039070 2024-03-28T18:00:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:49 smithi037 ceph-mon[31460]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:00:50.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:49 smithi037 ceph-mon[31460]: Updating smithi059:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:00:50.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:49 smithi037 ceph-mon[31460]: Updating smithi059:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:00:50.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:49 smithi037 ceph-mon[31460]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi170", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:00:50.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:49 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:50.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:49 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:50.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:49 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:51.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:50 smithi037 ceph-mon[31460]: Deploying cephadm binary to smithi170 2024-03-28T18:00:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:51 smithi037 ceph-mon[31460]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:00:53.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:53.341+0000 7fedce000700 1 -- 172.21.15.37:0/402555955 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fedd019ab30 con 0x7fedb8039070 2024-03-28T18:00:53.342 INFO:teuthology.orchestra.run.smithi037.stdout:Added host 'smithi170' with addr '172.21.15.170' 2024-03-28T18:00:53.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:53.344+0000 7fedd8600700 1 -- 172.21.15.37:0/402555955 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fedb8039070 msgr2=0x7fedb803b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:53.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:53.344+0000 7fedd8600700 1 --2- 172.21.15.37:0/402555955 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fedb8039070 0x7fedb803b530 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fedbc006fd0 tx=0x7fedbc006e40 comp rx=0 tx=0).stop 2024-03-28T18:00:53.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:53.344+0000 7fedd8600700 1 -- 172.21.15.37:0/402555955 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fedd0101df0 msgr2=0x7fedd019f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:53.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:53.344+0000 7fedd8600700 1 --2- 172.21.15.37:0/402555955 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fedd0101df0 0x7fedd019f440 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fedc40093e0 tx=0x7fedc4003ce0 comp rx=0 tx=0).stop 2024-03-28T18:00:53.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:53.345+0000 7fedd8600700 1 -- 172.21.15.37:0/402555955 shutdown_connections 2024-03-28T18:00:53.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:53.345+0000 7fedd8600700 1 --2- 172.21.15.37:0/402555955 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fedb8039070 0x7fedb803b530 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:53.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:53.345+0000 7fedd8600700 1 --2- 172.21.15.37:0/402555955 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fedd0101df0 0x7fedd019f440 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:53.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:53.345+0000 7fedd8600700 1 -- 172.21.15.37:0/402555955 >> 172.21.15.37:0/402555955 conn(0x7fedd00fd6a0 msgr2=0x7fedd0075d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:53.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:53.345+0000 7fedd8600700 1 -- 172.21.15.37:0/402555955 shutdown_connections 2024-03-28T18:00:53.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:53.345+0000 7fedd8600700 1 -- 172.21.15.37:0/402555955 wait complete. 2024-03-28T18:00:54.035 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:53 smithi037 ceph-mon[31460]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:00:54.035 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:54.035 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:00:54.087 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T18:00:54.545 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:00:54.981 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:54 smithi037 ceph-mon[31460]: Added host smithi170 2024-03-28T18:00:54.981 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:54 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:54.981 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:54 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:56.018 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:55 smithi037 ceph-mon[31460]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:00:56.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.479+0000 7f190b800700 1 -- 172.21.15.37:0/3081513814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1904102d20 msgr2=0x7f1904103100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:56.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.479+0000 7f190b800700 1 --2- 172.21.15.37:0/3081513814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1904102d20 0x7f1904103100 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f18f8009910 tx=0x7f18f8009c20 comp rx=0 tx=0).stop 2024-03-28T18:00:56.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.479+0000 7f190b800700 1 -- 172.21.15.37:0/3081513814 shutdown_connections 2024-03-28T18:00:56.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.479+0000 7f190b800700 1 --2- 172.21.15.37:0/3081513814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1904102d20 0x7f1904103100 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:56.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.479+0000 7f190b800700 1 -- 172.21.15.37:0/3081513814 >> 172.21.15.37:0/3081513814 conn(0x7f19040776d0 msgr2=0x7f19040fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:56.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.479+0000 7f190b800700 1 -- 172.21.15.37:0/3081513814 shutdown_connections 2024-03-28T18:00:56.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.479+0000 7f190b800700 1 -- 172.21.15.37:0/3081513814 wait complete. 2024-03-28T18:00:56.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.480+0000 7f190b800700 1 Processor -- start 2024-03-28T18:00:56.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.481+0000 7f190b800700 1 -- start start 2024-03-28T18:00:56.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.481+0000 7f190b800700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1904102d20 0x7f190419f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:56.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.481+0000 7f190b800700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190410c210 con 0x7f1904102d20 2024-03-28T18:00:56.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.481+0000 7f1908e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1904102d20 0x7f190419f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:56.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.481+0000 7f1908e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1904102d20 0x7f190419f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47720/0 (socket says 172.21.15.37:47720) 2024-03-28T18:00:56.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.481+0000 7f1908e00700 1 -- 172.21.15.37:0/4218210110 learned_addr learned my addr 172.21.15.37:0/4218210110 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:56.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.482+0000 7f1908e00700 1 -- 172.21.15.37:0/4218210110 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190419fbb0 con 0x7f1904102d20 2024-03-28T18:00:56.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.482+0000 7f1908e00700 1 --2- 172.21.15.37:0/4218210110 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1904102d20 0x7f190419f670 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f18f8006010 tx=0x7f18f8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:56.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.483+0000 7f1901600700 1 -- 172.21.15.37:0/4218210110 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18f8026070 con 0x7f1904102d20 2024-03-28T18:00:56.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.483+0000 7f1901600700 1 -- 172.21.15.37:0/4218210110 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18f800bc30 con 0x7f1904102d20 2024-03-28T18:00:56.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.483+0000 7f1901600700 1 -- 172.21.15.37:0/4218210110 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18f80039a0 con 0x7f1904102d20 2024-03-28T18:00:56.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.483+0000 7f190b800700 1 -- 172.21.15.37:0/4218210110 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f190419fe40 con 0x7f1904102d20 2024-03-28T18:00:56.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.483+0000 7f190b800700 1 -- 172.21.15.37:0/4218210110 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1904199740 con 0x7f1904102d20 2024-03-28T18:00:56.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.484+0000 7f190b800700 1 -- 172.21.15.37:0/4218210110 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1904069e60 con 0x7f1904102d20 2024-03-28T18:00:56.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.485+0000 7f1901600700 1 -- 172.21.15.37:0/4218210110 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f18f8029070 con 0x7f1904102d20 2024-03-28T18:00:56.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.485+0000 7f1901600700 1 --2- 172.21.15.37:0/4218210110 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f18ec038fc0 0x7f18ec03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:56.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.486+0000 7f1901600700 1 -- 172.21.15.37:0/4218210110 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f18f801e070 con 0x7f1904102d20 2024-03-28T18:00:56.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.486+0000 7f1903e00700 1 --2- 172.21.15.37:0/4218210110 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f18ec038fc0 0x7f18ec03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:56.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.487+0000 7f1903e00700 1 --2- 172.21.15.37:0/4218210110 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f18ec038fc0 0x7f18ec03b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f18f0006fd0 tx=0x7f18f0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:56.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.492+0000 7f1901600700 1 -- 172.21.15.37:0/4218210110 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f18f8013070 con 0x7f1904102d20 2024-03-28T18:00:56.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.729+0000 7f190b800700 1 -- 172.21.15.37:0/4218210110 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f19040637c0 con 0x7f18ec038fc0 2024-03-28T18:00:56.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.732+0000 7f1901600700 1 -- 172.21.15.37:0/4218210110 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+239 (secure 0 0 0) 0x7f19040637c0 con 0x7f18ec038fc0 2024-03-28T18:00:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:00:56.732 INFO:teuthology.orchestra.run.smithi037.stdout:[{"addr": "172.21.15.37", "hostname": "smithi037", "labels": [], "status": ""}, {"addr": "172.21.15.59", "hostname": "smithi059", "labels": [], "status": ""}, {"addr": "172.21.15.170", "hostname": "smithi170", "labels": [], "status": ""}] 2024-03-28T18:00:56.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.734+0000 7f190b800700 1 -- 172.21.15.37:0/4218210110 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f18ec038fc0 msgr2=0x7f18ec03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:56.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.734+0000 7f190b800700 1 --2- 172.21.15.37:0/4218210110 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f18ec038fc0 0x7f18ec03b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f18f0006fd0 tx=0x7f18f0006e40 comp rx=0 tx=0).stop 2024-03-28T18:00:56.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.734+0000 7f190b800700 1 -- 172.21.15.37:0/4218210110 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1904102d20 msgr2=0x7f190419f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:56.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.734+0000 7f190b800700 1 --2- 172.21.15.37:0/4218210110 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1904102d20 0x7f190419f670 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f18f8006010 tx=0x7f18f8005030 comp rx=0 tx=0).stop 2024-03-28T18:00:56.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.735+0000 7f190b800700 1 -- 172.21.15.37:0/4218210110 shutdown_connections 2024-03-28T18:00:56.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.735+0000 7f190b800700 1 --2- 172.21.15.37:0/4218210110 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f18ec038fc0 0x7f18ec03b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:56.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.735+0000 7f190b800700 1 --2- 172.21.15.37:0/4218210110 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1904102d20 0x7f190419f670 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:56.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.735+0000 7f190b800700 1 -- 172.21.15.37:0/4218210110 >> 172.21.15.37:0/4218210110 conn(0x7f19040776d0 msgr2=0x7f1904107ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:56.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.735+0000 7f190b800700 1 -- 172.21.15.37:0/4218210110 shutdown_connections 2024-03-28T18:00:56.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:56.736+0000 7f190b800700 1 -- 172.21.15.37:0/4218210110 wait complete. 2024-03-28T18:00:57.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:57 smithi037 ceph-mon[31460]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:00:57.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:00:57.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:57 smithi037 ceph-mon[31460]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:00:57.485 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-28T18:00:57.485 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-28T18:00:57.942 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:00:59.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:00:59 smithi037 ceph-mon[31460]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:00:59.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.917+0000 7f6bfc000700 1 -- 172.21.15.37:0/838814365 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6bf4103ff0 msgr2=0x7f6bf41043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:59.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.917+0000 7f6bfc000700 1 --2- 172.21.15.37:0/838814365 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6bf4103ff0 0x7f6bf41043d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6be800b0a0 tx=0x7f6be800b3b0 comp rx=0 tx=0).stop 2024-03-28T18:00:59.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.917+0000 7f6bfc000700 1 -- 172.21.15.37:0/838814365 shutdown_connections 2024-03-28T18:00:59.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.917+0000 7f6bfc000700 1 --2- 172.21.15.37:0/838814365 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6bf4103ff0 0x7f6bf41043d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:59.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.917+0000 7f6bfc000700 1 -- 172.21.15.37:0/838814365 >> 172.21.15.37:0/838814365 conn(0x7f6bf40ff880 msgr2=0x7f6bf4101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:59.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.917+0000 7f6bfc000700 1 -- 172.21.15.37:0/838814365 shutdown_connections 2024-03-28T18:00:59.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.918+0000 7f6bfc000700 1 -- 172.21.15.37:0/838814365 wait complete. 2024-03-28T18:00:59.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.919+0000 7f6bfc000700 1 Processor -- start 2024-03-28T18:00:59.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.919+0000 7f6bfc000700 1 -- start start 2024-03-28T18:00:59.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.919+0000 7f6bfc000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6bf4103ff0 0x7f6bf419f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:59.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.919+0000 7f6bfc000700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf410e3c0 con 0x7f6bf4103ff0 2024-03-28T18:00:59.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.920+0000 7f6bf9600700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6bf4103ff0 0x7f6bf419f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:59.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.920+0000 7f6bf9600700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6bf4103ff0 0x7f6bf419f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47746/0 (socket says 172.21.15.37:47746) 2024-03-28T18:00:59.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.920+0000 7f6bf9600700 1 -- 172.21.15.37:0/1513499265 learned_addr learned my addr 172.21.15.37:0/1513499265 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:00:59.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.920+0000 7f6bf9600700 1 -- 172.21.15.37:0/1513499265 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf419fac0 con 0x7f6bf4103ff0 2024-03-28T18:00:59.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.920+0000 7f6bf9600700 1 --2- 172.21.15.37:0/1513499265 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6bf4103ff0 0x7f6bf419f580 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6be80093e0 tx=0x7f6be8003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:59.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.921+0000 7f6bf2000700 1 -- 172.21.15.37:0/1513499265 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6be8017040 con 0x7f6bf4103ff0 2024-03-28T18:00:59.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.921+0000 7f6bf2000700 1 -- 172.21.15.37:0/1513499265 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6be8022070 con 0x7f6bf4103ff0 2024-03-28T18:00:59.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.921+0000 7f6bfc000700 1 -- 172.21.15.37:0/1513499265 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf419fd50 con 0x7f6bf4103ff0 2024-03-28T18:00:59.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.921+0000 7f6bf2000700 1 -- 172.21.15.37:0/1513499265 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6be801d440 con 0x7f6bf4103ff0 2024-03-28T18:00:59.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.921+0000 7f6bfc000700 1 -- 172.21.15.37:0/1513499265 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf4199650 con 0x7f6bf4103ff0 2024-03-28T18:00:59.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.922+0000 7f6bfc000700 1 -- 172.21.15.37:0/1513499265 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf41054d0 con 0x7f6bf4103ff0 2024-03-28T18:00:59.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.923+0000 7f6bf2000700 1 -- 172.21.15.37:0/1513499265 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f6be8004000 con 0x7f6bf4103ff0 2024-03-28T18:00:59.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.923+0000 7f6bf2000700 1 --2- 172.21.15.37:0/1513499265 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f6bdc039070 0x7f6bdc03b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:59.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.923+0000 7f6bf2000700 1 -- 172.21.15.37:0/1513499265 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6be800f050 con 0x7f6bf4103ff0 2024-03-28T18:00:59.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.924+0000 7f6bf8c00700 1 --2- 172.21.15.37:0/1513499265 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f6bdc039070 0x7f6bdc03b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:59.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.924+0000 7f6bf8c00700 1 --2- 172.21.15.37:0/1513499265 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f6bdc039070 0x7f6bdc03b530 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6be0006fd0 tx=0x7f6be0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:59.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:00:59.929+0000 7f6bf2000700 1 -- 172.21.15.37:0/1513499265 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6be8007e20 con 0x7f6bf4103ff0 2024-03-28T18:01:00.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:00.163+0000 7f6bfc000700 1 -- 172.21.15.37:0/1513499265 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f6bf419ab80 con 0x7f6bf4103ff0 2024-03-28T18:01:00.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:00.335+0000 7f6bf2000700 1 -- 172.21.15.37:0/1513499265 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6be8027460 con 0x7f6bf4103ff0 2024-03-28T18:01:00.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:00.339+0000 7f6bfc000700 1 -- 172.21.15.37:0/1513499265 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f6bdc039070 msgr2=0x7f6bdc03b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:00.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:00.339+0000 7f6bfc000700 1 --2- 172.21.15.37:0/1513499265 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f6bdc039070 0x7f6bdc03b530 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6be0006fd0 tx=0x7f6be0006e40 comp rx=0 tx=0).stop 2024-03-28T18:01:00.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:00.339+0000 7f6bfc000700 1 -- 172.21.15.37:0/1513499265 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6bf4103ff0 msgr2=0x7f6bf419f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:00.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:00.339+0000 7f6bfc000700 1 --2- 172.21.15.37:0/1513499265 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6bf4103ff0 0x7f6bf419f580 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6be80093e0 tx=0x7f6be8003ce0 comp rx=0 tx=0).stop 2024-03-28T18:01:00.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:00.339+0000 7f6bfc000700 1 -- 172.21.15.37:0/1513499265 shutdown_connections 2024-03-28T18:01:00.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:00.339+0000 7f6bfc000700 1 --2- 172.21.15.37:0/1513499265 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f6bdc039070 0x7f6bdc03b530 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:00.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:00.339+0000 7f6bfc000700 1 --2- 172.21.15.37:0/1513499265 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6bf4103ff0 0x7f6bf419f580 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:00.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:00.339+0000 7f6bfc000700 1 -- 172.21.15.37:0/1513499265 >> 172.21.15.37:0/1513499265 conn(0x7f6bf40ff880 msgr2=0x7f6bf4106cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:00.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:00.340+0000 7f6bfc000700 1 -- 172.21.15.37:0/1513499265 shutdown_connections 2024-03-28T18:01:00.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:00.340+0000 7f6bfc000700 1 -- 172.21.15.37:0/1513499265 wait complete. 2024-03-28T18:01:00.341 INFO:teuthology.orchestra.run.smithi037.stderr:adjusted tunables profile to default 2024-03-28T18:01:00.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:00 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:00.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:00 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:00.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:00 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:00.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:00 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:00.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:00 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:01:00.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:00 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:00.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:00 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:01:00.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:00 smithi037 ceph-mon[31460]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-28T18:01:00.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:00 smithi037 ceph-mon[31460]: Updating smithi170:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:01:00.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:00 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1513499265' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T18:01:01.099 INFO:tasks.cephadm:Adding mon.a on smithi037 2024-03-28T18:01:01.099 INFO:tasks.cephadm:Adding mon.b on smithi059 2024-03-28T18:01:01.099 INFO:tasks.cephadm:Adding mon.c on smithi170 2024-03-28T18:01:01.099 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi037:172.21.15.37=a;smithi059:172.21.15.59=b;smithi170:172.21.15.170=c' 2024-03-28T18:01:01.329 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:01:01.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:01 smithi037 ceph-mon[31460]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:01.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:01 smithi037 ceph-mon[31460]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:01:01.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:01 smithi037 ceph-mon[31460]: Updating smithi170:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:01:01.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:01 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1513499265' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T18:01:01.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:01 smithi037 ceph-mon[31460]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:01:01.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:01 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:01.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:01 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:01.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:01 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:02.789 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.788+0000 7fc982eda700 1 -- 172.21.15.170:0/1560607686 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc97c106a60 msgr2=0x7fc97c106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:02.789 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.788+0000 7fc982eda700 1 --2- 172.21.15.170:0/1560607686 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc97c106a60 0x7fc97c106e40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc97800b0a0 tx=0x7fc97800b3b0 comp rx=0 tx=0).stop 2024-03-28T18:01:02.789 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.788+0000 7fc982eda700 1 -- 172.21.15.170:0/1560607686 shutdown_connections 2024-03-28T18:01:02.789 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.788+0000 7fc982eda700 1 --2- 172.21.15.170:0/1560607686 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc97c106a60 0x7fc97c106e40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:02.789 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.788+0000 7fc982eda700 1 -- 172.21.15.170:0/1560607686 >> 172.21.15.170:0/1560607686 conn(0x7fc97c0776d0 msgr2=0x7fc97c0fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:02.789 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.788+0000 7fc982eda700 1 -- 172.21.15.170:0/1560607686 shutdown_connections 2024-03-28T18:01:02.789 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.788+0000 7fc982eda700 1 -- 172.21.15.170:0/1560607686 wait complete. 2024-03-28T18:01:02.789 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.788+0000 7fc982eda700 1 Processor -- start 2024-03-28T18:01:02.790 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.788+0000 7fc982eda700 1 -- start start 2024-03-28T18:01:02.790 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.789+0000 7fc982eda700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc97c106a60 0x7fc97c1975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:02.790 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.789+0000 7fc982eda700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c10b0a0 con 0x7fc97c106a60 2024-03-28T18:01:02.790 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.789+0000 7fc980c76700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc97c106a60 0x7fc97c1975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:02.790 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.789+0000 7fc980c76700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc97c106a60 0x7fc97c1975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.170:47034/0 (socket says 172.21.15.170:47034) 2024-03-28T18:01:02.791 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.789+0000 7fc980c76700 1 -- 172.21.15.170:0/3435173576 learned_addr learned my addr 172.21.15.170:0/3435173576 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T18:01:02.791 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.790+0000 7fc980c76700 1 -- 172.21.15.170:0/3435173576 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97c197b20 con 0x7fc97c106a60 2024-03-28T18:01:02.791 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.790+0000 7fc980c76700 1 --2- 172.21.15.170:0/3435173576 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc97c106a60 0x7fc97c1975e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc9780093e0 tx=0x7fc978003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:02.791 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.791+0000 7fc971ffb700 1 -- 172.21.15.170:0/3435173576 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc978017040 con 0x7fc97c106a60 2024-03-28T18:01:02.792 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.791+0000 7fc982eda700 1 -- 172.21.15.170:0/3435173576 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97c197db0 con 0x7fc97c106a60 2024-03-28T18:01:02.792 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.791+0000 7fc982eda700 1 -- 172.21.15.170:0/3435173576 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97c19beb0 con 0x7fc97c106a60 2024-03-28T18:01:02.792 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.791+0000 7fc971ffb700 1 -- 172.21.15.170:0/3435173576 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc978022070 con 0x7fc97c106a60 2024-03-28T18:01:02.792 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.791+0000 7fc971ffb700 1 -- 172.21.15.170:0/3435173576 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc978004000 con 0x7fc97c106a60 2024-03-28T18:01:02.792 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.792+0000 7fc982eda700 1 -- 172.21.15.170:0/3435173576 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc968005330 con 0x7fc97c106a60 2024-03-28T18:01:02.793 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.792+0000 7fc971ffb700 1 -- 172.21.15.170:0/3435173576 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fc978027460 con 0x7fc97c106a60 2024-03-28T18:01:02.793 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.792+0000 7fc971ffb700 1 --2- 172.21.15.170:0/3435173576 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc964039010 0x7fc96403b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:02.793 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.793+0000 7fc971ffb700 1 -- 172.21.15.170:0/3435173576 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc97800f050 con 0x7fc97c106a60 2024-03-28T18:01:02.793 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.793+0000 7fc973fff700 1 --2- 172.21.15.170:0/3435173576 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc964039010 0x7fc96403b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:02.794 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.794+0000 7fc973fff700 1 --2- 172.21.15.170:0/3435173576 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc964039010 0x7fc96403b4d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc96c006fd0 tx=0x7fc96c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:02.808 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:02.808+0000 7fc971ffb700 1 -- 172.21.15.170:0/3435173576 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc978007e20 con 0x7fc97c106a60 2024-03-28T18:01:03.015 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:03.015+0000 7fc982eda700 1 -- 172.21.15.170:0/3435173576 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi037:172.21.15.37=a;smithi059:172.21.15.59=b;smithi170:172.21.15.170=c", "target": ["mon-mgr", ""]}) v1 -- 0x7fc968000c80 con 0x7fc964039010 2024-03-28T18:01:03.031 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled mon update... 2024-03-28T18:01:03.031 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:03.031+0000 7fc971ffb700 1 -- 172.21.15.170:0/3435173576 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc968000c80 con 0x7fc964039010 2024-03-28T18:01:03.033 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:03.033+0000 7fc982eda700 1 -- 172.21.15.170:0/3435173576 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc964039010 msgr2=0x7fc96403b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:03.034 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:03.033+0000 7fc982eda700 1 --2- 172.21.15.170:0/3435173576 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc964039010 0x7fc96403b4d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc96c006fd0 tx=0x7fc96c006e40 comp rx=0 tx=0).stop 2024-03-28T18:01:03.034 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:03.033+0000 7fc982eda700 1 -- 172.21.15.170:0/3435173576 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc97c106a60 msgr2=0x7fc97c1975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:03.034 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:03.033+0000 7fc982eda700 1 --2- 172.21.15.170:0/3435173576 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc97c106a60 0x7fc97c1975e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc9780093e0 tx=0x7fc978003ce0 comp rx=0 tx=0).stop 2024-03-28T18:01:03.034 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:03.033+0000 7fc982eda700 1 -- 172.21.15.170:0/3435173576 shutdown_connections 2024-03-28T18:01:03.034 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:03.033+0000 7fc982eda700 1 --2- 172.21.15.170:0/3435173576 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc964039010 0x7fc96403b4d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:03.034 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:03.033+0000 7fc982eda700 1 --2- 172.21.15.170:0/3435173576 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc97c106a60 0x7fc97c1975e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:03.034 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:03.033+0000 7fc982eda700 1 -- 172.21.15.170:0/3435173576 >> 172.21.15.170:0/3435173576 conn(0x7fc97c0776d0 msgr2=0x7fc97c104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:03.034 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:03.033+0000 7fc982eda700 1 -- 172.21.15.170:0/3435173576 shutdown_connections 2024-03-28T18:01:03.034 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:03.033+0000 7fc982eda700 1 -- 172.21.15.170:0/3435173576 wait complete. 2024-03-28T18:01:03.615 DEBUG:teuthology.orchestra.run.smithi059:mon.b> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mon.b.service 2024-03-28T18:01:03.618 DEBUG:teuthology.orchestra.run.smithi170:mon.c> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mon.c.service 2024-03-28T18:01:03.620 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T18:01:03.620 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T18:01:03.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:03 smithi037 ceph-mon[31460]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:03.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:03.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:01:03.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:03.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:01:03.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:03.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:01:03.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:04.251 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:01:04.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:04 smithi037 ceph-mon[31460]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi037:172.21.15.37=a;smithi059:172.21.15.59=b;smithi170:172.21.15.170=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:01:04.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:04 smithi037 ceph-mon[31460]: Saving service mon spec with placement smithi037:172.21.15.37=a;smithi059:172.21.15.59=b;smithi170:172.21.15.170=c;count:3 2024-03-28T18:01:04.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:04 smithi037 ceph-mon[31460]: Deploying daemon mon.c on smithi170 2024-03-28T18:01:05.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:05 smithi037 ceph-mon[31460]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:07.042 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.028+0000 7faa68f3b700 1 -- 172.21.15.170:0/475536874 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa64106a60 msgr2=0x7faa64106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:07.042 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.028+0000 7faa68f3b700 1 --2- 172.21.15.170:0/475536874 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa64106a60 0x7faa64106e40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faa54009910 tx=0x7faa54009c20 comp rx=0 tx=0).stop 2024-03-28T18:01:07.042 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.028+0000 7faa68f3b700 1 -- 172.21.15.170:0/475536874 shutdown_connections 2024-03-28T18:01:07.042 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.028+0000 7faa68f3b700 1 --2- 172.21.15.170:0/475536874 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa64106a60 0x7faa64106e40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:07.042 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.028+0000 7faa68f3b700 1 -- 172.21.15.170:0/475536874 >> 172.21.15.170:0/475536874 conn(0x7faa640776d0 msgr2=0x7faa640fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:07.042 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.028+0000 7faa68f3b700 1 -- 172.21.15.170:0/475536874 shutdown_connections 2024-03-28T18:01:07.042 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.028+0000 7faa68f3b700 1 -- 172.21.15.170:0/475536874 wait complete. 2024-03-28T18:01:07.042 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.029+0000 7faa68f3b700 1 Processor -- start 2024-03-28T18:01:07.042 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.029+0000 7faa68f3b700 1 -- start start 2024-03-28T18:01:07.043 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.030+0000 7faa68f3b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa64106a60 0x7faa64198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:07.043 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.030+0000 7faa68f3b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6410b0a0 con 0x7faa64106a60 2024-03-28T18:01:07.043 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.030+0000 7faa6259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa64106a60 0x7faa64198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:07.043 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.030+0000 7faa6259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa64106a60 0x7faa64198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.170:53008/0 (socket says 172.21.15.170:53008) 2024-03-28T18:01:07.043 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.030+0000 7faa6259c700 1 -- 172.21.15.170:0/2349220465 learned_addr learned my addr 172.21.15.170:0/2349220465 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T18:01:07.043 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.031+0000 7faa6259c700 1 -- 172.21.15.170:0/2349220465 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa641989d0 con 0x7faa64106a60 2024-03-28T18:01:07.043 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.031+0000 7faa6259c700 1 --2- 172.21.15.170:0/2349220465 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa64106a60 0x7faa64198490 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7faa54006010 tx=0x7faa54005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:07.043 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.032+0000 7faa537fe700 1 -- 172.21.15.170:0/2349220465 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa54026070 con 0x7faa64106a60 2024-03-28T18:01:07.043 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.032+0000 7faa537fe700 1 -- 172.21.15.170:0/2349220465 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa5400bbd0 con 0x7faa64106a60 2024-03-28T18:01:07.043 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.032+0000 7faa68f3b700 1 -- 172.21.15.170:0/2349220465 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa64198cc0 con 0x7faa64106a60 2024-03-28T18:01:07.043 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.032+0000 7faa537fe700 1 -- 172.21.15.170:0/2349220465 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa54003910 con 0x7faa64106a60 2024-03-28T18:01:07.043 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.032+0000 7faa68f3b700 1 -- 172.21.15.170:0/2349220465 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa64199180 con 0x7faa64106a60 2024-03-28T18:01:07.043 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.033+0000 7faa68f3b700 1 -- 172.21.15.170:0/2349220465 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa641021a0 con 0x7faa64106a60 2024-03-28T18:01:07.044 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.033+0000 7faa537fe700 1 -- 172.21.15.170:0/2349220465 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7faa54029070 con 0x7faa64106a60 2024-03-28T18:01:07.044 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.034+0000 7faa537fe700 1 --2- 172.21.15.170:0/2349220465 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7faa4c038fc0 0x7faa4c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:07.044 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.034+0000 7faa537fe700 1 -- 172.21.15.170:0/2349220465 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7faa5401e070 con 0x7faa64106a60 2024-03-28T18:01:07.044 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.034+0000 7faa61d9b700 1 --2- 172.21.15.170:0/2349220465 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7faa4c038fc0 0x7faa4c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:07.044 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.035+0000 7faa61d9b700 1 --2- 172.21.15.170:0/2349220465 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7faa4c038fc0 0x7faa4c03b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7faa58006fd0 tx=0x7faa58006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:07.049 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.049+0000 7faa537fe700 1 -- 172.21.15.170:0/2349220465 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faa54013070 con 0x7faa64106a60 2024-03-28T18:01:07.344 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.344+0000 7faa68f3b700 1 -- 172.21.15.170:0/2349220465 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa640637c0 con 0x7faa64106a60 2024-03-28T18:01:07.345 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.345+0000 7faa537fe700 1 -- 172.21.15.170:0/2349220465 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+745 (secure 0 0 0) 0x7faa54030090 con 0x7faa64106a60 2024-03-28T18:01:07.345 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T18:01:07.346 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","modified":"2024-03-28T17:59:01.527622Z","created":"2024-03-28T17:59:01.527622Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-28T18:01:07.348 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.347+0000 7faa68f3b700 1 -- 172.21.15.170:0/2349220465 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7faa4c038fc0 msgr2=0x7faa4c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:07.348 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.347+0000 7faa68f3b700 1 --2- 172.21.15.170:0/2349220465 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7faa4c038fc0 0x7faa4c03b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7faa58006fd0 tx=0x7faa58006e40 comp rx=0 tx=0).stop 2024-03-28T18:01:07.348 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.347+0000 7faa68f3b700 1 -- 172.21.15.170:0/2349220465 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa64106a60 msgr2=0x7faa64198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:07.348 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.347+0000 7faa68f3b700 1 --2- 172.21.15.170:0/2349220465 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa64106a60 0x7faa64198490 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7faa54006010 tx=0x7faa54005030 comp rx=0 tx=0).stop 2024-03-28T18:01:07.348 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.347+0000 7faa68f3b700 1 -- 172.21.15.170:0/2349220465 shutdown_connections 2024-03-28T18:01:07.348 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.347+0000 7faa68f3b700 1 --2- 172.21.15.170:0/2349220465 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7faa4c038fc0 0x7faa4c03b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:07.348 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.347+0000 7faa68f3b700 1 --2- 172.21.15.170:0/2349220465 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa64106a60 0x7faa64198490 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:07.348 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.347+0000 7faa68f3b700 1 -- 172.21.15.170:0/2349220465 >> 172.21.15.170:0/2349220465 conn(0x7faa640776d0 msgr2=0x7faa64108580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:07.348 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.347+0000 7faa68f3b700 1 -- 172.21.15.170:0/2349220465 shutdown_connections 2024-03-28T18:01:07.348 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:07.347+0000 7faa68f3b700 1 -- 172.21.15.170:0/2349220465 wait complete. 2024-03-28T18:01:07.349 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2024-03-28T18:01:07.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:07 smithi037 ceph-mon[31460]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:07.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:07 smithi037 ceph-mon[31460]: from='client.? 172.21.15.170:0/2349220465' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T18:01:08.321 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: load: jerasure load: lrc 2024-03-28T18:01:08.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: RocksDB version: 7.9.2 2024-03-28T18:01:08.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Git sha 0 2024-03-28T18:01:08.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-28T18:01:08.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: DB SUMMARY 2024-03-28T18:01:08.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: DB Session ID: DX23X3ZMBDJIUOV7SETA 2024-03-28T18:01:08.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: CURRENT file: CURRENT 2024-03-28T18:01:08.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: IDENTITY file: IDENTITY 2024-03-28T18:01:08.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-28T18:01:08.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-28T18:01:08.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-28T18:01:08.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.error_if_exists: 0 2024-03-28T18:01:08.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.create_if_missing: 0 2024-03-28T18:01:08.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.paranoid_checks: 1 2024-03-28T18:01:08.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-28T18:01:08.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-28T18:01:08.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-28T18:01:08.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.env: 0x56183ac55b20 2024-03-28T18:01:08.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.fs: PosixFileSystem 2024-03-28T18:01:08.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.info_log: 0x56183ce83920 2024-03-28T18:01:08.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_file_opening_threads: 16 2024-03-28T18:01:08.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.statistics: (nil) 2024-03-28T18:01:08.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.use_fsync: 0 2024-03-28T18:01:08.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_log_file_size: 0 2024-03-28T18:01:08.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-28T18:01:08.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-28T18:01:08.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.keep_log_file_num: 1000 2024-03-28T18:01:08.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.recycle_log_file_num: 0 2024-03-28T18:01:08.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.allow_fallocate: 1 2024-03-28T18:01:08.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.allow_mmap_reads: 0 2024-03-28T18:01:08.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.allow_mmap_writes: 0 2024-03-28T18:01:08.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.use_direct_reads: 0 2024-03-28T18:01:08.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-28T18:01:08.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.create_missing_column_families: 0 2024-03-28T18:01:08.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.db_log_dir: 2024-03-28T18:01:08.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.wal_dir: 2024-03-28T18:01:08.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-28T18:01:08.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-28T18:01:08.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-28T18:01:08.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-28T18:01:08.790 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-28T18:01:08.790 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-28T18:01:08.791 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.advise_random_on_open: 1 2024-03-28T18:01:08.791 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.db_write_buffer_size: 0 2024-03-28T18:01:08.791 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.write_buffer_manager: 0x56183ce561e0 2024-03-28T18:01:08.791 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-28T18:01:08.791 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-28T18:01:08.791 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-28T18:01:08.791 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.rate_limiter: (nil) 2024-03-28T18:01:08.791 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-28T18:01:08.791 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.wal_recovery_mode: 2 2024-03-28T18:01:08.791 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.enable_thread_tracking: 0 2024-03-28T18:01:08.791 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.enable_pipelined_write: 0 2024-03-28T18:01:08.791 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.unordered_write: 0 2024-03-28T18:01:08.791 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-28T18:01:08.792 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-28T18:01:08.792 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-28T18:01:08.792 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-28T18:01:08.792 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.row_cache: None 2024-03-28T18:01:08.792 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.wal_filter: None 2024-03-28T18:01:08.792 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-28T18:01:08.792 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.allow_ingest_behind: 0 2024-03-28T18:01:08.792 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.two_write_queues: 0 2024-03-28T18:01:08.792 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.manual_wal_flush: 0 2024-03-28T18:01:08.792 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.wal_compression: 0 2024-03-28T18:01:08.792 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.atomic_flush: 0 2024-03-28T18:01:08.792 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-28T18:01:08.792 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-28T18:01:08.792 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-28T18:01:08.792 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.log_readahead_size: 0 2024-03-28T18:01:08.793 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-28T18:01:08.793 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.best_efforts_recovery: 0 2024-03-28T18:01:08.793 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-28T18:01:08.793 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-28T18:01:08.793 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.allow_data_in_errors: 0 2024-03-28T18:01:08.793 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.db_host_id: __hostname__ 2024-03-28T18:01:08.793 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-28T18:01:08.793 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_background_jobs: 2 2024-03-28T18:01:08.793 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_background_compactions: -1 2024-03-28T18:01:08.793 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_subcompactions: 1 2024-03-28T18:01:08.793 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-28T18:01:08.793 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-28T18:01:08.793 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-28T18:01:08.793 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_total_wal_size: 0 2024-03-28T18:01:08.793 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-28T18:01:08.794 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-28T18:01:08.794 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-28T18:01:08.794 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-28T18:01:08.794 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_open_files: -1 2024-03-28T18:01:08.794 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.bytes_per_sync: 0 2024-03-28T18:01:08.794 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-28T18:01:08.794 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-28T18:01:08.794 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compaction_readahead_size: 0 2024-03-28T18:01:08.794 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_background_flushes: -1 2024-03-28T18:01:08.794 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Compression algorithms supported: 2024-03-28T18:01:08.794 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-28T18:01:08.794 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: kZSTD supported: 0 2024-03-28T18:01:08.794 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: kXpressCompression supported: 0 2024-03-28T18:01:08.794 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: kLZ4HCCompression supported: 1 2024-03-28T18:01:08.795 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: kZlibCompression supported: 1 2024-03-28T18:01:08.795 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: kSnappyCompression supported: 1 2024-03-28T18:01:08.795 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: kLZ4Compression supported: 1 2024-03-28T18:01:08.795 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: kBZip2Compression supported: 0 2024-03-28T18:01:08.795 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-28T18:01:08.795 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-28T18:01:08.795 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-28T18:01:08.795 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-28T18:01:08.795 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-28T18:01:08.795 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.merge_operator: 2024-03-28T18:01:08.795 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compaction_filter: None 2024-03-28T18:01:08.795 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compaction_filter_factory: None 2024-03-28T18:01:08.795 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.sst_partitioner_factory: None 2024-03-28T18:01:08.795 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-28T18:01:08.795 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-28T18:01:08.796 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56183ce83a20) 2024-03-28T18:01:08.796 INFO:journalctl@ceph.mon.c.smithi170.stdout: cache_index_and_filter_blocks: 1 2024-03-28T18:01:08.796 INFO:journalctl@ceph.mon.c.smithi170.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-28T18:01:08.796 INFO:journalctl@ceph.mon.c.smithi170.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-28T18:01:08.796 INFO:journalctl@ceph.mon.c.smithi170.stdout: pin_top_level_index_and_filter: 1 2024-03-28T18:01:08.796 INFO:journalctl@ceph.mon.c.smithi170.stdout: index_type: 0 2024-03-28T18:01:08.796 INFO:journalctl@ceph.mon.c.smithi170.stdout: data_block_index_type: 0 2024-03-28T18:01:08.796 INFO:journalctl@ceph.mon.c.smithi170.stdout: index_shortening: 1 2024-03-28T18:01:08.796 INFO:journalctl@ceph.mon.c.smithi170.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-28T18:01:08.796 INFO:journalctl@ceph.mon.c.smithi170.stdout: checksum: 4 2024-03-28T18:01:08.796 INFO:journalctl@ceph.mon.c.smithi170.stdout: no_block_cache: 0 2024-03-28T18:01:08.796 INFO:journalctl@ceph.mon.c.smithi170.stdout: block_cache: 0x56183ce89090 2024-03-28T18:01:08.797 INFO:journalctl@ceph.mon.c.smithi170.stdout: block_cache_name: BinnedLRUCache 2024-03-28T18:01:08.797 INFO:journalctl@ceph.mon.c.smithi170.stdout: block_cache_options: 2024-03-28T18:01:08.797 INFO:journalctl@ceph.mon.c.smithi170.stdout: capacity : 536870912 2024-03-28T18:01:08.797 INFO:journalctl@ceph.mon.c.smithi170.stdout: num_shard_bits : 4 2024-03-28T18:01:08.797 INFO:journalctl@ceph.mon.c.smithi170.stdout: strict_capacity_limit : 0 2024-03-28T18:01:08.797 INFO:journalctl@ceph.mon.c.smithi170.stdout: high_pri_pool_ratio: 0.000 2024-03-28T18:01:08.797 INFO:journalctl@ceph.mon.c.smithi170.stdout: block_cache_compressed: (nil) 2024-03-28T18:01:08.797 INFO:journalctl@ceph.mon.c.smithi170.stdout: persistent_cache: (nil) 2024-03-28T18:01:08.797 INFO:journalctl@ceph.mon.c.smithi170.stdout: block_size: 4096 2024-03-28T18:01:08.797 INFO:journalctl@ceph.mon.c.smithi170.stdout: block_size_deviation: 10 2024-03-28T18:01:08.797 INFO:journalctl@ceph.mon.c.smithi170.stdout: block_restart_interval: 16 2024-03-28T18:01:08.797 INFO:journalctl@ceph.mon.c.smithi170.stdout: index_block_restart_interval: 1 2024-03-28T18:01:08.797 INFO:journalctl@ceph.mon.c.smithi170.stdout: metadata_block_size: 4096 2024-03-28T18:01:08.797 INFO:journalctl@ceph.mon.c.smithi170.stdout: partition_filters: 0 2024-03-28T18:01:08.797 INFO:journalctl@ceph.mon.c.smithi170.stdout: use_delta_encoding: 1 2024-03-28T18:01:08.798 INFO:journalctl@ceph.mon.c.smithi170.stdout: filter_policy: bloomfilter 2024-03-28T18:01:08.798 INFO:journalctl@ceph.mon.c.smithi170.stdout: whole_key_filtering: 1 2024-03-28T18:01:08.798 INFO:journalctl@ceph.mon.c.smithi170.stdout: verify_compression: 0 2024-03-28T18:01:08.798 INFO:journalctl@ceph.mon.c.smithi170.stdout: read_amp_bytes_per_bit: 0 2024-03-28T18:01:08.798 INFO:journalctl@ceph.mon.c.smithi170.stdout: format_version: 5 2024-03-28T18:01:08.798 INFO:journalctl@ceph.mon.c.smithi170.stdout: enable_index_compression: 1 2024-03-28T18:01:08.798 INFO:journalctl@ceph.mon.c.smithi170.stdout: block_align: 0 2024-03-28T18:01:08.798 INFO:journalctl@ceph.mon.c.smithi170.stdout: max_auto_readahead_size: 262144 2024-03-28T18:01:08.798 INFO:journalctl@ceph.mon.c.smithi170.stdout: prepopulate_block_cache: 0 2024-03-28T18:01:08.798 INFO:journalctl@ceph.mon.c.smithi170.stdout: initial_auto_readahead_size: 8192 2024-03-28T18:01:08.798 INFO:journalctl@ceph.mon.c.smithi170.stdout: num_file_reads_for_auto_readahead: 2 2024-03-28T18:01:08.798 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.write_buffer_size: 33554432 2024-03-28T18:01:08.798 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_write_buffer_number: 2 2024-03-28T18:01:08.798 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compression: NoCompression 2024-03-28T18:01:08.799 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.bottommost_compression: Disabled 2024-03-28T18:01:08.799 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.prefix_extractor: nullptr 2024-03-28T18:01:08.799 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-28T18:01:08.799 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.num_levels: 7 2024-03-28T18:01:08.799 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-28T18:01:08.799 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-28T18:01:08.799 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-28T18:01:08.799 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-28T18:01:08.799 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-28T18:01:08.799 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-28T18:01:08.799 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-28T18:01:08.799 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-28T18:01:08.799 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-28T18:01:08.799 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-28T18:01:08.799 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-28T18:01:08.799 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-28T18:01:08.800 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-28T18:01:08.800 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compression_opts.level: 32767 2024-03-28T18:01:08.800 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compression_opts.strategy: 0 2024-03-28T18:01:08.800 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-28T18:01:08.800 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-28T18:01:08.800 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-28T18:01:08.800 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-28T18:01:08.800 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compression_opts.enabled: false 2024-03-28T18:01:08.800 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-28T18:01:08.800 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-28T18:01:08.800 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-28T18:01:08.800 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-28T18:01:08.800 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.target_file_size_base: 67108864 2024-03-28T18:01:08.800 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-28T18:01:08.800 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-28T18:01:08.801 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-28T18:01:08.801 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-28T18:01:08.801 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-28T18:01:08.801 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-28T18:01:08.801 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-28T18:01:08.801 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-28T18:01:08.801 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-28T18:01:08.801 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-28T18:01:08.801 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-28T18:01:08.801 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-28T18:01:08.801 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-28T18:01:08.801 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-28T18:01:08.801 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.arena_block_size: 1048576 2024-03-28T18:01:08.801 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-28T18:01:08.801 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-28T18:01:08.801 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.disable_auto_compactions: 0 2024-03-28T18:01:08.802 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-28T18:01:08.802 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-28T18:01:08.802 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-28T18:01:08.802 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-28T18:01:08.802 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-28T18:01:08.802 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-28T18:01:08.802 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-28T18:01:08.802 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-28T18:01:08.802 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-28T18:01:08.802 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-28T18:01:08.802 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.table_properties_collectors: 2024-03-28T18:01:08.802 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.inplace_update_support: 0 2024-03-28T18:01:08.802 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-28T18:01:08.802 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-28T18:01:08.802 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-28T18:01:08.803 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-28T18:01:08.803 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.bloom_locality: 0 2024-03-28T18:01:08.803 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.max_successive_merges: 0 2024-03-28T18:01:08.803 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-28T18:01:08.803 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.paranoid_file_checks: 0 2024-03-28T18:01:08.803 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.force_consistency_checks: 1 2024-03-28T18:01:08.803 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.report_bg_io_stats: 0 2024-03-28T18:01:08.803 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.ttl: 2592000 2024-03-28T18:01:08.803 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-28T18:01:08.803 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-28T18:01:08.803 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-28T18:01:08.803 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.enable_blob_files: false 2024-03-28T18:01:08.803 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.min_blob_size: 0 2024-03-28T18:01:08.803 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.blob_file_size: 268435456 2024-03-28T18:01:08.803 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-28T18:01:08.803 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-28T18:01:08.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-28T18:01:08.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-28T18:01:08.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-28T18:01:08.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.blob_file_starting_level: 0 2024-03-28T18:01:08.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-28T18:01:08.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-28T18:01:08.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-28T18:01:08.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 562a1905-fb9c-4d25-a6b2-3320272f586e 2024-03-28T18:01:08.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711648868324524, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-28T18:01:08.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-28T18:01:08.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711648868324932, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711648868, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "562a1905-fb9c-4d25-a6b2-3320272f586e", "db_session_id": "DX23X3ZMBDJIUOV7SETA", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-28T18:01:08.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711648868325010, "job": 1, "event": "recovery_finished"} 2024-03-28T18:01:08.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-28T18:01:08.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-28T18:01:08.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56183cf7c000 2024-03-28T18:01:08.805 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: DB pointer 0x56183cf66000 2024-03-28T18:01:08.805 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-28T18:01:08.805 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-28T18:01:08.805 INFO:journalctl@ceph.mon.c.smithi170.stdout: ** DB Stats ** 2024-03-28T18:01:08.805 INFO:journalctl@ceph.mon.c.smithi170.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T18:01:08.805 INFO:journalctl@ceph.mon.c.smithi170.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-28T18:01:08.805 INFO:journalctl@ceph.mon.c.smithi170.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T18:01:08.805 INFO:journalctl@ceph.mon.c.smithi170.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T18:01:08.805 INFO:journalctl@ceph.mon.c.smithi170.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-28T18:01:08.805 INFO:journalctl@ceph.mon.c.smithi170.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T18:01:08.805 INFO:journalctl@ceph.mon.c.smithi170.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T18:01:08.805 INFO:journalctl@ceph.mon.c.smithi170.stdout: 2024-03-28T18:01:08.805 INFO:journalctl@ceph.mon.c.smithi170.stdout: ** Compaction Stats [default] ** 2024-03-28T18:01:08.805 INFO:journalctl@ceph.mon.c.smithi170.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T18:01:08.805 INFO:journalctl@ceph.mon.c.smithi170.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-28T18:01:08.805 INFO:journalctl@ceph.mon.c.smithi170.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T18:01:08.805 INFO:journalctl@ceph.mon.c.smithi170.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T18:01:08.806 INFO:journalctl@ceph.mon.c.smithi170.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T18:01:08.806 INFO:journalctl@ceph.mon.c.smithi170.stdout: 2024-03-28T18:01:08.806 INFO:journalctl@ceph.mon.c.smithi170.stdout: ** Compaction Stats [default] ** 2024-03-28T18:01:08.806 INFO:journalctl@ceph.mon.c.smithi170.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T18:01:08.806 INFO:journalctl@ceph.mon.c.smithi170.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-28T18:01:08.806 INFO:journalctl@ceph.mon.c.smithi170.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T18:01:08.806 INFO:journalctl@ceph.mon.c.smithi170.stdout: 2024-03-28T18:01:08.806 INFO:journalctl@ceph.mon.c.smithi170.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-28T18:01:08.806 INFO:journalctl@ceph.mon.c.smithi170.stdout: 2024-03-28T18:01:08.806 INFO:journalctl@ceph.mon.c.smithi170.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T18:01:08.806 INFO:journalctl@ceph.mon.c.smithi170.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-28T18:01:08.806 INFO:journalctl@ceph.mon.c.smithi170.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-28T18:01:08.806 INFO:journalctl@ceph.mon.c.smithi170.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-28T18:01:08.806 INFO:journalctl@ceph.mon.c.smithi170.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-28T18:01:08.807 INFO:journalctl@ceph.mon.c.smithi170.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-28T18:01:08.807 INFO:journalctl@ceph.mon.c.smithi170.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T18:01:08.807 INFO:journalctl@ceph.mon.c.smithi170.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T18:01:08.807 INFO:journalctl@ceph.mon.c.smithi170.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-28T18:01:08.807 INFO:journalctl@ceph.mon.c.smithi170.stdout: Block cache BinnedLRUCache@0x56183ce89090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-28T18:01:08.807 INFO:journalctl@ceph.mon.c.smithi170.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-28T18:01:08.807 INFO:journalctl@ceph.mon.c.smithi170.stdout: 2024-03-28T18:01:08.807 INFO:journalctl@ceph.mon.c.smithi170.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-28T18:01:08.807 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-28T18:01:08.807 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: using public_addr v2:172.21.15.170:0/0 -> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2024-03-28T18:01:08.807 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: starting mon.c rank -1 at public addrs [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] at bind addrs [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 2024-03-28T18:01:08.807 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: mon.c@-1(???) e0 preinit fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 2024-03-28T18:01:08.807 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: mon.c@-1(synchronizing).mds e1 new map 2024-03-28T18:01:08.807 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-28T18:01:08.807 INFO:journalctl@ceph.mon.c.smithi170.stdout: e1 2024-03-28T18:01:08.807 INFO:journalctl@ceph.mon.c.smithi170.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T18:01:08.808 INFO:journalctl@ceph.mon.c.smithi170.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T18:01:08.808 INFO:journalctl@ceph.mon.c.smithi170.stdout: legacy client fscid: -1 2024-03-28T18:01:08.808 INFO:journalctl@ceph.mon.c.smithi170.stdout: 2024-03-28T18:01:08.808 INFO:journalctl@ceph.mon.c.smithi170.stdout: No filesystems configured 2024-03-28T18:01:08.808 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-28T18:01:08.808 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-28T18:01:08.808 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-28T18:01:08.808 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-28T18:01:08.808 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-28T18:01:08.808 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-28T18:01:08.808 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T18:01:08.808 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T18:01:08.808 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T18:01:08.808 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T18:01:08.808 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: Added host smithi059 2024-03-28T18:01:08.809 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.809 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.809 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:08.809 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:08.809 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.809 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:01:08.809 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:08.809 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.809 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.809 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.809 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.809 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi059", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:01:08.809 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:08.809 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:01:08.809 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: Updating smithi059:/etc/ceph/ceph.conf 2024-03-28T18:01:08.809 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: Updating smithi059:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:01:08.809 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:08.810 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: Updating smithi059:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:01:08.810 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: Updating smithi059:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:01:08.810 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi170", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:01:08.810 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.810 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.810 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.810 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: Deploying cephadm binary to smithi170 2024-03-28T18:01:08.810 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:08.810 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:08.810 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.810 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:01:08.810 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: Added host smithi170 2024-03-28T18:01:08.810 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.810 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.810 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:08.811 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:08.811 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.811 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:01:08.811 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:08.811 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.811 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.811 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.811 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.811 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:01:08.811 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:08.811 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:01:08.811 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-28T18:01:08.811 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: Updating smithi170:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:01:08.811 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1513499265' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T18:01:08.811 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:08.811 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:01:08.811 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: Updating smithi170:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:01:08.812 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1513499265' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T18:01:08.812 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:01:08.812 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.812 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.812 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.812 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:08.812 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.812 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:01:08.812 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:08.812 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:01:08.812 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:08.812 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:01:08.812 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:08.812 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi037:172.21.15.37=a;smithi059:172.21.15.59=b;smithi170:172.21.15.170=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:01:08.812 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: Saving service mon spec with placement smithi037:172.21.15.37=a;smithi059:172.21.15.59=b;smithi170:172.21.15.170=c;count:3 2024-03-28T18:01:08.812 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: Deploying daemon mon.c on smithi170 2024-03-28T18:01:08.813 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:08.813 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:08.813 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: from='client.? 172.21.15.170:0/2349220465' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T18:01:08.813 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 ceph-mon[36280]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-28T18:01:08.813 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 bash[36260]: cc015f7b0d4f4bdf2abf7063fdf915ca885bce0fccc1e60ee3e8ced086a98854 2024-03-28T18:01:08.813 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:08 smithi170 systemd[1]: Started Ceph mon.c for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:01:08.826 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T18:01:08.826 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T18:01:09.062 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:01:13.463 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.462+0000 7f5f0eca3700 1 -- 172.21.15.170:0/270418640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ef40057d0 msgr2=0x7f5ef4007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:13.463 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.462+0000 7f5f0eca3700 1 --2- 172.21.15.170:0/270418640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ef40057d0 0x7f5ef4007c60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5f08100a70 tx=0x7f5f00009e80 comp rx=0 tx=0).stop 2024-03-28T18:01:13.463 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.462+0000 7f5f0eca3700 1 -- 172.21.15.170:0/270418640 shutdown_connections 2024-03-28T18:01:13.463 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.462+0000 7f5f0eca3700 1 --2- 172.21.15.170:0/270418640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ef40057d0 0x7f5ef4007c60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:13.463 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.462+0000 7f5f0eca3700 1 -- 172.21.15.170:0/270418640 >> 172.21.15.170:0/270418640 conn(0x7f5f080fc630 msgr2=0x7f5f080fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:13.463 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.462+0000 7f5f0eca3700 1 -- 172.21.15.170:0/270418640 shutdown_connections 2024-03-28T18:01:13.463 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.462+0000 7f5f0eca3700 1 -- 172.21.15.170:0/270418640 wait complete. 2024-03-28T18:01:13.463 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.462+0000 7f5f0eca3700 1 Processor -- start 2024-03-28T18:01:13.463 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.462+0000 7f5f0eca3700 1 -- start start 2024-03-28T18:01:13.464 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.463+0000 7f5f0eca3700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ef40057d0 0x7f5f081956d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:13.464 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.463+0000 7f5f0eca3700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f08195c10 0x7f5f08199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:13.464 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.463+0000 7f5f0eca3700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f08107ba0 con 0x7f5ef40057d0 2024-03-28T18:01:13.464 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.463+0000 7f5f0eca3700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f08107a20 con 0x7f5f08195c10 2024-03-28T18:01:13.464 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.463+0000 7f5f07fff700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f08195c10 0x7f5f08199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:13.464 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.463+0000 7f5f07fff700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f08195c10 0x7f5f08199f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:33938/0 (socket says 172.21.15.170:33938) 2024-03-28T18:01:13.464 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.463+0000 7f5f07fff700 1 -- 172.21.15.170:0/2873175792 learned_addr learned my addr 172.21.15.170:0/2873175792 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T18:01:13.464 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.463+0000 7f5f07fff700 1 -- 172.21.15.170:0/2873175792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f08195c10 msgr2=0x7f5f08199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-28T18:01:13.464 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.463+0000 7f5f07fff700 1 -- 172.21.15.170:0/2873175792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f08195c10 msgr2=0x7f5f08199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T18:01:13.464 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.463+0000 7f5f07fff700 1 --2- 172.21.15.170:0/2873175792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f08195c10 0x7f5f08199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T18:01:13.464 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.463+0000 7f5f07fff700 1 --2- 172.21.15.170:0/2873175792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f08195c10 0x7f5f08199f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:01:13.464 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.463+0000 7f5f0ca3f700 1 --2- 172.21.15.170:0/2873175792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ef40057d0 0x7f5f081956d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:13.465 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.464+0000 7f5f0ca3f700 1 -- 172.21.15.170:0/2873175792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f08195c10 msgr2=0x7f5f08199f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:01:13.465 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.464+0000 7f5f0ca3f700 1 --2- 172.21.15.170:0/2873175792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f08195c10 0x7f5f08199f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:13.465 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.464+0000 7f5f0ca3f700 1 -- 172.21.15.170:0/2873175792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f08196250 con 0x7f5ef40057d0 2024-03-28T18:01:13.466 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.464+0000 7f5f0ca3f700 1 --2- 172.21.15.170:0/2873175792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ef40057d0 0x7f5f081956d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5f00002230 tx=0x7f5f00017750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:13.466 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.465+0000 7f5f05ffb700 1 -- 172.21.15.170:0/2873175792 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f5f0002a610 con 0x7f5ef40057d0 2024-03-28T18:01:13.466 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.465+0000 7f5f05ffb700 1 -- 172.21.15.170:0/2873175792 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f00025070 con 0x7f5ef40057d0 2024-03-28T18:01:13.466 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.465+0000 7f5f05ffb700 1 -- 172.21.15.170:0/2873175792 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f5f00020a70 con 0x7f5ef40057d0 2024-03-28T18:01:13.466 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.465+0000 7f5f0eca3700 1 -- 172.21.15.170:0/2873175792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f0819a440 con 0x7f5ef40057d0 2024-03-28T18:01:13.466 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.465+0000 7f5f0eca3700 1 -- 172.21.15.170:0/2873175792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f0819a9a0 con 0x7f5ef40057d0 2024-03-28T18:01:13.466 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.466+0000 7f5f0eca3700 1 -- 172.21.15.170:0/2873175792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f08102280 con 0x7f5ef40057d0 2024-03-28T18:01:13.467 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.467+0000 7f5f05ffb700 1 -- 172.21.15.170:0/2873175792 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f5f0001e070 con 0x7f5ef40057d0 2024-03-28T18:01:13.467 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.467+0000 7f5f05ffb700 1 --2- 172.21.15.170:0/2873175792 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5ee8038dd0 0x7f5ee803b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:13.467 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.467+0000 7f5f05ffb700 1 -- 172.21.15.170:0/2873175792 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5f0001c070 con 0x7f5ef40057d0 2024-03-28T18:01:13.467 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.468+0000 7f5f07fff700 1 --2- 172.21.15.170:0/2873175792 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5ee8038dd0 0x7f5ee803b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:13.468 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.469+0000 7f5f07fff700 1 --2- 172.21.15.170:0/2873175792 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5ee8038dd0 0x7f5ee803b290 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8009180 tx=0x7f5ef8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:13.481 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.482+0000 7f5f05ffb700 1 -- 172.21.15.170:0/2873175792 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5f0001ee20 con 0x7f5ef40057d0 2024-03-28T18:01:13.753 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.754+0000 7f5f0eca3700 1 -- 172.21.15.170:0/2873175792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f0819ac50 con 0x7f5ef40057d0 2024-03-28T18:01:13.754 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:13.754 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:01:13.754 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: mon.a calling monitor election 2024-03-28T18:01:13.754 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:01:13.754 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:01:13.754 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:13.754 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:01:13.754 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: mon.c calling monitor election 2024-03-28T18:01:13.754 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:01:13.755 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:13.755 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:01:13.755 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:01:13.755 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T18:01:13.755 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: monmap e2: 2 mons at {a=[v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:01:13.755 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: fsmap 2024-03-28T18:01:13.755 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:01:13.755 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: mgrmap e13: x(active, since 59s) 2024-03-28T18:01:13.755 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: overall HEALTH_OK 2024-03-28T18:01:13.755 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:13.755 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:13.755 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:13.755 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:01:13.755 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:13 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:13.756 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.755+0000 7f5f05ffb700 1 -- 172.21.15.170:0/2873175792 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7f5f000302a0 con 0x7f5ef40057d0 2024-03-28T18:01:13.756 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T18:01:13.756 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":2,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","modified":"2024-03-28T18:01:08.358724Z","created":"2024-03-28T17:59:01.527622Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-28T18:01:13.757 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.757+0000 7f5f0eca3700 1 -- 172.21.15.170:0/2873175792 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5ee8038dd0 msgr2=0x7f5ee803b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:13.757 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.757+0000 7f5f0eca3700 1 --2- 172.21.15.170:0/2873175792 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5ee8038dd0 0x7f5ee803b290 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8009180 tx=0x7f5ef8008040 comp rx=0 tx=0).stop 2024-03-28T18:01:13.757 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.757+0000 7f5f0eca3700 1 -- 172.21.15.170:0/2873175792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ef40057d0 msgr2=0x7f5f081956d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:13.757 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.757+0000 7f5f0eca3700 1 --2- 172.21.15.170:0/2873175792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ef40057d0 0x7f5f081956d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5f00002230 tx=0x7f5f00017750 comp rx=0 tx=0).stop 2024-03-28T18:01:13.757 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.757+0000 7f5f0eca3700 1 -- 172.21.15.170:0/2873175792 shutdown_connections 2024-03-28T18:01:13.757 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.757+0000 7f5f0eca3700 1 --2- 172.21.15.170:0/2873175792 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5ee8038dd0 0x7f5ee803b290 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:13.757 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.757+0000 7f5f0eca3700 1 --2- 172.21.15.170:0/2873175792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ef40057d0 0x7f5f081956d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:13.757 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.757+0000 7f5f0eca3700 1 --2- 172.21.15.170:0/2873175792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f08195c10 0x7f5f08199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:13.757 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.757+0000 7f5f0eca3700 1 -- 172.21.15.170:0/2873175792 >> 172.21.15.170:0/2873175792 conn(0x7f5f080fc630 msgr2=0x7f5f080ffb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:13.757 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.757+0000 7f5f0eca3700 1 -- 172.21.15.170:0/2873175792 shutdown_connections 2024-03-28T18:01:13.757 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:13.757+0000 7f5f0eca3700 1 -- 172.21.15.170:0/2873175792 wait complete. 2024-03-28T18:01:13.758 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 2 2024-03-28T18:01:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:01:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: mon.a calling monitor election 2024-03-28T18:01:13.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:01:13.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:01:13.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:13.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:01:13.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: mon.c calling monitor election 2024-03-28T18:01:13.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:01:13.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:13.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:01:13.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:01:13.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T18:01:13.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: monmap e2: 2 mons at {a=[v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:01:13.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: fsmap 2024-03-28T18:01:13.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:01:13.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: mgrmap e13: x(active, since 59s) 2024-03-28T18:01:13.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: overall HEALTH_OK 2024-03-28T18:01:13.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:13.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:13.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:13.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:01:13.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:13 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:14.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:14 smithi170 ceph-mon[36280]: Deploying daemon mon.b on smithi059 2024-03-28T18:01:14.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:14 smithi170 ceph-mon[36280]: from='client.? 172.21.15.170:0/2873175792' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T18:01:14.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:01:14.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:14 smithi037 ceph-mon[31460]: Deploying daemon mon.b on smithi059 2024-03-28T18:01:14.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:14 smithi037 ceph-mon[31460]: from='client.? 172.21.15.170:0/2873175792' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T18:01:14.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:01:15.346 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T18:01:15.346 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T18:01:15.591 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:01:15.677 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:15 smithi170 ceph-mon[36280]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:15.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:15 smithi037 ceph-mon[31460]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:16.997 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.995+0000 7f41a7bb8700 1 -- 172.21.15.170:0/524468377 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41a0101ed0 msgr2=0x7f41a01022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:16.998 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.995+0000 7f41a7bb8700 1 --2- 172.21.15.170:0/524468377 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41a0101ed0 0x7f41a01022b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4190009910 tx=0x7f4190009c20 comp rx=0 tx=0).stop 2024-03-28T18:01:16.998 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.995+0000 7f41a7bb8700 1 -- 172.21.15.170:0/524468377 shutdown_connections 2024-03-28T18:01:16.998 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.995+0000 7f41a7bb8700 1 --2- 172.21.15.170:0/524468377 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41a0101ed0 0x7f41a01022b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:16.998 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.995+0000 7f41a7bb8700 1 -- 172.21.15.170:0/524468377 >> 172.21.15.170:0/524468377 conn(0x7f41a00fd760 msgr2=0x7f41a00ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:16.998 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.995+0000 7f41a7bb8700 1 -- 172.21.15.170:0/524468377 shutdown_connections 2024-03-28T18:01:16.998 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.995+0000 7f41a7bb8700 1 -- 172.21.15.170:0/524468377 wait complete. 2024-03-28T18:01:16.998 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.996+0000 7f41a7bb8700 1 Processor -- start 2024-03-28T18:01:16.998 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.996+0000 7f41a7bb8700 1 -- start start 2024-03-28T18:01:16.999 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.996+0000 7f41a7bb8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41a0101ed0 0x7f41a019d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:16.999 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.996+0000 7f41a7bb8700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41a019db60 0x7f41a0197790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:16.999 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.996+0000 7f41a7bb8700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a010c2a0 con 0x7f41a0101ed0 2024-03-28T18:01:16.999 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.996+0000 7f41a7bb8700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a010c120 con 0x7f41a019db60 2024-03-28T18:01:16.999 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.996+0000 7f41a5153700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41a019db60 0x7f41a0197790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:16.999 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.996+0000 7f41a5153700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41a019db60 0x7f41a0197790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:33946/0 (socket says 172.21.15.170:33946) 2024-03-28T18:01:16.999 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.996+0000 7f41a5153700 1 -- 172.21.15.170:0/3565892064 learned_addr learned my addr 172.21.15.170:0/3565892064 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T18:01:16.999 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.996+0000 7f41a5153700 1 -- 172.21.15.170:0/3565892064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41a019db60 msgr2=0x7f41a0197790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-28T18:01:16.999 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.996+0000 7f41a5153700 1 -- 172.21.15.170:0/3565892064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41a019db60 msgr2=0x7f41a0197790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T18:01:16.999 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.996+0000 7f41a5153700 1 --2- 172.21.15.170:0/3565892064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41a019db60 0x7f41a0197790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T18:01:16.999 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.996+0000 7f41a5153700 1 --2- 172.21.15.170:0/3565892064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41a019db60 0x7f41a0197790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:01:16.999 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.996+0000 7f41a5954700 1 --2- 172.21.15.170:0/3565892064 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41a0101ed0 0x7f41a019d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:17.000 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.997+0000 7f41a5954700 1 -- 172.21.15.170:0/3565892064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41a019db60 msgr2=0x7f41a0197790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:01:17.000 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.997+0000 7f41a5954700 1 --2- 172.21.15.170:0/3565892064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41a019db60 0x7f41a0197790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:17.000 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.997+0000 7f41a5954700 1 -- 172.21.15.170:0/3565892064 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a019e0e0 con 0x7f41a0101ed0 2024-03-28T18:01:17.000 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.997+0000 7f41a5954700 1 --2- 172.21.15.170:0/3565892064 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41a0101ed0 0x7f41a019d620 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4190014000 tx=0x7f4190005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:17.000 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.998+0000 7f4196ffd700 1 -- 172.21.15.170:0/3565892064 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f4190005dc0 con 0x7f41a0101ed0 2024-03-28T18:01:17.000 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.998+0000 7f41a7bb8700 1 -- 172.21.15.170:0/3565892064 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a0197d30 con 0x7f41a0101ed0 2024-03-28T18:01:17.000 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.998+0000 7f41a7bb8700 1 -- 172.21.15.170:0/3565892064 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a01982c0 con 0x7f41a0101ed0 2024-03-28T18:01:17.000 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.998+0000 7f4196ffd700 1 -- 172.21.15.170:0/3565892064 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f419000be40 con 0x7f41a0101ed0 2024-03-28T18:01:17.000 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.998+0000 7f4196ffd700 1 -- 172.21.15.170:0/3565892064 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f4190022990 con 0x7f41a0101ed0 2024-03-28T18:01:17.001 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.999+0000 7f41a7bb8700 1 -- 172.21.15.170:0/3565892064 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4184005330 con 0x7f41a0101ed0 2024-03-28T18:01:17.001 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.999+0000 7f4196ffd700 1 -- 172.21.15.170:0/3565892064 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f419002a070 con 0x7f41a0101ed0 2024-03-28T18:01:17.001 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:16.999+0000 7f4196ffd700 1 --2- 172.21.15.170:0/3565892064 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f418c038d60 0x7f418c03b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:17.001 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:17.000+0000 7f4196ffd700 1 -- 172.21.15.170:0/3565892064 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f419001e030 con 0x7f41a0101ed0 2024-03-28T18:01:17.001 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:17.000+0000 7f41a5153700 1 --2- 172.21.15.170:0/3565892064 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f418c038d60 0x7f418c03b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:17.001 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:17.001+0000 7f41a5153700 1 --2- 172.21.15.170:0/3565892064 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f418c038d60 0x7f418c03b220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f419c009180 tx=0x7f419c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:17.013 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:17.014+0000 7f4196ffd700 1 -- 172.21.15.170:0/3565892064 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4190013070 con 0x7f41a0101ed0 2024-03-28T18:01:17.288 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:17.288+0000 7f41a7bb8700 1 -- 172.21.15.170:0/3565892064 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f41840058f0 con 0x7f41a0101ed0 2024-03-28T18:01:17.289 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:17.289+0000 7f4196ffd700 1 -- 172.21.15.170:0/3565892064 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7f419002bc70 con 0x7f41a0101ed0 2024-03-28T18:01:17.289 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T18:01:17.289 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":2,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","modified":"2024-03-28T18:01:08.358724Z","created":"2024-03-28T17:59:01.527622Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-28T18:01:17.291 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:17.291+0000 7f41a7bb8700 1 -- 172.21.15.170:0/3565892064 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f418c038d60 msgr2=0x7f418c03b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:17.291 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:17.291+0000 7f41a7bb8700 1 --2- 172.21.15.170:0/3565892064 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f418c038d60 0x7f418c03b220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f419c009180 tx=0x7f419c008040 comp rx=0 tx=0).stop 2024-03-28T18:01:17.291 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:17.291+0000 7f41a7bb8700 1 -- 172.21.15.170:0/3565892064 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41a0101ed0 msgr2=0x7f41a019d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:17.291 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:17.291+0000 7f41a7bb8700 1 --2- 172.21.15.170:0/3565892064 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41a0101ed0 0x7f41a019d620 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4190014000 tx=0x7f4190005030 comp rx=0 tx=0).stop 2024-03-28T18:01:17.291 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:17.291+0000 7f41a7bb8700 1 -- 172.21.15.170:0/3565892064 shutdown_connections 2024-03-28T18:01:17.291 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:17.291+0000 7f41a7bb8700 1 --2- 172.21.15.170:0/3565892064 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f418c038d60 0x7f418c03b220 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:17.291 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:17.291+0000 7f41a7bb8700 1 --2- 172.21.15.170:0/3565892064 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41a0101ed0 0x7f41a019d620 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:17.291 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:17.291+0000 7f41a7bb8700 1 --2- 172.21.15.170:0/3565892064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41a019db60 0x7f41a0197790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:17.292 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:17.291+0000 7f41a7bb8700 1 -- 172.21.15.170:0/3565892064 >> 172.21.15.170:0/3565892064 conn(0x7f41a00fd760 msgr2=0x7f41a01038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:17.292 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:17.291+0000 7f41a7bb8700 1 -- 172.21.15.170:0/3565892064 shutdown_connections 2024-03-28T18:01:17.292 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:17.291+0000 7f41a7bb8700 1 -- 172.21.15.170:0/3565892064 wait complete. 2024-03-28T18:01:17.292 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 2 2024-03-28T18:01:17.721 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:17 smithi170 ceph-mon[36280]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:17.722 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:17 smithi170 ceph-mon[36280]: from='client.? 172.21.15.170:0/3565892064' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T18:01:17.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:17 smithi037 ceph-mon[31460]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:17.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:17 smithi037 ceph-mon[31460]: from='client.? 172.21.15.170:0/3565892064' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T18:01:18.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:17 smithi059 systemd[1]: Started Ceph mon.b for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:01:18.889 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T18:01:18.889 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T18:01:19.131 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:01:23.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: mon.c calling monitor election 2024-03-28T18:01:23.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:01:23.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: mon.a calling monitor election 2024-03-28T18:01:23.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:23.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:01:23.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:23.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:23.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:23.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:23.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:23.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:23.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:23.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:23.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T18:01:23.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: monmap e3: 3 mons at {a=[v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0],b=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:01:23.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: fsmap 2024-03-28T18:01:23.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:01:23.127 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: mgrmap e13: x(active, since 68s) 2024-03-28T18:01:23.127 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-28T18:01:23.127 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T18:01:23.127 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-28T18:01:23.127 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: mon.b (rank 2) addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] is down (out of quorum) 2024-03-28T18:01:23.127 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:23.127 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:22 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: mon.c calling monitor election 2024-03-28T18:01:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:01:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: mon.a calling monitor election 2024-03-28T18:01:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:01:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:23.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:23.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:23.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:23.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:23.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T18:01:23.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: monmap e3: 3 mons at {a=[v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0],b=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:01:23.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: fsmap 2024-03-28T18:01:23.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:01:23.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: mgrmap e13: x(active, since 68s) 2024-03-28T18:01:23.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-28T18:01:23.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T18:01:23.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-28T18:01:23.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: mon.b (rank 2) addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] is down (out of quorum) 2024-03-28T18:01:23.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:23.288 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:22 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:23.571 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.571+0000 7fafe4ba8700 1 -- 172.21.15.170:0/3111203412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe0100df0 msgr2=0x7fafcc005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:23.571 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.571+0000 7fafe4ba8700 1 --2- 172.21.15.170:0/3111203412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe0100df0 0x7fafcc005610 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fafd0005fa0 tx=0x7fafd000ffc0 comp rx=0 tx=0).stop 2024-03-28T18:01:23.572 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.571+0000 7fafe4ba8700 1 -- 172.21.15.170:0/3111203412 shutdown_connections 2024-03-28T18:01:23.572 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.571+0000 7fafe4ba8700 1 --2- 172.21.15.170:0/3111203412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe0100df0 0x7fafcc005610 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:23.572 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.571+0000 7fafe4ba8700 1 -- 172.21.15.170:0/3111203412 >> 172.21.15.170:0/3111203412 conn(0x7fafe00fc6a0 msgr2=0x7fafe00feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:23.572 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.571+0000 7fafe4ba8700 1 -- 172.21.15.170:0/3111203412 shutdown_connections 2024-03-28T18:01:23.572 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.571+0000 7fafe4ba8700 1 -- 172.21.15.170:0/3111203412 wait complete. 2024-03-28T18:01:23.574 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.572+0000 7fafe4ba8700 1 Processor -- start 2024-03-28T18:01:23.574 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.572+0000 7fafe4ba8700 1 -- start start 2024-03-28T18:01:23.574 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.573+0000 7fafe4ba8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe0100df0 0x7fafe01987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:23.574 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.573+0000 7fafe4ba8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe0198d10 0x7fafe0199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:23.574 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.573+0000 7fafe4ba8700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafe01a1840 0x7fafe01a3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:23.574 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.573+0000 7fafe4ba8700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe0108bd0 con 0x7fafe0100df0 2024-03-28T18:01:23.574 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.573+0000 7fafe4ba8700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe0108a50 con 0x7fafe0198d10 2024-03-28T18:01:23.574 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.573+0000 7fafe4ba8700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe0108d50 con 0x7fafe01a1840 2024-03-28T18:01:23.574 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.573+0000 7fafded9d700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafe01a1840 0x7fafe01a3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:23.574 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.573+0000 7fafded9d700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafe01a1840 0x7fafe01a3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:45974/0 (socket says 172.21.15.170:45974) 2024-03-28T18:01:23.574 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.573+0000 7fafded9d700 1 -- 172.21.15.170:0/1108370907 learned_addr learned my addr 172.21.15.170:0/1108370907 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T18:01:23.575 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.573+0000 7fafd7fff700 1 --2- 172.21.15.170:0/1108370907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe0198d10 0x7fafe0199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:23.575 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.573+0000 7fafde59c700 1 --2- 172.21.15.170:0/1108370907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe0100df0 0x7fafe01987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:23.575 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.573+0000 7fafded9d700 1 -- 172.21.15.170:0/1108370907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe0198d10 msgr2=0x7fafe0199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:23.575 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.573+0000 7fafded9d700 1 --2- 172.21.15.170:0/1108370907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe0198d10 0x7fafe0199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:23.575 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.573+0000 7fafded9d700 1 -- 172.21.15.170:0/1108370907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe0100df0 msgr2=0x7fafe01987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:23.575 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.573+0000 7fafded9d700 1 --2- 172.21.15.170:0/1108370907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe0100df0 0x7fafe01987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:23.575 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.573+0000 7fafded9d700 1 -- 172.21.15.170:0/1108370907 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe01a4400 con 0x7fafe01a1840 2024-03-28T18:01:23.575 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.573+0000 7fafd7fff700 1 --2- 172.21.15.170:0/1108370907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe0198d10 0x7fafe0199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:01:23.575 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.573+0000 7fafded9d700 1 --2- 172.21.15.170:0/1108370907 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafe01a1840 0x7fafe01a3c30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fafd800ba50 tx=0x7fafd800be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:23.575 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.574+0000 7fafd77fe700 1 -- 172.21.15.170:0/1108370907 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd8019070 con 0x7fafe01a1840 2024-03-28T18:01:23.575 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.574+0000 7fafd77fe700 1 -- 172.21.15.170:0/1108370907 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafd8014440 con 0x7fafe01a1840 2024-03-28T18:01:23.575 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.574+0000 7fafe4ba8700 1 -- 172.21.15.170:0/1108370907 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe01a46f0 con 0x7fafe01a1840 2024-03-28T18:01:23.575 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.574+0000 7fafe4ba8700 1 -- 172.21.15.170:0/1108370907 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe01a4be0 con 0x7fafe01a1840 2024-03-28T18:01:23.575 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.574+0000 7fafd77fe700 1 -- 172.21.15.170:0/1108370907 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd8007590 con 0x7fafe01a1840 2024-03-28T18:01:23.575 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.574+0000 7fafd77fe700 1 -- 172.21.15.170:0/1108370907 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fafd801c070 con 0x7fafe01a1840 2024-03-28T18:01:23.576 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.575+0000 7fafd77fe700 1 --2- 172.21.15.170:0/1108370907 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fafc0039450 0x7fafc003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:23.576 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.575+0000 7fafe4ba8700 1 -- 172.21.15.170:0/1108370907 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe00637c0 con 0x7fafe01a1840 2024-03-28T18:01:23.576 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.575+0000 7fafd77fe700 1 -- 172.21.15.170:0/1108370907 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fafd804c6c0 con 0x7fafe01a1840 2024-03-28T18:01:23.576 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.575+0000 7fafde59c700 1 --2- 172.21.15.170:0/1108370907 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fafc0039450 0x7fafc003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:23.577 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.576+0000 7fafde59c700 1 --2- 172.21.15.170:0/1108370907 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fafc0039450 0x7fafc003b910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fafd000fcb0 tx=0x7fafd0005900 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:23.579 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.580+0000 7fafd77fe700 1 -- 172.21.15.170:0/1108370907 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fafd8014910 con 0x7fafe01a1840 2024-03-28T18:01:23.883 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.883+0000 7fafe4ba8700 1 -- 172.21.15.170:0/1108370907 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fafe002da30 con 0x7fafe01a1840 2024-03-28T18:01:23.883 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.884+0000 7fafd77fe700 1 -- 172.21.15.170:0/1108370907 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1279 (secure 0 0 0) 0x7fafd80145f0 con 0x7fafe01a1840 2024-03-28T18:01:23.883 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T18:01:23.884 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":3,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","modified":"2024-03-28T18:01:17.748274Z","created":"2024-03-28T17:59:01.527622Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-28T18:01:23.885 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.885+0000 7fafe4ba8700 1 -- 172.21.15.170:0/1108370907 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fafc0039450 msgr2=0x7fafc003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:23.885 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.886+0000 7fafe4ba8700 1 --2- 172.21.15.170:0/1108370907 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fafc0039450 0x7fafc003b910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fafd000fcb0 tx=0x7fafd0005900 comp rx=0 tx=0).stop 2024-03-28T18:01:23.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.886+0000 7fafe4ba8700 1 -- 172.21.15.170:0/1108370907 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafe01a1840 msgr2=0x7fafe01a3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:23.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.886+0000 7fafe4ba8700 1 --2- 172.21.15.170:0/1108370907 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafe01a1840 0x7fafe01a3c30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fafd800ba50 tx=0x7fafd800be10 comp rx=0 tx=0).stop 2024-03-28T18:01:23.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.886+0000 7fafe4ba8700 1 -- 172.21.15.170:0/1108370907 shutdown_connections 2024-03-28T18:01:23.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.886+0000 7fafe4ba8700 1 --2- 172.21.15.170:0/1108370907 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fafc0039450 0x7fafc003b910 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:23.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.886+0000 7fafe4ba8700 1 --2- 172.21.15.170:0/1108370907 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafe01a1840 0x7fafe01a3c30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:23.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.886+0000 7fafe4ba8700 1 --2- 172.21.15.170:0/1108370907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe0100df0 0x7fafe01987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:23.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.886+0000 7fafe4ba8700 1 --2- 172.21.15.170:0/1108370907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe0198d10 0x7fafe0199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:23.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.886+0000 7fafe4ba8700 1 -- 172.21.15.170:0/1108370907 >> 172.21.15.170:0/1108370907 conn(0x7fafe00fc6a0 msgr2=0x7fafe00ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:23.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.886+0000 7fafe4ba8700 1 -- 172.21.15.170:0/1108370907 shutdown_connections 2024-03-28T18:01:23.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:23.886+0000 7fafe4ba8700 1 -- 172.21.15.170:0/1108370907 wait complete. 2024-03-28T18:01:23.887 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 3 2024-03-28T18:01:24.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:24.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:24.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:01:24.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:24.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:24.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:24.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:01:24.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:24.653 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-28T18:01:24.653 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-28T18:01:25.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: mon.c calling monitor election 2024-03-28T18:01:25.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:01:25.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: mon.a calling monitor election 2024-03-28T18:01:25.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:25.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:01:25.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:25.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:25.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:25.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:25.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:25.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:25.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:25.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:25.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T18:01:25.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: monmap e3: 3 mons at {a=[v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0],b=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:01:25.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: fsmap 2024-03-28T18:01:25.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:01:25.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: mgrmap e13: x(active, since 68s) 2024-03-28T18:01:25.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-28T18:01:25.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T18:01:25.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-28T18:01:25.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: mon.b (rank 2) addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] is down (out of quorum) 2024-03-28T18:01:25.037 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:25.037 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:25.037 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:25.037 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:25.037 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:01:25.037 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:25.120 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:01:25.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: mon.b calling monitor election 2024-03-28T18:01:25.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:25.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: mon.b calling monitor election 2024-03-28T18:01:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: mon.c calling monitor election 2024-03-28T18:01:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: mon.a calling monitor election 2024-03-28T18:01:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T18:01:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: monmap e3: 3 mons at {a=[v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0],b=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:01:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: fsmap 2024-03-28T18:01:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:01:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: mgrmap e13: x(active, since 70s) 2024-03-28T18:01:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-28T18:01:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:01:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: overall HEALTH_OK 2024-03-28T18:01:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:01:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:25.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:25.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:25.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:25 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: mon.b calling monitor election 2024-03-28T18:01:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: mon.b calling monitor election 2024-03-28T18:01:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: mon.c calling monitor election 2024-03-28T18:01:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: mon.a calling monitor election 2024-03-28T18:01:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T18:01:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: monmap e3: 3 mons at {a=[v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0],b=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:01:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: fsmap 2024-03-28T18:01:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:01:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: mgrmap e13: x(active, since 70s) 2024-03-28T18:01:26.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-28T18:01:26.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:01:26.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: overall HEALTH_OK 2024-03-28T18:01:26.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:26.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:26.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:26.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:01:26.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:26.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:26.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:26.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:25 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:26.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: mon.b calling monitor election 2024-03-28T18:01:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: mon.b calling monitor election 2024-03-28T18:01:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: mon.c calling monitor election 2024-03-28T18:01:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: mon.a calling monitor election 2024-03-28T18:01:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T18:01:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: monmap e3: 3 mons at {a=[v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0],b=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:01:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: fsmap 2024-03-28T18:01:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:01:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: mgrmap e13: x(active, since 70s) 2024-03-28T18:01:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-28T18:01:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:01:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: overall HEALTH_OK 2024-03-28T18:01:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:01:26.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:26.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:26.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:26.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:25 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:26.750 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:26 smithi037 ceph-mon[31460]: Updating smithi037:/etc/ceph/ceph.conf 2024-03-28T18:01:26.750 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:26 smithi037 ceph-mon[31460]: Updating smithi059:/etc/ceph/ceph.conf 2024-03-28T18:01:26.750 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:26 smithi037 ceph-mon[31460]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-28T18:01:26.750 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:26 smithi037 ceph-mon[31460]: Updating smithi059:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:01:26.750 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:26 smithi037 ceph-mon[31460]: Updating smithi170:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:01:26.750 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:26 smithi037 ceph-mon[31460]: Updating smithi037:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:01:26.750 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:26 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:26.751 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:26 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:26.751 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:26 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:26.751 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:26 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:26.937 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:26 smithi170 ceph-mon[36280]: Updating smithi037:/etc/ceph/ceph.conf 2024-03-28T18:01:26.937 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:26 smithi170 ceph-mon[36280]: Updating smithi059:/etc/ceph/ceph.conf 2024-03-28T18:01:26.937 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:26 smithi170 ceph-mon[36280]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-28T18:01:26.937 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:26 smithi170 ceph-mon[36280]: Updating smithi059:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:01:26.937 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:26 smithi170 ceph-mon[36280]: Updating smithi170:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:01:26.937 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:26 smithi170 ceph-mon[36280]: Updating smithi037:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:01:26.937 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:26 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:26.937 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:26 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:26.937 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:26 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:26.937 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:26 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:27.004 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:01:26 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:01:26.748+0000 7f9017400700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-28T18:01:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:26 smithi059 ceph-mon[36190]: Updating smithi037:/etc/ceph/ceph.conf 2024-03-28T18:01:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:26 smithi059 ceph-mon[36190]: Updating smithi059:/etc/ceph/ceph.conf 2024-03-28T18:01:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:26 smithi059 ceph-mon[36190]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-28T18:01:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:26 smithi059 ceph-mon[36190]: Updating smithi059:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:01:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:26 smithi059 ceph-mon[36190]: Updating smithi170:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:01:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:26 smithi059 ceph-mon[36190]: Updating smithi037:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:01:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:26 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:01:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:26 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:26 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:26 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:27.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.251+0000 7fbd58000700 1 -- 172.21.15.37:0/2096749820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd500745c0 msgr2=0x7fbd500749a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:27.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.251+0000 7fbd58000700 1 --2- 172.21.15.37:0/2096749820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd500745c0 0x7fbd500749a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbd44009910 tx=0x7fbd44009c20 comp rx=0 tx=0).stop 2024-03-28T18:01:27.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.251+0000 7fbd58000700 1 -- 172.21.15.37:0/2096749820 shutdown_connections 2024-03-28T18:01:27.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.251+0000 7fbd58000700 1 --2- 172.21.15.37:0/2096749820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd500745c0 0x7fbd500749a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:27.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.251+0000 7fbd58000700 1 -- 172.21.15.37:0/2096749820 >> 172.21.15.37:0/2096749820 conn(0x7fbd500fd670 msgr2=0x7fbd500ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:27.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.251+0000 7fbd58000700 1 -- 172.21.15.37:0/2096749820 shutdown_connections 2024-03-28T18:01:27.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.251+0000 7fbd58000700 1 -- 172.21.15.37:0/2096749820 wait complete. 2024-03-28T18:01:27.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.252+0000 7fbd58000700 1 Processor -- start 2024-03-28T18:01:27.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.253+0000 7fbd58000700 1 -- start start 2024-03-28T18:01:27.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.253+0000 7fbd58000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd500745c0 0x7fbd5019f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:27.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.253+0000 7fbd58000700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd5019fd30 0x7fbd501999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:27.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.253+0000 7fbd58000700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd50199ef0 0x7fbd5019a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:27.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.253+0000 7fbd58000700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5010e410 con 0x7fbd500745c0 2024-03-28T18:01:27.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.253+0000 7fbd58000700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5010e290 con 0x7fbd50199ef0 2024-03-28T18:01:27.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.253+0000 7fbd58000700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5010e590 con 0x7fbd5019fd30 2024-03-28T18:01:27.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.253+0000 7fbd55600700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd500745c0 0x7fbd5019f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:27.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.254+0000 7fbd55600700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd500745c0 0x7fbd5019f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:53174/0 (socket says 172.21.15.37:53174) 2024-03-28T18:01:27.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.254+0000 7fbd55600700 1 -- 172.21.15.37:0/2178065256 learned_addr learned my addr 172.21.15.37:0/2178065256 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:01:27.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.254+0000 7fbd54c00700 1 --2- 172.21.15.37:0/2178065256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd5019fd30 0x7fbd501999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:27.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.254+0000 7fbd56000700 1 --2- 172.21.15.37:0/2178065256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd50199ef0 0x7fbd5019a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:27.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.254+0000 7fbd55600700 1 -- 172.21.15.37:0/2178065256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd5019fd30 msgr2=0x7fbd501999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:27.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.254+0000 7fbd55600700 1 --2- 172.21.15.37:0/2178065256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd5019fd30 0x7fbd501999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:27.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.254+0000 7fbd55600700 1 -- 172.21.15.37:0/2178065256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd50199ef0 msgr2=0x7fbd5019a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:27.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.254+0000 7fbd55600700 1 --2- 172.21.15.37:0/2178065256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd50199ef0 0x7fbd5019a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:27.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.254+0000 7fbd55600700 1 -- 172.21.15.37:0/2178065256 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd501a66c0 con 0x7fbd500745c0 2024-03-28T18:01:27.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.254+0000 7fbd56000700 1 --2- 172.21.15.37:0/2178065256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd50199ef0 0x7fbd5019a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:01:27.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.255+0000 7fbd55600700 1 --2- 172.21.15.37:0/2178065256 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd500745c0 0x7fbd5019f7f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbd440092e0 tx=0x7fbd44004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:27.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.255+0000 7fbd4e000700 1 -- 172.21.15.37:0/2178065256 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd44026070 con 0x7fbd500745c0 2024-03-28T18:01:27.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.255+0000 7fbd4e000700 1 -- 172.21.15.37:0/2178065256 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd4400b750 con 0x7fbd500745c0 2024-03-28T18:01:27.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.255+0000 7fbd58000700 1 -- 172.21.15.37:0/2178065256 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd501a6890 con 0x7fbd500745c0 2024-03-28T18:01:27.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.255+0000 7fbd4e000700 1 -- 172.21.15.37:0/2178065256 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd44003970 con 0x7fbd500745c0 2024-03-28T18:01:27.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.255+0000 7fbd58000700 1 -- 172.21.15.37:0/2178065256 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd501a6dc0 con 0x7fbd500745c0 2024-03-28T18:01:27.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.257+0000 7fbd4e000700 1 -- 172.21.15.37:0/2178065256 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fbd44029070 con 0x7fbd500745c0 2024-03-28T18:01:27.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.257+0000 7fbd58000700 1 -- 172.21.15.37:0/2178065256 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd500637c0 con 0x7fbd500745c0 2024-03-28T18:01:27.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.257+0000 7fbd4e000700 1 --2- 172.21.15.37:0/2178065256 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fbd38039450 0x7fbd3803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:27.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.257+0000 7fbd4e000700 1 -- 172.21.15.37:0/2178065256 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbd4404cd20 con 0x7fbd500745c0 2024-03-28T18:01:27.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.257+0000 7fbd54c00700 1 --2- 172.21.15.37:0/2178065256 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fbd38039450 0x7fbd3803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:27.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.258+0000 7fbd54c00700 1 --2- 172.21.15.37:0/2178065256 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fbd38039450 0x7fbd3803b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c009ad0 tx=0x7fbd3c009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:27.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.263+0000 7fbd4e000700 1 -- 172.21.15.37:0/2178065256 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbd44013070 con 0x7fbd500745c0 2024-03-28T18:01:27.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.500+0000 7fbd58000700 1 -- 172.21.15.37:0/2178065256 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fbd501a70b0 con 0x7fbd500745c0 2024-03-28T18:01:27.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.501+0000 7fbd4e000700 1 -- 172.21.15.37:0/2178065256 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+273 (secure 0 0 0) 0x7fbd44030020 con 0x7fbd500745c0 2024-03-28T18:01:27.501 INFO:teuthology.orchestra.run.smithi037.stdout:# minimal ceph.conf for b5ad257c-ed2c-11ee-95d0-87774f69a715 2024-03-28T18:01:27.501 INFO:teuthology.orchestra.run.smithi037.stdout:[global] 2024-03-28T18:01:27.501 INFO:teuthology.orchestra.run.smithi037.stdout: fsid = b5ad257c-ed2c-11ee-95d0-87774f69a715 2024-03-28T18:01:27.501 INFO:teuthology.orchestra.run.smithi037.stdout: mon_host = [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2024-03-28T18:01:27.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.503+0000 7fbd58000700 1 -- 172.21.15.37:0/2178065256 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fbd38039450 msgr2=0x7fbd3803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:27.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.503+0000 7fbd58000700 1 --2- 172.21.15.37:0/2178065256 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fbd38039450 0x7fbd3803b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c009ad0 tx=0x7fbd3c009210 comp rx=0 tx=0).stop 2024-03-28T18:01:27.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.503+0000 7fbd58000700 1 -- 172.21.15.37:0/2178065256 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd500745c0 msgr2=0x7fbd5019f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:27.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.503+0000 7fbd58000700 1 --2- 172.21.15.37:0/2178065256 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd500745c0 0x7fbd5019f7f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbd440092e0 tx=0x7fbd44004fa0 comp rx=0 tx=0).stop 2024-03-28T18:01:27.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.503+0000 7fbd58000700 1 -- 172.21.15.37:0/2178065256 shutdown_connections 2024-03-28T18:01:27.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.503+0000 7fbd58000700 1 --2- 172.21.15.37:0/2178065256 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fbd38039450 0x7fbd3803b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:27.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.503+0000 7fbd58000700 1 --2- 172.21.15.37:0/2178065256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd50199ef0 0x7fbd5019a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:27.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.503+0000 7fbd58000700 1 --2- 172.21.15.37:0/2178065256 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd500745c0 0x7fbd5019f7f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:27.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.503+0000 7fbd58000700 1 --2- 172.21.15.37:0/2178065256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd5019fd30 0x7fbd501999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:27.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.503+0000 7fbd58000700 1 -- 172.21.15.37:0/2178065256 >> 172.21.15.37:0/2178065256 conn(0x7fbd500fd670 msgr2=0x7fbd50105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:27.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.504+0000 7fbd58000700 1 -- 172.21.15.37:0/2178065256 shutdown_connections 2024-03-28T18:01:27.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:27.504+0000 7fbd58000700 1 -- 172.21.15.37:0/2178065256 wait complete. 2024-03-28T18:01:27.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:27 smithi037 ceph-mon[31460]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:27.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:27 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:27 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:27 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:27 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:27 smithi037 ceph-mon[31460]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T18:01:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:27 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:01:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:27 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:01:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:27 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:27 smithi037 ceph-mon[31460]: Reconfiguring daemon mon.a on smithi037 2024-03-28T18:01:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:27 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2178065256' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:27.955 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:27 smithi170 ceph-mon[36280]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:27.955 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:27 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:27.955 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:27 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:27.955 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:27 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:27.955 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:27 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:27.955 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:27 smithi170 ceph-mon[36280]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T18:01:27.955 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:27 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:01:27.956 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:27 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:01:27.956 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:27 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:27.956 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:27 smithi170 ceph-mon[36280]: Reconfiguring daemon mon.a on smithi037 2024-03-28T18:01:27.956 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:27 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2178065256' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:27 smithi059 ceph-mon[36190]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:27 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:27 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:27 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:27 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:27 smithi059 ceph-mon[36190]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T18:01:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:27 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:01:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:27 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:01:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:27 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:27 smithi059 ceph-mon[36190]: Reconfiguring daemon mon.a on smithi037 2024-03-28T18:01:28.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:27 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2178065256' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:28.838 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-28T18:01:28.838 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-28T18:01:28.839 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T18:01:28.937 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-28T18:01:28.937 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:01:29.021 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-28T18:01:29.021 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T18:01:29.061 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-28T18:01:29.061 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:01:29.135 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T18:01:29.135 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T18:01:29.165 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T18:01:29.165 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:01:29.232 INFO:tasks.cephadm:Adding mgr.x on smithi037 2024-03-28T18:01:29.232 INFO:tasks.cephadm:Adding mgr.y on smithi059 2024-03-28T18:01:29.233 INFO:tasks.cephadm:Adding mgr.z on smithi170 2024-03-28T18:01:29.233 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph orch apply mgr '3;smithi037=x;smithi059=y;smithi170=z' 2024-03-28T18:01:29.501 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:01:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:29 smithi037 ceph-mon[31460]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:01:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:01:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:29.947 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:29 smithi170 ceph-mon[36280]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:29.947 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:29.947 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:29.948 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:01:29.948 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:01:29.948 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:29 smithi059 ceph-mon[36190]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:01:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:01:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:30.714 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:30 smithi059 ceph-mon[36190]: Reconfiguring mon.b (monmap changed)... 2024-03-28T18:01:30.714 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:30 smithi059 ceph-mon[36190]: Reconfiguring daemon mon.b on smithi059 2024-03-28T18:01:30.826 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:30 smithi170 ceph-mon[36280]: Reconfiguring mon.b (monmap changed)... 2024-03-28T18:01:30.826 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:30 smithi170 ceph-mon[36280]: Reconfiguring daemon mon.b on smithi059 2024-03-28T18:01:30.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:30 smithi037 ceph-mon[31460]: Reconfiguring mon.b (monmap changed)... 2024-03-28T18:01:30.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:30 smithi037 ceph-mon[31460]: Reconfiguring daemon mon.b on smithi059 2024-03-28T18:01:31.027 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.025+0000 7fd5db5f9700 1 -- 172.21.15.170:0/2311899502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd5d4100bb0 msgr2=0x7fd5d4100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:31.027 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.025+0000 7fd5db5f9700 1 --2- 172.21.15.170:0/2311899502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd5d4100bb0 0x7fd5d4100fd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc00b0a0 tx=0x7fd5cc00b3b0 comp rx=0 tx=0).stop 2024-03-28T18:01:31.027 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.025+0000 7fd5db5f9700 1 -- 172.21.15.170:0/2311899502 shutdown_connections 2024-03-28T18:01:31.028 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.025+0000 7fd5db5f9700 1 --2- 172.21.15.170:0/2311899502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd5d4100bb0 0x7fd5d4100fd0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:31.028 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.025+0000 7fd5db5f9700 1 -- 172.21.15.170:0/2311899502 >> 172.21.15.170:0/2311899502 conn(0x7fd5d40fc170 msgr2=0x7fd5d40fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:31.028 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.025+0000 7fd5db5f9700 1 -- 172.21.15.170:0/2311899502 shutdown_connections 2024-03-28T18:01:31.028 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.025+0000 7fd5db5f9700 1 -- 172.21.15.170:0/2311899502 wait complete. 2024-03-28T18:01:31.028 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.025+0000 7fd5db5f9700 1 Processor -- start 2024-03-28T18:01:31.028 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.026+0000 7fd5db5f9700 1 -- start start 2024-03-28T18:01:31.028 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.026+0000 7fd5db5f9700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5d4100bb0 0x7fd5d41996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:31.028 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.026+0000 7fd5db5f9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5d4199c30 0x7fd5d419eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:31.028 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.026+0000 7fd5db5f9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd5d419a0b0 0x7fd5d41a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:31.028 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.026+0000 7fd5db5f9700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d4106bf0 con 0x7fd5d419a0b0 2024-03-28T18:01:31.028 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.026+0000 7fd5db5f9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d4106a70 con 0x7fd5d4199c30 2024-03-28T18:01:31.028 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.026+0000 7fd5db5f9700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d4106d70 con 0x7fd5d4100bb0 2024-03-28T18:01:31.028 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.026+0000 7fd5d9395700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5d4100bb0 0x7fd5d41996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:31.028 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.026+0000 7fd5d9395700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5d4100bb0 0x7fd5d41996f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:45984/0 (socket says 172.21.15.170:45984) 2024-03-28T18:01:31.029 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.026+0000 7fd5d9395700 1 -- 172.21.15.170:0/1770451875 learned_addr learned my addr 172.21.15.170:0/1770451875 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T18:01:31.029 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.026+0000 7fd5d8b94700 1 --2- 172.21.15.170:0/1770451875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5d4199c30 0x7fd5d419eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:31.029 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.026+0000 7fd5d9395700 1 -- 172.21.15.170:0/1770451875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5d4199c30 msgr2=0x7fd5d419eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:31.029 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.026+0000 7fd5d9395700 1 --2- 172.21.15.170:0/1770451875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5d4199c30 0x7fd5d419eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:31.029 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.026+0000 7fd5d9395700 1 -- 172.21.15.170:0/1770451875 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd5d419a0b0 msgr2=0x7fd5d41a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:31.029 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.026+0000 7fd5d9395700 1 --2- 172.21.15.170:0/1770451875 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd5d419a0b0 0x7fd5d41a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:31.029 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.026+0000 7fd5d9395700 1 -- 172.21.15.170:0/1770451875 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d41a1b00 con 0x7fd5d4100bb0 2024-03-28T18:01:31.029 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.026+0000 7fd5d8b94700 1 --2- 172.21.15.170:0/1770451875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5d4199c30 0x7fd5d419eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:01:31.029 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.026+0000 7fd5d9395700 1 --2- 172.21.15.170:0/1770451875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5d4100bb0 0x7fd5d41996f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc000f80 tx=0x7fd5cc0095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:31.029 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.026+0000 7fd5ca7fc700 1 -- 172.21.15.170:0/1770451875 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc017040 con 0x7fd5d4100bb0 2024-03-28T18:01:31.029 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.027+0000 7fd5ca7fc700 1 -- 172.21.15.170:0/1770451875 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5cc0055b0 con 0x7fd5d4100bb0 2024-03-28T18:01:31.029 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.027+0000 7fd5db5f9700 1 -- 172.21.15.170:0/1770451875 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d41a1d90 con 0x7fd5d4100bb0 2024-03-28T18:01:31.029 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.027+0000 7fd5ca7fc700 1 -- 172.21.15.170:0/1770451875 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc012ba0 con 0x7fd5d4100bb0 2024-03-28T18:01:31.030 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.027+0000 7fd5db5f9700 1 -- 172.21.15.170:0/1770451875 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d41a22f0 con 0x7fd5d4100bb0 2024-03-28T18:01:31.030 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.027+0000 7fd5ca7fc700 1 -- 172.21.15.170:0/1770451875 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fd5cc01b420 con 0x7fd5d4100bb0 2024-03-28T18:01:31.030 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.027+0000 7fd5ca7fc700 1 --2- 172.21.15.170:0/1770451875 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd5c0039450 0x7fd5c003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:31.030 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.027+0000 7fd5db5f9700 1 -- 172.21.15.170:0/1770451875 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5d40637c0 con 0x7fd5d4100bb0 2024-03-28T18:01:31.030 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.028+0000 7fd5ca7fc700 1 -- 172.21.15.170:0/1770451875 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd5cc00f020 con 0x7fd5d4100bb0 2024-03-28T18:01:31.030 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.028+0000 7fd5d8b94700 1 --2- 172.21.15.170:0/1770451875 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd5c0039450 0x7fd5c003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:31.030 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.029+0000 7fd5d8b94700 1 --2- 172.21.15.170:0/1770451875 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd5c0039450 0x7fd5c003b910 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd5c40079b0 tx=0x7fd5c4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:31.032 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.033+0000 7fd5ca7fc700 1 -- 172.21.15.170:0/1770451875 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd5cc029020 con 0x7fd5d4100bb0 2024-03-28T18:01:31.244 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.245+0000 7fd5db5f9700 1 -- 172.21.15.170:0/1770451875 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi037=x;smithi059=y;smithi170=z", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5d40625d0 con 0x7fd5c0039450 2024-03-28T18:01:31.257 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.258+0000 7fd5ca7fc700 1 -- 172.21.15.170:0/1770451875 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd5d40625d0 con 0x7fd5c0039450 2024-03-28T18:01:31.258 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled mgr update... 2024-03-28T18:01:31.260 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.261+0000 7fd5db5f9700 1 -- 172.21.15.170:0/1770451875 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd5c0039450 msgr2=0x7fd5c003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:31.261 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.261+0000 7fd5db5f9700 1 --2- 172.21.15.170:0/1770451875 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd5c0039450 0x7fd5c003b910 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd5c40079b0 tx=0x7fd5c4008040 comp rx=0 tx=0).stop 2024-03-28T18:01:31.261 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.261+0000 7fd5db5f9700 1 -- 172.21.15.170:0/1770451875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5d4100bb0 msgr2=0x7fd5d41996f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:31.261 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.261+0000 7fd5db5f9700 1 --2- 172.21.15.170:0/1770451875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5d4100bb0 0x7fd5d41996f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc000f80 tx=0x7fd5cc0095c0 comp rx=0 tx=0).stop 2024-03-28T18:01:31.261 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.261+0000 7fd5db5f9700 1 -- 172.21.15.170:0/1770451875 shutdown_connections 2024-03-28T18:01:31.261 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.261+0000 7fd5db5f9700 1 --2- 172.21.15.170:0/1770451875 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd5c0039450 0x7fd5c003b910 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:31.261 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.261+0000 7fd5db5f9700 1 --2- 172.21.15.170:0/1770451875 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd5d419a0b0 0x7fd5d41a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:31.261 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.261+0000 7fd5db5f9700 1 --2- 172.21.15.170:0/1770451875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5d4100bb0 0x7fd5d41996f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:31.261 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.261+0000 7fd5db5f9700 1 --2- 172.21.15.170:0/1770451875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5d4199c30 0x7fd5d419eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:31.261 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.261+0000 7fd5db5f9700 1 -- 172.21.15.170:0/1770451875 >> 172.21.15.170:0/1770451875 conn(0x7fd5d40fc170 msgr2=0x7fd5d41906a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:31.261 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.261+0000 7fd5db5f9700 1 -- 172.21.15.170:0/1770451875 shutdown_connections 2024-03-28T18:01:31.261 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:01:31.261+0000 7fd5db5f9700 1 -- 172.21.15.170:0/1770451875 wait complete. 2024-03-28T18:01:31.789 DEBUG:teuthology.orchestra.run.smithi059:mgr.y> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mgr.y.service 2024-03-28T18:01:31.792 DEBUG:teuthology.orchestra.run.smithi170:mgr.z> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mgr.z.service 2024-03-28T18:01:31.795 INFO:tasks.cephadm:Deploying OSDs... 2024-03-28T18:01:31.795 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-28T18:01:31.795 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T18:01:31.855 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T18:01:31.855 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_1 2024-03-28T18:01:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:31 smithi037 ceph-mon[31460]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:31 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:31 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:31 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:01:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:31 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:01:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:31 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:31.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:31 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:31.909 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T18:01:31.909 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:01:31.909 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 631 Links: 1 2024-03-28T18:01:31.909 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:01:31.909 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:01:31.909 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-03-28 18:00:26.743675182 +0000 2024-03-28T18:01:31.909 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-03-28 17:55:18.688492383 +0000 2024-03-28T18:01:31.909 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-03-28 17:55:18.688492383 +0000 2024-03-28T18:01:31.909 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-03-28 17:55:18.688492383 +0000 2024-03-28T18:01:31.909 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T18:01:31.916 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:31 smithi170 ceph-mon[36280]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:31.916 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:31 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:31.916 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:31 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:31.916 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:31 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:01:31.916 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:31 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:01:31.916 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:31 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:31.916 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:31 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:31 smithi059 ceph-mon[36190]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:31 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:31 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:31 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:01:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:31 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:01:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:31 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:31 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:32.051 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-03-28T18:01:32.051 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-03-28T18:01:32.051 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000391197 s, 1.3 MB/s 2024-03-28T18:01:32.056 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T18:01:32.112 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_2 2024-03-28T18:01:32.167 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T18:01:32.167 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:01:32.167 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 650 Links: 1 2024-03-28T18:01:32.167 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:01:32.167 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:01:32.167 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-03-28 18:00:26.816674202 +0000 2024-03-28T18:01:32.167 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-03-28 17:55:19.278484658 +0000 2024-03-28T18:01:32.167 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-03-28 17:55:19.278484658 +0000 2024-03-28T18:01:32.167 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-03-28 17:55:19.278484658 +0000 2024-03-28T18:01:32.167 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T18:01:32.261 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-03-28T18:01:32.261 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-03-28T18:01:32.261 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000340294 s, 1.5 MB/s 2024-03-28T18:01:32.266 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T18:01:32.324 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_3 2024-03-28T18:01:32.375 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T18:01:32.375 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:01:32.375 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 672 Links: 1 2024-03-28T18:01:32.375 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:01:32.375 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:01:32.375 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-03-28 18:00:26.885673274 +0000 2024-03-28T18:01:32.375 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-03-28 17:55:19.900476514 +0000 2024-03-28T18:01:32.375 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-03-28 17:55:19.900476514 +0000 2024-03-28T18:01:32.375 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-03-28 17:55:19.900476514 +0000 2024-03-28T18:01:32.375 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T18:01:32.467 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-03-28T18:01:32.467 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-03-28T18:01:32.467 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000356447 s, 1.4 MB/s 2024-03-28T18:01:32.472 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T18:01:32.530 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_4 2024-03-28T18:01:32.583 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T18:01:32.583 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:01:32.583 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 688 Links: 1 2024-03-28T18:01:32.583 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:01:32.583 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:01:32.583 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-03-28 18:00:26.945672468 +0000 2024-03-28T18:01:32.583 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-03-28 17:55:20.488468813 +0000 2024-03-28T18:01:32.584 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-03-28 17:55:20.488468813 +0000 2024-03-28T18:01:32.584 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-03-28 17:55:20.488468813 +0000 2024-03-28T18:01:32.584 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T18:01:32.677 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-03-28T18:01:32.677 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-03-28T18:01:32.677 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000361494 s, 1.4 MB/s 2024-03-28T18:01:32.680 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T18:01:32.736 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-28T18:01:32.736 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T18:01:32.755 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T18:01:32.755 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_1 2024-03-28T18:01:32.814 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T18:01:32.814 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:01:32.814 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-28T18:01:32.814 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:01:32.814 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:01:32.814 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-28 18:01:17.137014234 +0000 2024-03-28T18:01:32.814 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-28 17:53:59.594851136 +0000 2024-03-28T18:01:32.814 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-28 17:53:59.594851136 +0000 2024-03-28T18:01:32.814 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-03-28 17:53:59.594851136 +0000 2024-03-28T18:01:32.815 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T18:01:32.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:32 smithi037 ceph-mon[31460]: Reconfiguring mon.c (monmap changed)... 2024-03-28T18:01:32.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:32 smithi037 ceph-mon[31460]: Reconfiguring daemon mon.c on smithi170 2024-03-28T18:01:32.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:32 smithi037 ceph-mon[31460]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi037=x;smithi059=y;smithi170=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:01:32.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:32 smithi037 ceph-mon[31460]: Saving service mgr spec with placement smithi037=x;smithi059=y;smithi170=z;count:3 2024-03-28T18:01:32.882 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:32 smithi059 ceph-mon[36190]: Reconfiguring mon.c (monmap changed)... 2024-03-28T18:01:32.882 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:32 smithi059 ceph-mon[36190]: Reconfiguring daemon mon.c on smithi170 2024-03-28T18:01:32.882 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:32 smithi059 ceph-mon[36190]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi037=x;smithi059=y;smithi170=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:01:32.882 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:32 smithi059 ceph-mon[36190]: Saving service mgr spec with placement smithi037=x;smithi059=y;smithi170=z;count:3 2024-03-28T18:01:32.885 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-03-28T18:01:32.885 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-03-28T18:01:32.885 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000273756 s, 1.9 MB/s 2024-03-28T18:01:32.887 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T18:01:32.943 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_2 2024-03-28T18:01:32.998 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T18:01:32.998 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:01:32.998 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-28T18:01:32.998 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:01:32.998 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:01:32.998 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-28 18:01:17.138014222 +0000 2024-03-28T18:01:32.999 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-28 17:53:59.941846973 +0000 2024-03-28T18:01:32.999 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-28 17:53:59.941846973 +0000 2024-03-28T18:01:32.999 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-03-28 17:53:59.941846973 +0000 2024-03-28T18:01:32.999 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T18:01:33.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:32 smithi170 ceph-mon[36280]: Reconfiguring mon.c (monmap changed)... 2024-03-28T18:01:33.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:32 smithi170 ceph-mon[36280]: Reconfiguring daemon mon.c on smithi170 2024-03-28T18:01:33.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:32 smithi170 ceph-mon[36280]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi037=x;smithi059=y;smithi170=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:01:33.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:32 smithi170 ceph-mon[36280]: Saving service mgr spec with placement smithi037=x;smithi059=y;smithi170=z;count:3 2024-03-28T18:01:33.067 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-03-28T18:01:33.067 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-03-28T18:01:33.067 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000202364 s, 2.5 MB/s 2024-03-28T18:01:33.069 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T18:01:33.124 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_3 2024-03-28T18:01:33.180 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T18:01:33.180 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:01:33.180 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-28T18:01:33.181 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:01:33.181 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:01:33.181 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-28 18:01:17.138014222 +0000 2024-03-28T18:01:33.181 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-28 17:54:00.294842738 +0000 2024-03-28T18:01:33.181 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-28 17:54:00.294842738 +0000 2024-03-28T18:01:33.181 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-03-28 17:54:00.294842738 +0000 2024-03-28T18:01:33.181 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T18:01:33.250 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-03-28T18:01:33.250 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-03-28T18:01:33.251 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000237932 s, 2.2 MB/s 2024-03-28T18:01:33.252 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T18:01:33.307 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_4 2024-03-28T18:01:33.363 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T18:01:33.363 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:01:33.363 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-03-28T18:01:33.363 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:01:33.363 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:01:33.363 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-28 18:01:17.138014222 +0000 2024-03-28T18:01:33.363 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-28 17:54:00.638838611 +0000 2024-03-28T18:01:33.363 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-28 17:54:00.638838611 +0000 2024-03-28T18:01:33.364 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-03-28 17:54:00.638838611 +0000 2024-03-28T18:01:33.364 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T18:01:33.432 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-03-28T18:01:33.433 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-03-28T18:01:33.433 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000247861 s, 2.1 MB/s 2024-03-28T18:01:33.434 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T18:01:33.489 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T18:01:33.490 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T18:01:33.505 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T18:01:33.505 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_1 2024-03-28T18:01:33.559 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T18:01:33.560 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:01:33.560 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 660 Links: 1 2024-03-28T18:01:33.560 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:01:33.560 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:01:33.560 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-28 18:01:07.449597635 +0000 2024-03-28T18:01:33.560 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-28 17:53:50.864313229 +0000 2024-03-28T18:01:33.560 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-28 17:53:50.864313229 +0000 2024-03-28T18:01:33.560 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-28 17:53:50.864313229 +0000 2024-03-28T18:01:33.560 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T18:01:33.626 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-28T18:01:33.626 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-28T18:01:33.627 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000109218 s, 4.7 MB/s 2024-03-28T18:01:33.628 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T18:01:33.684 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_2 2024-03-28T18:01:33.738 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T18:01:33.738 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:01:33.738 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-28T18:01:33.738 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:01:33.738 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:01:33.738 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-28 18:01:07.449597635 +0000 2024-03-28T18:01:33.738 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-28 17:53:51.193305591 +0000 2024-03-28T18:01:33.738 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-28 17:53:51.193305591 +0000 2024-03-28T18:01:33.739 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-28 17:53:51.193305591 +0000 2024-03-28T18:01:33.739 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T18:01:33.803 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-28T18:01:33.803 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-28T18:01:33.803 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 9.0637e-05 s, 5.6 MB/s 2024-03-28T18:01:33.805 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T18:01:33.860 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_3 2024-03-28T18:01:33.914 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T18:01:33.914 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:01:33.914 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-28T18:01:33.914 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:01:33.914 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:01:33.914 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-28 18:01:07.450597613 +0000 2024-03-28T18:01:33.914 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-28 17:53:51.506298325 +0000 2024-03-28T18:01:33.915 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-28 17:53:51.506298325 +0000 2024-03-28T18:01:33.915 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-28 17:53:51.506298325 +0000 2024-03-28T18:01:33.915 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T18:01:33.979 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-28T18:01:33.979 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-28T18:01:33.979 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 9.3551e-05 s, 5.5 MB/s 2024-03-28T18:01:33.980 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T18:01:34.035 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_4 2024-03-28T18:01:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:33 smithi170 ceph-mon[36280]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:33 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:33 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:33 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:01:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:33 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:33 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:01:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:33 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:33 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:01:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:33 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T18:01:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:33 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:01:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:33 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:34.089 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T18:01:34.089 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:01:34.089 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-28T18:01:34.089 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:01:34.090 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:01:34.090 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-28 18:01:07.450597613 +0000 2024-03-28T18:01:34.090 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-28 17:53:51.820291036 +0000 2024-03-28T18:01:34.090 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-28 17:53:51.820291036 +0000 2024-03-28T18:01:34.090 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-28 17:53:51.820291036 +0000 2024-03-28T18:01:34.090 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T18:01:34.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:33 smithi037 ceph-mon[31460]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:34.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:34.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:34.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:01:34.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:34.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:01:34.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:34.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:01:34.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T18:01:34.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:01:34.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:34.154 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-28T18:01:34.154 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-28T18:01:34.154 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000108368 s, 4.7 MB/s 2024-03-28T18:01:34.156 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T18:01:34.211 INFO:tasks.cephadm:Deploying osd.0 on smithi037 with /dev/vg_nvme/lv_4... 2024-03-28T18:01:34.211 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T18:01:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:33 smithi059 ceph-mon[36190]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:33 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:33 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:33 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:01:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:33 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:33 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:01:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:33 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:33 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:01:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:33 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T18:01:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:33 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:01:34.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:33 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:34.669 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:01:35.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:34 smithi170 ceph-mon[36280]: Deploying daemon mgr.z on smithi170 2024-03-28T18:01:35.036 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:34 smithi170 systemd[1]: Starting Ceph mgr.z for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:01:35.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:34 smithi037 ceph-mon[31460]: Deploying daemon mgr.z on smithi170 2024-03-28T18:01:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:34 smithi059 ceph-mon[36190]: Deploying daemon mgr.z on smithi170 2024-03-28T18:01:35.535 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:35 smithi170 podman[38292]: 2024-03-28 18:01:35.056234218 +0000 UTC m=+0.113131124 container create c20168783afc22e438231d335432974f56d97835d18e23e530a44f2f2582ac7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:01:35.535 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:35 smithi170 podman[38292]: 2024-03-28 18:01:34.960513592 +0000 UTC m=+0.017410516 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:01:35.536 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:35 smithi170 podman[38292]: 2024-03-28 18:01:35.23304065 +0000 UTC m=+0.289937563 container init c20168783afc22e438231d335432974f56d97835d18e23e530a44f2f2582ac7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-28T18:01:35.536 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:35 smithi170 podman[38292]: 2024-03-28 18:01:35.235661102 +0000 UTC m=+0.292558011 container start c20168783afc22e438231d335432974f56d97835d18e23e530a44f2f2582ac7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:01:35.536 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:35 smithi170 ceph-mgr[38311]: -- 172.21.15.170:0/1317052861 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56048366c5a0 con 0x560483640c00 2024-03-28T18:01:35.536 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:35 smithi170 bash[38292]: c20168783afc22e438231d335432974f56d97835d18e23e530a44f2f2582ac7a 2024-03-28T18:01:35.536 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:35 smithi170 systemd[1]: Started Ceph mgr.z for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:01:35.806 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:35 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:35.776+0000 7f3e91fb1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:01:36.030 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:35 smithi037 ceph-mon[31460]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:36.030 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:35 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:36.030 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:35 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:36.031 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:35 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:36.031 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:35 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:01:36.031 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:35 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T18:01:36.031 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:35 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:01:36.031 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:35 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:35 smithi059 ceph-mon[36190]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:35 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:35 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:35 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:35 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:01:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:35 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T18:01:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:35 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:01:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:35 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:36.285 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:35 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:35.869+0000 7f3e91fb1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:01:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:35 smithi170 ceph-mon[36280]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:35 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:35 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:35 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:35 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:01:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:35 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T18:01:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:35 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:01:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:35 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:36.785 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:36 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:36.507+0000 7f3e91fb1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:01:36.786 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:36 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:36.600+0000 7f3e91fb1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:01:36.786 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:36 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:36.740+0000 7f3e91fb1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:01:37.111 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:36 smithi059 ceph-mon[36190]: Deploying daemon mgr.y on smithi059 2024-03-28T18:01:37.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:36 smithi037 ceph-mon[31460]: Deploying daemon mgr.y on smithi059 2024-03-28T18:01:37.183 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:36 smithi170 ceph-mon[36280]: Deploying daemon mgr.y on smithi059 2024-03-28T18:01:37.183 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:36 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:36.835+0000 7f3e91fb1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:01:37.183 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:36 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:36.917+0000 7f3e91fb1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:01:37.183 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:37 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:37.182+0000 7f3e91fb1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:01:37.467 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:37 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:37.289+0000 7f3e91fb1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:01:37.467 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:37 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:37.373+0000 7f3e91fb1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:01:37.467 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:37 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:37.467+0000 7f3e91fb1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:01:37.539 INFO:teuthology.orchestra.run.smithi037.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T18:01:37.539 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T18:01:37.539 INFO:teuthology.orchestra.run.smithi037.stderr: stderr: 10+0 records in 2024-03-28T18:01:37.539 INFO:teuthology.orchestra.run.smithi037.stderr:10+0 records out 2024-03-28T18:01:37.539 INFO:teuthology.orchestra.run.smithi037.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0345161 s, 304 MB/s 2024-03-28T18:01:37.539 INFO:teuthology.orchestra.run.smithi037.stderr:--> Zapping successful for: 2024-03-28T18:01:37.785 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:37 smithi059 podman[37561]: 2024-03-28 18:01:37.722902221 +0000 UTC m=+0.123189461 container create 72e8a8cfa8e1a4159ff306bb6e80ed4b308bbf1917a4170b8f9194cee6243f4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-28T18:01:37.785 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:37 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:37.643+0000 7f3e91fb1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:01:38.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:37 smithi037 ceph-mon[31460]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:38.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:37 smithi059 ceph-mon[36190]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:38.285 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:37 smithi059 podman[37561]: 2024-03-28 18:01:37.950526973 +0000 UTC m=+0.350814223 container init 72e8a8cfa8e1a4159ff306bb6e80ed4b308bbf1917a4170b8f9194cee6243f4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:01:38.285 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:37 smithi059 podman[37561]: 2024-03-28 18:01:37.953202505 +0000 UTC m=+0.353489739 container start 72e8a8cfa8e1a4159ff306bb6e80ed4b308bbf1917a4170b8f9194cee6243f4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-28T18:01:38.285 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:37 smithi059 ceph-mgr[37580]: -- 172.21.15.59:0/3678442631 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5596be7565a0 con 0x5596be72a800 2024-03-28T18:01:38.285 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:38 smithi059 bash[37561]: 72e8a8cfa8e1a4159ff306bb6e80ed4b308bbf1917a4170b8f9194cee6243f4e 2024-03-28T18:01:38.285 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:38 smithi059 systemd[1]: Started Ceph mgr.y for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:01:38.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:37 smithi170 ceph-mon[36280]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:38.416 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi037:vg_nvme/lv_4 2024-03-28T18:01:38.785 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:38.514+0000 7f637601f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:01:38.785 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:38.612+0000 7f637601f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:01:38.897 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:01:39.035 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:38 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:38.624+0000 7f3e91fb1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:01:39.035 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:38 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:38.760+0000 7f3e91fb1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:01:39.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:39 smithi037 ceph-mon[31460]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:39.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:39 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:39.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:39 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:39.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:39 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:39.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:39 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:39.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:39 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:01:39.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:39 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:39.410 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:39 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:39.121+0000 7f3e91fb1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:01:39.410 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:39 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:39.209+0000 7f3e91fb1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:01:39.410 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:39 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:39.409+0000 7f3e91fb1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:01:39.411 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:39 smithi170 ceph-mon[36280]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:39.411 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:39 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:39.411 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:39 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:39.411 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:39 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:39.411 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:39 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:39.411 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:39 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:01:39.411 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:39 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:39.416 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:39 smithi059 ceph-mon[36190]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:39.416 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:39 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:39.416 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:39 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:39.416 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:39 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:39.416 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:39 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:39.416 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:39 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:01:39.416 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:39 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:39.416 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:39 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:39.317+0000 7f637601f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:01:39.416 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:39 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:39.415+0000 7f637601f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:01:39.749 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:39 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:39.564+0000 7f637601f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:01:39.749 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:39 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:39.663+0000 7f637601f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:01:39.749 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:39 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:39.748+0000 7f637601f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:01:39.785 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:39 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:39.602+0000 7f3e91fb1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:01:40.162 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:39 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:39.896+0000 7f3e91fb1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:01:40.162 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:40.161+0000 7f3e91fb1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:01:40.285 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:40.027+0000 7f637601f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:01:40.285 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:40.139+0000 7f637601f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:01:40.285 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:40.229+0000 7f637601f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:01:40.535 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:40.398+0000 7f3e91fb1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:01:40.785 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:40.326+0000 7f637601f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:01:40.785 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:40.506+0000 7f637601f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:01:40.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.920+0000 7f95a9a00700 1 -- 172.21.15.37:0/2711473401 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95a41021c0 msgr2=0x7f95a41025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:40.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.920+0000 7f95a9a00700 1 --2- 172.21.15.37:0/2711473401 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95a41021c0 0x7f95a41025e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f95940099c0 tx=0x7f9594009cd0 comp rx=0 tx=0).stop 2024-03-28T18:01:40.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.921+0000 7f95a9a00700 1 -- 172.21.15.37:0/2711473401 shutdown_connections 2024-03-28T18:01:40.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.921+0000 7f95a9a00700 1 --2- 172.21.15.37:0/2711473401 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95a4103d60 0x7f95a410a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:40.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.921+0000 7f95a9a00700 1 --2- 172.21.15.37:0/2711473401 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95a41021c0 0x7f95a41025e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:40.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.921+0000 7f95a9a00700 1 --2- 172.21.15.37:0/2711473401 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95a41033c0 0x7f95a4103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:40.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.921+0000 7f95a9a00700 1 -- 172.21.15.37:0/2711473401 >> 172.21.15.37:0/2711473401 conn(0x7f95a40fd7a0 msgr2=0x7f95a40ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:40.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.921+0000 7f95a9a00700 1 -- 172.21.15.37:0/2711473401 shutdown_connections 2024-03-28T18:01:40.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.921+0000 7f95a9a00700 1 -- 172.21.15.37:0/2711473401 wait complete. 2024-03-28T18:01:40.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.922+0000 7f95a9a00700 1 Processor -- start 2024-03-28T18:01:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.922+0000 7f95a9a00700 1 -- start start 2024-03-28T18:01:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.923+0000 7f95a9a00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95a41021c0 0x7f95a419a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.923+0000 7f95a9a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95a41033c0 0x7f95a419abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.923+0000 7f95a9a00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95a4103d60 0x7f95a41a1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.923+0000 7f95a9a00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a410ddd0 con 0x7f95a41033c0 2024-03-28T18:01:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.923+0000 7f95a9a00700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a410dc50 con 0x7f95a4103d60 2024-03-28T18:01:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.923+0000 7f95a9a00700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a410df50 con 0x7f95a41021c0 2024-03-28T18:01:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.923+0000 7f95a2a00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95a41021c0 0x7f95a419a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.923+0000 7f95a2a00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95a41021c0 0x7f95a419a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:44538/0 (socket says 172.21.15.37:44538) 2024-03-28T18:01:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.923+0000 7f95a2a00700 1 -- 172.21.15.37:0/3165839704 learned_addr learned my addr 172.21.15.37:0/3165839704 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:01:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.923+0000 7f95a3400700 1 --2- 172.21.15.37:0/3165839704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95a4103d60 0x7f95a41a1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.923+0000 7f95a2000700 1 --2- 172.21.15.37:0/3165839704 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95a41033c0 0x7f95a419abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.924+0000 7f95a2a00700 1 -- 172.21.15.37:0/3165839704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95a4103d60 msgr2=0x7f95a41a1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.924+0000 7f95a2a00700 1 --2- 172.21.15.37:0/3165839704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95a4103d60 0x7f95a41a1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.924+0000 7f95a2a00700 1 -- 172.21.15.37:0/3165839704 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95a41033c0 msgr2=0x7f95a419abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.924+0000 7f95a2a00700 1 --2- 172.21.15.37:0/3165839704 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95a41033c0 0x7f95a419abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.924+0000 7f95a2a00700 1 -- 172.21.15.37:0/3165839704 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a41a22c0 con 0x7f95a41021c0 2024-03-28T18:01:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.924+0000 7f95a2000700 1 --2- 172.21.15.37:0/3165839704 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95a41033c0 0x7f95a419abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:01:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.924+0000 7f95a2a00700 1 --2- 172.21.15.37:0/3165839704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95a41021c0 0x7f95a419a6b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9594005b40 tx=0x7f9594009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.924+0000 7f959b400700 1 -- 172.21.15.37:0/3165839704 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9594027070 con 0x7f95a41021c0 2024-03-28T18:01:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.924+0000 7f959b400700 1 -- 172.21.15.37:0/3165839704 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9594022470 con 0x7f95a41021c0 2024-03-28T18:01:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.924+0000 7f959b400700 1 -- 172.21.15.37:0/3165839704 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9594009e60 con 0x7f95a41021c0 2024-03-28T18:01:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.925+0000 7f95a9a00700 1 -- 172.21.15.37:0/3165839704 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a41a2550 con 0x7f95a41021c0 2024-03-28T18:01:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.925+0000 7f95a9a00700 1 -- 172.21.15.37:0/3165839704 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a41a2950 con 0x7f95a41021c0 2024-03-28T18:01:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.926+0000 7f959b400700 1 -- 172.21.15.37:0/3165839704 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f959402a070 con 0x7f95a41021c0 2024-03-28T18:01:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.926+0000 7f95a9a00700 1 -- 172.21.15.37:0/3165839704 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a4068e50 con 0x7f95a41021c0 2024-03-28T18:01:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.926+0000 7f959b400700 1 --2- 172.21.15.37:0/3165839704 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9588039200 0x7f958803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.926+0000 7f959b400700 1 -- 172.21.15.37:0/3165839704 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f959404cfa0 con 0x7f95a41021c0 2024-03-28T18:01:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.927+0000 7f95a2000700 1 --2- 172.21.15.37:0/3165839704 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9588039200 0x7f958803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.928+0000 7f95a2000700 1 --2- 172.21.15.37:0/3165839704 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9588039200 0x7f958803b6c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9590009cc0 tx=0x7f9590009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:40.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:40.932+0000 7f959b400700 1 -- 172.21.15.37:0/3165839704 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9594013070 con 0x7f95a41021c0 2024-03-28T18:01:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:40 smithi037 ceph-mon[31460]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:40 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:40 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:40 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:40 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:40 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:40 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:01:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:40 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:40 smithi037 ceph-mon[31460]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T18:01:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:40 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:01:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:40 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:01:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:40 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:41.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:40 smithi037 ceph-mon[31460]: Reconfiguring daemon mgr.x on smithi037 2024-03-28T18:01:41.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:41.185+0000 7f95a9a00700 1 -- 172.21.15.37:0/3165839704 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f95a4075ae0 con 0x7f9588039200 2024-03-28T18:01:41.193 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:40 smithi170 ceph-mon[36280]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:41.193 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:40 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:41.193 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:40 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:41.194 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:40 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:41.194 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:40 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:41.194 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:40 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:41.194 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:40 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:01:41.194 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:40 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:41.194 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:40 smithi170 ceph-mon[36280]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T18:01:41.194 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:40 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:01:41.194 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:40 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:01:41.194 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:40 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:41.194 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:40 smithi170 ceph-mon[36280]: Reconfiguring daemon mgr.x on smithi037 2024-03-28T18:01:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:40 smithi059 ceph-mon[36190]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:40 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:40 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:40 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:40 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:40 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:40 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:01:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:40 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:40 smithi059 ceph-mon[36190]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T18:01:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:40 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:01:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:40 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:01:41.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:40 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:41.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:40 smithi059 ceph-mon[36190]: Reconfiguring daemon mgr.x on smithi037 2024-03-28T18:01:41.535 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:41.192+0000 7f3e91fb1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:01:41.785 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:41 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:41.526+0000 7f637601f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:01:41.785 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:41 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:41.660+0000 7f637601f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:01:42.035 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:41.760+0000 7f3e91fb1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:01:42.035 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:41.857+0000 7f3e91fb1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:01:42.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:41 smithi170 ceph-mon[36280]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:01:42.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:41 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:01:42.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:41 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:01:42.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:41 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:42.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:41 smithi037 ceph-mon[31460]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:01:42.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:41 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:01:42.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:41 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:01:42.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:41 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:41 smithi059 ceph-mon[36190]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:01:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:41 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:01:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:41 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:01:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:41 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:42.285 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:42 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:42.002+0000 7f637601f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:01:42.285 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:42 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:42.093+0000 7f637601f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:01:42.535 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:01:42 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:01:42.072+0000 7f3e91fb1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:01:42.780 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:42 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:42.296+0000 7f637601f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:01:42.781 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:42 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:42.486+0000 7f637601f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:01:42.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:42.868+0000 7f959b400700 1 -- 172.21.15.37:0/3165839704 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mgrmap(e 14) v1 ==== 91698+0+0 (secure 0 0 0) 0x7f9594017650 con 0x7f95a41021c0 2024-03-28T18:01:43.035 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:42 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:42.780+0000 7f637601f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:01:43.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:42 smithi059 ceph-mon[36190]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:43.037 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:42 smithi059 ceph-mon[36190]: Standby manager daemon z started 2024-03-28T18:01:43.038 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:42 smithi059 ceph-mon[36190]: from='mgr.? 172.21.15.170:0/1873861318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T18:01:43.038 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:42 smithi059 ceph-mon[36190]: from='mgr.? 172.21.15.170:0/1873861318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:01:43.038 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:42 smithi059 ceph-mon[36190]: from='mgr.? 172.21.15.170:0/1873861318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T18:01:43.038 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:42 smithi059 ceph-mon[36190]: from='mgr.? 172.21.15.170:0/1873861318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:01:43.038 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:42 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:43.038 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:42 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:43.038 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:42 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:01:43.038 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:42 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:43.038 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:42 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:01:43.038 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:42 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:43.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:42 smithi037 ceph-mon[31460]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:43.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:42 smithi037 ceph-mon[31460]: Standby manager daemon z started 2024-03-28T18:01:43.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:42 smithi037 ceph-mon[31460]: from='mgr.? 172.21.15.170:0/1873861318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T18:01:43.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:42 smithi037 ceph-mon[31460]: from='mgr.? 172.21.15.170:0/1873861318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:01:43.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:42 smithi037 ceph-mon[31460]: from='mgr.? 172.21.15.170:0/1873861318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T18:01:43.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:42 smithi037 ceph-mon[31460]: from='mgr.? 172.21.15.170:0/1873861318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:01:43.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:42 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:43.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:42 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:43.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:42 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:01:43.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:42 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:43.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:42 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:01:43.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:42 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:43.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:42 smithi170 ceph-mon[36280]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:43.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:42 smithi170 ceph-mon[36280]: Standby manager daemon z started 2024-03-28T18:01:43.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:42 smithi170 ceph-mon[36280]: from='mgr.? 172.21.15.170:0/1873861318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T18:01:43.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:42 smithi170 ceph-mon[36280]: from='mgr.? 172.21.15.170:0/1873861318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:01:43.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:42 smithi170 ceph-mon[36280]: from='mgr.? 172.21.15.170:0/1873861318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T18:01:43.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:42 smithi170 ceph-mon[36280]: from='mgr.? 172.21.15.170:0/1873861318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:01:43.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:42 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:43.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:42 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:43.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:42 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:01:43.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:42 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:43.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:42 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:01:43.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:42 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:01:43.477 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:43.047+0000 7f637601f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:01:43.477 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:43.290+0000 7f637601f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:01:44.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:43 smithi037 ceph-mon[31460]: mgrmap e14: x(active, since 89s), standbys: z 2024-03-28T18:01:44.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:43 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T18:01:44.129 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:43 smithi059 ceph-mon[36190]: mgrmap e14: x(active, since 89s), standbys: z 2024-03-28T18:01:44.129 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:43 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T18:01:44.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:43 smithi170 ceph-mon[36280]: mgrmap e14: x(active, since 89s), standbys: z 2024-03-28T18:01:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:43 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T18:01:44.534 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:44 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:44.129+0000 7f637601f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:01:45.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:44 smithi059 ceph-mon[36190]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:45.035 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:44 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:44.719+0000 7f637601f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:01:45.035 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:44 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:44.821+0000 7f637601f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:01:45.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:44 smithi037 ceph-mon[31460]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:45.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:44 smithi170 ceph-mon[36280]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:45.535 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:01:45 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:01:45.046+0000 7f637601f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:01:45.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:01:45.896+0000 7f959b400700 1 -- 172.21.15.37:0/3165839704 <== mon.1 v2:172.21.15.170:3300/0 8 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f95940163a0 con 0x7f95a41021c0 2024-03-28T18:01:46.223 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:45 smithi037 ceph-mon[31460]: from='mgr.? 172.21.15.59:0/131087079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T18:01:46.223 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:45 smithi037 ceph-mon[31460]: from='mgr.? 172.21.15.59:0/131087079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:01:46.223 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:45 smithi037 ceph-mon[31460]: Standby manager daemon y started 2024-03-28T18:01:46.223 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:45 smithi037 ceph-mon[31460]: from='mgr.? 172.21.15.59:0/131087079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T18:01:46.224 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:45 smithi037 ceph-mon[31460]: from='mgr.? 172.21.15.59:0/131087079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:01:46.224 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:45 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3374884709' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db60618d-56f7-4130-8a90-84cbc480965b"}]: dispatch 2024-03-28T18:01:46.224 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:45 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db60618d-56f7-4130-8a90-84cbc480965b"}]: dispatch 2024-03-28T18:01:46.224 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:45 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db60618d-56f7-4130-8a90-84cbc480965b"}]': finished 2024-03-28T18:01:46.224 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:45 smithi037 ceph-mon[31460]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T18:01:46.224 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:45 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:01:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:45 smithi059 ceph-mon[36190]: from='mgr.? 172.21.15.59:0/131087079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T18:01:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:45 smithi059 ceph-mon[36190]: from='mgr.? 172.21.15.59:0/131087079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:01:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:45 smithi059 ceph-mon[36190]: Standby manager daemon y started 2024-03-28T18:01:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:45 smithi059 ceph-mon[36190]: from='mgr.? 172.21.15.59:0/131087079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T18:01:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:45 smithi059 ceph-mon[36190]: from='mgr.? 172.21.15.59:0/131087079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:01:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:45 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3374884709' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db60618d-56f7-4130-8a90-84cbc480965b"}]: dispatch 2024-03-28T18:01:46.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:45 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db60618d-56f7-4130-8a90-84cbc480965b"}]: dispatch 2024-03-28T18:01:46.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:45 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db60618d-56f7-4130-8a90-84cbc480965b"}]': finished 2024-03-28T18:01:46.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:45 smithi059 ceph-mon[36190]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T18:01:46.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:45 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:01:46.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:45 smithi170 ceph-mon[36280]: from='mgr.? 172.21.15.59:0/131087079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T18:01:46.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:45 smithi170 ceph-mon[36280]: from='mgr.? 172.21.15.59:0/131087079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:01:46.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:45 smithi170 ceph-mon[36280]: Standby manager daemon y started 2024-03-28T18:01:46.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:45 smithi170 ceph-mon[36280]: from='mgr.? 172.21.15.59:0/131087079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T18:01:46.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:45 smithi170 ceph-mon[36280]: from='mgr.? 172.21.15.59:0/131087079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:01:46.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:45 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3374884709' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db60618d-56f7-4130-8a90-84cbc480965b"}]: dispatch 2024-03-28T18:01:46.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:45 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db60618d-56f7-4130-8a90-84cbc480965b"}]: dispatch 2024-03-28T18:01:46.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:45 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db60618d-56f7-4130-8a90-84cbc480965b"}]': finished 2024-03-28T18:01:46.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:45 smithi170 ceph-mon[36280]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T18:01:46.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:45 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:01:47.199 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:46 smithi037 ceph-mon[31460]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:47.199 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:46 smithi037 ceph-mon[31460]: mgrmap e15: x(active, since 92s), standbys: y, z 2024-03-28T18:01:47.199 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:46 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T18:01:47.199 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:46 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3956631949' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:01:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:46 smithi059 ceph-mon[36190]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:46 smithi059 ceph-mon[36190]: mgrmap e15: x(active, since 92s), standbys: y, z 2024-03-28T18:01:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:46 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T18:01:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:46 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3956631949' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:01:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:46 smithi170 ceph-mon[36280]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:46 smithi170 ceph-mon[36280]: mgrmap e15: x(active, since 92s), standbys: y, z 2024-03-28T18:01:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:46 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T18:01:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:46 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3956631949' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:01:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:48 smithi059 ceph-mon[36190]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:48 smithi170 ceph-mon[36280]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:49.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:48 smithi037 ceph-mon[31460]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:51.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:50 smithi037 ceph-mon[31460]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:50 smithi059 ceph-mon[36190]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:50 smithi170 ceph-mon[36280]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:53.085 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:52 smithi037 ceph-mon[31460]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:52 smithi059 ceph-mon[36190]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:52 smithi170 ceph-mon[36280]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:54 smithi059 ceph-mon[36190]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:54 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T18:01:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:54 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:54 smithi059 ceph-mon[36190]: Deploying daemon osd.0 on smithi037 2024-03-28T18:01:55.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:54 smithi170 ceph-mon[36280]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:55.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:54 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T18:01:55.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:54 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:55.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:54 smithi170 ceph-mon[36280]: Deploying daemon osd.0 on smithi037 2024-03-28T18:01:55.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:54 smithi037 ceph-mon[31460]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:55.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:54 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T18:01:55.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:54 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:55.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:54 smithi037 ceph-mon[31460]: Deploying daemon osd.0 on smithi037 2024-03-28T18:01:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:56 smithi037 ceph-mon[31460]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:56 smithi059 ceph-mon[36190]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:56 smithi170 ceph-mon[36280]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:59.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:01:58 smithi037 ceph-mon[31460]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:59.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:01:58 smithi059 ceph-mon[36190]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:01:59.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:01:58 smithi170 ceph-mon[36280]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:02:01.190 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:00 smithi037 ceph-mon[31460]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:02:01.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:00 smithi059 ceph-mon[36190]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:02:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:00 smithi170 ceph-mon[36280]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:02:03.998 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:03 smithi037 ceph-mon[31460]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:02:03.998 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:03.998 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:03.998 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:03 smithi059 ceph-mon[36190]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:02:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:03 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:03 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:03 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:04.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:03 smithi170 ceph-mon[36280]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:02:04.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:03 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:04.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:03 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:04.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:03 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:05.975 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:05 smithi037 ceph-mon[31460]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:02:06.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:05 smithi059 ceph-mon[36190]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:02:06.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:05 smithi170 ceph-mon[36280]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:02:07.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:07 smithi037 ceph-mon[31460]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:02:07.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:07.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:07.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:07.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:07.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:07.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:07 smithi037 ceph-mon[31460]: from='osd.0 [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T18:02:07.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:07 smithi059 ceph-mon[36190]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:02:07.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:07.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:07.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:07.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:07.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:07.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:07 smithi059 ceph-mon[36190]: from='osd.0 [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T18:02:07.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:07 smithi170 ceph-mon[36280]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:02:07.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:07.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:07.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:07.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:07.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:07.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:07 smithi170 ceph-mon[36280]: from='osd.0 [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T18:02:07.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:02:07.543+0000 7f959b400700 1 -- 172.21.15.37:0/3165839704 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f95a4075ae0 con 0x7f9588039200 2024-03-28T18:02:07.543 INFO:teuthology.orchestra.run.smithi037.stdout:Created osd(s) 0 on host 'smithi037' 2024-03-28T18:02:07.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:02:07.546+0000 7f95a9a00700 1 -- 172.21.15.37:0/3165839704 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9588039200 msgr2=0x7f958803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:07.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:02:07.546+0000 7f95a9a00700 1 --2- 172.21.15.37:0/3165839704 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9588039200 0x7f958803b6c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9590009cc0 tx=0x7f9590009260 comp rx=0 tx=0).stop 2024-03-28T18:02:07.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:02:07.546+0000 7f95a9a00700 1 -- 172.21.15.37:0/3165839704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95a41021c0 msgr2=0x7f95a419a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:07.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:02:07.546+0000 7f95a9a00700 1 --2- 172.21.15.37:0/3165839704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95a41021c0 0x7f95a419a6b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9594005b40 tx=0x7f9594009580 comp rx=0 tx=0).stop 2024-03-28T18:02:07.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:02:07.546+0000 7f95a9a00700 1 -- 172.21.15.37:0/3165839704 shutdown_connections 2024-03-28T18:02:07.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:02:07.546+0000 7f95a9a00700 1 --2- 172.21.15.37:0/3165839704 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9588039200 0x7f958803b6c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:07.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:02:07.546+0000 7f95a9a00700 1 --2- 172.21.15.37:0/3165839704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95a4103d60 0x7f95a41a1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:07.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:02:07.546+0000 7f95a9a00700 1 --2- 172.21.15.37:0/3165839704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95a41021c0 0x7f95a419a6b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:07.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:02:07.546+0000 7f95a9a00700 1 --2- 172.21.15.37:0/3165839704 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95a41033c0 0x7f95a419abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:07.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:02:07.546+0000 7f95a9a00700 1 -- 172.21.15.37:0/3165839704 >> 172.21.15.37:0/3165839704 conn(0x7f95a40fd7a0 msgr2=0x7f95a40743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:02:07.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:02:07.547+0000 7f95a9a00700 1 -- 172.21.15.37:0/3165839704 shutdown_connections 2024-03-28T18:02:07.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:02:07.547+0000 7f95a9a00700 1 -- 172.21.15.37:0/3165839704 wait complete. 2024-03-28T18:02:08.333 DEBUG:teuthology.orchestra.run.smithi037:osd.0> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@osd.0.service 2024-03-28T18:02:08.337 INFO:tasks.cephadm:Deploying osd.1 on smithi059 with /dev/vg_nvme/lv_4... 2024-03-28T18:02:08.337 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T18:02:08.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:08 smithi059 ceph-mon[36190]: from='osd.0 [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T18:02:08.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:08 smithi059 ceph-mon[36190]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T18:02:08.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:08 smithi059 ceph-mon[36190]: from='osd.0 [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]: dispatch 2024-03-28T18:02:08.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:08 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:02:08.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:08 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:08.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:08 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:08.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:08 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:08.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:08 smithi170 ceph-mon[36280]: from='osd.0 [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T18:02:08.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:08 smithi170 ceph-mon[36280]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T18:02:08.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:08 smithi170 ceph-mon[36280]: from='osd.0 [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]: dispatch 2024-03-28T18:02:08.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:02:08.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:08.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:08.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:08.582 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:08 smithi037 ceph-mon[31460]: from='osd.0 [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T18:02:08.582 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:08 smithi037 ceph-mon[31460]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T18:02:08.582 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:08 smithi037 ceph-mon[31460]: from='osd.0 [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]: dispatch 2024-03-28T18:02:08.582 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:08 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:02:08.582 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:08 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:08.583 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:08 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:08.583 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:08 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:08.593 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:02:09.244 INFO:journalctl@ceph.osd.0.smithi037.stdout:Mar 28 18:02:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-0[43906]: 2024-03-28T18:02:08.975+0000 7f73e1800700 -1 osd.0 0 waiting for initial osdmap 2024-03-28T18:02:09.244 INFO:journalctl@ceph.osd.0.smithi037.stdout:Mar 28 18:02:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-0[43906]: 2024-03-28T18:02:08.989+0000 7f73d7600700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:02:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:09 smithi059 ceph-mon[36190]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:02:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:09 smithi059 ceph-mon[36190]: from='osd.0 [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]': finished 2024-03-28T18:02:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:09 smithi059 ceph-mon[36190]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T18:02:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:09 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:02:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:09 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:02:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:09 smithi059 ceph-mon[36190]: from='osd.0 [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035]' entity='osd.0' 2024-03-28T18:02:09.495 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:09 smithi037 ceph-mon[31460]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:02:09.495 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:09 smithi037 ceph-mon[31460]: from='osd.0 [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]': finished 2024-03-28T18:02:09.495 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:09 smithi037 ceph-mon[31460]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T18:02:09.495 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:09 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:02:09.495 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:09 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:02:09.495 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:09 smithi037 ceph-mon[31460]: from='osd.0 [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035]' entity='osd.0' 2024-03-28T18:02:09.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:09 smithi170 ceph-mon[36280]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:02:09.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:09 smithi170 ceph-mon[36280]: from='osd.0 [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]': finished 2024-03-28T18:02:09.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:09 smithi170 ceph-mon[36280]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T18:02:09.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:09 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:02:09.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:09 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:02:09.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:09 smithi170 ceph-mon[36280]: from='osd.0 [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035]' entity='osd.0' 2024-03-28T18:02:10.515 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T18:02:10.515 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T18:02:10.515 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10+0 records in 2024-03-28T18:02:10.515 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2024-03-28T18:02:10.515 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109739 s, 956 MB/s 2024-03-28T18:02:10.515 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping successful for: 2024-03-28T18:02:10.515 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:10 smithi059 ceph-mon[36190]: purged_snaps scrub starts 2024-03-28T18:02:10.515 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:10 smithi059 ceph-mon[36190]: purged_snaps scrub ok 2024-03-28T18:02:10.515 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:10 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:02:10.515 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:10 smithi059 ceph-mon[36190]: osd.0 [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035] boot 2024-03-28T18:02:10.515 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:10 smithi059 ceph-mon[36190]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T18:02:10.516 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:10 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:02:10.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:10 smithi170 ceph-mon[36280]: purged_snaps scrub starts 2024-03-28T18:02:10.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:10 smithi170 ceph-mon[36280]: purged_snaps scrub ok 2024-03-28T18:02:10.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:10 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:02:10.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:10 smithi170 ceph-mon[36280]: osd.0 [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035] boot 2024-03-28T18:02:10.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:10 smithi170 ceph-mon[36280]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T18:02:10.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:10 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:02:10.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:10 smithi037 ceph-mon[31460]: purged_snaps scrub starts 2024-03-28T18:02:10.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:10 smithi037 ceph-mon[31460]: purged_snaps scrub ok 2024-03-28T18:02:10.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:10 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:02:10.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:10 smithi037 ceph-mon[31460]: osd.0 [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035] boot 2024-03-28T18:02:10.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:10 smithi037 ceph-mon[31460]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T18:02:10.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:10 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:02:11.102 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi059:vg_nvme/lv_4 2024-03-28T18:02:11.352 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:02:11.473 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:11 smithi059 ceph-mon[36190]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:02:11.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:11 smithi170 ceph-mon[36280]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:02:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:11 smithi037 ceph-mon[31460]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:02:12.370 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:12 smithi059 ceph-mon[36190]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T18:02:12.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:12 smithi170 ceph-mon[36280]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T18:02:12.559 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:12 smithi037 ceph-mon[31460]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T18:02:12.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.930+0000 7f5b51cc1700 1 -- 172.21.15.59:0/2566142370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b4c1033e0 msgr2=0x7f5b4c103840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:12.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.930+0000 7f5b51cc1700 1 --2- 172.21.15.59:0/2566142370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b4c1033e0 0x7f5b4c103840 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5b400099c0 tx=0x7f5b40009cd0 comp rx=0 tx=0).stop 2024-03-28T18:02:12.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.930+0000 7f5b51cc1700 1 -- 172.21.15.59:0/2566142370 shutdown_connections 2024-03-28T18:02:12.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.930+0000 7f5b51cc1700 1 --2- 172.21.15.59:0/2566142370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b4c103d80 0x7f5b4c10a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:12.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.930+0000 7f5b51cc1700 1 --2- 172.21.15.59:0/2566142370 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5b4c1021e0 0x7f5b4c102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:12.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.930+0000 7f5b51cc1700 1 --2- 172.21.15.59:0/2566142370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b4c1033e0 0x7f5b4c103840 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:12.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.930+0000 7f5b51cc1700 1 -- 172.21.15.59:0/2566142370 >> 172.21.15.59:0/2566142370 conn(0x7f5b4c0fd7a0 msgr2=0x7f5b4c0ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:02:12.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.930+0000 7f5b51cc1700 1 -- 172.21.15.59:0/2566142370 shutdown_connections 2024-03-28T18:02:12.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.930+0000 7f5b51cc1700 1 -- 172.21.15.59:0/2566142370 wait complete. 2024-03-28T18:02:12.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.930+0000 7f5b51cc1700 1 Processor -- start 2024-03-28T18:02:12.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.930+0000 7f5b51cc1700 1 -- start start 2024-03-28T18:02:12.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b51cc1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b4c1021e0 0x7f5b4c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:12.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b51cc1700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5b4c1033e0 0x7f5b4c198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:12.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b51cc1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b4c103d80 0x7f5b4c19f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:12.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b51cc1700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4c10de30 con 0x7f5b4c103d80 2024-03-28T18:02:12.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b51cc1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4c10dcb0 con 0x7f5b4c1021e0 2024-03-28T18:02:12.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b51cc1700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4c10dfb0 con 0x7f5b4c1033e0 2024-03-28T18:02:12.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b4b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b4c1021e0 0x7f5b4c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:12.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b4b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b4c1021e0 0x7f5b4c198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43858/0 (socket says 172.21.15.59:43858) 2024-03-28T18:02:12.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b4b7fe700 1 -- 172.21.15.59:0/3803119504 learned_addr learned my addr 172.21.15.59:0/3803119504 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-28T18:02:12.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b4affd700 1 --2- 172.21.15.59:0/3803119504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5b4c1033e0 0x7f5b4c198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:12.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b4b7fe700 1 -- 172.21.15.59:0/3803119504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5b4c1033e0 msgr2=0x7f5b4c198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:12.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b4b7fe700 1 --2- 172.21.15.59:0/3803119504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5b4c1033e0 0x7f5b4c198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:12.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b4b7fe700 1 -- 172.21.15.59:0/3803119504 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b4c103d80 msgr2=0x7f5b4c19f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:12.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b4b7fe700 1 --2- 172.21.15.59:0/3803119504 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b4c103d80 0x7f5b4c19f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:12.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b4b7fe700 1 -- 172.21.15.59:0/3803119504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b4c1a0070 con 0x7f5b4c1021e0 2024-03-28T18:02:12.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b4b7fe700 1 --2- 172.21.15.59:0/3803119504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b4c1021e0 0x7f5b4c198410 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c00d980 tx=0x7f5b3c00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:02:12.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b48ff9700 1 -- 172.21.15.59:0/3803119504 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c017030 con 0x7f5b4c1021e0 2024-03-28T18:02:12.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b48ff9700 1 -- 172.21.15.59:0/3803119504 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b3c004500 con 0x7f5b4c1021e0 2024-03-28T18:02:12.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b51cc1700 1 -- 172.21.15.59:0/3803119504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b4c1a0360 con 0x7f5b4c1021e0 2024-03-28T18:02:12.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b48ff9700 1 -- 172.21.15.59:0/3803119504 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c003aa0 con 0x7f5b4c1021e0 2024-03-28T18:02:12.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.931+0000 7f5b51cc1700 1 -- 172.21.15.59:0/3803119504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4c1a0920 con 0x7f5b4c1021e0 2024-03-28T18:02:12.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.932+0000 7f5b51cc1700 1 -- 172.21.15.59:0/3803119504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b4c068e50 con 0x7f5b4c1021e0 2024-03-28T18:02:12.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.933+0000 7f5b48ff9700 1 -- 172.21.15.59:0/3803119504 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f5b3c01a070 con 0x7f5b4c1021e0 2024-03-28T18:02:12.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.934+0000 7f5b48ff9700 1 --2- 172.21.15.59:0/3803119504 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5b340a2800 0x7f5b340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:12.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.934+0000 7f5b48ff9700 1 -- 172.21.15.59:0/3803119504 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f5b3c0cc930 con 0x7f5b4c1021e0 2024-03-28T18:02:12.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.935+0000 7f5b4affd700 1 --2- 172.21.15.59:0/3803119504 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5b340a2800 0x7f5b340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:12.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.936+0000 7f5b4affd700 1 --2- 172.21.15.59:0/3803119504 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5b340a2800 0x7f5b340a4cc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5b40000c00 tx=0x7f5b40005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:02:12.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:12.939+0000 7f5b48ff9700 1 -- 172.21.15.59:0/3803119504 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5b3c0232f0 con 0x7f5b4c1021e0 2024-03-28T18:02:13.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:13.155+0000 7f5b51cc1700 1 -- 172.21.15.59:0/3803119504 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b4c0625d0 con 0x7f5b340a2800 2024-03-28T18:02:13.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:13 smithi037 ceph-mon[31460]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:13.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:13 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:02:13.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:13 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:02:13.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:13 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:13.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:13 smithi059 ceph-mon[36190]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:13.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:13 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:02:13.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:13 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:02:13.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:13 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:13.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:13 smithi170 ceph-mon[36280]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:13.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:13 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:02:13.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:13 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:02:13.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:13 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:14 smithi059 ceph-mon[36190]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:02:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:14 smithi059 ceph-mon[36190]: Detected new or changed devices on smithi037 2024-03-28T18:02:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:02:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:14 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi037 to 17095M 2024-03-28T18:02:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:14.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:14 smithi170 ceph-mon[36280]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:02:14.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:14 smithi170 ceph-mon[36280]: Detected new or changed devices on smithi037 2024-03-28T18:02:14.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:14.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:14.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:02:14.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:14 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi037 to 17095M 2024-03-28T18:02:14.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:14.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:14.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:14.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:14.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:14 smithi037 ceph-mon[31460]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:02:14.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:14 smithi037 ceph-mon[31460]: Detected new or changed devices on smithi037 2024-03-28T18:02:14.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:14.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:14.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:02:14.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:14 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi037 to 17095M 2024-03-28T18:02:14.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:14.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:14.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:14.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:15.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:15 smithi059 ceph-mon[36190]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:15.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:15 smithi170 ceph-mon[36280]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:15.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:15 smithi037 ceph-mon[31460]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:16.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:16 smithi059 ceph-mon[36190]: from='client.? 172.21.15.59:0/1866030223' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a4b7ad6-905e-4258-9574-693d4277dfc6"}]: dispatch 2024-03-28T18:02:16.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:16 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a4b7ad6-905e-4258-9574-693d4277dfc6"}]: dispatch 2024-03-28T18:02:16.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:16 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a4b7ad6-905e-4258-9574-693d4277dfc6"}]': finished 2024-03-28T18:02:16.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:16 smithi059 ceph-mon[36190]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T18:02:16.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:16 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:02:16.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:16 smithi059 ceph-mon[36190]: from='client.? 172.21.15.59:0/108970861' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:02:16.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:16 smithi170 ceph-mon[36280]: from='client.? 172.21.15.59:0/1866030223' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a4b7ad6-905e-4258-9574-693d4277dfc6"}]: dispatch 2024-03-28T18:02:16.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:16 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a4b7ad6-905e-4258-9574-693d4277dfc6"}]: dispatch 2024-03-28T18:02:16.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:16 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a4b7ad6-905e-4258-9574-693d4277dfc6"}]': finished 2024-03-28T18:02:16.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:16 smithi170 ceph-mon[36280]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T18:02:16.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:16 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:02:16.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:16 smithi170 ceph-mon[36280]: from='client.? 172.21.15.59:0/108970861' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:02:16.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:16 smithi037 ceph-mon[31460]: from='client.? 172.21.15.59:0/1866030223' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a4b7ad6-905e-4258-9574-693d4277dfc6"}]: dispatch 2024-03-28T18:02:16.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:16 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a4b7ad6-905e-4258-9574-693d4277dfc6"}]: dispatch 2024-03-28T18:02:16.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:16 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a4b7ad6-905e-4258-9574-693d4277dfc6"}]': finished 2024-03-28T18:02:16.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:16 smithi037 ceph-mon[31460]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T18:02:16.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:16 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:02:16.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:16 smithi037 ceph-mon[31460]: from='client.? 172.21.15.59:0/108970861' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:02:17.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:17 smithi059 ceph-mon[36190]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:17.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:17 smithi170 ceph-mon[36280]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:17.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:17 smithi037 ceph-mon[31460]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:19.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:19 smithi059 ceph-mon[36190]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:19.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:19 smithi170 ceph-mon[36280]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:19.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:19 smithi037 ceph-mon[31460]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:21.810 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:21 smithi059 ceph-mon[36190]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:21.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:21 smithi037 ceph-mon[31460]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:21 smithi170 ceph-mon[36280]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:23.853 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:23 smithi059 ceph-mon[36190]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:23.853 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:23 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T18:02:23.853 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:23 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:23.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:23 smithi037 ceph-mon[31460]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:23.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T18:02:23.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:23 smithi170 ceph-mon[36280]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T18:02:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:24.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:24 smithi037 ceph-mon[31460]: Deploying daemon osd.1 on smithi059 2024-03-28T18:02:24.875 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:24 smithi059 ceph-mon[36190]: Deploying daemon osd.1 on smithi059 2024-03-28T18:02:25.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:24 smithi170 ceph-mon[36280]: Deploying daemon osd.1 on smithi059 2024-03-28T18:02:25.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:25 smithi059 ceph-mon[36190]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:25.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:25 smithi037 ceph-mon[31460]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:26.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:25 smithi170 ceph-mon[36280]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:27.729 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:27 smithi059 ceph-mon[36190]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:27.838 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:27 smithi170 ceph-mon[36280]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:27.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:27 smithi037 ceph-mon[31460]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:29 smithi059 ceph-mon[36190]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:29.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:29 smithi037 ceph-mon[31460]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:29.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:29.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:29.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:29.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:29 smithi170 ceph-mon[36280]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:29.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:29.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:29.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:31 smithi059 ceph-mon[36190]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:31 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:31 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:31.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:31 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:31.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:31 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:31.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:31 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:31 smithi170 ceph-mon[36280]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:31 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:31 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:31.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:31 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:31.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:31 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:31.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:31 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:31 smithi037 ceph-mon[31460]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:31.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:31 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:31.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:31 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:31.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:31 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:31.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:31 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:31.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:31 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:32.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:32.048+0000 7f5b48ff9700 1 -- 172.21.15.59:0/3803119504 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5b4c0625d0 con 0x7f5b340a2800 2024-03-28T18:02:32.049 INFO:teuthology.orchestra.run.smithi059.stdout:Created osd(s) 1 on host 'smithi059' 2024-03-28T18:02:32.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:32.051+0000 7f5b51cc1700 1 -- 172.21.15.59:0/3803119504 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5b340a2800 msgr2=0x7f5b340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:32.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:32.051+0000 7f5b51cc1700 1 --2- 172.21.15.59:0/3803119504 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5b340a2800 0x7f5b340a4cc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5b40000c00 tx=0x7f5b40005fb0 comp rx=0 tx=0).stop 2024-03-28T18:02:32.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:32.051+0000 7f5b51cc1700 1 -- 172.21.15.59:0/3803119504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b4c1021e0 msgr2=0x7f5b4c198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:32.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:32.051+0000 7f5b51cc1700 1 --2- 172.21.15.59:0/3803119504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b4c1021e0 0x7f5b4c198410 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c00d980 tx=0x7f5b3c00dc90 comp rx=0 tx=0).stop 2024-03-28T18:02:32.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:32.051+0000 7f5b51cc1700 1 -- 172.21.15.59:0/3803119504 shutdown_connections 2024-03-28T18:02:32.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:32.051+0000 7f5b51cc1700 1 --2- 172.21.15.59:0/3803119504 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5b340a2800 0x7f5b340a4cc0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:32.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:32.051+0000 7f5b51cc1700 1 --2- 172.21.15.59:0/3803119504 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b4c103d80 0x7f5b4c19f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:32.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:32.051+0000 7f5b51cc1700 1 --2- 172.21.15.59:0/3803119504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b4c1021e0 0x7f5b4c198410 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:32.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:32.051+0000 7f5b51cc1700 1 --2- 172.21.15.59:0/3803119504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5b4c1033e0 0x7f5b4c198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:32.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:32.051+0000 7f5b51cc1700 1 -- 172.21.15.59:0/3803119504 >> 172.21.15.59:0/3803119504 conn(0x7f5b4c0fd7a0 msgr2=0x7f5b4c1066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:02:32.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:32.051+0000 7f5b51cc1700 1 -- 172.21.15.59:0/3803119504 shutdown_connections 2024-03-28T18:02:32.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:02:32.051+0000 7f5b51cc1700 1 -- 172.21.15.59:0/3803119504 wait complete. 2024-03-28T18:02:32.748 DEBUG:teuthology.orchestra.run.smithi059:osd.1> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@osd.1.service 2024-03-28T18:02:32.752 INFO:tasks.cephadm:Deploying osd.2 on smithi170 with /dev/vg_nvme/lv_4... 2024-03-28T18:02:32.752 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T18:02:32.998 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:02:33.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:33 smithi170 ceph-mon[36280]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:33 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:33 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:33 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:33 smithi170 ceph-mon[36280]: from='osd.1 [v2:172.21.15.59:6800/3326537211,v1:172.21.15.59:6801/3326537211]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T18:02:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:33 smithi170 ceph-mon[36280]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T18:02:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:33 smithi059 ceph-mon[36190]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:33 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:33 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:33 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:33 smithi059 ceph-mon[36190]: from='osd.1 [v2:172.21.15.59:6800/3326537211,v1:172.21.15.59:6801/3326537211]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T18:02:33.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:33 smithi059 ceph-mon[36190]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T18:02:33.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:33 smithi037 ceph-mon[31460]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:33.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:33.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:33.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:33.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:33 smithi037 ceph-mon[31460]: from='osd.1 [v2:172.21.15.59:6800/3326537211,v1:172.21.15.59:6801/3326537211]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T18:02:33.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:33 smithi037 ceph-mon[31460]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T18:02:34.138 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:34 smithi059 ceph-mon[36190]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T18:02:34.139 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:34 smithi059 ceph-mon[36190]: from='osd.1 [v2:172.21.15.59:6800/3326537211,v1:172.21.15.59:6801/3326537211]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:02:34.139 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:34 smithi059 ceph-mon[36190]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T18:02:34.139 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:34 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:02:34.139 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:34 smithi059 ceph-mon[36190]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:02:34.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 28 18:02:34 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-1[40791]: 2024-03-28T18:02:34.033+0000 7f41c2098700 -1 osd.1 0 waiting for initial osdmap 2024-03-28T18:02:34.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 28 18:02:34 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-1[40791]: 2024-03-28T18:02:34.037+0000 7f41bde8b700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:02:34.290 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:34 smithi170 ceph-mon[36280]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T18:02:34.290 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:34 smithi170 ceph-mon[36280]: from='osd.1 [v2:172.21.15.59:6800/3326537211,v1:172.21.15.59:6801/3326537211]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:02:34.290 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:34 smithi170 ceph-mon[36280]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T18:02:34.290 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:34 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:02:34.290 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:34 smithi170 ceph-mon[36280]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:02:34.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:34 smithi037 ceph-mon[31460]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T18:02:34.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:34 smithi037 ceph-mon[31460]: from='osd.1 [v2:172.21.15.59:6800/3326537211,v1:172.21.15.59:6801/3326537211]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:02:34.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:34 smithi037 ceph-mon[31460]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T18:02:34.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:34 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:02:34.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:34 smithi037 ceph-mon[31460]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:02:34.809 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T18:02:34.809 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T18:02:34.809 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-03-28T18:02:34.809 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-03-28T18:02:34.809 INFO:teuthology.orchestra.run.smithi170.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00848855 s, 1.2 GB/s 2024-03-28T18:02:34.809 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 2024-03-28T18:02:34.809 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-03-28T18:02:35.141 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:35 smithi170 ceph-mon[36280]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:35.141 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:35 smithi170 ceph-mon[36280]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-03-28T18:02:35.142 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:35 smithi170 ceph-mon[36280]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T18:02:35.142 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:35 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:02:35.142 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:35 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:02:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:35 smithi059 ceph-mon[36190]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:35 smithi059 ceph-mon[36190]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-03-28T18:02:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:35 smithi059 ceph-mon[36190]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T18:02:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:35 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:02:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:35 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:02:35.368 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi170:vg_nvme/lv_4 2024-03-28T18:02:35.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:35 smithi037 ceph-mon[31460]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:35.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:35 smithi037 ceph-mon[31460]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-03-28T18:02:35.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:35 smithi037 ceph-mon[31460]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T18:02:35.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:35 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:02:35.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:35 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:02:35.611 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:02:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:36 smithi037 ceph-mon[31460]: purged_snaps scrub starts 2024-03-28T18:02:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:36 smithi037 ceph-mon[31460]: purged_snaps scrub ok 2024-03-28T18:02:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:36 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:02:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:36 smithi037 ceph-mon[31460]: osd.1 [v2:172.21.15.59:6800/3326537211,v1:172.21.15.59:6801/3326537211] boot 2024-03-28T18:02:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:36 smithi037 ceph-mon[31460]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T18:02:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:36 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:02:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:36 smithi037 ceph-mon[31460]: Detected new or changed devices on smithi059 2024-03-28T18:02:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:36 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:36 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:36 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:02:36.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:36 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi059 to 17084M 2024-03-28T18:02:36.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:36 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:36.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:36 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:36.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:36 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:36.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:36 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:36.380 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:36 smithi170 ceph-mon[36280]: purged_snaps scrub starts 2024-03-28T18:02:36.380 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:36 smithi170 ceph-mon[36280]: purged_snaps scrub ok 2024-03-28T18:02:36.380 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:36 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:02:36.381 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:36 smithi170 ceph-mon[36280]: osd.1 [v2:172.21.15.59:6800/3326537211,v1:172.21.15.59:6801/3326537211] boot 2024-03-28T18:02:36.381 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:36 smithi170 ceph-mon[36280]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T18:02:36.381 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:36 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:02:36.381 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:36 smithi170 ceph-mon[36280]: Detected new or changed devices on smithi059 2024-03-28T18:02:36.381 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:36 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:36.381 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:36 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:36.381 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:36 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:02:36.381 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:36 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi059 to 17084M 2024-03-28T18:02:36.381 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:36 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:36.381 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:36 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:36.381 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:36 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:36.381 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:36 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:36 smithi059 ceph-mon[36190]: purged_snaps scrub starts 2024-03-28T18:02:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:36 smithi059 ceph-mon[36190]: purged_snaps scrub ok 2024-03-28T18:02:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:36 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:02:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:36 smithi059 ceph-mon[36190]: osd.1 [v2:172.21.15.59:6800/3326537211,v1:172.21.15.59:6801/3326537211] boot 2024-03-28T18:02:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:36 smithi059 ceph-mon[36190]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T18:02:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:36 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:02:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:36 smithi059 ceph-mon[36190]: Detected new or changed devices on smithi059 2024-03-28T18:02:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:36 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:36 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:36 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:02:36.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:36 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi059 to 17084M 2024-03-28T18:02:36.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:36 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:36.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:36 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:36.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:36 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:36.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:36 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:37.173 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.171+0000 7fb1265ac700 1 -- 172.21.15.170:0/474275758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb120103d60 msgr2=0x7fb12010a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:37.174 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.171+0000 7fb1265ac700 1 --2- 172.21.15.170:0/474275758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb120103d60 0x7fb12010a610 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb11800b0a0 tx=0x7fb11800b3b0 comp rx=0 tx=0).stop 2024-03-28T18:02:37.174 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.171+0000 7fb1265ac700 1 -- 172.21.15.170:0/474275758 shutdown_connections 2024-03-28T18:02:37.174 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.171+0000 7fb1265ac700 1 --2- 172.21.15.170:0/474275758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb120103d60 0x7fb12010a610 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:37.174 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.171+0000 7fb1265ac700 1 --2- 172.21.15.170:0/474275758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1201021c0 0x7fb1201025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:37.174 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.171+0000 7fb1265ac700 1 --2- 172.21.15.170:0/474275758 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1201033c0 0x7fb120103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:37.174 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.171+0000 7fb1265ac700 1 -- 172.21.15.170:0/474275758 >> 172.21.15.170:0/474275758 conn(0x7fb1200fd7a0 msgr2=0x7fb1200ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:02:37.174 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.171+0000 7fb1265ac700 1 -- 172.21.15.170:0/474275758 shutdown_connections 2024-03-28T18:02:37.174 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.171+0000 7fb1265ac700 1 -- 172.21.15.170:0/474275758 wait complete. 2024-03-28T18:02:37.174 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.171+0000 7fb1265ac700 1 Processor -- start 2024-03-28T18:02:37.174 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.171+0000 7fb1265ac700 1 -- start start 2024-03-28T18:02:37.175 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.172+0000 7fb1265ac700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1201021c0 0x7fb120075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:37.175 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.172+0000 7fb1265ac700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1201033c0 0x7fb120074580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:37.175 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.172+0000 7fb1265ac700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb120103d60 0x7fb120074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:37.175 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.172+0000 7fb1265ac700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12010dea0 con 0x7fb120103d60 2024-03-28T18:02:37.175 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.172+0000 7fb1265ac700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12010dd20 con 0x7fb1201033c0 2024-03-28T18:02:37.175 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.172+0000 7fb1265ac700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12010e020 con 0x7fb1201021c0 2024-03-28T18:02:37.175 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.172+0000 7fb11ffff700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1201021c0 0x7fb120075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:37.175 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.172+0000 7fb11ffff700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1201021c0 0x7fb120075f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:42658/0 (socket says 172.21.15.170:42658) 2024-03-28T18:02:37.175 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.172+0000 7fb11ffff700 1 -- 172.21.15.170:0/3053019455 learned_addr learned my addr 172.21.15.170:0/3053019455 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T18:02:37.175 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.172+0000 7fb11ffff700 1 -- 172.21.15.170:0/3053019455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1201033c0 msgr2=0x7fb120074580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:37.175 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.172+0000 7fb11ffff700 1 --2- 172.21.15.170:0/3053019455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1201033c0 0x7fb120074580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:37.176 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.172+0000 7fb11ffff700 1 -- 172.21.15.170:0/3053019455 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb120103d60 msgr2=0x7fb120074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:37.176 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.172+0000 7fb11ffff700 1 --2- 172.21.15.170:0/3053019455 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb120103d60 0x7fb120074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:37.176 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.172+0000 7fb11ffff700 1 -- 172.21.15.170:0/3053019455 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb120075370 con 0x7fb1201021c0 2024-03-28T18:02:37.176 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.172+0000 7fb11ffff700 1 --2- 172.21.15.170:0/3053019455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1201021c0 0x7fb120075f30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb11000b500 tx=0x7fb11000b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:02:37.176 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.172+0000 7fb11d7fa700 1 -- 172.21.15.170:0/3053019455 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb110015020 con 0x7fb1201021c0 2024-03-28T18:02:37.176 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.172+0000 7fb11d7fa700 1 -- 172.21.15.170:0/3053019455 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb110011d10 con 0x7fb1201021c0 2024-03-28T18:02:37.176 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.172+0000 7fb11d7fa700 1 -- 172.21.15.170:0/3053019455 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11000f2e0 con 0x7fb1201021c0 2024-03-28T18:02:37.176 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.173+0000 7fb1265ac700 1 -- 172.21.15.170:0/3053019455 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1201a2680 con 0x7fb1201021c0 2024-03-28T18:02:37.176 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.173+0000 7fb1265ac700 1 -- 172.21.15.170:0/3053019455 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1201a2bc0 con 0x7fb1201021c0 2024-03-28T18:02:37.177 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.174+0000 7fb1265ac700 1 -- 172.21.15.170:0/3053019455 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb120068e50 con 0x7fb1201021c0 2024-03-28T18:02:37.178 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.175+0000 7fb11d7fa700 1 -- 172.21.15.170:0/3053019455 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7fb110018020 con 0x7fb1201021c0 2024-03-28T18:02:37.178 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.175+0000 7fb11d7fa700 1 --2- 172.21.15.170:0/3053019455 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb1080a2880 0x7fb1080a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:37.178 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.175+0000 7fb11d7fa700 1 -- 172.21.15.170:0/3053019455 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7fb1100cc930 con 0x7fb1201021c0 2024-03-28T18:02:37.178 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.176+0000 7fb11f7fe700 1 --2- 172.21.15.170:0/3053019455 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb1080a2880 0x7fb1080a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:37.178 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.177+0000 7fb11f7fe700 1 --2- 172.21.15.170:0/3053019455 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb1080a2880 0x7fb1080a4d40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb1140097b0 tx=0x7fb114006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:02:37.180 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.180+0000 7fb11d7fa700 1 -- 172.21.15.170:0/3053019455 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb11001f020 con 0x7fb1201021c0 2024-03-28T18:02:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:37 smithi170 ceph-mon[36280]: pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:37 smithi170 ceph-mon[36280]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T18:02:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:37 smithi037 ceph-mon[31460]: pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:37.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:37 smithi037 ceph-mon[31460]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T18:02:37.388 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:37.387+0000 7fb1265ac700 1 -- 172.21.15.170:0/3053019455 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1200625d0 con 0x7fb1080a2880 2024-03-28T18:02:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:37 smithi059 ceph-mon[36190]: pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:02:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:37 smithi059 ceph-mon[36190]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T18:02:38.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:38 smithi170 ceph-mon[36280]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:02:38.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:38 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:02:38.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:38 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:02:38.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:38 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:38.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:38 smithi037 ceph-mon[31460]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:02:38.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:38 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:02:38.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:38 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:02:38.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:38 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:38.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:38 smithi059 ceph-mon[36190]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:02:38.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:38 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:02:38.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:38 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:02:38.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:38 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:39.140 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:39 smithi170 ceph-mon[36280]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:39.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:39 smithi037 ceph-mon[31460]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:39.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:39 smithi059 ceph-mon[36190]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:40.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:40 smithi037 ceph-mon[31460]: from='client.? 172.21.15.170:0/1702647782' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b76ec1c-46bf-4336-89b5-12d82906e8b7"}]: dispatch 2024-03-28T18:02:40.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:40 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b76ec1c-46bf-4336-89b5-12d82906e8b7"}]: dispatch 2024-03-28T18:02:40.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:40 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b76ec1c-46bf-4336-89b5-12d82906e8b7"}]': finished 2024-03-28T18:02:40.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:40 smithi037 ceph-mon[31460]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T18:02:40.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:40 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:02:40.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:40 smithi059 ceph-mon[36190]: from='client.? 172.21.15.170:0/1702647782' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b76ec1c-46bf-4336-89b5-12d82906e8b7"}]: dispatch 2024-03-28T18:02:40.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:40 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b76ec1c-46bf-4336-89b5-12d82906e8b7"}]: dispatch 2024-03-28T18:02:40.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:40 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b76ec1c-46bf-4336-89b5-12d82906e8b7"}]': finished 2024-03-28T18:02:40.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:40 smithi059 ceph-mon[36190]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T18:02:40.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:40 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:02:40.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:40 smithi170 ceph-mon[36280]: from='client.? 172.21.15.170:0/1702647782' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b76ec1c-46bf-4336-89b5-12d82906e8b7"}]: dispatch 2024-03-28T18:02:40.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:40 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b76ec1c-46bf-4336-89b5-12d82906e8b7"}]: dispatch 2024-03-28T18:02:40.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:40 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b76ec1c-46bf-4336-89b5-12d82906e8b7"}]': finished 2024-03-28T18:02:40.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:40 smithi170 ceph-mon[36280]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T18:02:40.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:40 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:02:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:41 smithi037 ceph-mon[31460]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:41.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:41 smithi037 ceph-mon[31460]: from='client.? 172.21.15.170:0/2178035764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:02:41.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:41 smithi059 ceph-mon[36190]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:41.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:41 smithi059 ceph-mon[36190]: from='client.? 172.21.15.170:0/2178035764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:02:41.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:41 smithi170 ceph-mon[36280]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:41.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:41 smithi170 ceph-mon[36280]: from='client.? 172.21.15.170:0/2178035764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:02:43.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:43 smithi037 ceph-mon[31460]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:43.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:43 smithi059 ceph-mon[36190]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:43.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:43 smithi170 ceph-mon[36280]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:45.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:45 smithi037 ceph-mon[31460]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:45.383 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:45 smithi170 ceph-mon[36280]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:45.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:45 smithi059 ceph-mon[36190]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:47.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:47 smithi037 ceph-mon[31460]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:47.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:47 smithi059 ceph-mon[36190]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:47.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:47 smithi170 ceph-mon[36280]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:48.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:48 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T18:02:48.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:48 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:48.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:48 smithi170 ceph-mon[36280]: Deploying daemon osd.2 on smithi170 2024-03-28T18:02:48.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:48 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T18:02:48.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:48 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:48.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:48 smithi037 ceph-mon[31460]: Deploying daemon osd.2 on smithi170 2024-03-28T18:02:48.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:48 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T18:02:48.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:48 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:48.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:48 smithi059 ceph-mon[36190]: Deploying daemon osd.2 on smithi170 2024-03-28T18:02:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:49 smithi170 ceph-mon[36280]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:49.370 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:49 smithi037 ceph-mon[31460]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:49.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:49 smithi059 ceph-mon[36190]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:51 smithi037 ceph-mon[31460]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:51.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:51 smithi059 ceph-mon[36190]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:51.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:51 smithi170 ceph-mon[36280]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:53.199 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:53 smithi170 ceph-mon[36280]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:53.199 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:53.199 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:53.199 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:53.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:53 smithi037 ceph-mon[31460]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:53.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:53.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:53.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:53.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:53 smithi059 ceph-mon[36190]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:53.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:53.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:53.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:55.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:55 smithi170 ceph-mon[36280]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:55.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:55 smithi059 ceph-mon[36190]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:55.622 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:55.623+0000 7fb11d7fa700 1 -- 172.21.15.170:0/3053019455 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb1200625d0 con 0x7fb1080a2880 2024-03-28T18:02:55.623 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 2 on host 'smithi170' 2024-03-28T18:02:55.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:55 smithi037 ceph-mon[31460]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:55.625 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:55.625+0000 7fb1265ac700 1 -- 172.21.15.170:0/3053019455 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb1080a2880 msgr2=0x7fb1080a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:55.625 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:55.625+0000 7fb1265ac700 1 --2- 172.21.15.170:0/3053019455 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb1080a2880 0x7fb1080a4d40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb1140097b0 tx=0x7fb114006d20 comp rx=0 tx=0).stop 2024-03-28T18:02:55.625 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:55.625+0000 7fb1265ac700 1 -- 172.21.15.170:0/3053019455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1201021c0 msgr2=0x7fb120075f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:55.625 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:55.625+0000 7fb1265ac700 1 --2- 172.21.15.170:0/3053019455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1201021c0 0x7fb120075f30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb11000b500 tx=0x7fb11000b8c0 comp rx=0 tx=0).stop 2024-03-28T18:02:55.625 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:55.626+0000 7fb1265ac700 1 -- 172.21.15.170:0/3053019455 shutdown_connections 2024-03-28T18:02:55.625 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:55.626+0000 7fb1265ac700 1 --2- 172.21.15.170:0/3053019455 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb1080a2880 0x7fb1080a4d40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:55.626 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:55.626+0000 7fb1265ac700 1 --2- 172.21.15.170:0/3053019455 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb120103d60 0x7fb120074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:55.626 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:55.626+0000 7fb1265ac700 1 --2- 172.21.15.170:0/3053019455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1201021c0 0x7fb120075f30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:55.626 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:55.626+0000 7fb1265ac700 1 --2- 172.21.15.170:0/3053019455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1201033c0 0x7fb120074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:55.626 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:55.626+0000 7fb1265ac700 1 -- 172.21.15.170:0/3053019455 >> 172.21.15.170:0/3053019455 conn(0x7fb1200fd7a0 msgr2=0x7fb1201066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:02:55.626 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:55.626+0000 7fb1265ac700 1 -- 172.21.15.170:0/3053019455 shutdown_connections 2024-03-28T18:02:55.626 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:02:55.626+0000 7fb1265ac700 1 -- 172.21.15.170:0/3053019455 wait complete. 2024-03-28T18:02:56.273 DEBUG:teuthology.orchestra.run.smithi170:osd.2> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@osd.2.service 2024-03-28T18:02:56.275 INFO:tasks.cephadm:Deploying osd.3 on smithi037 with /dev/vg_nvme/lv_3... 2024-03-28T18:02:56.276 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T18:02:56.518 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 28 18:02:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-2[41581]: 2024-03-28T18:02:56.427+0000 7ff88deab7c0 -1 osd.2 0 log_to_monitors true 2024-03-28T18:02:56.519 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:56 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:56.519 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:56 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:56.519 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:56 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:56.519 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:56 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:56.519 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:56 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:56.519 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:56 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:56.519 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:56 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:56.520 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:56 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:56 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:56 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:56 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:56 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:56 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:56.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:56 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:56.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:56 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:56.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:56 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:56.752 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:02:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:56 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:56 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:56 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:56 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:56 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:56 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:56.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:56 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:56.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:56 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:57.523 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:57 smithi170 ceph-mon[36280]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:57.523 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:57 smithi170 ceph-mon[36280]: from='osd.2 [v2:172.21.15.170:6800/1269398427,v1:172.21.15.170:6801/1269398427]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T18:02:57.523 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:57 smithi170 ceph-mon[36280]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T18:02:57.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:57 smithi037 ceph-mon[31460]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:57.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:57 smithi037 ceph-mon[31460]: from='osd.2 [v2:172.21.15.170:6800/1269398427,v1:172.21.15.170:6801/1269398427]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T18:02:57.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:57 smithi037 ceph-mon[31460]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T18:02:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:57 smithi059 ceph-mon[36190]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:57 smithi059 ceph-mon[36190]: from='osd.2 [v2:172.21.15.170:6800/1269398427,v1:172.21.15.170:6801/1269398427]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T18:02:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:57 smithi059 ceph-mon[36190]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T18:02:58.464 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:58 smithi037 ceph-mon[31460]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T18:02:58.464 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:58 smithi037 ceph-mon[31460]: from='osd.2 [v2:172.21.15.170:6800/1269398427,v1:172.21.15.170:6801/1269398427]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:02:58.464 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:58 smithi037 ceph-mon[31460]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T18:02:58.465 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:58 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:02:58.465 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:58 smithi037 ceph-mon[31460]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:02:58.465 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:58 smithi037 ceph-mon[31460]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-28T18:02:58.746 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:58 smithi170 ceph-mon[36280]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T18:02:58.746 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:58 smithi170 ceph-mon[36280]: from='osd.2 [v2:172.21.15.170:6800/1269398427,v1:172.21.15.170:6801/1269398427]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:02:58.746 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:58 smithi170 ceph-mon[36280]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T18:02:58.746 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:58 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:02:58.747 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:58 smithi170 ceph-mon[36280]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:02:58.747 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:58 smithi170 ceph-mon[36280]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-28T18:02:58.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:58 smithi059 ceph-mon[36190]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T18:02:58.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:58 smithi059 ceph-mon[36190]: from='osd.2 [v2:172.21.15.170:6800/1269398427,v1:172.21.15.170:6801/1269398427]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:02:58.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:58 smithi059 ceph-mon[36190]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T18:02:58.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:58 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:02:58.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:58 smithi059 ceph-mon[36190]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:02:58.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:58 smithi059 ceph-mon[36190]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-28T18:02:59.035 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 28 18:02:58 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-2[41581]: 2024-03-28T18:02:58.826+0000 7ff883d11700 -1 osd.2 0 waiting for initial osdmap 2024-03-28T18:02:59.035 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 28 18:02:58 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-2[41581]: 2024-03-28T18:02:58.830+0000 7ff87b2fb700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:02:59.570 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:59 smithi037 ceph-mon[31460]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:59.571 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:59 smithi037 ceph-mon[31460]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T18:02:59.571 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:59 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:02:59.571 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:59 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:02:59.571 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:59 smithi037 ceph-mon[31460]: from='osd.2 ' entity='osd.2' 2024-03-28T18:02:59.571 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:59 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:59.571 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:59 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:59.571 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:59 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:02:59.571 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:59 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:59.571 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:59 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:59.571 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:59 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:59.572 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:02:59 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:59.699 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:59 smithi170 ceph-mon[36280]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:59.699 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:59 smithi170 ceph-mon[36280]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T18:02:59.699 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:59 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:02:59.699 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:59 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:02:59.699 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:59 smithi170 ceph-mon[36280]: from='osd.2 ' entity='osd.2' 2024-03-28T18:02:59.699 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:59 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:59.699 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:59 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:59.700 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:59 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:02:59.700 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:59 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:59.700 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:59 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:59.700 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:59 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:59.700 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:02:59 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:59.752 INFO:teuthology.orchestra.run.smithi037.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T18:02:59.752 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T18:02:59.752 INFO:teuthology.orchestra.run.smithi037.stderr: stderr: 10+0 records in 2024-03-28T18:02:59.753 INFO:teuthology.orchestra.run.smithi037.stderr:10+0 records out 2024-03-28T18:02:59.753 INFO:teuthology.orchestra.run.smithi037.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0327248 s, 320 MB/s 2024-03-28T18:02:59.753 INFO:teuthology.orchestra.run.smithi037.stderr:--> Zapping successful for: 2024-03-28T18:02:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:59 smithi059 ceph-mon[36190]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:02:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:59 smithi059 ceph-mon[36190]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T18:02:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:59 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:02:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:59 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:02:59.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:59 smithi059 ceph-mon[36190]: from='osd.2 ' entity='osd.2' 2024-03-28T18:02:59.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:59 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:59.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:59 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:59.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:59 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:02:59.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:59 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:02:59.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:59 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:59.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:59 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:59.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:02:59 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:00.573 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi037:vg_nvme/lv_3 2024-03-28T18:03:00.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:00 smithi059 ceph-mon[36190]: purged_snaps scrub starts 2024-03-28T18:03:00.797 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:00 smithi059 ceph-mon[36190]: purged_snaps scrub ok 2024-03-28T18:03:00.797 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:00 smithi059 ceph-mon[36190]: Detected new or changed devices on smithi170 2024-03-28T18:03:00.797 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:00 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi170 to 17080M 2024-03-28T18:03:00.797 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:00 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:03:00.797 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:00 smithi059 ceph-mon[36190]: osd.2 [v2:172.21.15.170:6800/1269398427,v1:172.21.15.170:6801/1269398427] boot 2024-03-28T18:03:00.797 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:00 smithi059 ceph-mon[36190]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T18:03:00.798 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:00 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:03:00.798 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:00 smithi170 ceph-mon[36280]: purged_snaps scrub starts 2024-03-28T18:03:00.798 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:00 smithi170 ceph-mon[36280]: purged_snaps scrub ok 2024-03-28T18:03:00.798 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:00 smithi170 ceph-mon[36280]: Detected new or changed devices on smithi170 2024-03-28T18:03:00.798 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:00 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi170 to 17080M 2024-03-28T18:03:00.798 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:00 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:03:00.798 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:00 smithi170 ceph-mon[36280]: osd.2 [v2:172.21.15.170:6800/1269398427,v1:172.21.15.170:6801/1269398427] boot 2024-03-28T18:03:00.798 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:00 smithi170 ceph-mon[36280]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T18:03:00.799 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:00 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:03:00.814 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:00 smithi037 ceph-mon[31460]: purged_snaps scrub starts 2024-03-28T18:03:00.814 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:00 smithi037 ceph-mon[31460]: purged_snaps scrub ok 2024-03-28T18:03:00.814 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:00 smithi037 ceph-mon[31460]: Detected new or changed devices on smithi170 2024-03-28T18:03:00.814 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:00 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi170 to 17080M 2024-03-28T18:03:00.814 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:00 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:03:00.814 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:00 smithi037 ceph-mon[31460]: osd.2 [v2:172.21.15.170:6800/1269398427,v1:172.21.15.170:6801/1269398427] boot 2024-03-28T18:03:00.814 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:00 smithi037 ceph-mon[31460]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T18:03:00.814 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:00 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:03:00.999 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:03:01.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:01 smithi059 ceph-mon[36190]: pgmap v91: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:03:01.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:01 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:03:01.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:01 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T18:03:01.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:01 smithi059 ceph-mon[36190]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T18:03:01.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:01 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:03:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:01 smithi170 ceph-mon[36280]: pgmap v91: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:03:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:01 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:03:01.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:01 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T18:03:01.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:01 smithi170 ceph-mon[36280]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T18:03:01.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:01 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:03:01.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:01 smithi037 ceph-mon[31460]: pgmap v91: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:03:01.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:01 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:03:01.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:01 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T18:03:01.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:01 smithi037 ceph-mon[31460]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T18:03:01.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:01 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:03:02.870 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:02 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T18:03:02.870 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:02 smithi037 ceph-mon[31460]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T18:03:02.870 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:02 smithi037 ceph-mon[31460]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:02.870 INFO:journalctl@ceph.osd.0.smithi037.stdout:Mar 28 18:03:02 smithi037 sudo[48415]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T18:03:02.870 INFO:journalctl@ceph.osd.0.smithi037.stdout:Mar 28 18:03:02 smithi037 sudo[48415]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:03:02.870 INFO:journalctl@ceph.osd.0.smithi037.stdout:Mar 28 18:03:02 smithi037 sudo[48415]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:03:02.870 INFO:journalctl@ceph.osd.0.smithi037.stdout:Mar 28 18:03:02 smithi037 sudo[48415]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:03:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.151+0000 7f661dc00700 1 -- 172.21.15.37:0/2651288915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66181033e0 msgr2=0x7f6618103840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.151+0000 7f661dc00700 1 --2- 172.21.15.37:0/2651288915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66181033e0 0x7f6618103840 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6604009920 tx=0x7f6604009c30 comp rx=0 tx=0).stop 2024-03-28T18:03:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.151+0000 7f661dc00700 1 -- 172.21.15.37:0/2651288915 shutdown_connections 2024-03-28T18:03:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.151+0000 7f661dc00700 1 --2- 172.21.15.37:0/2651288915 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6618103d80 0x7f661810a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.151+0000 7f661dc00700 1 --2- 172.21.15.37:0/2651288915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66181021e0 0x7f6618102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.151+0000 7f661dc00700 1 --2- 172.21.15.37:0/2651288915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66181033e0 0x7f6618103840 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.151+0000 7f661dc00700 1 -- 172.21.15.37:0/2651288915 >> 172.21.15.37:0/2651288915 conn(0x7f66180fd740 msgr2=0x7f66180ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:03:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.151+0000 7f661dc00700 1 -- 172.21.15.37:0/2651288915 shutdown_connections 2024-03-28T18:03:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.152+0000 7f661dc00700 1 -- 172.21.15.37:0/2651288915 wait complete. 2024-03-28T18:03:03.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.153+0000 7f661dc00700 1 Processor -- start 2024-03-28T18:03:03.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.153+0000 7f661dc00700 1 -- start start 2024-03-28T18:03:03.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.153+0000 7f661dc00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66181021e0 0x7f661819a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:03.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.153+0000 7f661dc00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66181033e0 0x7f661819abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:03.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.153+0000 7f661dc00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6618103d80 0x7f66181a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:03.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.154+0000 7f661dc00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661810de20 con 0x7f66181033e0 2024-03-28T18:03:03.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.154+0000 7f661dc00700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661810dca0 con 0x7f66181021e0 2024-03-28T18:03:03.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.154+0000 7f661dc00700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661810dfa0 con 0x7f6618103d80 2024-03-28T18:03:03.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.154+0000 7f6616a00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66181021e0 0x7f661819a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:03.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.154+0000 7f6616a00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66181021e0 0x7f661819a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:38846/0 (socket says 172.21.15.37:38846) 2024-03-28T18:03:03.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.154+0000 7f6616a00700 1 -- 172.21.15.37:0/2780222186 learned_addr learned my addr 172.21.15.37:0/2780222186 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:03:03.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.154+0000 7f6616000700 1 --2- 172.21.15.37:0/2780222186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66181033e0 0x7f661819abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:03.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.154+0000 7f6617400700 1 --2- 172.21.15.37:0/2780222186 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6618103d80 0x7f66181a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:03.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.154+0000 7f6616a00700 1 -- 172.21.15.37:0/2780222186 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6618103d80 msgr2=0x7f66181a1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:03.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.154+0000 7f6616a00700 1 --2- 172.21.15.37:0/2780222186 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6618103d80 0x7f66181a1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:03.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.154+0000 7f6616a00700 1 -- 172.21.15.37:0/2780222186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66181033e0 msgr2=0x7f661819abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:03.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.154+0000 7f6616a00700 1 --2- 172.21.15.37:0/2780222186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66181033e0 0x7f661819abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:03.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.154+0000 7f6616a00700 1 -- 172.21.15.37:0/2780222186 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66181a2300 con 0x7f66181021e0 2024-03-28T18:03:03.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.154+0000 7f6616000700 1 --2- 172.21.15.37:0/2780222186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66181033e0 0x7f661819abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:03:03.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.155+0000 7f6616a00700 1 --2- 172.21.15.37:0/2780222186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66181021e0 0x7f661819a6a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f660800efc0 tx=0x7f660800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:03:03.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.155+0000 7f660f400700 1 -- 172.21.15.37:0/2780222186 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660800ca70 con 0x7f66181021e0 2024-03-28T18:03:03.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.155+0000 7f660f400700 1 -- 172.21.15.37:0/2780222186 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f660800cc10 con 0x7f66181021e0 2024-03-28T18:03:03.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.155+0000 7f660f400700 1 -- 172.21.15.37:0/2780222186 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6608007500 con 0x7f66181021e0 2024-03-28T18:03:03.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.155+0000 7f661dc00700 1 -- 172.21.15.37:0/2780222186 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66181a25f0 con 0x7f66181021e0 2024-03-28T18:03:03.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.155+0000 7f661dc00700 1 -- 172.21.15.37:0/2780222186 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66181a2b30 con 0x7f66181021e0 2024-03-28T18:03:03.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.156+0000 7f661dc00700 1 -- 172.21.15.37:0/2780222186 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6618068e50 con 0x7f66181021e0 2024-03-28T18:03:03.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.158+0000 7f660f400700 1 -- 172.21.15.37:0/2780222186 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f6608012050 con 0x7f66181021e0 2024-03-28T18:03:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.160+0000 7f660f400700 1 --2- 172.21.15.37:0/2780222186 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f65fc0a2880 0x7f65fc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.160+0000 7f660f400700 1 -- 172.21.15.37:0/2780222186 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3237+0+0 (secure 0 0 0) 0x7f66080cd6a0 con 0x7f66181021e0 2024-03-28T18:03:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.160+0000 7f6616000700 1 --2- 172.21.15.37:0/2780222186 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f65fc0a2880 0x7f65fc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.161+0000 7f6616000700 1 --2- 172.21.15.37:0/2780222186 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f65fc0a2880 0x7f65fc0a4d40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f660400fca0 tx=0x7f6604019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:03:03.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.162+0000 7f660f400700 1 -- 172.21.15.37:0/2780222186 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6608010040 con 0x7f66181021e0 2024-03-28T18:03:03.229 INFO:journalctl@ceph.osd.0.smithi037.stdout:Mar 28 18:03:03 smithi037 sudo[48445]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T18:03:03.229 INFO:journalctl@ceph.osd.0.smithi037.stdout:Mar 28 18:03:03 smithi037 sudo[48445]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:03:03.229 INFO:journalctl@ceph.osd.0.smithi037.stdout:Mar 28 18:03:03 smithi037 sudo[48445]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:03:03.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:02 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T18:03:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:02 smithi059 ceph-mon[36190]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T18:03:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:02 smithi059 ceph-mon[36190]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:03.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:02 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T18:03:03.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:02 smithi170 ceph-mon[36280]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T18:03:03.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:02 smithi170 ceph-mon[36280]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:03.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:03.409+0000 7f661dc00700 1 -- 172.21.15.37:0/2780222186 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f66181a2e50 con 0x7f65fc0a2880 2024-03-28T18:03:03.624 INFO:journalctl@ceph.osd.0.smithi037.stdout:Mar 28 18:03:03 smithi037 sudo[48445]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:03:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:03 smithi059 ceph-mon[36190]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T18:03:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:03 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:03 smithi059 ceph-mon[36190]: from='client.24176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:03:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:03 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:03:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:03 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:03:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:03 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:04.035 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 28 18:03:03 smithi059 sudo[44966]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T18:03:04.035 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 28 18:03:03 smithi059 sudo[44966]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:03:04.036 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 28 18:03:03 smithi059 sudo[44966]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:03:04.036 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 28 18:03:03 smithi059 sudo[44966]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:03:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:03 smithi037 ceph-mon[31460]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T18:03:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:03 smithi037 ceph-mon[31460]: from='client.24176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:03:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:03:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:03:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:04.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:03 smithi170 ceph-mon[36280]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T18:03:04.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:03 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:04.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:03 smithi170 ceph-mon[36280]: from='client.24176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:03:04.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:03 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:03:04.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:03 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:03:04.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:03 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:04.535 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 28 18:03:04 smithi059 sudo[44969]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T18:03:04.535 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 28 18:03:04 smithi059 sudo[44969]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:03:04.535 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 28 18:03:04 smithi059 sudo[44969]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:03:04.535 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 28 18:03:04 smithi059 sudo[44969]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:03:04.687 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 28 18:03:04 smithi170 sudo[45230]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T18:03:04.687 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 28 18:03:04 smithi170 sudo[45230]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:03:04.687 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 28 18:03:04 smithi170 sudo[45230]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:03:04.687 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 28 18:03:04 smithi170 sudo[45230]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:03:04.687 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 28 18:03:04 smithi170 sudo[45233]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T18:03:05.035 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 28 18:03:04 smithi170 sudo[45233]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:03:05.036 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 28 18:03:04 smithi170 sudo[45233]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:03:05.036 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 28 18:03:04 smithi170 sudo[45233]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:03:05.405 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:05 smithi037 sudo[48587]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T18:03:05.405 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:05 smithi037 sudo[48587]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:03:05.405 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:05 smithi037 sudo[48587]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:03:05.405 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:05 smithi037 sudo[48587]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:03:05.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:05.925+0000 7f660f400700 1 -- 172.21.15.37:0/2780222186 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6608004000 con 0x7f66181021e0 2024-03-28T18:03:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:05 smithi037 sudo[48611]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T18:03:06.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:05 smithi037 sudo[48611]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:03:06.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:05 smithi037 sudo[48611]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:03:06.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:05 smithi037 sudo[48611]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:03:06.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:05 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:06.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:05 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:05 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:05 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:06.396 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:06 smithi170 sudo[45236]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T18:03:06.397 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:06 smithi170 sudo[45236]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:03:06.397 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:06 smithi170 sudo[45236]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:03:06.397 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:06 smithi170 sudo[45236]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:03:06.397 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:06 smithi170 sudo[45239]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T18:03:06.397 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:06 smithi170 sudo[45239]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:03:06.397 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:06 smithi170 sudo[45239]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:03:06.397 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:06 smithi170 sudo[45239]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:03:06.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:06 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:06.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:06 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:06 smithi059 sudo[44972]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T18:03:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:06 smithi059 sudo[44972]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:03:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:06 smithi059 sudo[44972]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:03:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:06 smithi059 sudo[44972]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:03:07.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:07 smithi037 ceph-mon[31460]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:07.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:07 smithi037 ceph-mon[31460]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:03:07.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:07 smithi037 ceph-mon[31460]: pgmap v97: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:07.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:07 smithi037 ceph-mon[31460]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:03:07.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:03:07.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:03:07.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:03:07.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:07 smithi037 ceph-mon[31460]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:03:07.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:03:07.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:03:07.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:03:07.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:07 smithi037 ceph-mon[31460]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-03-28T18:03:07.627 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:07 smithi037 ceph-mon[31460]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:03:07.627 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:03:07.627 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:03:07.627 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:03:07.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:07 smithi059 sudo[44975]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T18:03:07.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:07 smithi059 sudo[44975]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:03:07.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:07 smithi059 sudo[44975]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:03:07.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:07 smithi059 sudo[44975]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:03:07.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:07 smithi059 ceph-mon[36190]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:07.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:07 smithi059 ceph-mon[36190]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:03:07.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:07 smithi059 ceph-mon[36190]: pgmap v97: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:07.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:07 smithi059 ceph-mon[36190]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:03:07.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:03:07.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:03:07.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:03:07.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:07 smithi059 ceph-mon[36190]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:03:07.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:03:07.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:03:07.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:03:07.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:07 smithi059 ceph-mon[36190]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-03-28T18:03:07.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:07 smithi059 ceph-mon[36190]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:03:07.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:03:07.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:03:07.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:03:07.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:07 smithi170 ceph-mon[36280]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:07.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:07 smithi170 ceph-mon[36280]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:03:07.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:07 smithi170 ceph-mon[36280]: pgmap v97: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:07.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:07 smithi170 ceph-mon[36280]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:03:07.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:03:07.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:03:07.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:03:07.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:07 smithi170 ceph-mon[36280]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:03:07.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:03:07.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:03:07.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:03:07.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:07 smithi170 ceph-mon[36280]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-03-28T18:03:07.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:07 smithi170 ceph-mon[36280]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:03:07.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:03:07.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:03:07.790 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:03:08.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:08 smithi037 ceph-mon[31460]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:03:08.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:08 smithi037 ceph-mon[31460]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:03:08.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:08 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/403517870' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66eca1da-2fbd-44e1-8bbf-5635a2c70ad5"}]: dispatch 2024-03-28T18:03:08.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:08 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66eca1da-2fbd-44e1-8bbf-5635a2c70ad5"}]: dispatch 2024-03-28T18:03:08.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:08 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "66eca1da-2fbd-44e1-8bbf-5635a2c70ad5"}]': finished 2024-03-28T18:03:08.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:08 smithi037 ceph-mon[31460]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T18:03:08.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:08 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:03:08.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:08 smithi059 ceph-mon[36190]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:03:08.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:08 smithi059 ceph-mon[36190]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:03:08.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:08 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/403517870' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66eca1da-2fbd-44e1-8bbf-5635a2c70ad5"}]: dispatch 2024-03-28T18:03:08.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:08 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66eca1da-2fbd-44e1-8bbf-5635a2c70ad5"}]: dispatch 2024-03-28T18:03:08.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:08 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "66eca1da-2fbd-44e1-8bbf-5635a2c70ad5"}]': finished 2024-03-28T18:03:08.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:08 smithi059 ceph-mon[36190]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T18:03:08.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:08 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:03:08.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:08 smithi170 ceph-mon[36280]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:03:08.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:08 smithi170 ceph-mon[36280]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:03:08.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:08 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/403517870' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66eca1da-2fbd-44e1-8bbf-5635a2c70ad5"}]: dispatch 2024-03-28T18:03:08.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:08 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66eca1da-2fbd-44e1-8bbf-5635a2c70ad5"}]: dispatch 2024-03-28T18:03:08.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:08 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "66eca1da-2fbd-44e1-8bbf-5635a2c70ad5"}]': finished 2024-03-28T18:03:08.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:08 smithi170 ceph-mon[36280]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T18:03:08.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:03:09.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:09 smithi170 ceph-mon[36280]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:09.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:09 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/4111612104' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:03:09.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:09 smithi037 ceph-mon[31460]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:09.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:09 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/4111612104' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:03:09.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:09 smithi059 ceph-mon[36190]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:09.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:09 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/4111612104' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:03:11.614 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:11 smithi037 ceph-mon[31460]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:11 smithi059 ceph-mon[36190]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:11.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:11 smithi170 ceph-mon[36280]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:13.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:13 smithi037 ceph-mon[31460]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 59 KiB/s wr, 5 op/s 2024-03-28T18:03:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:13 smithi059 ceph-mon[36190]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 59 KiB/s wr, 5 op/s 2024-03-28T18:03:13.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:13 smithi170 ceph-mon[36280]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 59 KiB/s wr, 5 op/s 2024-03-28T18:03:15.514 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:15 smithi037 ceph-mon[31460]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 40 KiB/s wr, 3 op/s 2024-03-28T18:03:15.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:15 smithi059 ceph-mon[36190]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 40 KiB/s wr, 3 op/s 2024-03-28T18:03:15.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:15 smithi170 ceph-mon[36280]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 40 KiB/s wr, 3 op/s 2024-03-28T18:03:17.590 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:17 smithi037 ceph-mon[31460]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 31 KiB/s wr, 2 op/s 2024-03-28T18:03:17.591 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:17 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T18:03:17.591 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:17 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:17.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:17 smithi059 ceph-mon[36190]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 31 KiB/s wr, 2 op/s 2024-03-28T18:03:17.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:17 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T18:03:17.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:17 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:17.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:17 smithi170 ceph-mon[36280]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 31 KiB/s wr, 2 op/s 2024-03-28T18:03:17.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:17 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T18:03:17.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:17 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:18.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:18 smithi037 ceph-mon[31460]: Deploying daemon osd.3 on smithi037 2024-03-28T18:03:18.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:18 smithi059 ceph-mon[36190]: Deploying daemon osd.3 on smithi037 2024-03-28T18:03:18.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:18 smithi170 ceph-mon[36280]: Deploying daemon osd.3 on smithi037 2024-03-28T18:03:19.578 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:19 smithi037 ceph-mon[31460]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 977 B/s rd, 25 KiB/s wr, 2 op/s 2024-03-28T18:03:19.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:19 smithi059 ceph-mon[36190]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 977 B/s rd, 25 KiB/s wr, 2 op/s 2024-03-28T18:03:19.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:19 smithi170 ceph-mon[36280]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 977 B/s rd, 25 KiB/s wr, 2 op/s 2024-03-28T18:03:21.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:21 smithi037 ceph-mon[31460]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T18:03:21.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:21 smithi059 ceph-mon[36190]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T18:03:21.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:21 smithi170 ceph-mon[36280]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T18:03:23.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:23 smithi037 ceph-mon[31460]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T18:03:23.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:23 smithi059 ceph-mon[36190]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T18:03:23.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:23 smithi170 ceph-mon[36280]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T18:03:25.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:25 smithi037 ceph-mon[31460]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:25.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:25 smithi059 ceph-mon[36190]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:25 smithi170 ceph-mon[36280]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:26.755 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:26 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:26.755 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:26 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:26.755 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:26 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:03:26.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:26 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:26.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:26 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:26.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:26 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:03:26.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:26 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:26.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:26 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:26.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:26 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:03:27.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:27 smithi037 ceph-mon[31460]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:27.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:27 smithi059 ceph-mon[36190]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:27.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:27 smithi170 ceph-mon[36280]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:29.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:29 smithi059 ceph-mon[36190]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:29.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:29.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:29.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:29.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:03:29.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:29.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:29 smithi170 ceph-mon[36280]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:29.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:29.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:29.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:29.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:03:29.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:29.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:29 smithi037 ceph-mon[31460]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:29.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:29.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:29.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:29.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:03:29.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:30.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:30.515+0000 7f660f400700 1 -- 172.21.15.37:0/2780222186 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f66181a2e50 con 0x7f65fc0a2880 2024-03-28T18:03:30.517 INFO:teuthology.orchestra.run.smithi037.stdout:Created osd(s) 3 on host 'smithi037' 2024-03-28T18:03:30.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:30.520+0000 7f661dc00700 1 -- 172.21.15.37:0/2780222186 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f65fc0a2880 msgr2=0x7f65fc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:30.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:30.520+0000 7f661dc00700 1 --2- 172.21.15.37:0/2780222186 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f65fc0a2880 0x7f65fc0a4d40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f660400fca0 tx=0x7f6604019040 comp rx=0 tx=0).stop 2024-03-28T18:03:30.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:30.520+0000 7f661dc00700 1 -- 172.21.15.37:0/2780222186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66181021e0 msgr2=0x7f661819a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:30.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:30.520+0000 7f661dc00700 1 --2- 172.21.15.37:0/2780222186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66181021e0 0x7f661819a6a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f660800efc0 tx=0x7f660800c3a0 comp rx=0 tx=0).stop 2024-03-28T18:03:30.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:30.520+0000 7f661dc00700 1 -- 172.21.15.37:0/2780222186 shutdown_connections 2024-03-28T18:03:30.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:30.520+0000 7f661dc00700 1 --2- 172.21.15.37:0/2780222186 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f65fc0a2880 0x7f65fc0a4d40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:30.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:30.520+0000 7f661dc00700 1 --2- 172.21.15.37:0/2780222186 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6618103d80 0x7f66181a1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:30.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:30.520+0000 7f661dc00700 1 --2- 172.21.15.37:0/2780222186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66181021e0 0x7f661819a6a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:30.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:30.520+0000 7f661dc00700 1 --2- 172.21.15.37:0/2780222186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66181033e0 0x7f661819abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:30.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:30.520+0000 7f661dc00700 1 -- 172.21.15.37:0/2780222186 >> 172.21.15.37:0/2780222186 conn(0x7f66180fd740 msgr2=0x7f66180743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:03:30.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:30.521+0000 7f661dc00700 1 -- 172.21.15.37:0/2780222186 shutdown_connections 2024-03-28T18:03:30.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:03:30.521+0000 7f661dc00700 1 -- 172.21.15.37:0/2780222186 wait complete. 2024-03-28T18:03:30.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:30 smithi059 ceph-mon[36190]: from='osd.3 [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T18:03:30.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:30 smithi170 ceph-mon[36280]: from='osd.3 [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T18:03:30.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:30 smithi037 ceph-mon[31460]: from='osd.3 [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T18:03:31.298 DEBUG:teuthology.orchestra.run.smithi037:osd.3> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@osd.3.service 2024-03-28T18:03:31.303 INFO:tasks.cephadm:Deploying osd.4 on smithi059 with /dev/vg_nvme/lv_3... 2024-03-28T18:03:31.304 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T18:03:31.483 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:31 smithi037 ceph-mon[31460]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:31.563 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:03:31.685 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:31 smithi059 ceph-mon[36190]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:31.685 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:31 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:03:31.685 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:31 smithi059 ceph-mon[36190]: from='osd.3 [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T18:03:31.685 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:31 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:31.685 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:31 smithi059 ceph-mon[36190]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T18:03:31.685 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:31 smithi059 ceph-mon[36190]: from='osd.3 [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]: dispatch 2024-03-28T18:03:31.685 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:31 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:03:31.685 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:31 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:31.743 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:31 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:03:31.743 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:31 smithi037 ceph-mon[31460]: from='osd.3 [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T18:03:31.743 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:31 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:31.743 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:31 smithi037 ceph-mon[31460]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T18:03:31.743 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:31 smithi037 ceph-mon[31460]: from='osd.3 [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]: dispatch 2024-03-28T18:03:31.743 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:31 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:03:31.743 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:31 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:31.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:31 smithi170 ceph-mon[36280]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:31.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:31 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:03:31.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:31 smithi170 ceph-mon[36280]: from='osd.3 [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T18:03:31.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:31 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:31.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:31 smithi170 ceph-mon[36280]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T18:03:31.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:31 smithi170 ceph-mon[36280]: from='osd.3 [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]: dispatch 2024-03-28T18:03:31.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:31 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:03:31.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:31 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:32.375 INFO:journalctl@ceph.osd.3.smithi037.stdout:Mar 28 18:03:32 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-3[50688]: 2024-03-28T18:03:32.207+0000 7f8fd1a00700 -1 osd.3 0 waiting for initial osdmap 2024-03-28T18:03:32.375 INFO:journalctl@ceph.osd.3.smithi037.stdout:Mar 28 18:03:32 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-3[50688]: 2024-03-28T18:03:32.226+0000 7f8fc6e00700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:03:32.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:32 smithi059 ceph-mon[36190]: from='osd.3 [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]': finished 2024-03-28T18:03:32.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:32 smithi059 ceph-mon[36190]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T18:03:32.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:32 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:03:32.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:32 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:03:32.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:32 smithi059 ceph-mon[36190]: from='osd.3 [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240]' entity='osd.3' 2024-03-28T18:03:32.758 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:32 smithi037 ceph-mon[31460]: from='osd.3 [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]': finished 2024-03-28T18:03:32.758 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:32 smithi037 ceph-mon[31460]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T18:03:32.759 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:32 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:03:32.759 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:32 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:03:32.759 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:32 smithi037 ceph-mon[31460]: from='osd.3 [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240]' entity='osd.3' 2024-03-28T18:03:32.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:32 smithi170 ceph-mon[36280]: from='osd.3 [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]': finished 2024-03-28T18:03:32.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:32 smithi170 ceph-mon[36280]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T18:03:32.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:32 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:03:32.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:32 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:03:32.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:32 smithi170 ceph-mon[36280]: from='osd.3 [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240]' entity='osd.3' 2024-03-28T18:03:33.674 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:33 smithi059 ceph-mon[36190]: purged_snaps scrub starts 2024-03-28T18:03:33.674 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:33 smithi059 ceph-mon[36190]: purged_snaps scrub ok 2024-03-28T18:03:33.674 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:33 smithi059 ceph-mon[36190]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:33.675 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:33 smithi059 ceph-mon[36190]: osd.3 [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240] boot 2024-03-28T18:03:33.675 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:33 smithi059 ceph-mon[36190]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T18:03:33.675 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:33 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:03:33.675 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:33 smithi059 ceph-mon[36190]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T18:03:33.675 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T18:03:33.675 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T18:03:33.675 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10+0 records in 2024-03-28T18:03:33.675 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2024-03-28T18:03:33.675 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110349 s, 950 MB/s 2024-03-28T18:03:33.675 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping successful for: 2024-03-28T18:03:33.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:33 smithi170 ceph-mon[36280]: purged_snaps scrub starts 2024-03-28T18:03:33.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:33 smithi170 ceph-mon[36280]: purged_snaps scrub ok 2024-03-28T18:03:33.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:33 smithi170 ceph-mon[36280]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:33.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:33 smithi170 ceph-mon[36280]: osd.3 [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240] boot 2024-03-28T18:03:33.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:33 smithi170 ceph-mon[36280]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T18:03:33.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:33 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:03:33.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:33 smithi170 ceph-mon[36280]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T18:03:33.800 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:33 smithi037 ceph-mon[31460]: purged_snaps scrub starts 2024-03-28T18:03:33.800 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:33 smithi037 ceph-mon[31460]: purged_snaps scrub ok 2024-03-28T18:03:33.800 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:33 smithi037 ceph-mon[31460]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:03:33.800 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:33 smithi037 ceph-mon[31460]: osd.3 [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240] boot 2024-03-28T18:03:33.801 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:33 smithi037 ceph-mon[31460]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T18:03:33.801 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:03:33.801 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:33 smithi037 ceph-mon[31460]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T18:03:34.341 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi059:vg_nvme/lv_3 2024-03-28T18:03:34.592 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:03:34.947 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:34 smithi059 ceph-mon[36190]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T18:03:34.968 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:34 smithi037 ceph-mon[31460]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T18:03:35.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:34 smithi170 ceph-mon[36280]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T18:03:35.701 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:35 smithi059 ceph-mon[36190]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:35.701 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:35 smithi059 ceph-mon[36190]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T18:03:36.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:35 smithi170 ceph-mon[36280]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:36.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:35 smithi170 ceph-mon[36280]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T18:03:36.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:35 smithi037 ceph-mon[31460]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:36.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:35 smithi037 ceph-mon[31460]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T18:03:36.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.241+0000 7efc87d0d700 1 -- 172.21.15.59:0/380451334 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc801022c0 msgr2=0x7efc80102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:36.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.241+0000 7efc87d0d700 1 --2- 172.21.15.59:0/380451334 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc801022c0 0x7efc80102720 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efc740099c0 tx=0x7efc74009cd0 comp rx=0 tx=0).stop 2024-03-28T18:03:36.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.241+0000 7efc87d0d700 1 -- 172.21.15.59:0/380451334 shutdown_connections 2024-03-28T18:03:36.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.241+0000 7efc87d0d700 1 --2- 172.21.15.59:0/380451334 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc80102c60 0x7efc80109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:36.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.241+0000 7efc87d0d700 1 --2- 172.21.15.59:0/380451334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc801010c0 0x7efc801014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:36.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.241+0000 7efc87d0d700 1 --2- 172.21.15.59:0/380451334 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc801022c0 0x7efc80102720 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:36.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.241+0000 7efc87d0d700 1 -- 172.21.15.59:0/380451334 >> 172.21.15.59:0/380451334 conn(0x7efc800fc660 msgr2=0x7efc800feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:03:36.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.241+0000 7efc87d0d700 1 -- 172.21.15.59:0/380451334 shutdown_connections 2024-03-28T18:03:36.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.241+0000 7efc87d0d700 1 -- 172.21.15.59:0/380451334 wait complete. 2024-03-28T18:03:36.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.242+0000 7efc87d0d700 1 Processor -- start 2024-03-28T18:03:36.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.242+0000 7efc87d0d700 1 -- start start 2024-03-28T18:03:36.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.242+0000 7efc87d0d700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc801010c0 0x7efc8006a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:36.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.242+0000 7efc87d0d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc801022c0 0x7efc8019d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:36.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.242+0000 7efc87d0d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc80102c60 0x7efc8019f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:36.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.242+0000 7efc87d0d700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8010cd70 con 0x7efc801022c0 2024-03-28T18:03:36.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.242+0000 7efc87d0d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8010cbf0 con 0x7efc80102c60 2024-03-28T18:03:36.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.242+0000 7efc87d0d700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8010cef0 con 0x7efc801010c0 2024-03-28T18:03:36.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.242+0000 7efc862aa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc80102c60 0x7efc8019f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:36.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.242+0000 7efc862aa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc80102c60 0x7efc8019f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60150/0 (socket says 172.21.15.59:60150) 2024-03-28T18:03:36.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.242+0000 7efc862aa700 1 -- 172.21.15.59:0/2781827890 learned_addr learned my addr 172.21.15.59:0/2781827890 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-28T18:03:36.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.243+0000 7efc852a8700 1 --2- 172.21.15.59:0/2781827890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc801022c0 0x7efc8019d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:36.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.243+0000 7efc85aa9700 1 --2- 172.21.15.59:0/2781827890 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc801010c0 0x7efc8006a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:36.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.243+0000 7efc862aa700 1 -- 172.21.15.59:0/2781827890 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc801010c0 msgr2=0x7efc8006a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:36.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.243+0000 7efc862aa700 1 --2- 172.21.15.59:0/2781827890 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc801010c0 0x7efc8006a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:36.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.243+0000 7efc862aa700 1 -- 172.21.15.59:0/2781827890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc801022c0 msgr2=0x7efc8019d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:36.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.243+0000 7efc862aa700 1 --2- 172.21.15.59:0/2781827890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc801022c0 0x7efc8019d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:36.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.243+0000 7efc862aa700 1 -- 172.21.15.59:0/2781827890 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc801a0180 con 0x7efc80102c60 2024-03-28T18:03:36.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.243+0000 7efc862aa700 1 --2- 172.21.15.59:0/2781827890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc80102c60 0x7efc8019f8b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efc7800b550 tx=0x7efc7800b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:03:36.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.243+0000 7efc7effd700 1 -- 172.21.15.59:0/2781827890 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc78015020 con 0x7efc80102c60 2024-03-28T18:03:36.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.243+0000 7efc87d0d700 1 -- 172.21.15.59:0/2781827890 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc801a0470 con 0x7efc80102c60 2024-03-28T18:03:36.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.243+0000 7efc7effd700 1 -- 172.21.15.59:0/2781827890 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc78011d10 con 0x7efc80102c60 2024-03-28T18:03:36.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.243+0000 7efc7effd700 1 -- 172.21.15.59:0/2781827890 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7800f2e0 con 0x7efc80102c60 2024-03-28T18:03:36.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.243+0000 7efc87d0d700 1 -- 172.21.15.59:0/2781827890 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc801a0a30 con 0x7efc80102c60 2024-03-28T18:03:36.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.244+0000 7efc87d0d700 1 -- 172.21.15.59:0/2781827890 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc80068e50 con 0x7efc80102c60 2024-03-28T18:03:36.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.245+0000 7efc7effd700 1 -- 172.21.15.59:0/2781827890 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7efc78018020 con 0x7efc80102c60 2024-03-28T18:03:36.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.246+0000 7efc7effd700 1 --2- 172.21.15.59:0/2781827890 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7efc680a2890 0x7efc680a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:36.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.246+0000 7efc7effd700 1 -- 172.21.15.59:0/2781827890 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3669+0+0 (secure 0 0 0) 0x7efc780ccfe0 con 0x7efc80102c60 2024-03-28T18:03:36.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.246+0000 7efc85aa9700 1 --2- 172.21.15.59:0/2781827890 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7efc680a2890 0x7efc680a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:36.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.247+0000 7efc85aa9700 1 --2- 172.21.15.59:0/2781827890 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7efc680a2890 0x7efc680a4d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efc700060b0 tx=0x7efc70007b80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:03:36.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.251+0000 7efc7effd700 1 -- 172.21.15.59:0/2781827890 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efc7801f020 con 0x7efc80102c60 2024-03-28T18:03:36.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:36.466+0000 7efc87d0d700 1 -- 172.21.15.59:0/2781827890 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7efc800625d0 con 0x7efc680a2890 2024-03-28T18:03:36.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:36 smithi059 ceph-mon[36190]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T18:03:36.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:36 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:03:36.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:36 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:03:36.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:36 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:37.026 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:36 smithi037 ceph-mon[31460]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T18:03:37.026 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:36 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:03:37.026 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:36 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:03:37.026 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:36 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:37.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:36 smithi170 ceph-mon[36280]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T18:03:37.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:36 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:03:37.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:36 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:03:37.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:36 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:37.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:37 smithi059 ceph-mon[36190]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:37.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:37 smithi059 ceph-mon[36190]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:03:38.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:37 smithi170 ceph-mon[36280]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:38.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:37 smithi170 ceph-mon[36280]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:03:38.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:37 smithi037 ceph-mon[31460]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:38.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:37 smithi037 ceph-mon[31460]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:03:39.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:38 smithi059 ceph-mon[36190]: Detected new or changed devices on smithi037 2024-03-28T18:03:39.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:38 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:39.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:38 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:39.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:38 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:03:39.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:38 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:03:39.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:38 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi037 to 8547M 2024-03-28T18:03:39.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:38 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:39.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:38 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:39.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:38 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:03:39.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:38 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:39.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:38 smithi170 ceph-mon[36280]: Detected new or changed devices on smithi037 2024-03-28T18:03:39.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:38 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:39.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:38 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:39.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:38 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:03:39.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:38 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:03:39.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:38 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi037 to 8547M 2024-03-28T18:03:39.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:38 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:39.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:38 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:39.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:38 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:03:39.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:38 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:39.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:38 smithi037 ceph-mon[31460]: Detected new or changed devices on smithi037 2024-03-28T18:03:39.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:38 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:39.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:38 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:39.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:38 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:03:39.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:38 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:03:39.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:38 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi037 to 8547M 2024-03-28T18:03:39.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:38 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:39.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:38 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:39.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:38 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:03:39.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:38 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:39 smithi059 ceph-mon[36190]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-03-28T18:03:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:39 smithi059 ceph-mon[36190]: from='client.? 172.21.15.59:0/301271274' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c877335-e42d-4d8e-9f48-a1d0a89ae438"}]: dispatch 2024-03-28T18:03:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:39 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c877335-e42d-4d8e-9f48-a1d0a89ae438"}]: dispatch 2024-03-28T18:03:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:39 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c877335-e42d-4d8e-9f48-a1d0a89ae438"}]': finished 2024-03-28T18:03:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:39 smithi059 ceph-mon[36190]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T18:03:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:39 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:03:40.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:39 smithi037 ceph-mon[31460]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-03-28T18:03:40.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:39 smithi037 ceph-mon[31460]: from='client.? 172.21.15.59:0/301271274' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c877335-e42d-4d8e-9f48-a1d0a89ae438"}]: dispatch 2024-03-28T18:03:40.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:39 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c877335-e42d-4d8e-9f48-a1d0a89ae438"}]: dispatch 2024-03-28T18:03:40.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:39 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c877335-e42d-4d8e-9f48-a1d0a89ae438"}]': finished 2024-03-28T18:03:40.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:39 smithi037 ceph-mon[31460]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T18:03:40.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:39 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:03:40.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:39 smithi170 ceph-mon[36280]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-03-28T18:03:40.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:39 smithi170 ceph-mon[36280]: from='client.? 172.21.15.59:0/301271274' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c877335-e42d-4d8e-9f48-a1d0a89ae438"}]: dispatch 2024-03-28T18:03:40.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:39 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c877335-e42d-4d8e-9f48-a1d0a89ae438"}]: dispatch 2024-03-28T18:03:40.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:39 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c877335-e42d-4d8e-9f48-a1d0a89ae438"}]': finished 2024-03-28T18:03:40.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:39 smithi170 ceph-mon[36280]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T18:03:40.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:39 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:03:41.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:40 smithi037 ceph-mon[31460]: from='client.? 172.21.15.59:0/505910755' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:03:41.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:40 smithi059 ceph-mon[36190]: from='client.? 172.21.15.59:0/505910755' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:03:41.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:40 smithi170 ceph-mon[36280]: from='client.? 172.21.15.59:0/505910755' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:03:42.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:41 smithi037 ceph-mon[31460]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:03:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:41 smithi059 ceph-mon[36190]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:03:42.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:41 smithi170 ceph-mon[36280]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:03:44.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:43 smithi037 ceph-mon[31460]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-28T18:03:44.209 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:43 smithi059 ceph-mon[36190]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-28T18:03:44.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:43 smithi170 ceph-mon[36280]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-28T18:03:46.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:45 smithi037 ceph-mon[31460]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-28T18:03:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:45 smithi059 ceph-mon[36190]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-28T18:03:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:45 smithi170 ceph-mon[36280]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-28T18:03:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:46 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T18:03:47.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:46 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:47.279 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:46 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T18:03:47.279 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:46 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:46 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T18:03:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:46 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:48.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:47 smithi059 ceph-mon[36190]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T18:03:48.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:47 smithi059 ceph-mon[36190]: Deploying daemon osd.4 on smithi059 2024-03-28T18:03:48.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:47 smithi037 ceph-mon[31460]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T18:03:48.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:47 smithi037 ceph-mon[31460]: Deploying daemon osd.4 on smithi059 2024-03-28T18:03:48.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:47 smithi170 ceph-mon[36280]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T18:03:48.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:47 smithi170 ceph-mon[36280]: Deploying daemon osd.4 on smithi059 2024-03-28T18:03:49.116 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:48 smithi037 ceph-mon[31460]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:48 smithi059 ceph-mon[36190]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:48 smithi170 ceph-mon[36280]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:51.195 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:50 smithi059 ceph-mon[36190]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:50 smithi170 ceph-mon[36280]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:50 smithi037 ceph-mon[31460]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:53.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:53 smithi037 ceph-mon[31460]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:53.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:53.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:03:53.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:53.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:53 smithi059 ceph-mon[36190]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:53.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:53.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:03:53.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:53.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:53 smithi170 ceph-mon[36280]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:53.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:53.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:03:53.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:55.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:55 smithi059 ceph-mon[36190]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:55.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:55 smithi037 ceph-mon[31460]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:55.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:55 smithi170 ceph-mon[36280]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:56.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:56.306+0000 7efc7effd700 1 -- 172.21.15.59:0/2781827890 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efc800625d0 con 0x7efc680a2890 2024-03-28T18:03:56.305 INFO:teuthology.orchestra.run.smithi059.stdout:Created osd(s) 4 on host 'smithi059' 2024-03-28T18:03:56.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:56.308+0000 7efc87d0d700 1 -- 172.21.15.59:0/2781827890 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7efc680a2890 msgr2=0x7efc680a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:56.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:56.308+0000 7efc87d0d700 1 --2- 172.21.15.59:0/2781827890 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7efc680a2890 0x7efc680a4d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efc700060b0 tx=0x7efc70007b80 comp rx=0 tx=0).stop 2024-03-28T18:03:56.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:56.308+0000 7efc87d0d700 1 -- 172.21.15.59:0/2781827890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc80102c60 msgr2=0x7efc8019f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:56.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:56.308+0000 7efc87d0d700 1 --2- 172.21.15.59:0/2781827890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc80102c60 0x7efc8019f8b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efc7800b550 tx=0x7efc7800b910 comp rx=0 tx=0).stop 2024-03-28T18:03:56.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:56.308+0000 7efc87d0d700 1 -- 172.21.15.59:0/2781827890 shutdown_connections 2024-03-28T18:03:56.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:56.308+0000 7efc87d0d700 1 --2- 172.21.15.59:0/2781827890 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7efc680a2890 0x7efc680a4d50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:56.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:56.308+0000 7efc87d0d700 1 --2- 172.21.15.59:0/2781827890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc80102c60 0x7efc8019f8b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:56.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:56.308+0000 7efc87d0d700 1 --2- 172.21.15.59:0/2781827890 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc801010c0 0x7efc8006a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:56.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:56.308+0000 7efc87d0d700 1 --2- 172.21.15.59:0/2781827890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc801022c0 0x7efc8019d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:56.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:56.308+0000 7efc87d0d700 1 -- 172.21.15.59:0/2781827890 >> 172.21.15.59:0/2781827890 conn(0x7efc800fc660 msgr2=0x7efc80105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:03:56.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:56.308+0000 7efc87d0d700 1 -- 172.21.15.59:0/2781827890 shutdown_connections 2024-03-28T18:03:56.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:03:56.308+0000 7efc87d0d700 1 -- 172.21.15.59:0/2781827890 wait complete. 2024-03-28T18:03:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:56 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:56 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:56 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:56 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:03:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:56 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:56 smithi059 ceph-mon[36190]: from='osd.4 [v2:172.21.15.59:6808/2410632623,v1:172.21.15.59:6809/2410632623]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T18:03:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:56 smithi059 ceph-mon[36190]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T18:03:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:56 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:56 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:56 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:03:56.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:56 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:56 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:56 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:56 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:03:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:56 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:56 smithi170 ceph-mon[36280]: from='osd.4 [v2:172.21.15.59:6808/2410632623,v1:172.21.15.59:6809/2410632623]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T18:03:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:56 smithi170 ceph-mon[36280]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T18:03:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:56 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:56 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:56 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:03:56.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:56 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:56.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:56 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:56.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:56 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:56.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:56 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:03:56.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:56 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:56.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:56 smithi037 ceph-mon[31460]: from='osd.4 [v2:172.21.15.59:6808/2410632623,v1:172.21.15.59:6809/2410632623]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T18:03:56.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:56 smithi037 ceph-mon[31460]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T18:03:56.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:56 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:56.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:56 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:03:56.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:56 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:03:57.011 DEBUG:teuthology.orchestra.run.smithi059:osd.4> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@osd.4.service 2024-03-28T18:03:57.013 INFO:tasks.cephadm:Deploying osd.5 on smithi170 with /dev/vg_nvme/lv_3... 2024-03-28T18:03:57.013 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T18:03:57.257 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:03:57.711 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:57 smithi170 ceph-mon[36280]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:57.711 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:57 smithi170 ceph-mon[36280]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T18:03:57.711 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:57 smithi170 ceph-mon[36280]: from='osd.4 [v2:172.21.15.59:6808/2410632623,v1:172.21.15.59:6809/2410632623]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:03:57.711 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:57 smithi170 ceph-mon[36280]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T18:03:57.711 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:57 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:03:57.711 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:57 smithi170 ceph-mon[36280]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:03:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:57 smithi059 ceph-mon[36190]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:57 smithi059 ceph-mon[36190]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T18:03:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:57 smithi059 ceph-mon[36190]: from='osd.4 [v2:172.21.15.59:6808/2410632623,v1:172.21.15.59:6809/2410632623]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:03:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:57 smithi059 ceph-mon[36190]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T18:03:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:57 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:03:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:57 smithi059 ceph-mon[36190]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:03:57.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:57 smithi037 ceph-mon[31460]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:57.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:57 smithi037 ceph-mon[31460]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T18:03:57.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:57 smithi037 ceph-mon[31460]: from='osd.4 [v2:172.21.15.59:6808/2410632623,v1:172.21.15.59:6809/2410632623]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:03:57.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:57 smithi037 ceph-mon[31460]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T18:03:57.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:03:57.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:57 smithi037 ceph-mon[31460]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:03:58.250 INFO:journalctl@ceph.osd.4.smithi059.stdout:Mar 28 18:03:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-4[47785]: 2024-03-28T18:03:57.887+0000 7f25110b7700 -1 osd.4 0 waiting for initial osdmap 2024-03-28T18:03:58.250 INFO:journalctl@ceph.osd.4.smithi059.stdout:Mar 28 18:03:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-4[47785]: 2024-03-28T18:03:57.893+0000 7f2508ea2700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:03:58.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:58 smithi059 ceph-mon[36190]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-03-28T18:03:58.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:58 smithi059 ceph-mon[36190]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T18:03:58.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:58 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:03:58.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:58 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:03:58.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:58 smithi059 ceph-mon[36190]: from='osd.4 ' entity='osd.4' 2024-03-28T18:03:58.659 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:58 smithi170 ceph-mon[36280]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-03-28T18:03:58.660 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:58 smithi170 ceph-mon[36280]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T18:03:58.660 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:58 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:03:58.660 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:58 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:03:58.660 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:58 smithi170 ceph-mon[36280]: from='osd.4 ' entity='osd.4' 2024-03-28T18:03:58.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:58 smithi037 ceph-mon[31460]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-03-28T18:03:58.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:58 smithi037 ceph-mon[31460]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T18:03:58.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:58 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:03:58.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:58 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:03:58.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:58 smithi037 ceph-mon[31460]: from='osd.4 ' entity='osd.4' 2024-03-28T18:03:59.329 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T18:03:59.329 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T18:03:59.330 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-03-28T18:03:59.330 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-03-28T18:03:59.330 INFO:teuthology.orchestra.run.smithi170.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00855319 s, 1.2 GB/s 2024-03-28T18:03:59.330 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-03-28T18:03:59.688 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:59 smithi170 ceph-mon[36280]: purged_snaps scrub starts 2024-03-28T18:03:59.688 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:59 smithi170 ceph-mon[36280]: purged_snaps scrub ok 2024-03-28T18:03:59.689 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:59 smithi170 ceph-mon[36280]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:59.689 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:59 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:03:59.689 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:59 smithi170 ceph-mon[36280]: osd.4 [v2:172.21.15.59:6808/2410632623,v1:172.21.15.59:6809/2410632623] boot 2024-03-28T18:03:59.689 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:59 smithi170 ceph-mon[36280]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T18:03:59.689 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:03:59 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:03:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:59 smithi059 ceph-mon[36190]: purged_snaps scrub starts 2024-03-28T18:03:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:59 smithi059 ceph-mon[36190]: purged_snaps scrub ok 2024-03-28T18:03:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:59 smithi059 ceph-mon[36190]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:59 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:03:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:59 smithi059 ceph-mon[36190]: osd.4 [v2:172.21.15.59:6808/2410632623,v1:172.21.15.59:6809/2410632623] boot 2024-03-28T18:03:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:59 smithi059 ceph-mon[36190]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T18:03:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:03:59 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:03:59.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:59 smithi037 ceph-mon[31460]: purged_snaps scrub starts 2024-03-28T18:03:59.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:59 smithi037 ceph-mon[31460]: purged_snaps scrub ok 2024-03-28T18:03:59.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:59 smithi037 ceph-mon[31460]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:03:59.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:59 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:03:59.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:59 smithi037 ceph-mon[31460]: osd.4 [v2:172.21.15.59:6808/2410632623,v1:172.21.15.59:6809/2410632623] boot 2024-03-28T18:03:59.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:59 smithi037 ceph-mon[31460]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T18:03:59.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:03:59 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:04:00.019 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi170:vg_nvme/lv_3 2024-03-28T18:04:00.252 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:04:01.030 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:00 smithi170 ceph-mon[36280]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:01.030 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:00 smithi170 ceph-mon[36280]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T18:04:01.030 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:00 smithi170 ceph-mon[36280]: Detected new or changed devices on smithi059 2024-03-28T18:04:01.030 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:00 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:01.030 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:00 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:01.030 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:00 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:04:01.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:00 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:04:01.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:00 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi059 to 8542M 2024-03-28T18:04:01.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:00 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:01.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:00 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:01.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:00 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:04:01.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:00 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:00 smithi059 ceph-mon[36190]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:00 smithi059 ceph-mon[36190]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T18:04:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:00 smithi059 ceph-mon[36190]: Detected new or changed devices on smithi059 2024-03-28T18:04:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:00 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:00 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:00 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:04:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:00 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:04:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:00 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi059 to 8542M 2024-03-28T18:04:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:00 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:00 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:00 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:04:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:00 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:00 smithi037 ceph-mon[31460]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:00 smithi037 ceph-mon[31460]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T18:04:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:00 smithi037 ceph-mon[31460]: Detected new or changed devices on smithi059 2024-03-28T18:04:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:00 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:00 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:00 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:04:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:00 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:04:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:00 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi059 to 8542M 2024-03-28T18:04:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:00 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:00 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:00 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:04:01.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:00 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:01.879 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.877+0000 7fbb39789700 1 -- 172.21.15.170:0/2614693661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb34105580 msgr2=0x7fbb34107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:01.879 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.877+0000 7fbb39789700 1 --2- 172.21.15.170:0/2614693661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb34105580 0x7fbb34107970 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c009990 tx=0x7fbb1c009ca0 comp rx=0 tx=0).stop 2024-03-28T18:04:01.879 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.877+0000 7fbb39789700 1 -- 172.21.15.170:0/2614693661 shutdown_connections 2024-03-28T18:04:01.879 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.877+0000 7fbb39789700 1 --2- 172.21.15.170:0/2614693661 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb34107eb0 0x7fbb3410a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:01.880 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.877+0000 7fbb39789700 1 --2- 172.21.15.170:0/2614693661 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb34102c50 0x7fbb34105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:01.880 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.877+0000 7fbb39789700 1 --2- 172.21.15.170:0/2614693661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb34105580 0x7fbb34107970 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:01.880 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.877+0000 7fbb39789700 1 -- 172.21.15.170:0/2614693661 >> 172.21.15.170:0/2614693661 conn(0x7fbb340fc660 msgr2=0x7fbb340feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:04:01.880 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.877+0000 7fbb39789700 1 -- 172.21.15.170:0/2614693661 shutdown_connections 2024-03-28T18:04:01.880 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.877+0000 7fbb39789700 1 -- 172.21.15.170:0/2614693661 wait complete. 2024-03-28T18:04:01.880 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.877+0000 7fbb39789700 1 Processor -- start 2024-03-28T18:04:01.880 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb39789700 1 -- start start 2024-03-28T18:04:01.880 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb39789700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb34102c50 0x7fbb341025b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:01.880 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb39789700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb34105580 0x7fbb34100c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:01.880 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb39789700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb34107eb0 0x7fbb34101140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:01.880 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb39789700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3410de10 con 0x7fbb34102c50 2024-03-28T18:04:01.880 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb39789700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3410dc90 con 0x7fbb34105580 2024-03-28T18:04:01.880 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb39789700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3410df90 con 0x7fbb34107eb0 2024-03-28T18:04:01.880 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb337fe700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb34107eb0 0x7fbb34101140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:01.881 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb337fe700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb34107eb0 0x7fbb34101140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:50540/0 (socket says 172.21.15.170:50540) 2024-03-28T18:04:01.881 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb337fe700 1 -- 172.21.15.170:0/3495210145 learned_addr learned my addr 172.21.15.170:0/3495210145 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T18:04:01.881 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb2a7fc700 1 --2- 172.21.15.170:0/3495210145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb34105580 0x7fbb34100c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:01.881 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb337fe700 1 -- 172.21.15.170:0/3495210145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb34105580 msgr2=0x7fbb34100c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:01.881 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb337fe700 1 --2- 172.21.15.170:0/3495210145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb34105580 0x7fbb34100c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:01.881 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb337fe700 1 -- 172.21.15.170:0/3495210145 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb34102c50 msgr2=0x7fbb341025b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:01.881 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb337fe700 1 --2- 172.21.15.170:0/3495210145 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb34102c50 0x7fbb341025b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:01.881 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb337fe700 1 -- 172.21.15.170:0/3495210145 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb341019f0 con 0x7fbb34107eb0 2024-03-28T18:04:01.881 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb2a7fc700 1 --2- 172.21.15.170:0/3495210145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb34105580 0x7fbb34100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:04:01.881 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb337fe700 1 --2- 172.21.15.170:0/3495210145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb34107eb0 0x7fbb34101140 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c0094f0 tx=0x7fbb2c007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:04:01.881 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb30ff9700 1 -- 172.21.15.170:0/3495210145 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c004db0 con 0x7fbb34107eb0 2024-03-28T18:04:01.881 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb30ff9700 1 -- 172.21.15.170:0/3495210145 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb2c00db60 con 0x7fbb34107eb0 2024-03-28T18:04:01.881 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb39789700 1 -- 172.21.15.170:0/3495210145 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb341a2610 con 0x7fbb34107eb0 2024-03-28T18:04:01.881 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.878+0000 7fbb30ff9700 1 -- 172.21.15.170:0/3495210145 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c015660 con 0x7fbb34107eb0 2024-03-28T18:04:01.882 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.879+0000 7fbb39789700 1 -- 172.21.15.170:0/3495210145 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb341a2b50 con 0x7fbb34107eb0 2024-03-28T18:04:01.882 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.879+0000 7fbb39789700 1 -- 172.21.15.170:0/3495210145 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb34068e50 con 0x7fbb34107eb0 2024-03-28T18:04:01.883 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.880+0000 7fbb30ff9700 1 -- 172.21.15.170:0/3495210145 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbb2c005020 con 0x7fbb34107eb0 2024-03-28T18:04:01.883 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.881+0000 7fbb30ff9700 1 --2- 172.21.15.170:0/3495210145 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fbb200a2890 0x7fbb200a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:01.883 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.881+0000 7fbb30ff9700 1 -- 172.21.15.170:0/3495210145 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7fbb2c0cd5b0 con 0x7fbb34107eb0 2024-03-28T18:04:01.883 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.882+0000 7fbb32ffd700 1 --2- 172.21.15.170:0/3495210145 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fbb200a2890 0x7fbb200a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:01.883 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.883+0000 7fbb32ffd700 1 --2- 172.21.15.170:0/3495210145 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fbb200a2890 0x7fbb200a4d50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbb240097b0 tx=0x7fbb24006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:04:01.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:01.886+0000 7fbb30ff9700 1 -- 172.21.15.170:0/3495210145 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbb2c010070 con 0x7fbb34107eb0 2024-03-28T18:04:02.091 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:02.091+0000 7fbb39789700 1 -- 172.21.15.170:0/3495210145 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb340625d0 con 0x7fbb200a2890 2024-03-28T18:04:02.995 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:02 smithi170 ceph-mon[36280]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:02.995 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:02 smithi170 ceph-mon[36280]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:04:02.995 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:02 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:04:02.995 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:02 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:04:02.995 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:02 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:02 smithi059 ceph-mon[36190]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:02 smithi059 ceph-mon[36190]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:04:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:02 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:04:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:02 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:04:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:02 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:03.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:02 smithi037 ceph-mon[31460]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:03.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:02 smithi037 ceph-mon[31460]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:04:03.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:02 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:04:03.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:02 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:04:03.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:02 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:05.180 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:04 smithi170 ceph-mon[36280]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:05.181 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:04 smithi170 ceph-mon[36280]: from='client.? 172.21.15.170:0/109916145' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b884842a-86e2-455b-9570-ff163b3bc0cf"}]: dispatch 2024-03-28T18:04:05.181 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:04 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b884842a-86e2-455b-9570-ff163b3bc0cf"}]: dispatch 2024-03-28T18:04:05.181 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:04 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b884842a-86e2-455b-9570-ff163b3bc0cf"}]': finished 2024-03-28T18:04:05.181 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:04 smithi170 ceph-mon[36280]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T18:04:05.181 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:04 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:04:05.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:04 smithi059 ceph-mon[36190]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:05.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:04 smithi059 ceph-mon[36190]: from='client.? 172.21.15.170:0/109916145' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b884842a-86e2-455b-9570-ff163b3bc0cf"}]: dispatch 2024-03-28T18:04:05.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:04 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b884842a-86e2-455b-9570-ff163b3bc0cf"}]: dispatch 2024-03-28T18:04:05.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:04 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b884842a-86e2-455b-9570-ff163b3bc0cf"}]': finished 2024-03-28T18:04:05.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:04 smithi059 ceph-mon[36190]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T18:04:05.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:04 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:04:05.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:04 smithi037 ceph-mon[31460]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:05.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:04 smithi037 ceph-mon[31460]: from='client.? 172.21.15.170:0/109916145' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b884842a-86e2-455b-9570-ff163b3bc0cf"}]: dispatch 2024-03-28T18:04:05.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:04 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b884842a-86e2-455b-9570-ff163b3bc0cf"}]: dispatch 2024-03-28T18:04:05.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:04 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b884842a-86e2-455b-9570-ff163b3bc0cf"}]': finished 2024-03-28T18:04:05.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:04 smithi037 ceph-mon[31460]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T18:04:05.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:04 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:04:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:05 smithi059 ceph-mon[36190]: from='client.? 172.21.15.170:0/3427022795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:04:06.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:05 smithi170 ceph-mon[36280]: from='client.? 172.21.15.170:0/3427022795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:04:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:05 smithi037 ceph-mon[31460]: from='client.? 172.21.15.170:0/3427022795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:04:07.276 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:06 smithi037 ceph-mon[31460]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:06 smithi059 ceph-mon[36190]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:07.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:06 smithi170 ceph-mon[36280]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:09.124 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:08 smithi170 ceph-mon[36280]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:08 smithi059 ceph-mon[36190]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:09.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:08 smithi037 ceph-mon[31460]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:11.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:10 smithi170 ceph-mon[36280]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:10 smithi059 ceph-mon[36190]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:10 smithi037 ceph-mon[31460]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:12.990 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:12 smithi170 ceph-mon[36280]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:12.990 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:12 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T18:04:12.990 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:12 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:12.990 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:12 smithi170 ceph-mon[36280]: Deploying daemon osd.5 on smithi170 2024-03-28T18:04:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:12 smithi059 ceph-mon[36190]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:12 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T18:04:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:12 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:12 smithi059 ceph-mon[36190]: Deploying daemon osd.5 on smithi170 2024-03-28T18:04:13.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:12 smithi037 ceph-mon[31460]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:13.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:12 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T18:04:13.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:12 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:13.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:12 smithi037 ceph-mon[31460]: Deploying daemon osd.5 on smithi170 2024-03-28T18:04:15.223 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:14 smithi170 ceph-mon[36280]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:15.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:14 smithi059 ceph-mon[36190]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:15.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:14 smithi037 ceph-mon[31460]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:17.083 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:16 smithi170 ceph-mon[36280]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:17.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:16 smithi059 ceph-mon[36190]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:17.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:16 smithi037 ceph-mon[31460]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:18 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:18 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:04:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:18 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:18 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:18 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:04:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:18 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:19.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:18 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:19.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:18 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:04:19.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:18 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:19.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:19 smithi170 ceph-mon[36280]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:19 smithi059 ceph-mon[36190]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:20.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:19 smithi037 ceph-mon[31460]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:21.578 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:21.579+0000 7fbb30ff9700 1 -- 172.21.15.170:0/3495210145 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbb340625d0 con 0x7fbb200a2890 2024-03-28T18:04:21.579 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 5 on host 'smithi170' 2024-03-28T18:04:21.582 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:21.582+0000 7fbb39789700 1 -- 172.21.15.170:0/3495210145 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fbb200a2890 msgr2=0x7fbb200a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:21.582 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:21.582+0000 7fbb39789700 1 --2- 172.21.15.170:0/3495210145 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fbb200a2890 0x7fbb200a4d50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbb240097b0 tx=0x7fbb24006d20 comp rx=0 tx=0).stop 2024-03-28T18:04:21.582 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:21.582+0000 7fbb39789700 1 -- 172.21.15.170:0/3495210145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb34107eb0 msgr2=0x7fbb34101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:21.582 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:21.582+0000 7fbb39789700 1 --2- 172.21.15.170:0/3495210145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb34107eb0 0x7fbb34101140 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c0094f0 tx=0x7fbb2c007f90 comp rx=0 tx=0).stop 2024-03-28T18:04:21.582 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:21.582+0000 7fbb39789700 1 -- 172.21.15.170:0/3495210145 shutdown_connections 2024-03-28T18:04:21.582 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:21.582+0000 7fbb39789700 1 --2- 172.21.15.170:0/3495210145 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fbb200a2890 0x7fbb200a4d50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:21.582 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:21.582+0000 7fbb39789700 1 --2- 172.21.15.170:0/3495210145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb34107eb0 0x7fbb34101140 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:21.582 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:21.582+0000 7fbb39789700 1 --2- 172.21.15.170:0/3495210145 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb34102c50 0x7fbb341025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:21.582 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:21.582+0000 7fbb39789700 1 --2- 172.21.15.170:0/3495210145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb34105580 0x7fbb34100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:21.583 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:21.582+0000 7fbb39789700 1 -- 172.21.15.170:0/3495210145 >> 172.21.15.170:0/3495210145 conn(0x7fbb340fc660 msgr2=0x7fbb34104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:04:21.583 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:21.582+0000 7fbb39789700 1 -- 172.21.15.170:0/3495210145 shutdown_connections 2024-03-28T18:04:21.583 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:04:21.582+0000 7fbb39789700 1 -- 172.21.15.170:0/3495210145 wait complete. 2024-03-28T18:04:21.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:21 smithi059 ceph-mon[36190]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:21.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:21 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:21.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:21 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:21.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:21 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:21.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:21 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:04:21.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:21 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:21.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:21 smithi170 ceph-mon[36280]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:21.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:21 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:21.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:21 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:21.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:21 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:21.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:21 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:04:21.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:21 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:21.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:21 smithi037 ceph-mon[31460]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:21 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:21 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:21 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:21 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:04:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:21 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:22.156 DEBUG:teuthology.orchestra.run.smithi170:osd.5> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@osd.5.service 2024-03-28T18:04:22.158 INFO:tasks.cephadm:Deploying osd.6 on smithi037 with /dev/vg_nvme/lv_2... 2024-03-28T18:04:22.159 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T18:04:22.620 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:04:22.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:22 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:22.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:22 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:22.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:22 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:04:22.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:22 smithi059 ceph-mon[36190]: from='osd.5 [v2:172.21.15.170:6808/1218943899,v1:172.21.15.170:6809/1218943899]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T18:04:22.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:22 smithi059 ceph-mon[36190]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T18:04:22.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:22 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:22.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:22 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:22.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:22 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:04:22.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:22 smithi170 ceph-mon[36280]: from='osd.5 [v2:172.21.15.170:6808/1218943899,v1:172.21.15.170:6809/1218943899]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T18:04:22.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:22 smithi170 ceph-mon[36280]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T18:04:22.806 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:22 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:22.806 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:22 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:22.806 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:22 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:04:22.806 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:22 smithi037 ceph-mon[31460]: from='osd.5 [v2:172.21.15.170:6808/1218943899,v1:172.21.15.170:6809/1218943899]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T18:04:22.806 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:22 smithi037 ceph-mon[31460]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T18:04:23.658 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:23 smithi170 ceph-mon[36280]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:23.658 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:23 smithi170 ceph-mon[36280]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T18:04:23.658 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:23 smithi170 ceph-mon[36280]: from='osd.5 [v2:172.21.15.170:6808/1218943899,v1:172.21.15.170:6809/1218943899]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:04:23.658 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:23 smithi170 ceph-mon[36280]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T18:04:23.658 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:04:23.658 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:23 smithi170 ceph-mon[36280]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:04:23.817 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:23 smithi037 ceph-mon[31460]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:23.818 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:23 smithi037 ceph-mon[31460]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T18:04:23.818 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:23 smithi037 ceph-mon[31460]: from='osd.5 [v2:172.21.15.170:6808/1218943899,v1:172.21.15.170:6809/1218943899]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:04:23.818 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:23 smithi037 ceph-mon[31460]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T18:04:23.818 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:04:23.818 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:23 smithi037 ceph-mon[31460]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:04:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:23 smithi059 ceph-mon[36190]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:23 smithi059 ceph-mon[36190]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T18:04:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:23 smithi059 ceph-mon[36190]: from='osd.5 [v2:172.21.15.170:6808/1218943899,v1:172.21.15.170:6809/1218943899]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:04:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:23 smithi059 ceph-mon[36190]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T18:04:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:23 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:04:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:23 smithi059 ceph-mon[36190]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:04:24.286 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 28 18:04:24 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-5[48127]: 2024-03-28T18:04:24.031+0000 7f52e70a2700 -1 osd.5 0 waiting for initial osdmap 2024-03-28T18:04:24.286 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 28 18:04:24 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-5[48127]: 2024-03-28T18:04:24.037+0000 7f52dee8d700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:04:24.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:24 smithi037 ceph-mon[31460]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-28T18:04:24.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:24 smithi037 ceph-mon[31460]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T18:04:24.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:24 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:04:24.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:24 smithi037 ceph-mon[31460]: from='osd.5 ' entity='osd.5' 2024-03-28T18:04:24.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:24 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:04:24.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:24 smithi170 ceph-mon[36280]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-28T18:04:24.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:24 smithi170 ceph-mon[36280]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T18:04:24.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:24 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:04:24.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:24 smithi170 ceph-mon[36280]: from='osd.5 ' entity='osd.5' 2024-03-28T18:04:24.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:24 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:04:25.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:24 smithi059 ceph-mon[36190]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-28T18:04:25.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:24 smithi059 ceph-mon[36190]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T18:04:25.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:04:25.037 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:24 smithi059 ceph-mon[36190]: from='osd.5 ' entity='osd.5' 2024-03-28T18:04:25.037 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:04:25.577 INFO:teuthology.orchestra.run.smithi037.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T18:04:25.577 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T18:04:25.577 INFO:teuthology.orchestra.run.smithi037.stderr: stderr: 10+0 records in 2024-03-28T18:04:25.577 INFO:teuthology.orchestra.run.smithi037.stderr:10+0 records out 2024-03-28T18:04:25.577 INFO:teuthology.orchestra.run.smithi037.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0382501 s, 274 MB/s 2024-03-28T18:04:25.577 INFO:teuthology.orchestra.run.smithi037.stderr:--> Zapping successful for: 2024-03-28T18:04:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:25 smithi037 ceph-mon[31460]: purged_snaps scrub starts 2024-03-28T18:04:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:25 smithi037 ceph-mon[31460]: purged_snaps scrub ok 2024-03-28T18:04:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:25 smithi037 ceph-mon[31460]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:25 smithi037 ceph-mon[31460]: osd.5 [v2:172.21.15.170:6808/1218943899,v1:172.21.15.170:6809/1218943899] boot 2024-03-28T18:04:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:25 smithi037 ceph-mon[31460]: osdmap e38: 6 total, 6 up, 6 in 2024-03-28T18:04:25.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:25 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:04:25.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:25 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:25.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:25 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:25.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:25 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:04:25.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:25 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:04:25.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:25 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:25.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:25 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:25.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:25 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:04:25.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:25 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:25 smithi059 ceph-mon[36190]: purged_snaps scrub starts 2024-03-28T18:04:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:25 smithi059 ceph-mon[36190]: purged_snaps scrub ok 2024-03-28T18:04:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:25 smithi059 ceph-mon[36190]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:25 smithi059 ceph-mon[36190]: osd.5 [v2:172.21.15.170:6808/1218943899,v1:172.21.15.170:6809/1218943899] boot 2024-03-28T18:04:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:25 smithi059 ceph-mon[36190]: osdmap e38: 6 total, 6 up, 6 in 2024-03-28T18:04:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:25 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:04:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:25 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:25 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:25 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:04:26.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:25 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:04:26.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:25 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:26.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:25 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:26.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:25 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:04:26.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:25 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:26.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:25 smithi170 ceph-mon[36280]: purged_snaps scrub starts 2024-03-28T18:04:26.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:25 smithi170 ceph-mon[36280]: purged_snaps scrub ok 2024-03-28T18:04:26.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:25 smithi170 ceph-mon[36280]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:04:26.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:25 smithi170 ceph-mon[36280]: osd.5 [v2:172.21.15.170:6808/1218943899,v1:172.21.15.170:6809/1218943899] boot 2024-03-28T18:04:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:25 smithi170 ceph-mon[36280]: osdmap e38: 6 total, 6 up, 6 in 2024-03-28T18:04:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:25 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:04:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:25 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:25 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:25 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:04:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:25 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:04:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:25 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:25 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:25 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:04:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:25 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:26.592 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi037:vg_nvme/lv_2 2024-03-28T18:04:26.836 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:26 smithi037 ceph-mon[31460]: Detected new or changed devices on smithi170 2024-03-28T18:04:26.836 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:26 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi170 to 8540M 2024-03-28T18:04:26.836 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:26 smithi037 ceph-mon[31460]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T18:04:27.012 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:04:27.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:26 smithi059 ceph-mon[36190]: Detected new or changed devices on smithi170 2024-03-28T18:04:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:26 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi170 to 8540M 2024-03-28T18:04:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:26 smithi059 ceph-mon[36190]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T18:04:27.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:26 smithi170 ceph-mon[36280]: Detected new or changed devices on smithi170 2024-03-28T18:04:27.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:26 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi170 to 8540M 2024-03-28T18:04:27.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:26 smithi170 ceph-mon[36280]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T18:04:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:27 smithi037 ceph-mon[31460]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:27 smithi037 ceph-mon[31460]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T18:04:27.937 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:27 smithi170 ceph-mon[36280]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:27.937 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:27 smithi170 ceph-mon[36280]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T18:04:28.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:27 smithi059 ceph-mon[36190]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:27 smithi059 ceph-mon[36190]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T18:04:28.856 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:28 smithi037 ceph-mon[31460]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T18:04:29.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:28 smithi059 ceph-mon[36190]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T18:04:29.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:28 smithi170 ceph-mon[36280]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T18:04:29.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.393+0000 7fba74600700 1 -- 172.21.15.37:0/2578808036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba6c105570 msgr2=0x7fba6c107960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:29.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.393+0000 7fba74600700 1 --2- 172.21.15.37:0/2578808036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba6c105570 0x7fba6c107960 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fba58009990 tx=0x7fba58009ca0 comp rx=0 tx=0).stop 2024-03-28T18:04:29.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.394+0000 7fba74600700 1 -- 172.21.15.37:0/2578808036 shutdown_connections 2024-03-28T18:04:29.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.394+0000 7fba74600700 1 --2- 172.21.15.37:0/2578808036 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba6c107ea0 0x7fba6c10a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:29.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.394+0000 7fba74600700 1 --2- 172.21.15.37:0/2578808036 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba6c102c40 0x7fba6c105030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:29.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.394+0000 7fba74600700 1 --2- 172.21.15.37:0/2578808036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba6c105570 0x7fba6c107960 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:29.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.394+0000 7fba74600700 1 -- 172.21.15.37:0/2578808036 >> 172.21.15.37:0/2578808036 conn(0x7fba6c0fc650 msgr2=0x7fba6c0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:04:29.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.394+0000 7fba74600700 1 -- 172.21.15.37:0/2578808036 shutdown_connections 2024-03-28T18:04:29.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.394+0000 7fba74600700 1 -- 172.21.15.37:0/2578808036 wait complete. 2024-03-28T18:04:29.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.395+0000 7fba74600700 1 Processor -- start 2024-03-28T18:04:29.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.395+0000 7fba74600700 1 -- start start 2024-03-28T18:04:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.395+0000 7fba74600700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba6c102c40 0x7fba6c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.395+0000 7fba74600700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba6c105570 0x7fba6c198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.396+0000 7fba74600700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba6c107ea0 0x7fba6c19f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.396+0000 7fba74600700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c10de10 con 0x7fba6c107ea0 2024-03-28T18:04:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.396+0000 7fba74600700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c10dc90 con 0x7fba6c102c40 2024-03-28T18:04:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.396+0000 7fba74600700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c10df90 con 0x7fba6c105570 2024-03-28T18:04:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.396+0000 7fba72600700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba6c107ea0 0x7fba6c19f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.396+0000 7fba72600700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba6c107ea0 0x7fba6c19f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44470/0 (socket says 172.21.15.37:44470) 2024-03-28T18:04:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.396+0000 7fba72600700 1 -- 172.21.15.37:0/3407715136 learned_addr learned my addr 172.21.15.37:0/3407715136 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:04:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.396+0000 7fba71c00700 1 --2- 172.21.15.37:0/3407715136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba6c102c40 0x7fba6c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.396+0000 7fba71200700 1 --2- 172.21.15.37:0/3407715136 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba6c105570 0x7fba6c198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.396+0000 7fba71c00700 1 -- 172.21.15.37:0/3407715136 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba6c105570 msgr2=0x7fba6c198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:29.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.396+0000 7fba71c00700 1 --2- 172.21.15.37:0/3407715136 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba6c105570 0x7fba6c198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:29.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.396+0000 7fba71c00700 1 -- 172.21.15.37:0/3407715136 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba6c107ea0 msgr2=0x7fba6c19f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:29.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.396+0000 7fba71c00700 1 --2- 172.21.15.37:0/3407715136 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba6c107ea0 0x7fba6c19f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:29.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.396+0000 7fba71c00700 1 -- 172.21.15.37:0/3407715136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6c1a0070 con 0x7fba6c102c40 2024-03-28T18:04:29.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.396+0000 7fba71200700 1 --2- 172.21.15.37:0/3407715136 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba6c105570 0x7fba6c198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:04:29.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.396+0000 7fba72600700 1 --2- 172.21.15.37:0/3407715136 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba6c107ea0 0x7fba6c19f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:04:29.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.397+0000 7fba71c00700 1 --2- 172.21.15.37:0/3407715136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba6c102c40 0x7fba6c198410 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fba6000e9f0 tx=0x7fba6000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:04:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.397+0000 7fba6a000700 1 -- 172.21.15.37:0/3407715136 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6000cbd0 con 0x7fba6c102c40 2024-03-28T18:04:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.397+0000 7fba6a000700 1 -- 172.21.15.37:0/3407715136 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba6000cd70 con 0x7fba6c102c40 2024-03-28T18:04:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.397+0000 7fba74600700 1 -- 172.21.15.37:0/3407715136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba6c1a0360 con 0x7fba6c102c40 2024-03-28T18:04:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.397+0000 7fba6a000700 1 -- 172.21.15.37:0/3407715136 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba600187a0 con 0x7fba6c102c40 2024-03-28T18:04:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.397+0000 7fba74600700 1 -- 172.21.15.37:0/3407715136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba6c1a08f0 con 0x7fba6c102c40 2024-03-28T18:04:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.398+0000 7fba74600700 1 -- 172.21.15.37:0/3407715136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba6c068e50 con 0x7fba6c102c40 2024-03-28T18:04:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.400+0000 7fba6a000700 1 -- 172.21.15.37:0/3407715136 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fba600109a0 con 0x7fba6c102c40 2024-03-28T18:04:29.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.402+0000 7fba6a000700 1 --2- 172.21.15.37:0/3407715136 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fba540a2910 0x7fba540a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:29.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.402+0000 7fba6a000700 1 -- 172.21.15.37:0/3407715136 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4533+0+0 (secure 0 0 0) 0x7fba600cdd30 con 0x7fba6c102c40 2024-03-28T18:04:29.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.402+0000 7fba71200700 1 --2- 172.21.15.37:0/3407715136 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fba540a2910 0x7fba540a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:29.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.403+0000 7fba71200700 1 --2- 172.21.15.37:0/3407715136 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fba540a2910 0x7fba540a4dd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fba5800b2e0 tx=0x7fba5801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:04:29.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.406+0000 7fba6a000700 1 -- 172.21.15.37:0/3407715136 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fba6009bbf0 con 0x7fba6c102c40 2024-03-28T18:04:29.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:29.647+0000 7fba74600700 1 -- 172.21.15.37:0/3407715136 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fba6c0625d0 con 0x7fba540a2910 2024-03-28T18:04:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:29 smithi059 ceph-mon[36190]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:29 smithi059 ceph-mon[36190]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T18:04:30.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:29 smithi170 ceph-mon[36280]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:30.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:29 smithi170 ceph-mon[36280]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T18:04:30.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:29 smithi037 ceph-mon[31460]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:30.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:29 smithi037 ceph-mon[31460]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T18:04:30.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:30 smithi037 ceph-mon[31460]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:04:30.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:30 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:04:30.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:30 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:04:30.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:30 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:31.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:30 smithi059 ceph-mon[36190]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:04:31.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:30 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:04:31.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:30 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:04:31.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:30 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:31.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:30 smithi170 ceph-mon[36280]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:04:31.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:30 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:04:31.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:30 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:04:31.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:30 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:31 smithi037 ceph-mon[31460]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:31.936 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:31 smithi059 ceph-mon[36190]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:32.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:31 smithi170 ceph-mon[36280]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:34.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:33 smithi059 ceph-mon[36190]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:04:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:33 smithi170 ceph-mon[36280]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:04:34.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:33 smithi037 ceph-mon[31460]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:04:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:34 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2949919437' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "249a1200-77f4-44ad-981f-1d352f732b12"}]: dispatch 2024-03-28T18:04:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:34 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "249a1200-77f4-44ad-981f-1d352f732b12"}]: dispatch 2024-03-28T18:04:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:34 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "249a1200-77f4-44ad-981f-1d352f732b12"}]': finished 2024-03-28T18:04:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:34 smithi059 ceph-mon[36190]: osdmap e43: 7 total, 6 up, 7 in 2024-03-28T18:04:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:34 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:04:35.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:34 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2949919437' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "249a1200-77f4-44ad-981f-1d352f732b12"}]: dispatch 2024-03-28T18:04:35.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:34 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "249a1200-77f4-44ad-981f-1d352f732b12"}]: dispatch 2024-03-28T18:04:35.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:34 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "249a1200-77f4-44ad-981f-1d352f732b12"}]': finished 2024-03-28T18:04:35.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:34 smithi170 ceph-mon[36280]: osdmap e43: 7 total, 6 up, 7 in 2024-03-28T18:04:35.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:34 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:04:35.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:34 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2949919437' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "249a1200-77f4-44ad-981f-1d352f732b12"}]: dispatch 2024-03-28T18:04:35.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:34 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "249a1200-77f4-44ad-981f-1d352f732b12"}]: dispatch 2024-03-28T18:04:35.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:34 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "249a1200-77f4-44ad-981f-1d352f732b12"}]': finished 2024-03-28T18:04:35.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:34 smithi037 ceph-mon[31460]: osdmap e43: 7 total, 6 up, 7 in 2024-03-28T18:04:35.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:34 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:04:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:35 smithi059 ceph-mon[36190]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-28T18:04:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:35 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/681135323' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:04:36.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:35 smithi170 ceph-mon[36280]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-28T18:04:36.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:35 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/681135323' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:04:36.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:35 smithi037 ceph-mon[31460]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-28T18:04:36.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:35 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/681135323' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:04:38.000 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:37 smithi037 ceph-mon[31460]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:04:38.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:37 smithi059 ceph-mon[36190]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:04:38.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:37 smithi170 ceph-mon[36280]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:04:39.955 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:39 smithi037 ceph-mon[31460]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T18:04:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:39 smithi059 ceph-mon[36190]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T18:04:40.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:39 smithi170 ceph-mon[36280]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T18:04:41.923 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:41 smithi037 ceph-mon[31460]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T18:04:42.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:41 smithi059 ceph-mon[36190]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T18:04:42.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:41 smithi170 ceph-mon[36280]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T18:04:43.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:43 smithi037 ceph-mon[31460]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:43.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:43 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T18:04:43.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:43 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:44.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:43 smithi059 ceph-mon[36190]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:44.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:43 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T18:04:44.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:43 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:44.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:43 smithi170 ceph-mon[36280]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:44.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:43 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T18:04:44.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:43 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:44.864 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:44 smithi037 ceph-mon[31460]: Deploying daemon osd.6 on smithi037 2024-03-28T18:04:45.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:44 smithi059 ceph-mon[36190]: Deploying daemon osd.6 on smithi037 2024-03-28T18:04:45.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:44 smithi170 ceph-mon[36280]: Deploying daemon osd.6 on smithi037 2024-03-28T18:04:45.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:45 smithi037 ceph-mon[31460]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:46.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:45 smithi059 ceph-mon[36190]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:46.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:45 smithi170 ceph-mon[36280]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:47.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:47 smithi037 ceph-mon[31460]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:48.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:47 smithi059 ceph-mon[36190]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:48.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:47 smithi170 ceph-mon[36280]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:49.825 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:49 smithi037 ceph-mon[31460]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:50.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:49 smithi059 ceph-mon[36190]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:49 smithi170 ceph-mon[36280]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:51.863 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:51 smithi037 ceph-mon[31460]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:52.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:51 smithi059 ceph-mon[36190]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:51 smithi170 ceph-mon[36280]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:53.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:52 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:04:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:52 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:52 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:53.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:52 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:04:53.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:52 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:53.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:52 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:52 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:04:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:52 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:52 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:54.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:53 smithi059 ceph-mon[36190]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:54.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:53 smithi170 ceph-mon[36280]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:54.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:53 smithi037 ceph-mon[31460]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:55.872 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:55 smithi037 ceph-mon[31460]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:55 smithi059 ceph-mon[36190]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:56.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:55 smithi170 ceph-mon[36280]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:57.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:57 smithi037 ceph-mon[31460]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:57.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:57.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:57.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:57 smithi037 ceph-mon[31460]: from='osd.6 [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T18:04:57.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:57.661+0000 7fba6a000700 1 -- 172.21.15.37:0/3407715136 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fba6c0625d0 con 0x7fba540a2910 2024-03-28T18:04:57.663 INFO:teuthology.orchestra.run.smithi037.stdout:Created osd(s) 6 on host 'smithi037' 2024-03-28T18:04:57.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:57.664+0000 7fba74600700 1 -- 172.21.15.37:0/3407715136 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fba540a2910 msgr2=0x7fba540a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:57.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:57.664+0000 7fba74600700 1 --2- 172.21.15.37:0/3407715136 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fba540a2910 0x7fba540a4dd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fba5800b2e0 tx=0x7fba5801a040 comp rx=0 tx=0).stop 2024-03-28T18:04:57.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:57.664+0000 7fba74600700 1 -- 172.21.15.37:0/3407715136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba6c102c40 msgr2=0x7fba6c198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:57.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:57.664+0000 7fba74600700 1 --2- 172.21.15.37:0/3407715136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba6c102c40 0x7fba6c198410 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fba6000e9f0 tx=0x7fba6000ed00 comp rx=0 tx=0).stop 2024-03-28T18:04:57.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:57.664+0000 7fba74600700 1 -- 172.21.15.37:0/3407715136 shutdown_connections 2024-03-28T18:04:57.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:57.664+0000 7fba74600700 1 --2- 172.21.15.37:0/3407715136 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fba540a2910 0x7fba540a4dd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:57.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:57.664+0000 7fba74600700 1 --2- 172.21.15.37:0/3407715136 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba6c107ea0 0x7fba6c19f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:57.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:57.664+0000 7fba74600700 1 --2- 172.21.15.37:0/3407715136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba6c102c40 0x7fba6c198410 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:57.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:57.664+0000 7fba74600700 1 --2- 172.21.15.37:0/3407715136 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba6c105570 0x7fba6c198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:57.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:57.664+0000 7fba74600700 1 -- 172.21.15.37:0/3407715136 >> 172.21.15.37:0/3407715136 conn(0x7fba6c0fc650 msgr2=0x7fba6c1036c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:04:57.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:57.665+0000 7fba74600700 1 -- 172.21.15.37:0/3407715136 shutdown_connections 2024-03-28T18:04:57.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:04:57.665+0000 7fba74600700 1 -- 172.21.15.37:0/3407715136 wait complete. 2024-03-28T18:04:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:57 smithi059 ceph-mon[36190]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:57 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:57 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:57 smithi059 ceph-mon[36190]: from='osd.6 [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T18:04:57.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:57 smithi170 ceph-mon[36280]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:57.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:57 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:57.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:57 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:57.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:57 smithi170 ceph-mon[36280]: from='osd.6 [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T18:04:58.661 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:58 smithi037 ceph-mon[31460]: from='osd.6 [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T18:04:58.662 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:58 smithi037 ceph-mon[31460]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T18:04:58.662 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:58 smithi037 ceph-mon[31460]: from='osd.6 [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]: dispatch 2024-03-28T18:04:58.662 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:58 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:04:58.662 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:58 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:58.662 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:58 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:58.668 DEBUG:teuthology.orchestra.run.smithi037:osd.6> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@osd.6.service 2024-03-28T18:04:58.673 INFO:tasks.cephadm:Deploying osd.7 on smithi059 with /dev/vg_nvme/lv_2... 2024-03-28T18:04:58.673 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T18:04:58.708 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:58 smithi059 ceph-mon[36190]: from='osd.6 [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T18:04:58.708 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:58 smithi059 ceph-mon[36190]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T18:04:58.708 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:58 smithi059 ceph-mon[36190]: from='osd.6 [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]: dispatch 2024-03-28T18:04:58.708 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:58 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:04:58.708 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:58 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:58.708 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:58 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:58.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:58 smithi170 ceph-mon[36280]: from='osd.6 [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T18:04:58.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:58 smithi170 ceph-mon[36280]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T18:04:58.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:58 smithi170 ceph-mon[36280]: from='osd.6 [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]: dispatch 2024-03-28T18:04:58.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:58 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:04:58.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:58 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:58.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:58 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:04:58.932 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:04:59.356 INFO:journalctl@ceph.osd.6.smithi037.stdout:Mar 28 18:04:59 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-6[57676]: 2024-03-28T18:04:59.041+0000 7f30a2a00700 -1 osd.6 0 waiting for initial osdmap 2024-03-28T18:04:59.356 INFO:journalctl@ceph.osd.6.smithi037.stdout:Mar 28 18:04:59 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-6[57676]: 2024-03-28T18:04:59.075+0000 7f3098800700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:04:59.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:59 smithi037 ceph-mon[31460]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:59.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:59 smithi037 ceph-mon[31460]: from='osd.6 [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]': finished 2024-03-28T18:04:59.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:59 smithi037 ceph-mon[31460]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T18:04:59.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:59 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:04:59.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:59 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:04:59.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:04:59 smithi037 ceph-mon[31460]: from='osd.6 [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046]' entity='osd.6' 2024-03-28T18:04:59.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:59 smithi059 ceph-mon[36190]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:59 smithi059 ceph-mon[36190]: from='osd.6 [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]': finished 2024-03-28T18:04:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:59 smithi059 ceph-mon[36190]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T18:04:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:59 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:04:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:59 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:04:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:04:59 smithi059 ceph-mon[36190]: from='osd.6 [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046]' entity='osd.6' 2024-03-28T18:04:59.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:59 smithi170 ceph-mon[36280]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:04:59.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:59 smithi170 ceph-mon[36280]: from='osd.6 [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]': finished 2024-03-28T18:04:59.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:59 smithi170 ceph-mon[36280]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T18:04:59.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:59 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:04:59.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:59 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:04:59.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:04:59 smithi170 ceph-mon[36280]: from='osd.6 [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046]' entity='osd.6' 2024-03-28T18:05:00.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:00 smithi037 ceph-mon[31460]: purged_snaps scrub starts 2024-03-28T18:05:00.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:00 smithi037 ceph-mon[31460]: purged_snaps scrub ok 2024-03-28T18:05:00.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:00 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:05:00.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:00 smithi037 ceph-mon[31460]: osd.6 [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046] boot 2024-03-28T18:05:00.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:00 smithi037 ceph-mon[31460]: osdmap e46: 7 total, 7 up, 7 in 2024-03-28T18:05:00.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:00 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:05:00.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:00 smithi059 ceph-mon[36190]: purged_snaps scrub starts 2024-03-28T18:05:00.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:00 smithi059 ceph-mon[36190]: purged_snaps scrub ok 2024-03-28T18:05:00.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:00 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:05:00.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:00 smithi059 ceph-mon[36190]: osd.6 [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046] boot 2024-03-28T18:05:00.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:00 smithi059 ceph-mon[36190]: osdmap e46: 7 total, 7 up, 7 in 2024-03-28T18:05:00.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:00 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:05:00.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:00 smithi170 ceph-mon[36280]: purged_snaps scrub starts 2024-03-28T18:05:00.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:00 smithi170 ceph-mon[36280]: purged_snaps scrub ok 2024-03-28T18:05:00.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:00 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:05:00.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:00 smithi170 ceph-mon[36280]: osd.6 [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046] boot 2024-03-28T18:05:00.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:00 smithi170 ceph-mon[36280]: osdmap e46: 7 total, 7 up, 7 in 2024-03-28T18:05:00.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:00 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:05:00.962 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T18:05:00.963 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T18:05:00.963 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10+0 records in 2024-03-28T18:05:00.963 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2024-03-28T18:05:00.963 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112628 s, 931 MB/s 2024-03-28T18:05:00.963 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping successful for: 2024-03-28T18:05:01.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:01 smithi059 ceph-mon[36190]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:05:01.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:01 smithi037 ceph-mon[31460]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:05:01.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:01 smithi170 ceph-mon[36280]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:05:01.790 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi059:vg_nvme/lv_2 2024-03-28T18:05:02.084 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:05:02.462 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:02 smithi059 ceph-mon[36190]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T18:05:02.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:02 smithi170 ceph-mon[36280]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T18:05:02.793 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:02 smithi037 ceph-mon[31460]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T18:05:03.503 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:03 smithi059 ceph-mon[36190]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:03.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:03 smithi037 ceph-mon[31460]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:03.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.738+0000 7fd6ec1e7700 1 -- 172.21.15.59:0/3718045672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6e41022c0 msgr2=0x7fd6e4102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:03.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.738+0000 7fd6ec1e7700 1 --2- 172.21.15.59:0/3718045672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6e41022c0 0x7fd6e4102720 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd6e00099c0 tx=0x7fd6e0009cd0 comp rx=0 tx=0).stop 2024-03-28T18:05:03.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.738+0000 7fd6ec1e7700 1 -- 172.21.15.59:0/3718045672 shutdown_connections 2024-03-28T18:05:03.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.738+0000 7fd6ec1e7700 1 --2- 172.21.15.59:0/3718045672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd6e4102c60 0x7fd6e4109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:03.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.738+0000 7fd6ec1e7700 1 --2- 172.21.15.59:0/3718045672 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6e41010c0 0x7fd6e41014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:03.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.738+0000 7fd6ec1e7700 1 --2- 172.21.15.59:0/3718045672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6e41022c0 0x7fd6e4102720 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:03.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.738+0000 7fd6ec1e7700 1 -- 172.21.15.59:0/3718045672 >> 172.21.15.59:0/3718045672 conn(0x7fd6e40fc680 msgr2=0x7fd6e40feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:05:03.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.738+0000 7fd6ec1e7700 1 -- 172.21.15.59:0/3718045672 shutdown_connections 2024-03-28T18:05:03.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.738+0000 7fd6ec1e7700 1 -- 172.21.15.59:0/3718045672 wait complete. 2024-03-28T18:05:03.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.738+0000 7fd6ec1e7700 1 Processor -- start 2024-03-28T18:05:03.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.738+0000 7fd6ec1e7700 1 -- start start 2024-03-28T18:05:03.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.739+0000 7fd6ec1e7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6e41010c0 0x7fd6e4198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:03.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.739+0000 7fd6ec1e7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd6e41022c0 0x7fd6e41989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:03.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.739+0000 7fd6ec1e7700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6e4102c60 0x7fd6e419f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:03.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.739+0000 7fd6ec1e7700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e410ccf0 con 0x7fd6e41022c0 2024-03-28T18:05:03.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.739+0000 7fd6ec1e7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e410cb70 con 0x7fd6e41010c0 2024-03-28T18:05:03.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.739+0000 7fd6ec1e7700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e410ce70 con 0x7fd6e4102c60 2024-03-28T18:05:03.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.739+0000 7fd6e9f83700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6e41010c0 0x7fd6e4198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:03.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.739+0000 7fd6e9f83700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6e41010c0 0x7fd6e4198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58052/0 (socket says 172.21.15.59:58052) 2024-03-28T18:05:03.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.739+0000 7fd6e9f83700 1 -- 172.21.15.59:0/3421647973 learned_addr learned my addr 172.21.15.59:0/3421647973 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-28T18:05:03.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.739+0000 7fd6e9f83700 1 -- 172.21.15.59:0/3421647973 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6e4102c60 msgr2=0x7fd6e419f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:03.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.739+0000 7fd6e9f83700 1 --2- 172.21.15.59:0/3421647973 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6e4102c60 0x7fd6e419f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:03.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.739+0000 7fd6e9f83700 1 -- 172.21.15.59:0/3421647973 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd6e41022c0 msgr2=0x7fd6e41989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:03.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.739+0000 7fd6e9f83700 1 --2- 172.21.15.59:0/3421647973 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd6e41022c0 0x7fd6e41989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:03.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.739+0000 7fd6e9f83700 1 -- 172.21.15.59:0/3421647973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e41a0070 con 0x7fd6e41010c0 2024-03-28T18:05:03.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.739+0000 7fd6e9f83700 1 --2- 172.21.15.59:0/3421647973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6e41010c0 0x7fd6e4198460 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400d980 tx=0x7fd6d400dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:05:03.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.739+0000 7fd6daffd700 1 -- 172.21.15.59:0/3421647973 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4017070 con 0x7fd6e41010c0 2024-03-28T18:05:03.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.739+0000 7fd6ec1e7700 1 -- 172.21.15.59:0/3421647973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e41a0360 con 0x7fd6e41010c0 2024-03-28T18:05:03.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.739+0000 7fd6daffd700 1 -- 172.21.15.59:0/3421647973 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6d4009b40 con 0x7fd6e41010c0 2024-03-28T18:05:03.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.739+0000 7fd6daffd700 1 -- 172.21.15.59:0/3421647973 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d400f620 con 0x7fd6e41010c0 2024-03-28T18:05:03.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.740+0000 7fd6ec1e7700 1 -- 172.21.15.59:0/3421647973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e41a0850 con 0x7fd6e41010c0 2024-03-28T18:05:03.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.740+0000 7fd6ec1e7700 1 -- 172.21.15.59:0/3421647973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e4068e80 con 0x7fd6e41010c0 2024-03-28T18:05:03.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.741+0000 7fd6daffd700 1 -- 172.21.15.59:0/3421647973 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd6d4010460 con 0x7fd6e41010c0 2024-03-28T18:05:03.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.742+0000 7fd6daffd700 1 --2- 172.21.15.59:0/3421647973 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd6d00a2890 0x7fd6d00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:03.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.742+0000 7fd6daffd700 1 -- 172.21.15.59:0/3421647973 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4965+0+0 (secure 0 0 0) 0x7fd6d40cd9f0 con 0x7fd6e41010c0 2024-03-28T18:05:03.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.743+0000 7fd6e9782700 1 --2- 172.21.15.59:0/3421647973 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd6d00a2890 0x7fd6d00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:03.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.744+0000 7fd6e9782700 1 --2- 172.21.15.59:0/3421647973 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd6d00a2890 0x7fd6d00a4d50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd6e00098e0 tx=0x7fd6e0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:05:03.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.747+0000 7fd6daffd700 1 -- 172.21.15.59:0/3421647973 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd6e4068e80 con 0x7fd6e41010c0 2024-03-28T18:05:03.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:03 smithi170 ceph-mon[36280]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:03.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:03.963+0000 7fd6ec1e7700 1 -- 172.21.15.59:0/3421647973 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6e40fe870 con 0x7fd6d00a2890 2024-03-28T18:05:04.959 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:04 smithi059 ceph-mon[36190]: Detected new or changed devices on smithi037 2024-03-28T18:05:04.959 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:04 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:04.959 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:04 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:04.959 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:04 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:04.959 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:04 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:04.959 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:04 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:04.959 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:04 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi037 to 5698M 2024-03-28T18:05:04.959 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:04 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:04.959 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:04 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:04.959 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:04 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:04.960 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:04 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:04.960 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:04 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:05:04.960 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:04 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:05:04.960 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:04 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:04.960 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:04 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:05:04.960 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:04 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:04.960 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:04 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:04.960 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:04 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:05.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:04 smithi037 ceph-mon[31460]: Detected new or changed devices on smithi037 2024-03-28T18:05:05.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:04 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:05.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:04 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:05.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:04 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:05.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:04 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:05.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:04 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:05.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:04 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi037 to 5698M 2024-03-28T18:05:05.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:04 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:05.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:04 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:05.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:04 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:05.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:04 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:05.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:04 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:05:05.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:04 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:05:05.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:04 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:05.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:04 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:05:05.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:04 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:05.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:04 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:05.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:04 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:05.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:04 smithi170 ceph-mon[36280]: Detected new or changed devices on smithi037 2024-03-28T18:05:05.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:04 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:05.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:04 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:05.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:04 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:05.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:04 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:05.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:04 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:05.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:04 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi037 to 5698M 2024-03-28T18:05:05.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:04 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:05.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:04 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:05.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:04 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:05.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:04 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:05.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:04 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:05:05.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:04 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:05:05.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:04 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:05.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:04 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:05:05.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:04 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:05.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:04 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:05.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:04 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:05.948 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:05 smithi059 ceph-mon[36190]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:05.948 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:05 smithi059 ceph-mon[36190]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:05:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:05 smithi037 ceph-mon[31460]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:06.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:05 smithi037 ceph-mon[31460]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:05:06.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:05 smithi170 ceph-mon[36280]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:06.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:05 smithi170 ceph-mon[36280]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:05:07.026 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:06 smithi037 ceph-mon[31460]: from='client.? 172.21.15.59:0/938696036' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c73338e6-f505-4517-a84b-f35e04750a99"}]: dispatch 2024-03-28T18:05:07.026 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:06 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c73338e6-f505-4517-a84b-f35e04750a99"}]: dispatch 2024-03-28T18:05:07.026 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:06 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c73338e6-f505-4517-a84b-f35e04750a99"}]': finished 2024-03-28T18:05:07.026 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:06 smithi037 ceph-mon[31460]: osdmap e48: 8 total, 7 up, 8 in 2024-03-28T18:05:07.026 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:06 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:05:07.120 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:06 smithi059 ceph-mon[36190]: from='client.? 172.21.15.59:0/938696036' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c73338e6-f505-4517-a84b-f35e04750a99"}]: dispatch 2024-03-28T18:05:07.121 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:06 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c73338e6-f505-4517-a84b-f35e04750a99"}]: dispatch 2024-03-28T18:05:07.121 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:06 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c73338e6-f505-4517-a84b-f35e04750a99"}]': finished 2024-03-28T18:05:07.121 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:06 smithi059 ceph-mon[36190]: osdmap e48: 8 total, 7 up, 8 in 2024-03-28T18:05:07.121 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:06 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:05:07.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:06 smithi170 ceph-mon[36280]: from='client.? 172.21.15.59:0/938696036' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c73338e6-f505-4517-a84b-f35e04750a99"}]: dispatch 2024-03-28T18:05:07.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:06 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c73338e6-f505-4517-a84b-f35e04750a99"}]: dispatch 2024-03-28T18:05:07.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:06 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c73338e6-f505-4517-a84b-f35e04750a99"}]': finished 2024-03-28T18:05:07.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:06 smithi170 ceph-mon[36280]: osdmap e48: 8 total, 7 up, 8 in 2024-03-28T18:05:07.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:06 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:05:08.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:07 smithi037 ceph-mon[31460]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:08.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:07 smithi037 ceph-mon[31460]: from='client.? 172.21.15.59:0/3877350179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:05:08.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:07 smithi059 ceph-mon[36190]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:08.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:07 smithi059 ceph-mon[36190]: from='client.? 172.21.15.59:0/3877350179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:05:08.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:07 smithi170 ceph-mon[36280]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:08.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:07 smithi170 ceph-mon[36280]: from='client.? 172.21.15.59:0/3877350179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:05:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:09 smithi037 ceph-mon[31460]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:10.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:09 smithi059 ceph-mon[36190]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:10.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:09 smithi170 ceph-mon[36280]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:12.016 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:11 smithi059 ceph-mon[36190]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:11 smithi037 ceph-mon[31460]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:12.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:11 smithi170 ceph-mon[36280]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:13 smithi037 ceph-mon[31460]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:14.195 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:13 smithi059 ceph-mon[36190]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:14.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:13 smithi170 ceph-mon[36280]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:15.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T18:05:15.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:15.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T18:05:15.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:15.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T18:05:15.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:15.918 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:15 smithi059 ceph-mon[36190]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:15.918 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:15 smithi059 ceph-mon[36190]: Deploying daemon osd.7 on smithi059 2024-03-28T18:05:16.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:15 smithi037 ceph-mon[31460]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:16.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:15 smithi037 ceph-mon[31460]: Deploying daemon osd.7 on smithi059 2024-03-28T18:05:16.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:15 smithi170 ceph-mon[36280]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:16.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:15 smithi170 ceph-mon[36280]: Deploying daemon osd.7 on smithi059 2024-03-28T18:05:17.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:16 smithi037 ceph-mon[31460]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:17.139 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:16 smithi059 ceph-mon[36190]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:16 smithi170 ceph-mon[36280]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:19.208 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:18 smithi037 ceph-mon[31460]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:19.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:18 smithi059 ceph-mon[36190]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:19.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:18 smithi170 ceph-mon[36280]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:21.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:21 smithi037 ceph-mon[31460]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:21.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:21 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:21.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:21 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:21.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:21 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:05:21.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:21 smithi059 ceph-mon[36190]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:21 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:21 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:21 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:05:21.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:21 smithi170 ceph-mon[36280]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:21.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:21 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:21.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:21 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:21.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:21 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:05:23.170 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:23 smithi059 ceph-mon[36190]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:23.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:23 smithi037 ceph-mon[31460]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:23.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:23 smithi170 ceph-mon[36280]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:24.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:24.352+0000 7fd6daffd700 1 -- 172.21.15.59:0/3421647973 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd6e40fe870 con 0x7fd6d00a2890 2024-03-28T18:05:24.351 INFO:teuthology.orchestra.run.smithi059.stdout:Created osd(s) 7 on host 'smithi059' 2024-03-28T18:05:24.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:24.354+0000 7fd6ec1e7700 1 -- 172.21.15.59:0/3421647973 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd6d00a2890 msgr2=0x7fd6d00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:24.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:24.354+0000 7fd6ec1e7700 1 --2- 172.21.15.59:0/3421647973 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd6d00a2890 0x7fd6d00a4d50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd6e00098e0 tx=0x7fd6e0000bc0 comp rx=0 tx=0).stop 2024-03-28T18:05:24.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:24.354+0000 7fd6ec1e7700 1 -- 172.21.15.59:0/3421647973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6e41010c0 msgr2=0x7fd6e4198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:24.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:24.354+0000 7fd6ec1e7700 1 --2- 172.21.15.59:0/3421647973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6e41010c0 0x7fd6e4198460 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400d980 tx=0x7fd6d400dd40 comp rx=0 tx=0).stop 2024-03-28T18:05:24.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:24.354+0000 7fd6ec1e7700 1 -- 172.21.15.59:0/3421647973 shutdown_connections 2024-03-28T18:05:24.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:24.354+0000 7fd6ec1e7700 1 --2- 172.21.15.59:0/3421647973 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd6d00a2890 0x7fd6d00a4d50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:24.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:24.354+0000 7fd6ec1e7700 1 --2- 172.21.15.59:0/3421647973 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6e4102c60 0x7fd6e419f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:24.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:24.354+0000 7fd6ec1e7700 1 --2- 172.21.15.59:0/3421647973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6e41010c0 0x7fd6e4198460 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:24.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:24.354+0000 7fd6ec1e7700 1 --2- 172.21.15.59:0/3421647973 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd6e41022c0 0x7fd6e41989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:24.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:24.354+0000 7fd6ec1e7700 1 -- 172.21.15.59:0/3421647973 >> 172.21.15.59:0/3421647973 conn(0x7fd6e40fc680 msgr2=0x7fd6e40fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:05:24.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:24.354+0000 7fd6ec1e7700 1 -- 172.21.15.59:0/3421647973 shutdown_connections 2024-03-28T18:05:24.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:05:24.354+0000 7fd6ec1e7700 1 -- 172.21.15.59:0/3421647973 wait complete. 2024-03-28T18:05:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:24 smithi059 ceph-mon[36190]: from='osd.7 [v2:172.21.15.59:6816/1343675690,v1:172.21.15.59:6817/1343675690]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T18:05:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:24 smithi059 ceph-mon[36190]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T18:05:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:05:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:24.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:24 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:24.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:24 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:24.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:24 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:24.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:24 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:24.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:24 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:24.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:24 smithi170 ceph-mon[36280]: from='osd.7 [v2:172.21.15.59:6816/1343675690,v1:172.21.15.59:6817/1343675690]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T18:05:24.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:24 smithi170 ceph-mon[36280]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T18:05:24.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:24 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:24.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:24 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:05:24.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:24 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:24 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:24 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:24 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:24 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:24 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:24 smithi037 ceph-mon[31460]: from='osd.7 [v2:172.21.15.59:6816/1343675690,v1:172.21.15.59:6817/1343675690]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T18:05:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:24 smithi037 ceph-mon[31460]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T18:05:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:24 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:24.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:24 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:05:24.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:24 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:25.138 DEBUG:teuthology.orchestra.run.smithi059:osd.7> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@osd.7.service 2024-03-28T18:05:25.140 INFO:tasks.cephadm:Deploying osd.8 on smithi170 with /dev/vg_nvme/lv_2... 2024-03-28T18:05:25.140 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T18:05:25.386 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:05:25.752 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:25 smithi059 ceph-mon[36190]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:25.752 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:25 smithi059 ceph-mon[36190]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T18:05:25.752 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:25 smithi059 ceph-mon[36190]: from='osd.7 [v2:172.21.15.59:6816/1343675690,v1:172.21.15.59:6817/1343675690]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:05:25.752 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:25 smithi059 ceph-mon[36190]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T18:05:25.752 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:25 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:05:25.752 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:25 smithi059 ceph-mon[36190]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:05:25.760 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:25 smithi170 ceph-mon[36280]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:25.761 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:25 smithi170 ceph-mon[36280]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T18:05:25.761 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:25 smithi170 ceph-mon[36280]: from='osd.7 [v2:172.21.15.59:6816/1343675690,v1:172.21.15.59:6817/1343675690]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:05:25.761 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:25 smithi170 ceph-mon[36280]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T18:05:25.761 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:25 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:05:25.761 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:25 smithi170 ceph-mon[36280]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:05:25.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:25 smithi037 ceph-mon[31460]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:25.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:25 smithi037 ceph-mon[31460]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T18:05:25.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:25 smithi037 ceph-mon[31460]: from='osd.7 [v2:172.21.15.59:6816/1343675690,v1:172.21.15.59:6817/1343675690]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:05:25.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:25 smithi037 ceph-mon[31460]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T18:05:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:25 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:05:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:25 smithi037 ceph-mon[31460]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:05:26.034 INFO:journalctl@ceph.osd.7.smithi059.stdout:Mar 28 18:05:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-7[54751]: 2024-03-28T18:05:25.926+0000 7fadb36e6700 -1 osd.7 0 waiting for initial osdmap 2024-03-28T18:05:26.035 INFO:journalctl@ceph.osd.7.smithi059.stdout:Mar 28 18:05:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-7[54751]: 2024-03-28T18:05:25.939+0000 7fadab4d1700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:05:26.641 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:26 smithi059 ceph-mon[36190]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-03-28T18:05:26.641 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:26 smithi059 ceph-mon[36190]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T18:05:26.641 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:26 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:05:26.641 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:26 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:05:26.641 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:26 smithi059 ceph-mon[36190]: from='osd.7 ' entity='osd.7' 2024-03-28T18:05:26.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:26 smithi170 ceph-mon[36280]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-03-28T18:05:26.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:26 smithi170 ceph-mon[36280]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T18:05:26.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:26 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:05:26.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:26 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:05:26.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:26 smithi170 ceph-mon[36280]: from='osd.7 ' entity='osd.7' 2024-03-28T18:05:26.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:26 smithi037 ceph-mon[31460]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-03-28T18:05:26.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:26 smithi037 ceph-mon[31460]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T18:05:26.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:26 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:05:26.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:26 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:05:26.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:26 smithi037 ceph-mon[31460]: from='osd.7 ' entity='osd.7' 2024-03-28T18:05:27.400 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T18:05:27.400 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T18:05:27.400 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-03-28T18:05:27.400 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-03-28T18:05:27.400 INFO:teuthology.orchestra.run.smithi170.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00838995 s, 1.2 GB/s 2024-03-28T18:05:27.400 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-03-28T18:05:27.714 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:27 smithi170 ceph-mon[36280]: purged_snaps scrub starts 2024-03-28T18:05:27.714 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:27 smithi170 ceph-mon[36280]: purged_snaps scrub ok 2024-03-28T18:05:27.714 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:27 smithi170 ceph-mon[36280]: pgmap v196: 1 pgs: 1 unknown; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:27.714 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:27 smithi170 ceph-mon[36280]: osd.7 [v2:172.21.15.59:6816/1343675690,v1:172.21.15.59:6817/1343675690] boot 2024-03-28T18:05:27.714 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:27 smithi170 ceph-mon[36280]: osdmap e51: 8 total, 8 up, 8 in 2024-03-28T18:05:27.714 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:27 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:05:27.714 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:27 smithi170 ceph-mon[36280]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-28T18:05:27.715 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:27 smithi170 ceph-mon[36280]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T18:05:27.780 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:27 smithi059 ceph-mon[36190]: purged_snaps scrub starts 2024-03-28T18:05:27.780 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:27 smithi059 ceph-mon[36190]: purged_snaps scrub ok 2024-03-28T18:05:27.780 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:27 smithi059 ceph-mon[36190]: pgmap v196: 1 pgs: 1 unknown; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:27.780 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:27 smithi059 ceph-mon[36190]: osd.7 [v2:172.21.15.59:6816/1343675690,v1:172.21.15.59:6817/1343675690] boot 2024-03-28T18:05:27.780 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:27 smithi059 ceph-mon[36190]: osdmap e51: 8 total, 8 up, 8 in 2024-03-28T18:05:27.780 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:27 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:05:27.780 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:27 smithi059 ceph-mon[36190]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-28T18:05:27.780 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:27 smithi059 ceph-mon[36190]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T18:05:27.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:27 smithi037 ceph-mon[31460]: purged_snaps scrub starts 2024-03-28T18:05:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:27 smithi037 ceph-mon[31460]: purged_snaps scrub ok 2024-03-28T18:05:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:27 smithi037 ceph-mon[31460]: pgmap v196: 1 pgs: 1 unknown; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:05:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:27 smithi037 ceph-mon[31460]: osd.7 [v2:172.21.15.59:6816/1343675690,v1:172.21.15.59:6817/1343675690] boot 2024-03-28T18:05:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:27 smithi037 ceph-mon[31460]: osdmap e51: 8 total, 8 up, 8 in 2024-03-28T18:05:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:27 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:05:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:27 smithi037 ceph-mon[31460]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-28T18:05:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:27 smithi037 ceph-mon[31460]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T18:05:28.192 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi170:vg_nvme/lv_2 2024-03-28T18:05:28.437 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:05:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:29 smithi059 ceph-mon[36190]: pgmap v199: 1 pgs: 1 unknown; 577 KiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:29 smithi059 ceph-mon[36190]: Detected new or changed devices on smithi059 2024-03-28T18:05:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:29.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:29 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi059 to 5694M 2024-03-28T18:05:29.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:29.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:29.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:29.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:29 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:29.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:29 smithi059 ceph-mon[36190]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T18:05:29.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:29 smithi170 ceph-mon[36280]: pgmap v199: 1 pgs: 1 unknown; 577 KiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:29.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:29 smithi170 ceph-mon[36280]: Detected new or changed devices on smithi059 2024-03-28T18:05:29.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:29.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:29.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:29.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:29.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:29.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:29 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi059 to 5694M 2024-03-28T18:05:29.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:29.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:29.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:29.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:29 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:29.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:29 smithi170 ceph-mon[36280]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T18:05:29.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:29 smithi037 ceph-mon[31460]: pgmap v199: 1 pgs: 1 unknown; 577 KiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:29.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:29 smithi037 ceph-mon[31460]: Detected new or changed devices on smithi059 2024-03-28T18:05:29.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:29.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:29.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:29.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:29.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:29.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:29 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi059 to 5694M 2024-03-28T18:05:29.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:29.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:29.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:29.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:29 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:29.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:29 smithi037 ceph-mon[31460]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T18:05:30.063 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.061+0000 7fc8371f0700 1 -- 172.21.15.170:0/3164764565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc830101d30 msgr2=0x7fc8301095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:30.063 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.061+0000 7fc8371f0700 1 --2- 172.21.15.170:0/3164764565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc830101d30 0x7fc8301095f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc82800b0d0 tx=0x7fc82800b3e0 comp rx=0 tx=0).stop 2024-03-28T18:05:30.063 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.061+0000 7fc8371f0700 1 -- 172.21.15.170:0/3164764565 shutdown_connections 2024-03-28T18:05:30.063 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.061+0000 7fc8371f0700 1 --2- 172.21.15.170:0/3164764565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc830101d30 0x7fc8301095f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:30.064 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.061+0000 7fc8371f0700 1 --2- 172.21.15.170:0/3164764565 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc830102650 0x7fc830102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:30.064 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.061+0000 7fc8371f0700 1 --2- 172.21.15.170:0/3164764565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc830101390 0x7fc8301017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:30.064 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.061+0000 7fc8371f0700 1 -- 172.21.15.170:0/3164764565 >> 172.21.15.170:0/3164764565 conn(0x7fc8300fbe00 msgr2=0x7fc8300fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:05:30.064 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.061+0000 7fc8371f0700 1 -- 172.21.15.170:0/3164764565 shutdown_connections 2024-03-28T18:05:30.064 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.061+0000 7fc8371f0700 1 -- 172.21.15.170:0/3164764565 wait complete. 2024-03-28T18:05:30.064 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.061+0000 7fc8371f0700 1 Processor -- start 2024-03-28T18:05:30.064 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.061+0000 7fc8371f0700 1 -- start start 2024-03-28T18:05:30.064 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.062+0000 7fc8371f0700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc830101390 0x7fc830198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:30.064 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.062+0000 7fc8371f0700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc830101d30 0x7fc830198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:30.064 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.062+0000 7fc8371f0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc830102650 0x7fc83019f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:30.064 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.062+0000 7fc8371f0700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83010cd20 con 0x7fc830101d30 2024-03-28T18:05:30.064 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.062+0000 7fc8371f0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83010cba0 con 0x7fc830102650 2024-03-28T18:05:30.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.062+0000 7fc8371f0700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83010cea0 con 0x7fc830101390 2024-03-28T18:05:30.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.062+0000 7fc834f8c700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc830101390 0x7fc830198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:30.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.062+0000 7fc834f8c700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc830101390 0x7fc830198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:50928/0 (socket says 172.21.15.170:50928) 2024-03-28T18:05:30.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.062+0000 7fc834f8c700 1 -- 172.21.15.170:0/1066420272 learned_addr learned my addr 172.21.15.170:0/1066420272 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T18:05:30.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.062+0000 7fc83578d700 1 --2- 172.21.15.170:0/1066420272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc830102650 0x7fc83019f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:30.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.062+0000 7fc834f8c700 1 -- 172.21.15.170:0/1066420272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc830102650 msgr2=0x7fc83019f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:30.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.062+0000 7fc834f8c700 1 --2- 172.21.15.170:0/1066420272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc830102650 0x7fc83019f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:30.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.062+0000 7fc834f8c700 1 -- 172.21.15.170:0/1066420272 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc830101d30 msgr2=0x7fc830198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:30.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.062+0000 7fc834f8c700 1 --2- 172.21.15.170:0/1066420272 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc830101d30 0x7fc830198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:30.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.062+0000 7fc834f8c700 1 -- 172.21.15.170:0/1066420272 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8301a0010 con 0x7fc830101390 2024-03-28T18:05:30.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.062+0000 7fc834f8c700 1 --2- 172.21.15.170:0/1066420272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc830101390 0x7fc830198400 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc82400b530 tx=0x7fc82400b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:05:30.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.062+0000 7fc82dffb700 1 -- 172.21.15.170:0/1066420272 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc824015020 con 0x7fc830101390 2024-03-28T18:05:30.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.062+0000 7fc8371f0700 1 -- 172.21.15.170:0/1066420272 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8301a0300 con 0x7fc830101390 2024-03-28T18:05:30.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.062+0000 7fc82dffb700 1 -- 172.21.15.170:0/1066420272 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc824011d10 con 0x7fc830101390 2024-03-28T18:05:30.066 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.062+0000 7fc82dffb700 1 -- 172.21.15.170:0/1066420272 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82400f2e0 con 0x7fc830101390 2024-03-28T18:05:30.066 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.063+0000 7fc8371f0700 1 -- 172.21.15.170:0/1066420272 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8301a0910 con 0x7fc830101390 2024-03-28T18:05:30.066 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.064+0000 7fc8371f0700 1 -- 172.21.15.170:0/1066420272 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc830068e50 con 0x7fc830101390 2024-03-28T18:05:30.067 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.065+0000 7fc82dffb700 1 -- 172.21.15.170:0/1066420272 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc824018020 con 0x7fc830101390 2024-03-28T18:05:30.067 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.065+0000 7fc82dffb700 1 --2- 172.21.15.170:0/1066420272 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc8200a2890 0x7fc8200a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:30.067 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.066+0000 7fc82dffb700 1 -- 172.21.15.170:0/1066420272 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5397+0+0 (secure 0 0 0) 0x7fc8240cd610 con 0x7fc830101390 2024-03-28T18:05:30.067 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.066+0000 7fc82ffff700 1 --2- 172.21.15.170:0/1066420272 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc8200a2890 0x7fc8200a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:30.067 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.067+0000 7fc82ffff700 1 --2- 172.21.15.170:0/1066420272 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc8200a2890 0x7fc8200a4d50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc81c00a990 tx=0x7fc81c005c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:05:30.070 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.070+0000 7fc82dffb700 1 -- 172.21.15.170:0/1066420272 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc82401f020 con 0x7fc830101390 2024-03-28T18:05:30.274 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:30.273+0000 7fc8371f0700 1 -- 172.21.15.170:0/1066420272 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8300fdf10 con 0x7fc8200a2890 2024-03-28T18:05:30.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:30 smithi170 ceph-mon[36280]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T18:05:30.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:30 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:05:30.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:30 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:05:30.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:30 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:30.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:30 smithi059 ceph-mon[36190]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T18:05:30.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:30 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:05:30.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:30 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:05:30.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:30 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:30.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:30 smithi037 ceph-mon[31460]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T18:05:30.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:30 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:05:30.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:30 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:05:30.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:30 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:31.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:31 smithi059 ceph-mon[36190]: pgmap v202: 1 pgs: 1 unknown; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:31.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:31 smithi059 ceph-mon[36190]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:05:31.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:31 smithi059 ceph-mon[36190]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T18:05:31.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:31 smithi170 ceph-mon[36280]: pgmap v202: 1 pgs: 1 unknown; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:31.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:31 smithi170 ceph-mon[36280]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:05:31.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:31 smithi170 ceph-mon[36280]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T18:05:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:31 smithi037 ceph-mon[31460]: pgmap v202: 1 pgs: 1 unknown; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:31 smithi037 ceph-mon[31460]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:05:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:31 smithi037 ceph-mon[31460]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T18:05:32.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:32 smithi037 ceph-mon[31460]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-28T18:05:32.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:32 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:05:32.978 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:32 smithi059 ceph-mon[36190]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-28T18:05:32.978 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:32 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:05:33.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:32 smithi170 ceph-mon[36280]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-28T18:05:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:32 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:05:33.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:33 smithi037 ceph-mon[31460]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:33.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:33 smithi037 ceph-mon[31460]: from='client.? 172.21.15.170:0/1553965484' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "149b8297-ff2d-4a88-9b1e-0def021cc516"}]: dispatch 2024-03-28T18:05:33.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:33 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "149b8297-ff2d-4a88-9b1e-0def021cc516"}]: dispatch 2024-03-28T18:05:33.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:33 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "149b8297-ff2d-4a88-9b1e-0def021cc516"}]': finished 2024-03-28T18:05:33.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:33 smithi037 ceph-mon[31460]: osdmap e56: 9 total, 8 up, 9 in 2024-03-28T18:05:33.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:05:33.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:33 smithi037 ceph-mon[31460]: from='client.? 172.21.15.170:0/2487233128' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:05:33.952 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:33 smithi059 ceph-mon[36190]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:33.952 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:33 smithi059 ceph-mon[36190]: from='client.? 172.21.15.170:0/1553965484' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "149b8297-ff2d-4a88-9b1e-0def021cc516"}]: dispatch 2024-03-28T18:05:33.953 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:33 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "149b8297-ff2d-4a88-9b1e-0def021cc516"}]: dispatch 2024-03-28T18:05:33.953 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:33 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "149b8297-ff2d-4a88-9b1e-0def021cc516"}]': finished 2024-03-28T18:05:33.953 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:33 smithi059 ceph-mon[36190]: osdmap e56: 9 total, 8 up, 9 in 2024-03-28T18:05:33.953 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:33 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:05:33.953 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:33 smithi059 ceph-mon[36190]: from='client.? 172.21.15.170:0/2487233128' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:05:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:33 smithi170 ceph-mon[36280]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:33 smithi170 ceph-mon[36280]: from='client.? 172.21.15.170:0/1553965484' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "149b8297-ff2d-4a88-9b1e-0def021cc516"}]: dispatch 2024-03-28T18:05:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:33 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "149b8297-ff2d-4a88-9b1e-0def021cc516"}]: dispatch 2024-03-28T18:05:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:33 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "149b8297-ff2d-4a88-9b1e-0def021cc516"}]': finished 2024-03-28T18:05:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:33 smithi170 ceph-mon[36280]: osdmap e56: 9 total, 8 up, 9 in 2024-03-28T18:05:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:33 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:05:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:33 smithi170 ceph-mon[36280]: from='client.? 172.21.15.170:0/2487233128' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:05:35.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:35 smithi037 ceph-mon[31460]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:36.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:35 smithi059 ceph-mon[36190]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:36.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:35 smithi170 ceph-mon[36280]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:37.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:37 smithi037 ceph-mon[31460]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:37.892 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:37 smithi170 ceph-mon[36280]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:38.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:37 smithi059 ceph-mon[36190]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:39.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:39 smithi037 ceph-mon[31460]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:39.921 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:39 smithi170 ceph-mon[36280]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:40.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:39 smithi059 ceph-mon[36190]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:40.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:40 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T18:05:40.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:40 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:40.907 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:40 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T18:05:40.908 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:40 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:41.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:40 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T18:05:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:40 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:41 smithi037 ceph-mon[31460]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:41 smithi037 ceph-mon[31460]: Deploying daemon osd.8 on smithi170 2024-03-28T18:05:41.896 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:41 smithi170 ceph-mon[36280]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:41.896 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:41 smithi170 ceph-mon[36280]: Deploying daemon osd.8 on smithi170 2024-03-28T18:05:42.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:41 smithi059 ceph-mon[36190]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:42.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:41 smithi059 ceph-mon[36190]: Deploying daemon osd.8 on smithi170 2024-03-28T18:05:43.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:43 smithi059 ceph-mon[36190]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:43.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:43 smithi037 ceph-mon[31460]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:44.023 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:43 smithi170 ceph-mon[36280]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:45.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:45 smithi037 ceph-mon[31460]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:45.916 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:45 smithi170 ceph-mon[36280]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:46.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:45 smithi059 ceph-mon[36190]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:47.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:46 smithi059 ceph-mon[36190]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:46 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:46 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:46 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:05:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:46 smithi170 ceph-mon[36280]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:46 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:46 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:46 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:05:47.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:46 smithi037 ceph-mon[31460]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:47.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:46 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:47.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:46 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:47.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:46 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:05:49.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:48 smithi170 ceph-mon[36280]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:49.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:48 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:49.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:48 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:49.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:48 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:49.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:48 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:49.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:48 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:48 smithi059 ceph-mon[36190]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:48 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:48 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:48 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:48 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:48 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:49.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:48 smithi037 ceph-mon[31460]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:49.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:48 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:49.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:48 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:49.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:48 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:49.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:48 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:49.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:48 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:49.811 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:49.811+0000 7fc82dffb700 1 -- 172.21.15.170:0/1066420272 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc8300fdf10 con 0x7fc8200a2890 2024-03-28T18:05:49.811 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 8 on host 'smithi170' 2024-03-28T18:05:49.813 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:49.814+0000 7fc8371f0700 1 -- 172.21.15.170:0/1066420272 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc8200a2890 msgr2=0x7fc8200a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:49.814 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:49.814+0000 7fc8371f0700 1 --2- 172.21.15.170:0/1066420272 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc8200a2890 0x7fc8200a4d50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc81c00a990 tx=0x7fc81c005c80 comp rx=0 tx=0).stop 2024-03-28T18:05:49.814 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:49.814+0000 7fc8371f0700 1 -- 172.21.15.170:0/1066420272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc830101390 msgr2=0x7fc830198400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:49.814 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:49.814+0000 7fc8371f0700 1 --2- 172.21.15.170:0/1066420272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc830101390 0x7fc830198400 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc82400b530 tx=0x7fc82400b840 comp rx=0 tx=0).stop 2024-03-28T18:05:49.814 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:49.814+0000 7fc8371f0700 1 -- 172.21.15.170:0/1066420272 shutdown_connections 2024-03-28T18:05:49.814 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:49.814+0000 7fc8371f0700 1 --2- 172.21.15.170:0/1066420272 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc8200a2890 0x7fc8200a4d50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:49.814 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:49.814+0000 7fc8371f0700 1 --2- 172.21.15.170:0/1066420272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc830102650 0x7fc83019f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:49.814 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:49.814+0000 7fc8371f0700 1 --2- 172.21.15.170:0/1066420272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc830101390 0x7fc830198400 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:49.814 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:49.814+0000 7fc8371f0700 1 --2- 172.21.15.170:0/1066420272 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc830101d30 0x7fc830198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:49.814 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:49.814+0000 7fc8371f0700 1 -- 172.21.15.170:0/1066420272 >> 172.21.15.170:0/1066420272 conn(0x7fc8300fbe00 msgr2=0x7fc8300fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:05:49.814 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:49.814+0000 7fc8371f0700 1 -- 172.21.15.170:0/1066420272 shutdown_connections 2024-03-28T18:05:49.814 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:05:49.814+0000 7fc8371f0700 1 -- 172.21.15.170:0/1066420272 wait complete. 2024-03-28T18:05:50.395 DEBUG:teuthology.orchestra.run.smithi170:osd.8> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@osd.8.service 2024-03-28T18:05:50.397 INFO:tasks.cephadm:Deploying osd.9 on smithi037 with /dev/vg_nvme/lv_1... 2024-03-28T18:05:50.397 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T18:05:50.862 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:05:51.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:50 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:51.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:50 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:51.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:50 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:05:51.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:50 smithi059 ceph-mon[36190]: from='osd.8 [v2:172.21.15.170:6816/3235794556,v1:172.21.15.170:6817/3235794556]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T18:05:51.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:50 smithi059 ceph-mon[36190]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T18:05:51.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:50 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:51.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:50 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:51.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:50 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:05:51.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:50 smithi170 ceph-mon[36280]: from='osd.8 [v2:172.21.15.170:6816/3235794556,v1:172.21.15.170:6817/3235794556]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T18:05:51.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:50 smithi170 ceph-mon[36280]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T18:05:51.036 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:50 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:51.036 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:50 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:51.036 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:50 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:05:51.036 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:50 smithi037 ceph-mon[31460]: from='osd.8 [v2:172.21.15.170:6816/3235794556,v1:172.21.15.170:6817/3235794556]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T18:05:51.036 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:50 smithi037 ceph-mon[31460]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T18:05:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:51 smithi170 ceph-mon[36280]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:51 smithi170 ceph-mon[36280]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T18:05:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:51 smithi170 ceph-mon[36280]: from='osd.8 [v2:172.21.15.170:6816/3235794556,v1:172.21.15.170:6817/3235794556]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:05:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:51 smithi170 ceph-mon[36280]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T18:05:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:51 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:05:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:51 smithi170 ceph-mon[36280]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:05:52.036 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 28 18:05:51 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-8[54703]: 2024-03-28T18:05:51.792+0000 7fe6f4440700 -1 osd.8 0 waiting for initial osdmap 2024-03-28T18:05:52.036 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 28 18:05:51 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-8[54703]: 2024-03-28T18:05:51.803+0000 7fe6f0233700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:05:52.048 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:51 smithi037 ceph-mon[31460]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:52.048 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:51 smithi037 ceph-mon[31460]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T18:05:52.048 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:51 smithi037 ceph-mon[31460]: from='osd.8 [v2:172.21.15.170:6816/3235794556,v1:172.21.15.170:6817/3235794556]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:05:52.048 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:51 smithi037 ceph-mon[31460]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T18:05:52.048 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:51 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:05:52.048 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:51 smithi037 ceph-mon[31460]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:05:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:51 smithi059 ceph-mon[36190]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:51 smithi059 ceph-mon[36190]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T18:05:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:51 smithi059 ceph-mon[36190]: from='osd.8 [v2:172.21.15.170:6816/3235794556,v1:172.21.15.170:6817/3235794556]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:05:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:51 smithi059 ceph-mon[36190]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T18:05:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:51 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:05:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:51 smithi059 ceph-mon[36190]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:05:52.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:52 smithi037 ceph-mon[31460]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-28T18:05:52.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:52 smithi037 ceph-mon[31460]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T18:05:52.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:52 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:05:52.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:52 smithi037 ceph-mon[31460]: osd.8 [v2:172.21.15.170:6816/3235794556,v1:172.21.15.170:6817/3235794556] boot 2024-03-28T18:05:52.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:52 smithi037 ceph-mon[31460]: osdmap e59: 9 total, 9 up, 9 in 2024-03-28T18:05:52.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:52 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:05:53.154 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:52 smithi170 ceph-mon[36280]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-28T18:05:53.154 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:52 smithi170 ceph-mon[36280]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T18:05:53.154 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:52 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:05:53.154 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:52 smithi170 ceph-mon[36280]: osd.8 [v2:172.21.15.170:6816/3235794556,v1:172.21.15.170:6817/3235794556] boot 2024-03-28T18:05:53.155 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:52 smithi170 ceph-mon[36280]: osdmap e59: 9 total, 9 up, 9 in 2024-03-28T18:05:53.155 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:52 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:05:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:52 smithi059 ceph-mon[36190]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-28T18:05:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:52 smithi059 ceph-mon[36190]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T18:05:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:52 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:05:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:52 smithi059 ceph-mon[36190]: osd.8 [v2:172.21.15.170:6816/3235794556,v1:172.21.15.170:6817/3235794556] boot 2024-03-28T18:05:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:52 smithi059 ceph-mon[36190]: osdmap e59: 9 total, 9 up, 9 in 2024-03-28T18:05:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:52 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:05:53.874 INFO:teuthology.orchestra.run.smithi037.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T18:05:53.874 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T18:05:53.874 INFO:teuthology.orchestra.run.smithi037.stderr: stderr: 10+0 records in 2024-03-28T18:05:53.874 INFO:teuthology.orchestra.run.smithi037.stderr:10+0 records out 2024-03-28T18:05:53.874 INFO:teuthology.orchestra.run.smithi037.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0350681 s, 299 MB/s 2024-03-28T18:05:53.874 INFO:teuthology.orchestra.run.smithi037.stderr: stderr: 2024-03-28T18:05:53.874 INFO:teuthology.orchestra.run.smithi037.stderr:--> Zapping successful for: 2024-03-28T18:05:53.911 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:53 smithi037 ceph-mon[31460]: purged_snaps scrub starts 2024-03-28T18:05:53.912 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:53 smithi037 ceph-mon[31460]: purged_snaps scrub ok 2024-03-28T18:05:53.912 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:53 smithi037 ceph-mon[31460]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:53.912 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:53.912 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:53.912 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:53.912 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:53.912 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:53.912 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:53.912 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:53.912 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:53.912 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:53.912 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:53 smithi037 ceph-mon[31460]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T18:05:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:53 smithi059 ceph-mon[36190]: purged_snaps scrub starts 2024-03-28T18:05:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:53 smithi059 ceph-mon[36190]: purged_snaps scrub ok 2024-03-28T18:05:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:53 smithi059 ceph-mon[36190]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:54.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:54.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:54.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:54.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:54.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:53 smithi059 ceph-mon[36190]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T18:05:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:53 smithi170 ceph-mon[36280]: purged_snaps scrub starts 2024-03-28T18:05:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:53 smithi170 ceph-mon[36280]: purged_snaps scrub ok 2024-03-28T18:05:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:53 smithi170 ceph-mon[36280]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:05:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:54.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:54.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:54.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:54.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:05:54.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:54.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:54.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:54.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:05:54.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:53 smithi170 ceph-mon[36280]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T18:05:54.866 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi037:vg_nvme/lv_1 2024-03-28T18:05:55.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:54 smithi037 ceph-mon[31460]: Detected new or changed devices on smithi170 2024-03-28T18:05:55.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:54 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi170 to 5693M 2024-03-28T18:05:55.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:54 smithi037 ceph-mon[31460]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T18:05:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:54 smithi059 ceph-mon[36190]: Detected new or changed devices on smithi170 2024-03-28T18:05:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:54 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi170 to 5693M 2024-03-28T18:05:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:54 smithi059 ceph-mon[36190]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T18:05:55.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:54 smithi170 ceph-mon[36280]: Detected new or changed devices on smithi170 2024-03-28T18:05:55.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:54 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi170 to 5693M 2024-03-28T18:05:55.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:54 smithi170 ceph-mon[36280]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T18:05:55.286 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:05:56.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:55 smithi037 ceph-mon[31460]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:05:56.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:55 smithi037 ceph-mon[31460]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T18:05:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:55 smithi059 ceph-mon[36190]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:05:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:55 smithi059 ceph-mon[36190]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T18:05:56.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:55 smithi170 ceph-mon[36280]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:05:56.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:55 smithi170 ceph-mon[36280]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T18:05:57.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.655+0000 7fbbb1400700 1 -- 172.21.15.37:0/1515186250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbac10a2a0 msgr2=0x7fbbac110b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:57.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.655+0000 7fbbb1400700 1 --2- 172.21.15.37:0/1515186250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbac10a2a0 0x7fbbac110b50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbba400b0d0 tx=0x7fbba400b3e0 comp rx=0 tx=0).stop 2024-03-28T18:05:57.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.655+0000 7fbbb1400700 1 -- 172.21.15.37:0/1515186250 shutdown_connections 2024-03-28T18:05:57.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.655+0000 7fbbb1400700 1 --2- 172.21.15.37:0/1515186250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbac10a2a0 0x7fbbac110b50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:57.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.655+0000 7fbbb1400700 1 --2- 172.21.15.37:0/1515186250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbbac108700 0x7fbbac108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:57.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.655+0000 7fbbb1400700 1 --2- 172.21.15.37:0/1515186250 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbbac109900 0x7fbbac109d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:57.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.655+0000 7fbbb1400700 1 -- 172.21.15.37:0/1515186250 >> 172.21.15.37:0/1515186250 conn(0x7fbbac0776d0 msgr2=0x7fbbac079b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:05:57.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.655+0000 7fbbb1400700 1 -- 172.21.15.37:0/1515186250 shutdown_connections 2024-03-28T18:05:57.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.655+0000 7fbbb1400700 1 -- 172.21.15.37:0/1515186250 wait complete. 2024-03-28T18:05:57.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.656+0000 7fbbb1400700 1 Processor -- start 2024-03-28T18:05:57.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.656+0000 7fbbb1400700 1 -- start start 2024-03-28T18:05:57.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.657+0000 7fbbb1400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbac108700 0x7fbbac19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:57.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.657+0000 7fbbb1400700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbbac109900 0x7fbbac19ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:57.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.657+0000 7fbbb1400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbbac10a2a0 0x7fbbac1a5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:57.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.657+0000 7fbbb1400700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac114320 con 0x7fbbac10a2a0 2024-03-28T18:05:57.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.657+0000 7fbbb1400700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac1141a0 con 0x7fbbac108700 2024-03-28T18:05:57.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.657+0000 7fbbb1400700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac1144a0 con 0x7fbbac109900 2024-03-28T18:05:57.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.657+0000 7fbbabe00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbac108700 0x7fbbac19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:57.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.657+0000 7fbbabe00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbac108700 0x7fbbac19e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:37776/0 (socket says 172.21.15.37:37776) 2024-03-28T18:05:57.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.657+0000 7fbbabe00700 1 -- 172.21.15.37:0/2850777905 learned_addr learned my addr 172.21.15.37:0/2850777905 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:05:57.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.657+0000 7fbbab400700 1 --2- 172.21.15.37:0/2850777905 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbbac109900 0x7fbbac19ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:57.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.657+0000 7fbbb0a00700 1 --2- 172.21.15.37:0/2850777905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbbac10a2a0 0x7fbbac1a5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:57.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.658+0000 7fbbabe00700 1 -- 172.21.15.37:0/2850777905 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbbac109900 msgr2=0x7fbbac19ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:57.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.658+0000 7fbbabe00700 1 --2- 172.21.15.37:0/2850777905 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbbac109900 0x7fbbac19ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:57.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.658+0000 7fbbabe00700 1 -- 172.21.15.37:0/2850777905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbbac10a2a0 msgr2=0x7fbbac1a5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:57.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.658+0000 7fbbabe00700 1 --2- 172.21.15.37:0/2850777905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbbac10a2a0 0x7fbbac1a5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:57.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.658+0000 7fbbabe00700 1 -- 172.21.15.37:0/2850777905 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbac1a6550 con 0x7fbbac108700 2024-03-28T18:05:57.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.658+0000 7fbbabe00700 1 --2- 172.21.15.37:0/2850777905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbac108700 0x7fbbac19e940 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c00ea50 tx=0x7fbb9c00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:05:57.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.658+0000 7fbba8c00700 1 -- 172.21.15.37:0/2850777905 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c00cbd0 con 0x7fbbac108700 2024-03-28T18:05:57.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.658+0000 7fbba8c00700 1 -- 172.21.15.37:0/2850777905 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb9c00cd70 con 0x7fbbac108700 2024-03-28T18:05:57.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.658+0000 7fbba8c00700 1 -- 172.21.15.37:0/2850777905 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c0187a0 con 0x7fbbac108700 2024-03-28T18:05:57.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.658+0000 7fbbb1400700 1 -- 172.21.15.37:0/2850777905 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbac1a6840 con 0x7fbbac108700 2024-03-28T18:05:57.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.659+0000 7fbbb1400700 1 -- 172.21.15.37:0/2850777905 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbac1a6d80 con 0x7fbbac108700 2024-03-28T18:05:57.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.660+0000 7fbbb1400700 1 -- 172.21.15.37:0/2850777905 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbac068e50 con 0x7fbbac108700 2024-03-28T18:05:57.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.662+0000 7fbba8c00700 1 -- 172.21.15.37:0/2850777905 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbb9c01f020 con 0x7fbbac108700 2024-03-28T18:05:57.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.663+0000 7fbba8c00700 1 --2- 172.21.15.37:0/2850777905 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fbb900a2890 0x7fbb900a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:57.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.663+0000 7fbba8c00700 1 -- 172.21.15.37:0/2850777905 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5829+0+0 (secure 0 0 0) 0x7fbb9c0ce330 con 0x7fbbac108700 2024-03-28T18:05:57.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.663+0000 7fbbab400700 1 --2- 172.21.15.37:0/2850777905 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fbb900a2890 0x7fbb900a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:57.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.664+0000 7fbbab400700 1 --2- 172.21.15.37:0/2850777905 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fbb900a2890 0x7fbb900a4d50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbb98009680 tx=0x7fbb98009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:05:57.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.666+0000 7fbba8c00700 1 -- 172.21.15.37:0/2850777905 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbb9c09bce0 con 0x7fbbac108700 2024-03-28T18:05:57.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:57 smithi037 ceph-mon[31460]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:05:57.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:57 smithi037 ceph-mon[31460]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T18:05:57.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:05:57.910+0000 7fbbb1400700 1 -- 172.21.15.37:0/2850777905 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbac0625d0 con 0x7fbb900a2890 2024-03-28T18:05:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:57 smithi059 ceph-mon[36190]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:05:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:57 smithi059 ceph-mon[36190]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T18:05:58.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:57 smithi170 ceph-mon[36280]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:05:58.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:57 smithi170 ceph-mon[36280]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T18:05:58.998 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:58 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:05:58.998 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:58 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:05:58.998 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:58 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:59.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:58 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:05:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:58 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:05:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:58 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:59.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:58 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:05:59.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:58 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:05:59.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:58 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:59 smithi059 ceph-mon[36190]: pgmap v225: 1 pgs: 1 peering; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-28T18:06:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:05:59 smithi059 ceph-mon[36190]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:06:00.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:59 smithi170 ceph-mon[36280]: pgmap v225: 1 pgs: 1 peering; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-28T18:06:00.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:05:59 smithi170 ceph-mon[36280]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:06:00.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:59 smithi037 ceph-mon[31460]: pgmap v225: 1 pgs: 1 peering; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-28T18:06:00.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:05:59 smithi037 ceph-mon[31460]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:06:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:01 smithi059 ceph-mon[36190]: pgmap v226: 1 pgs: 1 peering; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:06:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:01 smithi170 ceph-mon[36280]: pgmap v226: 1 pgs: 1 peering; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:06:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:01 smithi037 ceph-mon[31460]: pgmap v226: 1 pgs: 1 peering; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:06:03.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:02 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/92192294' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55c27cc2-c771-44b7-90fe-8f0feca44162"}]: dispatch 2024-03-28T18:06:03.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:02 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55c27cc2-c771-44b7-90fe-8f0feca44162"}]: dispatch 2024-03-28T18:06:03.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:02 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55c27cc2-c771-44b7-90fe-8f0feca44162"}]': finished 2024-03-28T18:06:03.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:02 smithi059 ceph-mon[36190]: osdmap e64: 10 total, 9 up, 10 in 2024-03-28T18:06:03.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:02 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:06:03.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:02 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/92192294' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55c27cc2-c771-44b7-90fe-8f0feca44162"}]: dispatch 2024-03-28T18:06:03.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:02 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55c27cc2-c771-44b7-90fe-8f0feca44162"}]: dispatch 2024-03-28T18:06:03.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:02 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55c27cc2-c771-44b7-90fe-8f0feca44162"}]': finished 2024-03-28T18:06:03.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:02 smithi170 ceph-mon[36280]: osdmap e64: 10 total, 9 up, 10 in 2024-03-28T18:06:03.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:02 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:06:03.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:02 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/92192294' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55c27cc2-c771-44b7-90fe-8f0feca44162"}]: dispatch 2024-03-28T18:06:03.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:02 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55c27cc2-c771-44b7-90fe-8f0feca44162"}]: dispatch 2024-03-28T18:06:03.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:02 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55c27cc2-c771-44b7-90fe-8f0feca44162"}]': finished 2024-03-28T18:06:03.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:02 smithi037 ceph-mon[31460]: osdmap e64: 10 total, 9 up, 10 in 2024-03-28T18:06:03.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:02 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:06:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:03 smithi059 ceph-mon[36190]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-28T18:06:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:03 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2823609238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:06:04.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:03 smithi170 ceph-mon[36280]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-28T18:06:04.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:03 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2823609238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:06:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:03 smithi037 ceph-mon[31460]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-28T18:06:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:03 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2823609238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:06:06.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:05 smithi059 ceph-mon[36190]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:06:06.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:05 smithi170 ceph-mon[36280]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:06:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:05 smithi037 ceph-mon[31460]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:06:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:07 smithi059 ceph-mon[36190]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:08.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:07 smithi170 ceph-mon[36280]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:08.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:07 smithi037 ceph-mon[31460]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:10.033 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:09 smithi037 ceph-mon[31460]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:10.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:09 smithi059 ceph-mon[36190]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:10.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:09 smithi170 ceph-mon[36280]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:11 smithi059 ceph-mon[36190]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:11 smithi170 ceph-mon[36280]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:11 smithi037 ceph-mon[31460]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:13.011 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:12 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T18:06:13.011 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:12 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:13.011 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:12 smithi037 ceph-mon[31460]: Deploying daemon osd.9 on smithi037 2024-03-28T18:06:13.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:12 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T18:06:13.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:12 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:13.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:12 smithi059 ceph-mon[36190]: Deploying daemon osd.9 on smithi037 2024-03-28T18:06:13.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:12 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T18:06:13.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:12 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:13.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:12 smithi170 ceph-mon[36280]: Deploying daemon osd.9 on smithi037 2024-03-28T18:06:14.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:13 smithi059 ceph-mon[36190]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:14.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:13 smithi170 ceph-mon[36280]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:14.074 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:13 smithi037 ceph-mon[31460]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:15.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:15 smithi037 ceph-mon[31460]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:15 smithi059 ceph-mon[36190]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:16.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:15 smithi170 ceph-mon[36280]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:18.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:17 smithi059 ceph-mon[36190]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:18.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:17 smithi170 ceph-mon[36280]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:18.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:17 smithi037 ceph-mon[31460]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:20.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:19 smithi037 ceph-mon[31460]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:19 smithi059 ceph-mon[36190]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:20.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:19 smithi170 ceph-mon[36280]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:21.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:21 smithi037 ceph-mon[31460]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:21 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:21 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:21 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:06:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:21 smithi059 ceph-mon[36190]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:21 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:21 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:21 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:06:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:21 smithi170 ceph-mon[36280]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:21 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:22.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:21 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:22.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:21 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:06:23.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:23 smithi037 ceph-mon[31460]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:23 smithi059 ceph-mon[36190]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:23 smithi170 ceph-mon[36280]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:25.742 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:25 smithi037 ceph-mon[31460]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:25.742 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:25 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:25.742 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:25 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:25.742 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:25 smithi037 ceph-mon[31460]: from='osd.9 [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T18:06:25.742 INFO:teuthology.orchestra.run.smithi037.stdout:Created osd(s) 9 on host 'smithi037' 2024-03-28T18:06:25.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:06:25.739+0000 7fbba8c00700 1 -- 172.21.15.37:0/2850777905 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbbac0625d0 con 0x7fbb900a2890 2024-03-28T18:06:25.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:06:25.743+0000 7fbbb1400700 1 -- 172.21.15.37:0/2850777905 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fbb900a2890 msgr2=0x7fbb900a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:06:25.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:06:25.743+0000 7fbbb1400700 1 --2- 172.21.15.37:0/2850777905 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fbb900a2890 0x7fbb900a4d50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbb98009680 tx=0x7fbb98009260 comp rx=0 tx=0).stop 2024-03-28T18:06:25.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:06:25.744+0000 7fbbb1400700 1 -- 172.21.15.37:0/2850777905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbac108700 msgr2=0x7fbbac19e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:06:25.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:06:25.744+0000 7fbbb1400700 1 --2- 172.21.15.37:0/2850777905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbac108700 0x7fbbac19e940 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c00ea50 tx=0x7fbb9c00ed60 comp rx=0 tx=0).stop 2024-03-28T18:06:25.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:06:25.744+0000 7fbbb1400700 1 -- 172.21.15.37:0/2850777905 shutdown_connections 2024-03-28T18:06:25.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:06:25.744+0000 7fbbb1400700 1 --2- 172.21.15.37:0/2850777905 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fbb900a2890 0x7fbb900a4d50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:06:25.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:06:25.744+0000 7fbbb1400700 1 --2- 172.21.15.37:0/2850777905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbbac10a2a0 0x7fbbac1a5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:06:25.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:06:25.744+0000 7fbbb1400700 1 --2- 172.21.15.37:0/2850777905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbac108700 0x7fbbac19e940 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:06:25.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:06:25.744+0000 7fbbb1400700 1 --2- 172.21.15.37:0/2850777905 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbbac109900 0x7fbbac19ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:06:25.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:06:25.744+0000 7fbbb1400700 1 -- 172.21.15.37:0/2850777905 >> 172.21.15.37:0/2850777905 conn(0x7fbbac0776d0 msgr2=0x7fbbac10cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:06:25.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:06:25.744+0000 7fbbb1400700 1 -- 172.21.15.37:0/2850777905 shutdown_connections 2024-03-28T18:06:25.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:06:25.744+0000 7fbbb1400700 1 -- 172.21.15.37:0/2850777905 wait complete. 2024-03-28T18:06:25.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:25 smithi059 ceph-mon[36190]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:25.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:25 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:25.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:25 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:25.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:25 smithi059 ceph-mon[36190]: from='osd.9 [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T18:06:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:25 smithi170 ceph-mon[36280]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:25 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:25 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:25 smithi170 ceph-mon[36280]: from='osd.9 [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T18:06:26.512 DEBUG:teuthology.orchestra.run.smithi037:osd.9> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@osd.9.service 2024-03-28T18:06:26.518 INFO:tasks.cephadm:Deploying osd.10 on smithi059 with /dev/vg_nvme/lv_1... 2024-03-28T18:06:26.518 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T18:06:26.766 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:26 smithi037 ceph-mon[31460]: from='osd.9 [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T18:06:26.766 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:26 smithi037 ceph-mon[31460]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T18:06:26.766 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:26 smithi037 ceph-mon[31460]: from='osd.9 [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]: dispatch 2024-03-28T18:06:26.766 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:26 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:06:26.766 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:26 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:26.766 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:26 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:26.767 INFO:journalctl@ceph.osd.9.smithi037.stdout:Mar 28 18:06:26 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-9[64290]: 2024-03-28T18:06:26.654+0000 7f70f4600700 -1 osd.9 0 waiting for initial osdmap 2024-03-28T18:06:26.767 INFO:journalctl@ceph.osd.9.smithi037.stdout:Mar 28 18:06:26 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-9[64290]: 2024-03-28T18:06:26.672+0000 7f70ef400700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:06:26.779 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:06:26.894 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:26 smithi059 ceph-mon[36190]: from='osd.9 [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T18:06:26.894 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:26 smithi059 ceph-mon[36190]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T18:06:26.894 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:26 smithi059 ceph-mon[36190]: from='osd.9 [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]: dispatch 2024-03-28T18:06:26.894 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:26 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:06:26.894 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:26 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:26.894 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:26 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:27.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:26 smithi170 ceph-mon[36280]: from='osd.9 [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T18:06:27.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:26 smithi170 ceph-mon[36280]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T18:06:27.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:26 smithi170 ceph-mon[36280]: from='osd.9 [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]: dispatch 2024-03-28T18:06:27.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:26 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:06:27.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:26 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:27.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:26 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:27 smithi059 ceph-mon[36190]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:27 smithi059 ceph-mon[36190]: from='osd.9 [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]': finished 2024-03-28T18:06:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:27 smithi059 ceph-mon[36190]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T18:06:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:27 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:06:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:27 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:06:28.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:27 smithi170 ceph-mon[36280]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:28.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:27 smithi170 ceph-mon[36280]: from='osd.9 [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]': finished 2024-03-28T18:06:28.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:27 smithi170 ceph-mon[36280]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T18:06:28.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:27 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:06:28.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:27 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:06:28.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:27 smithi037 ceph-mon[31460]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:28.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:27 smithi037 ceph-mon[31460]: from='osd.9 [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]': finished 2024-03-28T18:06:28.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:27 smithi037 ceph-mon[31460]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T18:06:28.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:27 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:06:28.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:27 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:06:28.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:28 smithi059 ceph-mon[36190]: purged_snaps scrub starts 2024-03-28T18:06:28.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:28 smithi059 ceph-mon[36190]: purged_snaps scrub ok 2024-03-28T18:06:28.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:28 smithi059 ceph-mon[36190]: osd.9 [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008] boot 2024-03-28T18:06:28.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:28 smithi059 ceph-mon[36190]: osdmap e67: 10 total, 10 up, 10 in 2024-03-28T18:06:28.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:28 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:06:28.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:28 smithi037 ceph-mon[31460]: purged_snaps scrub starts 2024-03-28T18:06:28.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:28 smithi037 ceph-mon[31460]: purged_snaps scrub ok 2024-03-28T18:06:28.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:28 smithi037 ceph-mon[31460]: osd.9 [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008] boot 2024-03-28T18:06:28.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:28 smithi037 ceph-mon[31460]: osdmap e67: 10 total, 10 up, 10 in 2024-03-28T18:06:28.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:28 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:06:28.936 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:28 smithi170 ceph-mon[36280]: purged_snaps scrub starts 2024-03-28T18:06:28.936 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:28 smithi170 ceph-mon[36280]: purged_snaps scrub ok 2024-03-28T18:06:28.936 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:28 smithi170 ceph-mon[36280]: osd.9 [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008] boot 2024-03-28T18:06:28.936 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:28 smithi170 ceph-mon[36280]: osdmap e67: 10 total, 10 up, 10 in 2024-03-28T18:06:28.936 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:28 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:06:29.003 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T18:06:29.003 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T18:06:29.003 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10+0 records in 2024-03-28T18:06:29.003 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2024-03-28T18:06:29.003 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111227 s, 943 MB/s 2024-03-28T18:06:29.003 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping successful for: 2024-03-28T18:06:29.696 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi059:vg_nvme/lv_1 2024-03-28T18:06:29.947 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:06:30.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:29 smithi059 ceph-mon[36190]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:29 smithi059 ceph-mon[36190]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T18:06:30.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:29 smithi170 ceph-mon[36280]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:30.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:29 smithi170 ceph-mon[36280]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T18:06:30.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:29 smithi037 ceph-mon[31460]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:06:30.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:29 smithi037 ceph-mon[31460]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T18:06:31.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.682+0000 7f02040fa700 1 -- 172.21.15.59:0/4040597576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01fc1021c0 msgr2=0x7f01fc1025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:06:31.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.682+0000 7f02040fa700 1 --2- 172.21.15.59:0/4040597576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01fc1021c0 0x7f01fc1025e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f01ec00b0d0 tx=0x7f01ec00b3e0 comp rx=0 tx=0).stop 2024-03-28T18:06:31.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.682+0000 7f02040fa700 1 -- 172.21.15.59:0/4040597576 shutdown_connections 2024-03-28T18:06:31.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.682+0000 7f02040fa700 1 --2- 172.21.15.59:0/4040597576 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f01fc103d60 0x7f01fc10a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:06:31.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.682+0000 7f02040fa700 1 --2- 172.21.15.59:0/4040597576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01fc1021c0 0x7f01fc1025e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:06:31.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.682+0000 7f02040fa700 1 --2- 172.21.15.59:0/4040597576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01fc1033c0 0x7f01fc103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:06:31.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.682+0000 7f02040fa700 1 -- 172.21.15.59:0/4040597576 >> 172.21.15.59:0/4040597576 conn(0x7f01fc0fd7a0 msgr2=0x7f01fc0ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:06:31.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.682+0000 7f02040fa700 1 -- 172.21.15.59:0/4040597576 shutdown_connections 2024-03-28T18:06:31.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.682+0000 7f02040fa700 1 -- 172.21.15.59:0/4040597576 wait complete. 2024-03-28T18:06:31.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.683+0000 7f02040fa700 1 Processor -- start 2024-03-28T18:06:31.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.683+0000 7f02040fa700 1 -- start start 2024-03-28T18:06:31.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.683+0000 7f02040fa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01fc1021c0 0x7f01fc19a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:06:31.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.683+0000 7f02040fa700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f01fc1033c0 0x7f01fc19ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:06:31.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.683+0000 7f02040fa700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01fc103d60 0x7f01fc1a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:06:31.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.683+0000 7f02040fa700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01fc10ded0 con 0x7f01fc1033c0 2024-03-28T18:06:31.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.683+0000 7f02040fa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01fc10dd50 con 0x7f01fc1021c0 2024-03-28T18:06:31.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.683+0000 7f02040fa700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01fc10e050 con 0x7f01fc103d60 2024-03-28T18:06:31.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.683+0000 7f0201e96700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01fc1021c0 0x7f01fc19a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:06:31.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.683+0000 7f0201e96700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01fc1021c0 0x7f01fc19a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60412/0 (socket says 172.21.15.59:60412) 2024-03-28T18:06:31.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.683+0000 7f0201e96700 1 -- 172.21.15.59:0/395706299 learned_addr learned my addr 172.21.15.59:0/395706299 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-28T18:06:31.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.683+0000 7f0202697700 1 --2- 172.21.15.59:0/395706299 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01fc103d60 0x7f01fc1a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:06:31.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.683+0000 7f0201e96700 1 -- 172.21.15.59:0/395706299 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01fc103d60 msgr2=0x7f01fc1a1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:06:31.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.683+0000 7f0201e96700 1 --2- 172.21.15.59:0/395706299 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01fc103d60 0x7f01fc1a1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:06:31.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.683+0000 7f0201e96700 1 -- 172.21.15.59:0/395706299 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f01fc1033c0 msgr2=0x7f01fc19ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:06:31.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.683+0000 7f0201e96700 1 --2- 172.21.15.59:0/395706299 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f01fc1033c0 0x7f01fc19ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:06:31.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.683+0000 7f0201e96700 1 -- 172.21.15.59:0/395706299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01fc1a2410 con 0x7f01fc1021c0 2024-03-28T18:06:31.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.684+0000 7f0201e96700 1 --2- 172.21.15.59:0/395706299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01fc1021c0 0x7f01fc19a800 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f01ec0062a0 tx=0x7f01ec007e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:06:31.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.684+0000 7f01faffd700 1 -- 172.21.15.59:0/395706299 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ec003c00 con 0x7f01fc1021c0 2024-03-28T18:06:31.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.684+0000 7f02040fa700 1 -- 172.21.15.59:0/395706299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01fc1a26a0 con 0x7f01fc1021c0 2024-03-28T18:06:31.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.684+0000 7f01faffd700 1 -- 172.21.15.59:0/395706299 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01ec003da0 con 0x7f01fc1021c0 2024-03-28T18:06:31.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.684+0000 7f01faffd700 1 -- 172.21.15.59:0/395706299 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ec005560 con 0x7f01fc1021c0 2024-03-28T18:06:31.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.684+0000 7f02040fa700 1 -- 172.21.15.59:0/395706299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01fc1a2c00 con 0x7f01fc1021c0 2024-03-28T18:06:31.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.685+0000 7f02040fa700 1 -- 172.21.15.59:0/395706299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01fc068e50 con 0x7f01fc1021c0 2024-03-28T18:06:31.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.686+0000 7f01faffd700 1 -- 172.21.15.59:0/395706299 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f01ec005700 con 0x7f01fc1021c0 2024-03-28T18:06:31.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.687+0000 7f01faffd700 1 --2- 172.21.15.59:0/395706299 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f01e40a2890 0x7f01e40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:06:31.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.687+0000 7f01faffd700 1 -- 172.21.15.59:0/395706299 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f01ec00f020 con 0x7f01fc1021c0 2024-03-28T18:06:31.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.687+0000 7f0201695700 1 --2- 172.21.15.59:0/395706299 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f01e40a2890 0x7f01e40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:06:31.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.688+0000 7f0201695700 1 --2- 172.21.15.59:0/395706299 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f01e40a2890 0x7f01e40a4d50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f01f00097b0 tx=0x7f01f0006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:06:31.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.692+0000 7f01faffd700 1 -- 172.21.15.59:0/395706299 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f01ec02b020 con 0x7f01fc1021c0 2024-03-28T18:06:31.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:31.905+0000 7f02040fa700 1 -- 172.21.15.59:0/395706299 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f01fc075ae0 con 0x7f01e40a2890 2024-03-28T18:06:31.905 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:31 smithi059 ceph-mon[36190]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:32.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:31 smithi170 ceph-mon[36280]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:32.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:31 smithi037 ceph-mon[31460]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:32.859 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:32 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:06:32.859 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:32 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:32.859 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:32 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:06:32.859 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:32 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:32.859 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:32 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:32.859 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:32 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:32.859 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:32 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:32.859 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:32 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:32.859 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:32 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:32.859 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:32 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:32.859 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:32 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:32.860 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:32 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:06:32.860 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:32 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:32.860 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:32 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:06:32.860 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:32 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:32.860 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:32 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:06:32.860 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:32 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:33.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:32 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:06:33.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:32 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:33.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:32 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:06:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:32 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:32 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:32 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:32 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:32 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:32 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:32 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:32 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:32 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:06:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:32 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:32 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:06:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:32 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:32 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:06:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:32 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:33.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:32 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:06:33.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:32 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:33.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:32 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:06:33.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:32 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:33.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:32 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:33.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:32 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:33.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:32 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:33.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:32 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:33.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:32 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:33.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:32 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:33.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:32 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:33.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:32 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:06:33.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:32 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:33.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:32 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:06:33.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:32 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:33.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:32 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:06:33.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:32 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:33.883 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:33 smithi059 ceph-mon[36190]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:06:33.883 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:33 smithi059 ceph-mon[36190]: Detected new or changed devices on smithi037 2024-03-28T18:06:33.883 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:33 smithi059 ceph-mon[36190]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:33.883 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:33 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi037 to 4273M 2024-03-28T18:06:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:33 smithi170 ceph-mon[36280]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:06:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:33 smithi170 ceph-mon[36280]: Detected new or changed devices on smithi037 2024-03-28T18:06:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:33 smithi170 ceph-mon[36280]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:33 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi037 to 4273M 2024-03-28T18:06:34.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:33 smithi037 ceph-mon[31460]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:06:34.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:33 smithi037 ceph-mon[31460]: Detected new or changed devices on smithi037 2024-03-28T18:06:34.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:33 smithi037 ceph-mon[31460]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:34.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:33 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi037 to 4273M 2024-03-28T18:06:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:35 smithi059 ceph-mon[36190]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:35 smithi059 ceph-mon[36190]: from='client.? 172.21.15.59:0/3331438719' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0595135-ab81-49d3-8c23-19f960c03e18"}]: dispatch 2024-03-28T18:06:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:35 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0595135-ab81-49d3-8c23-19f960c03e18"}]: dispatch 2024-03-28T18:06:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:35 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0595135-ab81-49d3-8c23-19f960c03e18"}]': finished 2024-03-28T18:06:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:35 smithi059 ceph-mon[36190]: osdmap e69: 11 total, 10 up, 11 in 2024-03-28T18:06:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:35 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:06:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:35 smithi059 ceph-mon[36190]: from='client.? 172.21.15.59:0/712069878' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:06:36.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:35 smithi170 ceph-mon[36280]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:36.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:35 smithi170 ceph-mon[36280]: from='client.? 172.21.15.59:0/3331438719' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0595135-ab81-49d3-8c23-19f960c03e18"}]: dispatch 2024-03-28T18:06:36.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:35 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0595135-ab81-49d3-8c23-19f960c03e18"}]: dispatch 2024-03-28T18:06:36.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:35 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0595135-ab81-49d3-8c23-19f960c03e18"}]': finished 2024-03-28T18:06:36.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:35 smithi170 ceph-mon[36280]: osdmap e69: 11 total, 10 up, 11 in 2024-03-28T18:06:36.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:35 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:06:36.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:35 smithi170 ceph-mon[36280]: from='client.? 172.21.15.59:0/712069878' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:06:36.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:35 smithi037 ceph-mon[31460]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:36.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:35 smithi037 ceph-mon[31460]: from='client.? 172.21.15.59:0/3331438719' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0595135-ab81-49d3-8c23-19f960c03e18"}]: dispatch 2024-03-28T18:06:36.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:35 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0595135-ab81-49d3-8c23-19f960c03e18"}]: dispatch 2024-03-28T18:06:36.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:35 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0595135-ab81-49d3-8c23-19f960c03e18"}]': finished 2024-03-28T18:06:36.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:35 smithi037 ceph-mon[31460]: osdmap e69: 11 total, 10 up, 11 in 2024-03-28T18:06:36.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:35 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:06:36.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:35 smithi037 ceph-mon[31460]: from='client.? 172.21.15.59:0/712069878' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:06:37.952 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:37 smithi037 ceph-mon[31460]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:38.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:37 smithi059 ceph-mon[36190]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:38.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:37 smithi170 ceph-mon[36280]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:39.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:39 smithi037 ceph-mon[31460]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:40.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:39 smithi059 ceph-mon[36190]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:40.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:39 smithi170 ceph-mon[36280]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:41.999 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:41 smithi059 ceph-mon[36190]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:42.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:41 smithi170 ceph-mon[36280]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:42.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:41 smithi037 ceph-mon[31460]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:43.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:42 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T18:06:43.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:42 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:43.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:42 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T18:06:43.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:42 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:43.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:42 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T18:06:43.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:42 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:44.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:43 smithi059 ceph-mon[36190]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:44.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:43 smithi059 ceph-mon[36190]: Deploying daemon osd.10 on smithi059 2024-03-28T18:06:44.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:43 smithi170 ceph-mon[36280]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:44.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:43 smithi170 ceph-mon[36280]: Deploying daemon osd.10 on smithi059 2024-03-28T18:06:44.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:43 smithi037 ceph-mon[31460]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:44.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:43 smithi037 ceph-mon[31460]: Deploying daemon osd.10 on smithi059 2024-03-28T18:06:45.849 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:45 smithi059 ceph-mon[36190]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:46.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:45 smithi170 ceph-mon[36280]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:46.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:45 smithi037 ceph-mon[31460]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:48.017 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:47 smithi059 ceph-mon[36190]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:48.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:47 smithi170 ceph-mon[36280]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:48.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:47 smithi037 ceph-mon[31460]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:50.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:49 smithi059 ceph-mon[36190]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:50.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:49 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:50.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:49 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:50.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:49 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:06:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:49 smithi170 ceph-mon[36280]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:50.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:49 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:50.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:49 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:50.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:49 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:06:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:49 smithi037 ceph-mon[31460]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:49 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:49 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:50.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:49 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:06:51.939 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:51 smithi059 ceph-mon[36190]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:51 smithi170 ceph-mon[36280]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:51 smithi037 ceph-mon[31460]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:52.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:52.702+0000 7f01faffd700 1 -- 172.21.15.59:0/395706299 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f01fc075ae0 con 0x7f01e40a2890 2024-03-28T18:06:52.702 INFO:teuthology.orchestra.run.smithi059.stdout:Created osd(s) 10 on host 'smithi059' 2024-03-28T18:06:52.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:52.704+0000 7f02040fa700 1 -- 172.21.15.59:0/395706299 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f01e40a2890 msgr2=0x7f01e40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:06:52.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:52.704+0000 7f02040fa700 1 --2- 172.21.15.59:0/395706299 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f01e40a2890 0x7f01e40a4d50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f01f00097b0 tx=0x7f01f0006d20 comp rx=0 tx=0).stop 2024-03-28T18:06:52.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:52.704+0000 7f02040fa700 1 -- 172.21.15.59:0/395706299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01fc1021c0 msgr2=0x7f01fc19a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:06:52.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:52.704+0000 7f02040fa700 1 --2- 172.21.15.59:0/395706299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01fc1021c0 0x7f01fc19a800 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f01ec0062a0 tx=0x7f01ec007e40 comp rx=0 tx=0).stop 2024-03-28T18:06:52.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:52.704+0000 7f02040fa700 1 -- 172.21.15.59:0/395706299 shutdown_connections 2024-03-28T18:06:52.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:52.704+0000 7f02040fa700 1 --2- 172.21.15.59:0/395706299 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f01e40a2890 0x7f01e40a4d50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:06:52.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:52.704+0000 7f02040fa700 1 --2- 172.21.15.59:0/395706299 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01fc103d60 0x7f01fc1a1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:06:52.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:52.704+0000 7f02040fa700 1 --2- 172.21.15.59:0/395706299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01fc1021c0 0x7f01fc19a800 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:06:52.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:52.704+0000 7f02040fa700 1 --2- 172.21.15.59:0/395706299 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f01fc1033c0 0x7f01fc19ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:06:52.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:52.704+0000 7f02040fa700 1 -- 172.21.15.59:0/395706299 >> 172.21.15.59:0/395706299 conn(0x7f01fc0fd7a0 msgr2=0x7f01fc0743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:06:52.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:52.704+0000 7f02040fa700 1 -- 172.21.15.59:0/395706299 shutdown_connections 2024-03-28T18:06:52.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-28T18:06:52.704+0000 7f02040fa700 1 -- 172.21.15.59:0/395706299 wait complete. 2024-03-28T18:06:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:52 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:52 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:52 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:52 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:06:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:52 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:52 smithi059 ceph-mon[36190]: from='osd.10 [v2:172.21.15.59:6824/2113830275,v1:172.21.15.59:6825/2113830275]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T18:06:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:52 smithi059 ceph-mon[36190]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T18:06:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:52 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:06:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:52 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:52 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:53.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:52 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:53.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:52 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:53.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:52 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:53.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:52 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:06:53.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:52 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:53.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:52 smithi037 ceph-mon[31460]: from='osd.10 [v2:172.21.15.59:6824/2113830275,v1:172.21.15.59:6825/2113830275]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T18:06:53.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:52 smithi037 ceph-mon[31460]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T18:06:53.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:52 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:06:53.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:52 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:53.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:52 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:52 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:53.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:52 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:53.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:52 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:53.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:52 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:06:53.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:52 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:53.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:52 smithi170 ceph-mon[36280]: from='osd.10 [v2:172.21.15.59:6824/2113830275,v1:172.21.15.59:6825/2113830275]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T18:06:53.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:52 smithi170 ceph-mon[36280]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T18:06:53.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:52 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:06:53.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:52 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:53.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:52 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:53.407 DEBUG:teuthology.orchestra.run.smithi059:osd.10> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@osd.10.service 2024-03-28T18:06:53.410 INFO:tasks.cephadm:Deploying osd.11 on smithi170 with /dev/vg_nvme/lv_1... 2024-03-28T18:06:53.410 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T18:06:53.655 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:06:53.925 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:53 smithi059 ceph-mon[36190]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:53.925 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:53 smithi059 ceph-mon[36190]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T18:06:53.925 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:53 smithi059 ceph-mon[36190]: from='osd.10 [v2:172.21.15.59:6824/2113830275,v1:172.21.15.59:6825/2113830275]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:06:53.925 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:53 smithi059 ceph-mon[36190]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T18:06:53.925 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:06:53.925 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:53 smithi059 ceph-mon[36190]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:06:54.028 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:53 smithi170 ceph-mon[36280]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:54.028 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:53 smithi170 ceph-mon[36280]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T18:06:54.028 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:53 smithi170 ceph-mon[36280]: from='osd.10 [v2:172.21.15.59:6824/2113830275,v1:172.21.15.59:6825/2113830275]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:06:54.028 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:53 smithi170 ceph-mon[36280]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T18:06:54.028 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:06:54.028 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:53 smithi170 ceph-mon[36280]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:06:54.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:53 smithi037 ceph-mon[31460]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:53 smithi037 ceph-mon[31460]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T18:06:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:53 smithi037 ceph-mon[31460]: from='osd.10 [v2:172.21.15.59:6824/2113830275,v1:172.21.15.59:6825/2113830275]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:06:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:53 smithi037 ceph-mon[31460]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T18:06:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:06:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:53 smithi037 ceph-mon[31460]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-28T18:06:54.622 INFO:journalctl@ceph.osd.10.smithi059.stdout:Mar 28 18:06:54 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-10[61544]: 2024-03-28T18:06:54.302+0000 7f1f976e9700 -1 osd.10 0 waiting for initial osdmap 2024-03-28T18:06:54.622 INFO:journalctl@ceph.osd.10.smithi059.stdout:Mar 28 18:06:54 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-10[61544]: 2024-03-28T18:06:54.315+0000 7f1f8f4d4700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:06:54.961 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:54 smithi059 ceph-mon[36190]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-03-28T18:06:54.961 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:54 smithi059 ceph-mon[36190]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T18:06:54.961 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:54 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:06:54.961 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:54 smithi059 ceph-mon[36190]: from='osd.10 ' entity='osd.10' 2024-03-28T18:06:55.108 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:54 smithi170 ceph-mon[36280]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-03-28T18:06:55.109 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:54 smithi170 ceph-mon[36280]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T18:06:55.109 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:54 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:06:55.109 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:54 smithi170 ceph-mon[36280]: from='osd.10 ' entity='osd.10' 2024-03-28T18:06:55.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:54 smithi037 ceph-mon[31460]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-03-28T18:06:55.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:54 smithi037 ceph-mon[31460]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T18:06:55.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:54 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:06:55.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:54 smithi037 ceph-mon[31460]: from='osd.10 ' entity='osd.10' 2024-03-28T18:06:55.793 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T18:06:55.794 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T18:06:55.794 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-03-28T18:06:55.794 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-03-28T18:06:55.794 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00850969 s, 1.2 GB/s 2024-03-28T18:06:55.794 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-03-28T18:06:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:55 smithi059 ceph-mon[36190]: purged_snaps scrub starts 2024-03-28T18:06:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:55 smithi059 ceph-mon[36190]: purged_snaps scrub ok 2024-03-28T18:06:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:55 smithi059 ceph-mon[36190]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:55 smithi059 ceph-mon[36190]: osd.10 [v2:172.21.15.59:6824/2113830275,v1:172.21.15.59:6825/2113830275] boot 2024-03-28T18:06:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:55 smithi059 ceph-mon[36190]: osdmap e72: 11 total, 11 up, 11 in 2024-03-28T18:06:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:55 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:06:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:55 smithi059 ceph-mon[36190]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T18:06:56.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:55 smithi037 ceph-mon[31460]: purged_snaps scrub starts 2024-03-28T18:06:56.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:55 smithi037 ceph-mon[31460]: purged_snaps scrub ok 2024-03-28T18:06:56.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:55 smithi037 ceph-mon[31460]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:56.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:55 smithi037 ceph-mon[31460]: osd.10 [v2:172.21.15.59:6824/2113830275,v1:172.21.15.59:6825/2113830275] boot 2024-03-28T18:06:56.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:55 smithi037 ceph-mon[31460]: osdmap e72: 11 total, 11 up, 11 in 2024-03-28T18:06:56.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:55 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:06:56.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:55 smithi037 ceph-mon[31460]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T18:06:56.156 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:55 smithi170 ceph-mon[36280]: purged_snaps scrub starts 2024-03-28T18:06:56.156 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:55 smithi170 ceph-mon[36280]: purged_snaps scrub ok 2024-03-28T18:06:56.156 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:55 smithi170 ceph-mon[36280]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:06:56.156 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:55 smithi170 ceph-mon[36280]: osd.10 [v2:172.21.15.59:6824/2113830275,v1:172.21.15.59:6825/2113830275] boot 2024-03-28T18:06:56.157 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:55 smithi170 ceph-mon[36280]: osdmap e72: 11 total, 11 up, 11 in 2024-03-28T18:06:56.157 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:55 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:06:56.157 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:55 smithi170 ceph-mon[36280]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T18:06:56.479 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi170:vg_nvme/lv_1 2024-03-28T18:06:56.727 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:06:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:57 smithi059 ceph-mon[36190]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:06:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:57 smithi059 ceph-mon[36190]: Detected new or changed devices on smithi059 2024-03-28T18:06:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:57 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:57 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:58.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:57 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:58.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:57 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:58.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:57 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:58.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:57 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:58.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:57 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi059 to 4271M 2024-03-28T18:06:58.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:57 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:58.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:57 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:58.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:57 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:06:58.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:57 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:58.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:57 smithi059 ceph-mon[36190]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T18:06:58.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:57 smithi170 ceph-mon[36280]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:06:58.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:57 smithi170 ceph-mon[36280]: Detected new or changed devices on smithi059 2024-03-28T18:06:58.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:57 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:58.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:57 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:58.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:57 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:58.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:57 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:58.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:57 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:58.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:57 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:58.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:57 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi059 to 4271M 2024-03-28T18:06:58.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:57 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:58.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:57 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:58.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:57 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:06:58.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:57 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:58.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:57 smithi170 ceph-mon[36280]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T18:06:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:57 smithi037 ceph-mon[31460]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:06:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:57 smithi037 ceph-mon[31460]: Detected new or changed devices on smithi059 2024-03-28T18:06:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:06:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:57 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi059 to 4271M 2024-03-28T18:06:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:06:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:06:58.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:57 smithi037 ceph-mon[31460]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T18:06:58.305 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.304+0000 7f724729c700 1 -- 172.21.15.170:0/3098878522 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72401033c0 msgr2=0x7f7240103820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:06:58.305 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.304+0000 7f724729c700 1 --2- 172.21.15.170:0/3098878522 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72401033c0 0x7f7240103820 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7230009990 tx=0x7f7230009ca0 comp rx=0 tx=0).stop 2024-03-28T18:06:58.305 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.304+0000 7f724729c700 1 -- 172.21.15.170:0/3098878522 shutdown_connections 2024-03-28T18:06:58.305 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.304+0000 7f724729c700 1 --2- 172.21.15.170:0/3098878522 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7240103d60 0x7f724010a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:06:58.306 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.304+0000 7f724729c700 1 --2- 172.21.15.170:0/3098878522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72401021c0 0x7f72401025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:06:58.306 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.304+0000 7f724729c700 1 --2- 172.21.15.170:0/3098878522 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72401033c0 0x7f7240103820 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:06:58.306 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.304+0000 7f724729c700 1 -- 172.21.15.170:0/3098878522 >> 172.21.15.170:0/3098878522 conn(0x7f72400fd7a0 msgr2=0x7f72400ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:06:58.306 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.305+0000 7f724729c700 1 -- 172.21.15.170:0/3098878522 shutdown_connections 2024-03-28T18:06:58.306 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.305+0000 7f724729c700 1 -- 172.21.15.170:0/3098878522 wait complete. 2024-03-28T18:06:58.308 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.305+0000 7f724729c700 1 Processor -- start 2024-03-28T18:06:58.308 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.305+0000 7f724729c700 1 -- start start 2024-03-28T18:06:58.308 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.306+0000 7f724729c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72401021c0 0x7f7240198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:06:58.308 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.306+0000 7f724729c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f72401033c0 0x7f7240198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:06:58.308 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.306+0000 7f724729c700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7240103d60 0x7f724019fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:06:58.308 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.306+0000 7f724729c700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724010dea0 con 0x7f72401033c0 2024-03-28T18:06:58.308 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.306+0000 7f724729c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724010dd20 con 0x7f72401021c0 2024-03-28T18:06:58.309 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.306+0000 7f724729c700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724010e020 con 0x7f7240103d60 2024-03-28T18:06:58.309 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.306+0000 7f7245839700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7240103d60 0x7f724019fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:06:58.309 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.306+0000 7f7245839700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7240103d60 0x7f724019fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:39940/0 (socket says 172.21.15.170:39940) 2024-03-28T18:06:58.309 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.306+0000 7f7245839700 1 -- 172.21.15.170:0/2129905413 learned_addr learned my addr 172.21.15.170:0/2129905413 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T18:06:58.309 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.306+0000 7f7245038700 1 --2- 172.21.15.170:0/2129905413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72401021c0 0x7f7240198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:06:58.309 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.306+0000 7f7245839700 1 -- 172.21.15.170:0/2129905413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72401021c0 msgr2=0x7f7240198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:06:58.309 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.306+0000 7f7245839700 1 --2- 172.21.15.170:0/2129905413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72401021c0 0x7f7240198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:06:58.309 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.306+0000 7f7245839700 1 -- 172.21.15.170:0/2129905413 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f72401033c0 msgr2=0x7f7240198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:06:58.309 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.306+0000 7f7245839700 1 --2- 172.21.15.170:0/2129905413 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f72401033c0 0x7f7240198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:06:58.309 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.306+0000 7f7245839700 1 -- 172.21.15.170:0/2129905413 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72401a01a0 con 0x7f7240103d60 2024-03-28T18:06:58.309 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.306+0000 7f7245839700 1 --2- 172.21.15.170:0/2129905413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7240103d60 0x7f724019fb00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f723800b9a0 tx=0x7f723800bcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:06:58.309 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.306+0000 7f72367fc700 1 -- 172.21.15.170:0/2129905413 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7238019070 con 0x7f7240103d60 2024-03-28T18:06:58.309 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.306+0000 7f724729c700 1 -- 172.21.15.170:0/2129905413 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72401a0490 con 0x7f7240103d60 2024-03-28T18:06:58.310 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.307+0000 7f72367fc700 1 -- 172.21.15.170:0/2129905413 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f723800ce10 con 0x7f7240103d60 2024-03-28T18:06:58.310 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.307+0000 7f72367fc700 1 -- 172.21.15.170:0/2129905413 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7238007390 con 0x7f7240103d60 2024-03-28T18:06:58.310 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.307+0000 7f724729c700 1 -- 172.21.15.170:0/2129905413 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72401a0aa0 con 0x7f7240103d60 2024-03-28T18:06:58.310 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.307+0000 7f724729c700 1 -- 172.21.15.170:0/2129905413 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7240068e50 con 0x7f7240103d60 2024-03-28T18:06:58.311 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.309+0000 7f72367fc700 1 -- 172.21.15.170:0/2129905413 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f723801c070 con 0x7f7240103d60 2024-03-28T18:06:58.311 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.309+0000 7f72367fc700 1 --2- 172.21.15.170:0/2129905413 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f722c0a2910 0x7f722c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:06:58.311 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.309+0000 7f72367fc700 1 -- 172.21.15.170:0/2129905413 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6694+0+0 (secure 0 0 0) 0x7f72380cdba0 con 0x7f7240103d60 2024-03-28T18:06:58.311 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.310+0000 7f7245038700 1 --2- 172.21.15.170:0/2129905413 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f722c0a2910 0x7f722c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:06:58.311 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.311+0000 7f7245038700 1 --2- 172.21.15.170:0/2129905413 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f722c0a2910 0x7f722c0a4dd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7240103220 tx=0x7f723c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:06:58.313 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.313+0000 7f72367fc700 1 -- 172.21.15.170:0/2129905413 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f723809b290 con 0x7f7240103d60 2024-03-28T18:06:58.517 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:06:58.517+0000 7f724729c700 1 -- 172.21.15.170:0/2129905413 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f72400625d0 con 0x7f722c0a2910 2024-03-28T18:06:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:58 smithi059 ceph-mon[36190]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T18:06:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:58 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:06:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:58 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:06:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:58 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:58 smithi059 ceph-mon[36190]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T18:06:59.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:58 smithi170 ceph-mon[36280]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T18:06:59.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:58 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:06:59.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:58 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:06:59.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:58 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:59.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:58 smithi170 ceph-mon[36280]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T18:06:59.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:58 smithi037 ceph-mon[31460]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T18:06:59.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:58 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:06:59.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:58 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:06:59.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:58 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:59.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:58 smithi037 ceph-mon[31460]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T18:07:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:59 smithi059 ceph-mon[36190]: pgmap v267: 1 pgs: 1 remapped+peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:06:59 smithi059 ceph-mon[36190]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:07:00.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:59 smithi170 ceph-mon[36280]: pgmap v267: 1 pgs: 1 remapped+peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:00.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:06:59 smithi170 ceph-mon[36280]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:07:00.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:59 smithi037 ceph-mon[31460]: pgmap v267: 1 pgs: 1 remapped+peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:00.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:06:59 smithi037 ceph-mon[31460]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:07:02.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:01 smithi059 ceph-mon[36190]: pgmap v269: 1 pgs: 1 remapped+peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:01 smithi059 ceph-mon[36190]: from='client.? 172.21.15.170:0/1095244364' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5871e177-c17b-4637-a797-4fa5125920dd"}]: dispatch 2024-03-28T18:07:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:01 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5871e177-c17b-4637-a797-4fa5125920dd"}]: dispatch 2024-03-28T18:07:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:01 smithi059 ceph-mon[36190]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5871e177-c17b-4637-a797-4fa5125920dd"}]': finished 2024-03-28T18:07:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:01 smithi059 ceph-mon[36190]: osdmap e77: 12 total, 11 up, 12 in 2024-03-28T18:07:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:01 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:07:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:01 smithi170 ceph-mon[36280]: pgmap v269: 1 pgs: 1 remapped+peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:01 smithi170 ceph-mon[36280]: from='client.? 172.21.15.170:0/1095244364' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5871e177-c17b-4637-a797-4fa5125920dd"}]: dispatch 2024-03-28T18:07:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:01 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5871e177-c17b-4637-a797-4fa5125920dd"}]: dispatch 2024-03-28T18:07:02.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:01 smithi170 ceph-mon[36280]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5871e177-c17b-4637-a797-4fa5125920dd"}]': finished 2024-03-28T18:07:02.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:01 smithi170 ceph-mon[36280]: osdmap e77: 12 total, 11 up, 12 in 2024-03-28T18:07:02.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:01 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:07:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:01 smithi037 ceph-mon[31460]: pgmap v269: 1 pgs: 1 remapped+peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:01 smithi037 ceph-mon[31460]: from='client.? 172.21.15.170:0/1095244364' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5871e177-c17b-4637-a797-4fa5125920dd"}]: dispatch 2024-03-28T18:07:02.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:01 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5871e177-c17b-4637-a797-4fa5125920dd"}]: dispatch 2024-03-28T18:07:02.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:01 smithi037 ceph-mon[31460]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5871e177-c17b-4637-a797-4fa5125920dd"}]': finished 2024-03-28T18:07:02.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:01 smithi037 ceph-mon[31460]: osdmap e77: 12 total, 11 up, 12 in 2024-03-28T18:07:02.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:01 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:07:03.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:02 smithi059 ceph-mon[36190]: from='client.? 172.21.15.170:0/3587292179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:07:03.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:02 smithi170 ceph-mon[36280]: from='client.? 172.21.15.170:0/3587292179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:07:03.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:02 smithi037 ceph-mon[31460]: from='client.? 172.21.15.170:0/3587292179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:07:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:03 smithi059 ceph-mon[36190]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-28T18:07:04.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:03 smithi170 ceph-mon[36280]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-28T18:07:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:03 smithi037 ceph-mon[31460]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-28T18:07:06.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:05 smithi059 ceph-mon[36190]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-28T18:07:06.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:05 smithi170 ceph-mon[36280]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-28T18:07:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:05 smithi037 ceph-mon[31460]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-28T18:07:08.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:07 smithi170 ceph-mon[36280]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:07:08.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:07 smithi037 ceph-mon[31460]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:07:08.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:07 smithi059 ceph-mon[36190]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:07:09.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T18:07:09.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:08 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:09.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:08 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T18:07:09.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:08 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:09.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:08 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T18:07:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:08 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:09.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:09 smithi037 ceph-mon[31460]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-28T18:07:09.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:09 smithi037 ceph-mon[31460]: Deploying daemon osd.11 on smithi170 2024-03-28T18:07:10.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:09 smithi170 ceph-mon[36280]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-28T18:07:10.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:09 smithi170 ceph-mon[36280]: Deploying daemon osd.11 on smithi170 2024-03-28T18:07:10.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:09 smithi059 ceph-mon[36190]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-28T18:07:10.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:09 smithi059 ceph-mon[36190]: Deploying daemon osd.11 on smithi170 2024-03-28T18:07:11.852 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:11 smithi170 ceph-mon[36280]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T18:07:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:11 smithi037 ceph-mon[31460]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T18:07:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:11 smithi059 ceph-mon[36190]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T18:07:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:13 smithi037 ceph-mon[31460]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-03-28T18:07:14.158 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:13 smithi170 ceph-mon[36280]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-03-28T18:07:14.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:13 smithi059 ceph-mon[36190]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-03-28T18:07:15.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:15 smithi059 ceph-mon[36190]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:15.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:15 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:15.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:15 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:07:15.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:15 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:15.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:15 smithi170 ceph-mon[36280]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:15.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:15 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:15.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:15 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:07:15.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:15 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:15.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:15 smithi037 ceph-mon[31460]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:15.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:15 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:15.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:15 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:07:15.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:15 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:17.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:17 smithi059 ceph-mon[36190]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:17.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:17 smithi170 ceph-mon[36280]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:17.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:17 smithi037 ceph-mon[31460]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:18.057 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:18.057+0000 7f72367fc700 1 -- 172.21.15.170:0/2129905413 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f72400625d0 con 0x7f722c0a2910 2024-03-28T18:07:18.058 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 11 on host 'smithi170' 2024-03-28T18:07:18.060 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:18.060+0000 7f724729c700 1 -- 172.21.15.170:0/2129905413 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f722c0a2910 msgr2=0x7f722c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:18.060 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:18.060+0000 7f724729c700 1 --2- 172.21.15.170:0/2129905413 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f722c0a2910 0x7f722c0a4dd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7240103220 tx=0x7f723c009190 comp rx=0 tx=0).stop 2024-03-28T18:07:18.060 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:18.060+0000 7f724729c700 1 -- 172.21.15.170:0/2129905413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7240103d60 msgr2=0x7f724019fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:18.060 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:18.060+0000 7f724729c700 1 --2- 172.21.15.170:0/2129905413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7240103d60 0x7f724019fb00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f723800b9a0 tx=0x7f723800bcb0 comp rx=0 tx=0).stop 2024-03-28T18:07:18.060 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:18.060+0000 7f724729c700 1 -- 172.21.15.170:0/2129905413 shutdown_connections 2024-03-28T18:07:18.060 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:18.060+0000 7f724729c700 1 --2- 172.21.15.170:0/2129905413 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f722c0a2910 0x7f722c0a4dd0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:18.060 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:18.060+0000 7f724729c700 1 --2- 172.21.15.170:0/2129905413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7240103d60 0x7f724019fb00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:18.060 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:18.060+0000 7f724729c700 1 --2- 172.21.15.170:0/2129905413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72401021c0 0x7f7240198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:18.060 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:18.060+0000 7f724729c700 1 --2- 172.21.15.170:0/2129905413 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f72401033c0 0x7f7240198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:18.060 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:18.060+0000 7f724729c700 1 -- 172.21.15.170:0/2129905413 >> 172.21.15.170:0/2129905413 conn(0x7f72400fd7a0 msgr2=0x7f7240106740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:18.060 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:18.060+0000 7f724729c700 1 -- 172.21.15.170:0/2129905413 shutdown_connections 2024-03-28T18:07:18.060 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:18.060+0000 7f724729c700 1 -- 172.21.15.170:0/2129905413 wait complete. 2024-03-28T18:07:18.777 DEBUG:teuthology.orchestra.run.smithi170:osd.11> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@osd.11.service 2024-03-28T18:07:18.779 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-28T18:07:18.779 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-28T18:07:19.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:18 smithi170 ceph-mon[36280]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:19.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:18 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:19.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:18 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:18 smithi170 ceph-mon[36280]: from='osd.11 [v2:172.21.15.170:6824/3082439360,v1:172.21.15.170:6825/3082439360]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T18:07:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:18 smithi170 ceph-mon[36280]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T18:07:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:18 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:18 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:19.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:18 smithi059 ceph-mon[36190]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:19.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:18 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:19.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:18 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:19.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:18 smithi059 ceph-mon[36190]: from='osd.11 [v2:172.21.15.170:6824/3082439360,v1:172.21.15.170:6825/3082439360]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T18:07:19.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:18 smithi059 ceph-mon[36190]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T18:07:19.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:18 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:19.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:18 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:19.290 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:07:19.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:18 smithi037 ceph-mon[31460]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:19.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:18 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:19.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:18 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:19.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:18 smithi037 ceph-mon[31460]: from='osd.11 [v2:172.21.15.170:6824/3082439360,v1:172.21.15.170:6825/3082439360]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T18:07:19.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:18 smithi037 ceph-mon[31460]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T18:07:19.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:18 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:19.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:18 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:20 smithi059 ceph-mon[36190]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T18:07:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:20 smithi059 ceph-mon[36190]: from='osd.11 [v2:172.21.15.170:6824/3082439360,v1:172.21.15.170:6825/3082439360]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:07:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:20 smithi059 ceph-mon[36190]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T18:07:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:20 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:07:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:20 smithi059 ceph-mon[36190]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:07:20.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:20 smithi170 ceph-mon[36280]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T18:07:20.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:20 smithi170 ceph-mon[36280]: from='osd.11 [v2:172.21.15.170:6824/3082439360,v1:172.21.15.170:6825/3082439360]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:07:20.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:20 smithi170 ceph-mon[36280]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T18:07:20.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:20 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:07:20.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:20 smithi170 ceph-mon[36280]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:07:20.286 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 28 18:07:20 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-11[61340]: 2024-03-28T18:07:20.018+0000 7f3c0d7e2700 -1 osd.11 0 waiting for initial osdmap 2024-03-28T18:07:20.286 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 28 18:07:20 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-osd-11[61340]: 2024-03-28T18:07:20.032+0000 7f3c095d5700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:07:20.347 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:20 smithi037 ceph-mon[31460]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T18:07:20.347 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:20 smithi037 ceph-mon[31460]: from='osd.11 [v2:172.21.15.170:6824/3082439360,v1:172.21.15.170:6825/3082439360]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:07:20.347 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:20 smithi037 ceph-mon[31460]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T18:07:20.347 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:20 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:07:20.347 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:20 smithi037 ceph-mon[31460]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-28T18:07:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:21 smithi059 ceph-mon[36190]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:21 smithi059 ceph-mon[36190]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-28T18:07:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:21 smithi059 ceph-mon[36190]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T18:07:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:21 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:07:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:21 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:07:21.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:21 smithi170 ceph-mon[36280]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:21.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:21 smithi170 ceph-mon[36280]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-28T18:07:21.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:21 smithi170 ceph-mon[36280]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T18:07:21.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:21 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:07:21.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:21 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:07:21.326 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:21 smithi037 ceph-mon[31460]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:21.326 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:21 smithi037 ceph-mon[31460]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-28T18:07:21.326 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:21 smithi037 ceph-mon[31460]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T18:07:21.326 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:21 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:07:21.326 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:21 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:07:21.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.655+0000 7f49fbe00700 1 -- 172.21.15.37:0/2439325951 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f49fc074bf0 msgr2=0x7f49fc075070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:21.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.655+0000 7f49fbe00700 1 --2- 172.21.15.37:0/2439325951 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f49fc074bf0 0x7f49fc075070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f49e4009990 tx=0x7f49e4009ca0 comp rx=0 tx=0).stop 2024-03-28T18:07:21.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.655+0000 7f49fbe00700 1 -- 172.21.15.37:0/2439325951 shutdown_connections 2024-03-28T18:07:21.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.655+0000 7f49fbe00700 1 --2- 172.21.15.37:0/2439325951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49fc10e6e0 0x7f49fc110ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:21.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.655+0000 7f49fbe00700 1 --2- 172.21.15.37:0/2439325951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49fc0761c0 0x7f49fc074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:21.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.655+0000 7f49fbe00700 1 --2- 172.21.15.37:0/2439325951 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f49fc074bf0 0x7f49fc075070 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:21.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.655+0000 7f49fbe00700 1 -- 172.21.15.37:0/2439325951 >> 172.21.15.37:0/2439325951 conn(0x7f49fc0fd6a0 msgr2=0x7f49fc0ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:21.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.655+0000 7f49fbe00700 1 -- 172.21.15.37:0/2439325951 shutdown_connections 2024-03-28T18:07:21.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.655+0000 7f49fbe00700 1 -- 172.21.15.37:0/2439325951 wait complete. 2024-03-28T18:07:21.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.656+0000 7f49fbe00700 1 Processor -- start 2024-03-28T18:07:21.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.656+0000 7f49fbe00700 1 -- start start 2024-03-28T18:07:21.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.656+0000 7f49fbe00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49fc074bf0 0x7f49fc19e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:21.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.656+0000 7f49fbe00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49fc0761c0 0x7f49fc19eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:21.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.656+0000 7f49fbe00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f49fc10e6e0 0x7f49fc1a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:21.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.657+0000 7f49fbe00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49fc114330 con 0x7f49fc10e6e0 2024-03-28T18:07:21.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.657+0000 7f49fbe00700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49fc1141b0 con 0x7f49fc0761c0 2024-03-28T18:07:21.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.657+0000 7f49fbe00700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49fc1144b0 con 0x7f49fc074bf0 2024-03-28T18:07:21.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.657+0000 7f49faa00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49fc074bf0 0x7f49fc19e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:21.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.657+0000 7f49faa00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49fc074bf0 0x7f49fc19e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:36606/0 (socket says 172.21.15.37:36606) 2024-03-28T18:07:21.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.657+0000 7f49faa00700 1 -- 172.21.15.37:0/3362071038 learned_addr learned my addr 172.21.15.37:0/3362071038 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:07:21.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.657+0000 7f49fb400700 1 --2- 172.21.15.37:0/3362071038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f49fc10e6e0 0x7f49fc1a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:21.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.657+0000 7f49f2000700 1 --2- 172.21.15.37:0/3362071038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49fc0761c0 0x7f49fc19eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:21.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.657+0000 7f49faa00700 1 -- 172.21.15.37:0/3362071038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49fc0761c0 msgr2=0x7f49fc19eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:21.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.657+0000 7f49faa00700 1 --2- 172.21.15.37:0/3362071038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49fc0761c0 0x7f49fc19eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:21.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.657+0000 7f49faa00700 1 -- 172.21.15.37:0/3362071038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f49fc10e6e0 msgr2=0x7f49fc1a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:21.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.657+0000 7f49faa00700 1 --2- 172.21.15.37:0/3362071038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f49fc10e6e0 0x7f49fc1a5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:21.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.657+0000 7f49faa00700 1 -- 172.21.15.37:0/3362071038 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49fc1a65b0 con 0x7f49fc074bf0 2024-03-28T18:07:21.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.658+0000 7f49faa00700 1 --2- 172.21.15.37:0/3362071038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49fc074bf0 0x7f49fc19e9a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f49ec009f40 tx=0x7f49ec00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:21.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.658+0000 7f49f3e00700 1 -- 172.21.15.37:0/3362071038 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ec00c9b0 con 0x7f49fc074bf0 2024-03-28T18:07:21.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.658+0000 7f49f3e00700 1 -- 172.21.15.37:0/3362071038 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49ec019070 con 0x7f49fc074bf0 2024-03-28T18:07:21.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.658+0000 7f49fbe00700 1 -- 172.21.15.37:0/3362071038 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49fc1a6840 con 0x7f49fc074bf0 2024-03-28T18:07:21.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.658+0000 7f49f3e00700 1 -- 172.21.15.37:0/3362071038 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ec014480 con 0x7f49fc074bf0 2024-03-28T18:07:21.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.658+0000 7f49fbe00700 1 -- 172.21.15.37:0/3362071038 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49fc1a6d80 con 0x7f49fc074bf0 2024-03-28T18:07:21.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.659+0000 7f49fbe00700 1 -- 172.21.15.37:0/3362071038 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49fc068e50 con 0x7f49fc074bf0 2024-03-28T18:07:21.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.661+0000 7f49f3e00700 1 -- 172.21.15.37:0/3362071038 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f49ec012050 con 0x7f49fc074bf0 2024-03-28T18:07:21.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.662+0000 7f49f3e00700 1 --2- 172.21.15.37:0/3362071038 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f49e00a2890 0x7f49e00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:21.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.663+0000 7f49f3e00700 1 -- 172.21.15.37:0/3362071038 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f49ec0ce520 con 0x7f49fc074bf0 2024-03-28T18:07:21.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.663+0000 7f49f2000700 1 --2- 172.21.15.37:0/3362071038 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f49e00a2890 0x7f49e00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:21.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.663+0000 7f49f2000700 1 --2- 172.21.15.37:0/3362071038 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f49e00a2890 0x7f49e00a4d50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f49e400b7b0 tx=0x7f49e4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:21.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.667+0000 7f49f3e00700 1 -- 172.21.15.37:0/3362071038 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f49ec010040 con 0x7f49fc074bf0 2024-03-28T18:07:21.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.897+0000 7f49fbe00700 1 -- 172.21.15.37:0/3362071038 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f49fc062da0 con 0x7f49fc074bf0 2024-03-28T18:07:21.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.898+0000 7f49f3e00700 1 -- 172.21.15.37:0/3362071038 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) v1 ==== 74+0+133 (secure 0 0 0) 0x7f49ec09b9c0 con 0x7f49fc074bf0 2024-03-28T18:07:21.898 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:07:21.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.901+0000 7f49fbe00700 1 -- 172.21.15.37:0/3362071038 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f49e00a2890 msgr2=0x7f49e00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:21.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.901+0000 7f49fbe00700 1 --2- 172.21.15.37:0/3362071038 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f49e00a2890 0x7f49e00a4d50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f49e400b7b0 tx=0x7f49e4005fd0 comp rx=0 tx=0).stop 2024-03-28T18:07:21.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.901+0000 7f49fbe00700 1 -- 172.21.15.37:0/3362071038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49fc074bf0 msgr2=0x7f49fc19e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:21.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.901+0000 7f49fbe00700 1 --2- 172.21.15.37:0/3362071038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49fc074bf0 0x7f49fc19e9a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f49ec009f40 tx=0x7f49ec00c3a0 comp rx=0 tx=0).stop 2024-03-28T18:07:21.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.901+0000 7f49fbe00700 1 -- 172.21.15.37:0/3362071038 shutdown_connections 2024-03-28T18:07:21.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.901+0000 7f49fbe00700 1 --2- 172.21.15.37:0/3362071038 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f49e00a2890 0x7f49e00a4d50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:21.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.901+0000 7f49fbe00700 1 --2- 172.21.15.37:0/3362071038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f49fc10e6e0 0x7f49fc1a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:21.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.901+0000 7f49fbe00700 1 --2- 172.21.15.37:0/3362071038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49fc074bf0 0x7f49fc19e9a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:21.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.901+0000 7f49fbe00700 1 --2- 172.21.15.37:0/3362071038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49fc0761c0 0x7f49fc19eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:21.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.901+0000 7f49fbe00700 1 -- 172.21.15.37:0/3362071038 >> 172.21.15.37:0/3362071038 conn(0x7f49fc0fd6a0 msgr2=0x7f49fc101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:21.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.901+0000 7f49fbe00700 1 -- 172.21.15.37:0/3362071038 shutdown_connections 2024-03-28T18:07:21.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:21.901+0000 7f49fbe00700 1 -- 172.21.15.37:0/3362071038 wait complete. 2024-03-28T18:07:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:22 smithi170 ceph-mon[36280]: purged_snaps scrub starts 2024-03-28T18:07:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:22 smithi170 ceph-mon[36280]: purged_snaps scrub ok 2024-03-28T18:07:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:22 smithi170 ceph-mon[36280]: osd.11 [v2:172.21.15.170:6824/3082439360,v1:172.21.15.170:6825/3082439360] boot 2024-03-28T18:07:22.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:22 smithi170 ceph-mon[36280]: osdmap e80: 12 total, 12 up, 12 in 2024-03-28T18:07:22.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:22 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:07:22.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:22 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3362071038' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T18:07:22.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:22 smithi037 ceph-mon[31460]: purged_snaps scrub starts 2024-03-28T18:07:22.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:22 smithi037 ceph-mon[31460]: purged_snaps scrub ok 2024-03-28T18:07:22.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:22 smithi037 ceph-mon[31460]: osd.11 [v2:172.21.15.170:6824/3082439360,v1:172.21.15.170:6825/3082439360] boot 2024-03-28T18:07:22.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:22 smithi037 ceph-mon[31460]: osdmap e80: 12 total, 12 up, 12 in 2024-03-28T18:07:22.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:22 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:07:22.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:22 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3362071038' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T18:07:22.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:22 smithi059 ceph-mon[36190]: purged_snaps scrub starts 2024-03-28T18:07:22.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:22 smithi059 ceph-mon[36190]: purged_snaps scrub ok 2024-03-28T18:07:22.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:22 smithi059 ceph-mon[36190]: osd.11 [v2:172.21.15.170:6824/3082439360,v1:172.21.15.170:6825/3082439360] boot 2024-03-28T18:07:22.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:22 smithi059 ceph-mon[36190]: osdmap e80: 12 total, 12 up, 12 in 2024-03-28T18:07:22.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:22 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:07:22.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:22 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3362071038' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T18:07:22.853 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1711649241,"num_in_osds":12,"osd_in_since":1711649221,"num_remapped_pgs":0} 2024-03-28T18:07:22.854 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T18:07:23.322 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:07:23.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:23 smithi037 ceph-mon[31460]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:23 smithi037 ceph-mon[31460]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T18:07:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:23 smithi037 ceph-mon[31460]: Detected new or changed devices on smithi170 2024-03-28T18:07:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:07:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:07:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:07:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:07:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:23 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi170 to 4270M 2024-03-28T18:07:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:07:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:07:23.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:23 smithi059 ceph-mon[36190]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:23.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:23 smithi059 ceph-mon[36190]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T18:07:23.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:23 smithi059 ceph-mon[36190]: Detected new or changed devices on smithi170 2024-03-28T18:07:23.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:23 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:23.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:23 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:23.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:23 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:07:23.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:23 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:07:23.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:23 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:07:23.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:23 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:07:23.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:23 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi170 to 4270M 2024-03-28T18:07:23.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:23 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:23.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:23 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:23.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:23 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:07:23.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:23 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:23.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:23 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:07:23.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:23 smithi170 ceph-mon[36280]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:07:23.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:23 smithi170 ceph-mon[36280]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T18:07:23.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:23 smithi170 ceph-mon[36280]: Detected new or changed devices on smithi170 2024-03-28T18:07:23.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:23.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:23.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:07:23.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:07:23.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:07:23.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:07:23.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:23 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi170 to 4270M 2024-03-28T18:07:23.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:23.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:23.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:07:23.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:23.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:07:24.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:24 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:24.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:24 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:07:24.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:24 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:07:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:24 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:24.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:24 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:24.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:24 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:07:24.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:24 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:25.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.618+0000 7fe933600700 1 -- 172.21.15.37:0/628693226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe92c06aa90 msgr2=0x7fe92c06ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:25.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.618+0000 7fe933600700 1 --2- 172.21.15.37:0/628693226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe92c06aa90 0x7fe92c06ae70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe91c0099c0 tx=0x7fe91c009cd0 comp rx=0 tx=0).stop 2024-03-28T18:07:25.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.618+0000 7fe933600700 1 -- 172.21.15.37:0/628693226 shutdown_connections 2024-03-28T18:07:25.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.618+0000 7fe933600700 1 --2- 172.21.15.37:0/628693226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe92c107500 0x7fe92c10f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:25.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.618+0000 7fe933600700 1 --2- 172.21.15.37:0/628693226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe92c06aa90 0x7fe92c06ae70 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:25.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.618+0000 7fe933600700 1 --2- 172.21.15.37:0/628693226 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe92c06b440 0x7fe92c106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:25.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.618+0000 7fe933600700 1 -- 172.21.15.37:0/628693226 >> 172.21.15.37:0/628693226 conn(0x7fe92c078000 msgr2=0x7fe92c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:25.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.618+0000 7fe933600700 1 -- 172.21.15.37:0/628693226 shutdown_connections 2024-03-28T18:07:25.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.618+0000 7fe933600700 1 -- 172.21.15.37:0/628693226 wait complete. 2024-03-28T18:07:25.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.619+0000 7fe933600700 1 Processor -- start 2024-03-28T18:07:25.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.619+0000 7fe933600700 1 -- start start 2024-03-28T18:07:25.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.620+0000 7fe933600700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe92c06aa90 0x7fe92c1a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:25.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.620+0000 7fe933600700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe92c06b440 0x7fe92c1a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:25.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.620+0000 7fe933600700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe92c107500 0x7fe92c19a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:25.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.620+0000 7fe933600700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c112490 con 0x7fe92c06aa90 2024-03-28T18:07:25.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.620+0000 7fe933600700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c112310 con 0x7fe92c107500 2024-03-28T18:07:25.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.620+0000 7fe933600700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c112610 con 0x7fe92c06b440 2024-03-28T18:07:25.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.620+0000 7fe930c00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe92c06aa90 0x7fe92c1a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:25.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.620+0000 7fe930c00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe92c06aa90 0x7fe92c1a07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:38758/0 (socket says 172.21.15.37:38758) 2024-03-28T18:07:25.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.620+0000 7fe930c00700 1 -- 172.21.15.37:0/3898838663 learned_addr learned my addr 172.21.15.37:0/3898838663 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:07:25.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.620+0000 7fe923e00700 1 --2- 172.21.15.37:0/3898838663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe92c06b440 0x7fe92c1a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:25.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.621+0000 7fe931600700 1 --2- 172.21.15.37:0/3898838663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe92c107500 0x7fe92c19a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:25.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.621+0000 7fe930c00700 1 -- 172.21.15.37:0/3898838663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe92c06b440 msgr2=0x7fe92c1a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:25.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.621+0000 7fe930c00700 1 --2- 172.21.15.37:0/3898838663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe92c06b440 0x7fe92c1a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:25.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.621+0000 7fe930c00700 1 -- 172.21.15.37:0/3898838663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe92c107500 msgr2=0x7fe92c19a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:25.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.621+0000 7fe930c00700 1 --2- 172.21.15.37:0/3898838663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe92c107500 0x7fe92c19a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:25.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.621+0000 7fe930c00700 1 -- 172.21.15.37:0/3898838663 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92c19b0d0 con 0x7fe92c06aa90 2024-03-28T18:07:25.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.621+0000 7fe923e00700 1 --2- 172.21.15.37:0/3898838663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe92c06b440 0x7fe92c1a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:07:25.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.621+0000 7fe930c00700 1 --2- 172.21.15.37:0/3898838663 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe92c06aa90 0x7fe92c1a07b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe91c006010 tx=0x7fe91c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:25.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.621+0000 7fe92a000700 1 -- 172.21.15.37:0/3898838663 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c003680 con 0x7fe92c06aa90 2024-03-28T18:07:25.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.621+0000 7fe92a000700 1 -- 172.21.15.37:0/3898838663 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe91c0055b0 con 0x7fe92c06aa90 2024-03-28T18:07:25.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.621+0000 7fe933600700 1 -- 172.21.15.37:0/3898838663 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92c19b360 con 0x7fe92c06aa90 2024-03-28T18:07:25.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.621+0000 7fe92a000700 1 -- 172.21.15.37:0/3898838663 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c0173f0 con 0x7fe92c06aa90 2024-03-28T18:07:25.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.622+0000 7fe933600700 1 -- 172.21.15.37:0/3898838663 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92c1a7680 con 0x7fe92c06aa90 2024-03-28T18:07:25.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.623+0000 7fe933600700 1 -- 172.21.15.37:0/3898838663 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe92c1038b0 con 0x7fe92c06aa90 2024-03-28T18:07:25.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.625+0000 7fe92a000700 1 -- 172.21.15.37:0/3898838663 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe91c00fca0 con 0x7fe92c06aa90 2024-03-28T18:07:25.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.626+0000 7fe92a000700 1 --2- 172.21.15.37:0/3898838663 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe9100a2a30 0x7fe9100a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:25.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.626+0000 7fe92a000700 1 -- 172.21.15.37:0/3898838663 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe91c0cfcd0 con 0x7fe92c06aa90 2024-03-28T18:07:25.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.627+0000 7fe923e00700 1 --2- 172.21.15.37:0/3898838663 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe9100a2a30 0x7fe9100a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:25.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.627+0000 7fe923e00700 1 --2- 172.21.15.37:0/3898838663 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe9100a2a30 0x7fe9100a4ef0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe92c10ef00 tx=0x7fe914009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:25.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.629+0000 7fe92a000700 1 -- 172.21.15.37:0/3898838663 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe91c013070 con 0x7fe92c06aa90 2024-03-28T18:07:25.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:25 smithi059 ceph-mon[36190]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:25 smithi170 ceph-mon[36280]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:25.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.858+0000 7fe933600700 1 -- 172.21.15.37:0/3898838663 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe92c068e50 con 0x7fe92c06aa90 2024-03-28T18:07:25.859 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:25 smithi037 ceph-mon[31460]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:25.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.860+0000 7fe92a000700 1 -- 172.21.15.37:0/3898838663 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18433 (secure 0 0 0) 0x7fe91c09d320 con 0x7fe92c06aa90 2024-03-28T18:07:25.862 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:07:25.862 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":81,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","created":"2024-03-28T17:59:05.949847+0000","modified":"2024-03-28T18:07:22.031148+0000","last_up_change":"2024-03-28T18:07:21.021030+0000","last_in_change":"2024-03-28T18:07:01.416192+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:03:00.517967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db60618d-56f7-4130-8a90-84cbc480965b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6803","nonce":3562602035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6805","nonce":3562602035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6809","nonce":3562602035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6807","nonce":3562602035}]},"public_addr":"172.21.15.37:6803/3562602035","cluster_addr":"172.21.15.37:6805/3562602035","heartbeat_back_addr":"172.21.15.37:6809/3562602035","heartbeat_front_addr":"172.21.15.37:6807/3562602035","state":["exists","up"]},{"osd":1,"uuid":"7a4b7ad6-905e-4258-9574-693d4277dfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6801","nonce":3326537211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6803","nonce":3326537211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6807","nonce":3326537211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6805","nonce":3326537211}]},"public_addr":"172.21.15.59:6801/3326537211","cluster_addr":"172.21.15.59:6803/3326537211","heartbeat_back_addr":"172.21.15.59:6807/3326537211","heartbeat_front_addr":"172.21.15.59:6805/3326537211","state":["exists","up"]},{"osd":2,"uuid":"8b76ec1c-46bf-4336-89b5-12d82906e8b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6801","nonce":1269398427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6803","nonce":1269398427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6807","nonce":1269398427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6805","nonce":1269398427}]},"public_addr":"172.21.15.170:6801/1269398427","cluster_addr":"172.21.15.170:6803/1269398427","heartbeat_back_addr":"172.21.15.170:6807/1269398427","heartbeat_front_addr":"172.21.15.170:6805/1269398427","state":["exists","up"]},{"osd":3,"uuid":"66eca1da-2fbd-44e1-8bbf-5635a2c70ad5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6811","nonce":317805240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6813","nonce":317805240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6817","nonce":317805240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6815","nonce":317805240}]},"public_addr":"172.21.15.37:6811/317805240","cluster_addr":"172.21.15.37:6813/317805240","heartbeat_back_addr":"172.21.15.37:6817/317805240","heartbeat_front_addr":"172.21.15.37:6815/317805240","state":["exists","up"]},{"osd":4,"uuid":"4c877335-e42d-4d8e-9f48-a1d0a89ae438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6809","nonce":2410632623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6811","nonce":2410632623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6815","nonce":2410632623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6813","nonce":2410632623}]},"public_addr":"172.21.15.59:6809/2410632623","cluster_addr":"172.21.15.59:6811/2410632623","heartbeat_back_addr":"172.21.15.59:6815/2410632623","heartbeat_front_addr":"172.21.15.59:6813/2410632623","state":["exists","up"]},{"osd":5,"uuid":"b884842a-86e2-455b-9570-ff163b3bc0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6809","nonce":1218943899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6811","nonce":1218943899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6815","nonce":1218943899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6813","nonce":1218943899}]},"public_addr":"172.21.15.170:6809/1218943899","cluster_addr":"172.21.15.170:6811/1218943899","heartbeat_back_addr":"172.21.15.170:6815/1218943899","heartbeat_front_addr":"172.21.15.170:6813/1218943899","state":["exists","up"]},{"osd":6,"uuid":"249a1200-77f4-44ad-981f-1d352f732b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6819","nonce":1899652046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6821","nonce":1899652046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6825","nonce":1899652046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6823","nonce":1899652046}]},"public_addr":"172.21.15.37:6819/1899652046","cluster_addr":"172.21.15.37:6821/1899652046","heartbeat_back_addr":"172.21.15.37:6825/1899652046","heartbeat_front_addr":"172.21.15.37:6823/1899652046","state":["exists","up"]},{"osd":7,"uuid":"c73338e6-f505-4517-a84b-f35e04750a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6817","nonce":1343675690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6819","nonce":1343675690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6823","nonce":1343675690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6821","nonce":1343675690}]},"public_addr":"172.21.15.59:6817/1343675690","cluster_addr":"172.21.15.59:6819/1343675690","heartbeat_back_addr":"172.21.15.59:6823/1343675690","heartbeat_front_addr":"172.21.15.59:6821/1343675690","state":["exists","up"]},{"osd":8,"uuid":"149b8297-ff2d-4a88-9b1e-0def021cc516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6817","nonce":3235794556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6819","nonce":3235794556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6823","nonce":3235794556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6821","nonce":3235794556}]},"public_addr":"172.21.15.170:6817/3235794556","cluster_addr":"172.21.15.170:6819/3235794556","heartbeat_back_addr":"172.21.15.170:6823/3235794556","heartbeat_front_addr":"172.21.15.170:6821/3235794556","state":["exists","up"]},{"osd":9,"uuid":"55c27cc2-c771-44b7-90fe-8f0feca44162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6827","nonce":2307153008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6829","nonce":2307153008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6833","nonce":2307153008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6831","nonce":2307153008}]},"public_addr":"172.21.15.37:6827/2307153008","cluster_addr":"172.21.15.37:6829/2307153008","heartbeat_back_addr":"172.21.15.37:6833/2307153008","heartbeat_front_addr":"172.21.15.37:6831/2307153008","state":["exists","up"]},{"osd":10,"uuid":"c0595135-ab81-49d3-8c23-19f960c03e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6825","nonce":2113830275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6827","nonce":2113830275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6831","nonce":2113830275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6829","nonce":2113830275}]},"public_addr":"172.21.15.59:6825/2113830275","cluster_addr":"172.21.15.59:6827/2113830275","heartbeat_back_addr":"172.21.15.59:6831/2113830275","heartbeat_front_addr":"172.21.15.59:6829/2113830275","state":["exists","up"]},{"osd":11,"uuid":"5871e177-c17b-4637-a797-4fa5125920dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6825","nonce":3082439360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6827","nonce":3082439360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6831","nonce":3082439360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6829","nonce":3082439360}]},"public_addr":"172.21.15.170:6825/3082439360","cluster_addr":"172.21.15.170:6827/3082439360","heartbeat_back_addr":"172.21.15.170:6831/3082439360","heartbeat_front_addr":"172.21.15.170:6829/3082439360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:07.690545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:33.183396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:57.400263+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:30.597211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:57.259044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:22.627823+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:57.585863+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:25.000476+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:50.876688+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:25.859761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:53.563146+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:07:19.304115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3023379428":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1443710231":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6800/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6801/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6801/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6800/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1007625490":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/2895698584":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/497313034":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/3293943475":"2024-03-29T17:59:39.162038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:07:25.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.864+0000 7fe933600700 1 -- 172.21.15.37:0/3898838663 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe9100a2a30 msgr2=0x7fe9100a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:25.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.864+0000 7fe933600700 1 --2- 172.21.15.37:0/3898838663 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe9100a2a30 0x7fe9100a4ef0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe92c10ef00 tx=0x7fe914009260 comp rx=0 tx=0).stop 2024-03-28T18:07:25.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.864+0000 7fe933600700 1 -- 172.21.15.37:0/3898838663 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe92c06aa90 msgr2=0x7fe92c1a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:25.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.864+0000 7fe933600700 1 --2- 172.21.15.37:0/3898838663 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe92c06aa90 0x7fe92c1a07b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe91c006010 tx=0x7fe91c00f690 comp rx=0 tx=0).stop 2024-03-28T18:07:25.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.864+0000 7fe933600700 1 -- 172.21.15.37:0/3898838663 shutdown_connections 2024-03-28T18:07:25.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.864+0000 7fe933600700 1 --2- 172.21.15.37:0/3898838663 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe9100a2a30 0x7fe9100a4ef0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:25.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.864+0000 7fe933600700 1 --2- 172.21.15.37:0/3898838663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe92c107500 0x7fe92c19a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:25.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.864+0000 7fe933600700 1 --2- 172.21.15.37:0/3898838663 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe92c06aa90 0x7fe92c1a07b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:25.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.864+0000 7fe933600700 1 --2- 172.21.15.37:0/3898838663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe92c06b440 0x7fe92c1a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:25.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.864+0000 7fe933600700 1 -- 172.21.15.37:0/3898838663 >> 172.21.15.37:0/3898838663 conn(0x7fe92c078000 msgr2=0x7fe92c10d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:25.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.865+0000 7fe933600700 1 -- 172.21.15.37:0/3898838663 shutdown_connections 2024-03-28T18:07:25.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:25.865+0000 7fe933600700 1 -- 172.21.15.37:0/3898838663 wait complete. 2024-03-28T18:07:26.577 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:26 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3898838663' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:07:26.654 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:03:00.517967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:07:26.655 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-28T18:07:26.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:26 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3898838663' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:07:26.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:26 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3898838663' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:07:27.153 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:07:27.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:27 smithi037 ceph-mon[31460]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:27.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:27 smithi059 ceph-mon[36190]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:27.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:27 smithi170 ceph-mon[36280]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:29.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.336+0000 7f9ceae00700 1 -- 172.21.15.37:0/706611995 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ce4108290 msgr2=0x7f9ce4108670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:29.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.336+0000 7f9ceae00700 1 --2- 172.21.15.37:0/706611995 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ce4108290 0x7f9ce4108670 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4009990 tx=0x7f9cd4009ca0 comp rx=0 tx=0).stop 2024-03-28T18:07:29.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.336+0000 7f9ceae00700 1 -- 172.21.15.37:0/706611995 shutdown_connections 2024-03-28T18:07:29.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.336+0000 7f9ceae00700 1 --2- 172.21.15.37:0/706611995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce4102ba0 0x7f9ce410f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:29.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.336+0000 7f9ceae00700 1 --2- 172.21.15.37:0/706611995 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ce4108290 0x7f9ce4108670 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:29.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.336+0000 7f9ceae00700 1 --2- 172.21.15.37:0/706611995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ce41021e0 0x7f9ce4102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:29.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.336+0000 7f9ceae00700 1 -- 172.21.15.37:0/706611995 >> 172.21.15.37:0/706611995 conn(0x7f9ce40792c0 msgr2=0x7f9ce40796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:29.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.336+0000 7f9ceae00700 1 -- 172.21.15.37:0/706611995 shutdown_connections 2024-03-28T18:07:29.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.336+0000 7f9ceae00700 1 -- 172.21.15.37:0/706611995 wait complete. 2024-03-28T18:07:29.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.337+0000 7f9ceae00700 1 Processor -- start 2024-03-28T18:07:29.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.337+0000 7f9ceae00700 1 -- start start 2024-03-28T18:07:29.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.338+0000 7f9ceae00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ce41021e0 0x7f9ce4074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:29.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.338+0000 7f9ceae00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce4102ba0 0x7f9ce40749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:29.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.338+0000 7f9ceae00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ce4108290 0x7f9ce4074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:29.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.338+0000 7f9ceae00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce41124f0 con 0x7f9ce4108290 2024-03-28T18:07:29.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.338+0000 7f9ceae00700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4112370 con 0x7f9ce4102ba0 2024-03-28T18:07:29.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.338+0000 7f9ceae00700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4112670 con 0x7f9ce41021e0 2024-03-28T18:07:29.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.338+0000 7f9ce3400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce4102ba0 0x7f9ce40749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:29.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.338+0000 7f9ce3400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce4102ba0 0x7f9ce40749d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:38230/0 (socket says 172.21.15.37:38230) 2024-03-28T18:07:29.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.338+0000 7f9ce3400700 1 -- 172.21.15.37:0/3376869610 learned_addr learned my addr 172.21.15.37:0/3376869610 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:07:29.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.338+0000 7f9ce8e00700 1 --2- 172.21.15.37:0/3376869610 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ce4108290 0x7f9ce4074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:29.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.338+0000 7f9ce3e00700 1 --2- 172.21.15.37:0/3376869610 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ce41021e0 0x7f9ce4074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:29.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.338+0000 7f9ce3400700 1 -- 172.21.15.37:0/3376869610 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ce41021e0 msgr2=0x7f9ce4074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:29.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.338+0000 7f9ce3400700 1 --2- 172.21.15.37:0/3376869610 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ce41021e0 0x7f9ce4074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:29.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.338+0000 7f9ce3400700 1 -- 172.21.15.37:0/3376869610 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ce4108290 msgr2=0x7f9ce4074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:29.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.338+0000 7f9ce3400700 1 --2- 172.21.15.37:0/3376869610 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ce4108290 0x7f9ce4074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:29.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.338+0000 7f9ce3400700 1 -- 172.21.15.37:0/3376869610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce41ab390 con 0x7f9ce4102ba0 2024-03-28T18:07:29.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.339+0000 7f9ce3400700 1 --2- 172.21.15.37:0/3376869610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce4102ba0 0x7f9ce40749d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9cd000e9a0 tx=0x7f9cd000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:29.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.339+0000 7f9ce0c00700 1 -- 172.21.15.37:0/3376869610 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd000ca10 con 0x7f9ce4102ba0 2024-03-28T18:07:29.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.339+0000 7f9ceae00700 1 -- 172.21.15.37:0/3376869610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce41ab620 con 0x7f9ce4102ba0 2024-03-28T18:07:29.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.339+0000 7f9ce0c00700 1 -- 172.21.15.37:0/3376869610 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9cd000cbb0 con 0x7f9ce4102ba0 2024-03-28T18:07:29.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.339+0000 7f9ce0c00700 1 -- 172.21.15.37:0/3376869610 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0018680 con 0x7f9ce4102ba0 2024-03-28T18:07:29.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.339+0000 7f9ceae00700 1 -- 172.21.15.37:0/3376869610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce41abb60 con 0x7f9ce4102ba0 2024-03-28T18:07:29.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.340+0000 7f9ceae00700 1 -- 172.21.15.37:0/3376869610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce41038d0 con 0x7f9ce4102ba0 2024-03-28T18:07:29.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.342+0000 7f9ce0c00700 1 -- 172.21.15.37:0/3376869610 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9cd0010430 con 0x7f9ce4102ba0 2024-03-28T18:07:29.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.343+0000 7f9ce0c00700 1 --2- 172.21.15.37:0/3376869610 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9cc80a2890 0x7f9cc80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:29.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.344+0000 7f9ce0c00700 1 -- 172.21.15.37:0/3376869610 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9cd00ce8a0 con 0x7f9ce4102ba0 2024-03-28T18:07:29.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.344+0000 7f9ce3e00700 1 --2- 172.21.15.37:0/3376869610 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9cc80a2890 0x7f9cc80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:29.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.344+0000 7f9ce3e00700 1 --2- 172.21.15.37:0/3376869610 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9cc80a2890 0x7f9cc80a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4006010 tx=0x7f9cd400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:29.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.348+0000 7f9ce0c00700 1 -- 172.21.15.37:0/3376869610 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9cd009bd40 con 0x7f9ce4102ba0 2024-03-28T18:07:29.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.581+0000 7f9ceae00700 1 -- 172.21.15.37:0/3376869610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9ce402da30 con 0x7f9ce4102ba0 2024-03-28T18:07:29.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.582+0000 7f9ce0c00700 1 -- 172.21.15.37:0/3376869610 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9cd0097030 con 0x7f9ce4102ba0 2024-03-28T18:07:29.583 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 1 2024-03-28T18:07:29.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.585+0000 7f9ceae00700 1 -- 172.21.15.37:0/3376869610 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9cc80a2890 msgr2=0x7f9cc80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:29.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.585+0000 7f9ceae00700 1 --2- 172.21.15.37:0/3376869610 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9cc80a2890 0x7f9cc80a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4006010 tx=0x7f9cd400b260 comp rx=0 tx=0).stop 2024-03-28T18:07:29.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.585+0000 7f9ceae00700 1 -- 172.21.15.37:0/3376869610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce4102ba0 msgr2=0x7f9ce40749d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:29.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.585+0000 7f9ceae00700 1 --2- 172.21.15.37:0/3376869610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce4102ba0 0x7f9ce40749d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9cd000e9a0 tx=0x7f9cd000ecb0 comp rx=0 tx=0).stop 2024-03-28T18:07:29.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.585+0000 7f9ceae00700 1 -- 172.21.15.37:0/3376869610 shutdown_connections 2024-03-28T18:07:29.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.585+0000 7f9ceae00700 1 --2- 172.21.15.37:0/3376869610 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9cc80a2890 0x7f9cc80a4d50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:29.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.585+0000 7f9ceae00700 1 --2- 172.21.15.37:0/3376869610 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ce4108290 0x7f9ce4074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:29.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.585+0000 7f9ceae00700 1 --2- 172.21.15.37:0/3376869610 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ce41021e0 0x7f9ce4074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:29.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.585+0000 7f9ceae00700 1 --2- 172.21.15.37:0/3376869610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce4102ba0 0x7f9ce40749d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:29.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.585+0000 7f9ceae00700 1 -- 172.21.15.37:0/3376869610 >> 172.21.15.37:0/3376869610 conn(0x7f9ce40792c0 msgr2=0x7f9ce4100f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:29.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.585+0000 7f9ceae00700 1 -- 172.21.15.37:0/3376869610 shutdown_connections 2024-03-28T18:07:29.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:29.585+0000 7f9ceae00700 1 -- 172.21.15.37:0/3376869610 wait complete. 2024-03-28T18:07:29.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:29 smithi059 ceph-mon[36190]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:29.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:29 smithi170 ceph-mon[36280]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:29.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:29 smithi037 ceph-mon[31460]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:30.418 INFO:tasks.cephadm:Adding mds.a on smithi037 2024-03-28T18:07:30.418 INFO:tasks.cephadm:Adding mds.d on smithi037 2024-03-28T18:07:30.418 INFO:tasks.cephadm:Adding mds.g on smithi037 2024-03-28T18:07:30.418 INFO:tasks.cephadm:Adding mds.j on smithi037 2024-03-28T18:07:30.418 INFO:tasks.cephadm:Adding mds.b on smithi059 2024-03-28T18:07:30.418 INFO:tasks.cephadm:Adding mds.e on smithi059 2024-03-28T18:07:30.419 INFO:tasks.cephadm:Adding mds.h on smithi059 2024-03-28T18:07:30.419 INFO:tasks.cephadm:Adding mds.k on smithi059 2024-03-28T18:07:30.419 INFO:tasks.cephadm:Adding mds.c on smithi170 2024-03-28T18:07:30.419 INFO:tasks.cephadm:Adding mds.f on smithi170 2024-03-28T18:07:30.419 INFO:tasks.cephadm:Adding mds.i on smithi170 2024-03-28T18:07:30.419 INFO:tasks.cephadm:Adding mds.l on smithi170 2024-03-28T18:07:30.419 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph orch apply mds all '12;smithi037=a;smithi037=d;smithi037=g;smithi037=j;smithi059=b;smithi059=e;smithi059=h;smithi059=k;smithi170=c;smithi170=f;smithi170=i;smithi170=l' 2024-03-28T18:07:30.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:30 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3376869610' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:07:30.657 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:07:30.780 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:30 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3376869610' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:07:30.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:30 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3376869610' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:07:31.686 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:31 smithi059 ceph-mon[36190]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:31.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:31 smithi170 ceph-mon[36280]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:31 smithi037 ceph-mon[31460]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:32.330 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.329+0000 7fa0a86eb700 1 -- 172.21.15.170:0/1370761196 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0a01022c0 msgr2=0x7fa0a0102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:32.330 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.329+0000 7fa0a86eb700 1 --2- 172.21.15.170:0/1370761196 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0a01022c0 0x7fa0a0102720 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa090009990 tx=0x7fa090009ca0 comp rx=0 tx=0).stop 2024-03-28T18:07:32.330 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.330+0000 7fa0a86eb700 1 -- 172.21.15.170:0/1370761196 shutdown_connections 2024-03-28T18:07:32.330 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.330+0000 7fa0a86eb700 1 --2- 172.21.15.170:0/1370761196 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0a0102c60 0x7fa0a0109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:32.330 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.330+0000 7fa0a86eb700 1 --2- 172.21.15.170:0/1370761196 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0a01010c0 0x7fa0a01014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:32.331 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.330+0000 7fa0a86eb700 1 --2- 172.21.15.170:0/1370761196 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0a01022c0 0x7fa0a0102720 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:32.331 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.330+0000 7fa0a86eb700 1 -- 172.21.15.170:0/1370761196 >> 172.21.15.170:0/1370761196 conn(0x7fa0a00fc680 msgr2=0x7fa0a00feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:32.331 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.330+0000 7fa0a86eb700 1 -- 172.21.15.170:0/1370761196 shutdown_connections 2024-03-28T18:07:32.331 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.330+0000 7fa0a86eb700 1 -- 172.21.15.170:0/1370761196 wait complete. 2024-03-28T18:07:32.333 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.330+0000 7fa0a86eb700 1 Processor -- start 2024-03-28T18:07:32.333 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.330+0000 7fa0a86eb700 1 -- start start 2024-03-28T18:07:32.333 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0a86eb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0a01010c0 0x7fa0a006b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:32.333 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0a86eb700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0a01022c0 0x7fa0a019d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:32.333 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0a86eb700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0a0102c60 0x7fa0a019f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:32.333 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0a86eb700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a010cda0 con 0x7fa0a0102c60 2024-03-28T18:07:32.334 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0a86eb700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a010cc20 con 0x7fa0a01010c0 2024-03-28T18:07:32.334 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0a86eb700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a010cf20 con 0x7fa0a01022c0 2024-03-28T18:07:32.335 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0a5c86700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0a01022c0 0x7fa0a019d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:32.335 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0a5c86700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0a01022c0 0x7fa0a019d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:45642/0 (socket says 172.21.15.170:45642) 2024-03-28T18:07:32.335 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0a5c86700 1 -- 172.21.15.170:0/3960755596 learned_addr learned my addr 172.21.15.170:0/3960755596 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T18:07:32.335 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0a6487700 1 --2- 172.21.15.170:0/3960755596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0a01010c0 0x7fa0a006b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:32.335 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0a5c86700 1 -- 172.21.15.170:0/3960755596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0a01010c0 msgr2=0x7fa0a006b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:32.335 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0a5c86700 1 --2- 172.21.15.170:0/3960755596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0a01010c0 0x7fa0a006b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:32.335 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0a5c86700 1 -- 172.21.15.170:0/3960755596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0a0102c60 msgr2=0x7fa0a019f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:32.335 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0a5c86700 1 --2- 172.21.15.170:0/3960755596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0a0102c60 0x7fa0a019f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:32.335 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0a5c86700 1 -- 172.21.15.170:0/3960755596 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a01a0180 con 0x7fa0a01022c0 2024-03-28T18:07:32.335 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0a5c86700 1 --2- 172.21.15.170:0/3960755596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0a01022c0 0x7fa0a019d260 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa0900098e0 tx=0x7fa090009ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:32.335 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0977fe700 1 -- 172.21.15.170:0/3960755596 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa090027070 con 0x7fa0a01022c0 2024-03-28T18:07:32.335 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0977fe700 1 -- 172.21.15.170:0/3960755596 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa090004150 con 0x7fa0a01022c0 2024-03-28T18:07:32.335 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0a86eb700 1 -- 172.21.15.170:0/3960755596 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a01a0410 con 0x7fa0a01022c0 2024-03-28T18:07:32.335 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0977fe700 1 -- 172.21.15.170:0/3960755596 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09000f6c0 con 0x7fa0a01022c0 2024-03-28T18:07:32.336 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.331+0000 7fa0a86eb700 1 -- 172.21.15.170:0/3960755596 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a01a0940 con 0x7fa0a01022c0 2024-03-28T18:07:32.336 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.332+0000 7fa0a86eb700 1 -- 172.21.15.170:0/3960755596 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a0068e50 con 0x7fa0a01022c0 2024-03-28T18:07:32.336 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.333+0000 7fa0977fe700 1 -- 172.21.15.170:0/3960755596 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa09002a070 con 0x7fa0a01022c0 2024-03-28T18:07:32.336 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.334+0000 7fa0977fe700 1 --2- 172.21.15.170:0/3960755596 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa08c0a2910 0x7fa08c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:32.337 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.334+0000 7fa0977fe700 1 -- 172.21.15.170:0/3960755596 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa0900cf6c0 con 0x7fa0a01022c0 2024-03-28T18:07:32.337 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.335+0000 7fa0a6487700 1 --2- 172.21.15.170:0/3960755596 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa08c0a2910 0x7fa08c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:32.337 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.336+0000 7fa0a6487700 1 --2- 172.21.15.170:0/3960755596 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa08c0a2910 0x7fa08c0a4dd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa09c006fd0 tx=0x7fa09c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:32.339 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.339+0000 7fa0977fe700 1 -- 172.21.15.170:0/3960755596 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa09005e740 con 0x7fa0a01022c0 2024-03-28T18:07:32.545 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.544+0000 7fa0a86eb700 1 -- 172.21.15.170:0/3960755596 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi037=a;smithi037=d;smithi037=g;smithi037=j;smithi059=b;smithi059=e;smithi059=h;smithi059=k;smithi170=c;smithi170=f;smithi170=i;smithi170=l", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0a00626d0 con 0x7fa08c0a2910 2024-03-28T18:07:32.561 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.561+0000 7fa0977fe700 1 -- 172.21.15.170:0/3960755596 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fa0a00626d0 con 0x7fa08c0a2910 2024-03-28T18:07:32.561 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled mds.all update... 2024-03-28T18:07:32.564 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.563+0000 7fa0a86eb700 1 -- 172.21.15.170:0/3960755596 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa08c0a2910 msgr2=0x7fa08c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:32.564 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.563+0000 7fa0a86eb700 1 --2- 172.21.15.170:0/3960755596 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa08c0a2910 0x7fa08c0a4dd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa09c006fd0 tx=0x7fa09c008040 comp rx=0 tx=0).stop 2024-03-28T18:07:32.564 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.563+0000 7fa0a86eb700 1 -- 172.21.15.170:0/3960755596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0a01022c0 msgr2=0x7fa0a019d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:32.564 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.563+0000 7fa0a86eb700 1 --2- 172.21.15.170:0/3960755596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0a01022c0 0x7fa0a019d260 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa0900098e0 tx=0x7fa090009ee0 comp rx=0 tx=0).stop 2024-03-28T18:07:32.564 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.563+0000 7fa0a86eb700 1 -- 172.21.15.170:0/3960755596 shutdown_connections 2024-03-28T18:07:32.564 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.563+0000 7fa0a86eb700 1 --2- 172.21.15.170:0/3960755596 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa08c0a2910 0x7fa08c0a4dd0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:32.564 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.563+0000 7fa0a86eb700 1 --2- 172.21.15.170:0/3960755596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0a0102c60 0x7fa0a019f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:32.564 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.563+0000 7fa0a86eb700 1 --2- 172.21.15.170:0/3960755596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0a01010c0 0x7fa0a006b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:32.564 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.563+0000 7fa0a86eb700 1 --2- 172.21.15.170:0/3960755596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0a01022c0 0x7fa0a019d260 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:32.564 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.563+0000 7fa0a86eb700 1 -- 172.21.15.170:0/3960755596 >> 172.21.15.170:0/3960755596 conn(0x7fa0a00fc680 msgr2=0x7fa0a01056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:32.564 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.563+0000 7fa0a86eb700 1 -- 172.21.15.170:0/3960755596 shutdown_connections 2024-03-28T18:07:32.564 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T18:07:32.563+0000 7fa0a86eb700 1 -- 172.21.15.170:0/3960755596 wait complete. 2024-03-28T18:07:33.303 DEBUG:teuthology.orchestra.run.smithi037:mds.a> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.a.service 2024-03-28T18:07:33.308 DEBUG:teuthology.orchestra.run.smithi037:mds.d> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service 2024-03-28T18:07:33.352 DEBUG:teuthology.orchestra.run.smithi037:mds.g> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service 2024-03-28T18:07:33.397 DEBUG:teuthology.orchestra.run.smithi037:mds.j> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service 2024-03-28T18:07:33.442 DEBUG:teuthology.orchestra.run.smithi059:mds.b> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.b.service 2024-03-28T18:07:33.445 DEBUG:teuthology.orchestra.run.smithi059:mds.e> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service 2024-03-28T18:07:33.488 DEBUG:teuthology.orchestra.run.smithi059:mds.h> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.h.service 2024-03-28T18:07:33.530 DEBUG:teuthology.orchestra.run.smithi059:mds.k> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service 2024-03-28T18:07:33.574 DEBUG:teuthology.orchestra.run.smithi170:mds.c> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.c.service 2024-03-28T18:07:33.577 DEBUG:teuthology.orchestra.run.smithi170:mds.f> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service 2024-03-28T18:07:33.619 DEBUG:teuthology.orchestra.run.smithi170:mds.i> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.i.service 2024-03-28T18:07:33.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:33 smithi037 ceph-mon[31460]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:33.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:33 smithi037 ceph-mon[31460]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi037=a;smithi037=d;smithi037=g;smithi037=j;smithi059=b;smithi059=e;smithi059=h;smithi059=k;smithi170=c;smithi170=f;smithi170=i;smithi170=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:07:33.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:33 smithi037 ceph-mon[31460]: Saving service mds.all spec with placement smithi037=a;smithi037=d;smithi037=g;smithi037=j;smithi059=b;smithi059=e;smithi059=h;smithi059=k;smithi170=c;smithi170=f;smithi170=i;smithi170=l;count:12 2024-03-28T18:07:33.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:33.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:33 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:07:33.662 DEBUG:teuthology.orchestra.run.smithi170:mds.l> sudo journalctl -f -n 0 -u ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service 2024-03-28T18:07:33.705 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-28T18:07:33.705 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:07:33.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:33 smithi059 ceph-mon[36190]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:33.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:33 smithi059 ceph-mon[36190]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi037=a;smithi037=d;smithi037=g;smithi037=j;smithi059=b;smithi059=e;smithi059=h;smithi059=k;smithi170=c;smithi170=f;smithi170=i;smithi170=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:07:33.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:33 smithi059 ceph-mon[36190]: Saving service mds.all spec with placement smithi037=a;smithi037=d;smithi037=g;smithi037=j;smithi059=b;smithi059=e;smithi059=h;smithi059=k;smithi170=c;smithi170=f;smithi170=i;smithi170=l;count:12 2024-03-28T18:07:33.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:33 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:33.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:33 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:07:33.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.944+0000 7f5672600640 1 -- 172.21.15.37:0/4013877664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f566c1016c0 msgr2=0x7f566c101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:33.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.944+0000 7f5672600640 1 --2- 172.21.15.37:0/4013877664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f566c1016c0 0x7f566c101ae0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f565c009900 tx=0x7f565c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:07:33.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.944+0000 7f5672600640 1 -- 172.21.15.37:0/4013877664 shutdown_connections 2024-03-28T18:07:33.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.944+0000 7f5672600640 1 --2- 172.21.15.37:0/4013877664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f566c1028b0 0x7f566c102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:33.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.944+0000 7f5672600640 1 --2- 172.21.15.37:0/4013877664 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f566c103270 0x7f566c109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:33.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.944+0000 7f5672600640 1 --2- 172.21.15.37:0/4013877664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f566c1016c0 0x7f566c101ae0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:33.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.944+0000 7f5672600640 1 -- 172.21.15.37:0/4013877664 >> 172.21.15.37:0/4013877664 conn(0x7f566c0fcc50 msgr2=0x7f566c0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:33.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.945+0000 7f5672600640 1 -- 172.21.15.37:0/4013877664 shutdown_connections 2024-03-28T18:07:33.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.945+0000 7f5672600640 1 -- 172.21.15.37:0/4013877664 wait complete. 2024-03-28T18:07:33.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.946+0000 7f5672600640 1 Processor -- start 2024-03-28T18:07:33.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.946+0000 7f5672600640 1 -- start start 2024-03-28T18:07:33.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.946+0000 7f5672600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f566c1016c0 0x7f566c199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:33.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.946+0000 7f5672600640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f566c1028b0 0x7f566c19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:33.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.946+0000 7f5672600640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f566c103270 0x7f566c1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:33.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.946+0000 7f5672600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c10d360 con 0x7f566c1016c0 2024-03-28T18:07:33.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.946+0000 7f5672600640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c10d1e0 con 0x7f566c1028b0 2024-03-28T18:07:33.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.946+0000 7f5672600640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c10d4e0 con 0x7f566c103270 2024-03-28T18:07:33.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.947+0000 7f566b400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f566c1016c0 0x7f566c199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:33.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.947+0000 7f566aa00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f566c1028b0 0x7f566c19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:33.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.947+0000 7f566be00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f566c103270 0x7f566c1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:33.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.947+0000 7f566aa00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f566c1028b0 0x7f566c19a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:46460/0 (socket says 172.21.15.37:46460) 2024-03-28T18:07:33.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.947+0000 7f566be00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f566c103270 0x7f566c1a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:43104/0 (socket says 172.21.15.37:43104) 2024-03-28T18:07:33.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.947+0000 7f566b400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f566c1016c0 0x7f566c199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51784/0 (socket says 172.21.15.37:51784) 2024-03-28T18:07:33.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.947+0000 7f566aa00640 1 -- 172.21.15.37:0/3249840153 learned_addr learned my addr 172.21.15.37:0/3249840153 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:07:33.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.947+0000 7f566aa00640 1 -- 172.21.15.37:0/3249840153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f566c103270 msgr2=0x7f566c1a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:33.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.947+0000 7f566aa00640 1 --2- 172.21.15.37:0/3249840153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f566c103270 0x7f566c1a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:33.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.947+0000 7f566aa00640 1 -- 172.21.15.37:0/3249840153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f566c1016c0 msgr2=0x7f566c199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:33.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.947+0000 7f566aa00640 1 --2- 172.21.15.37:0/3249840153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f566c1016c0 0x7f566c199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:33.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.947+0000 7f566aa00640 1 -- 172.21.15.37:0/3249840153 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566c1a1810 con 0x7f566c1028b0 2024-03-28T18:07:33.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.947+0000 7f566aa00640 1 --2- 172.21.15.37:0/3249840153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f566c1028b0 0x7f566c19a130 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f565800ea60 tx=0x7f565800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:33.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.948+0000 7f5663e00640 1 -- 172.21.15.37:0/3249840153 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565800cde0 con 0x7f566c1028b0 2024-03-28T18:07:33.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.948+0000 7f5663e00640 1 -- 172.21.15.37:0/3249840153 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56580035f0 con 0x7f566c1028b0 2024-03-28T18:07:33.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.948+0000 7f5663e00640 1 -- 172.21.15.37:0/3249840153 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5658010670 con 0x7f566c1028b0 2024-03-28T18:07:33.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.948+0000 7f5672600640 1 -- 172.21.15.37:0/3249840153 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f566c1a1b00 con 0x7f566c1028b0 2024-03-28T18:07:33.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.948+0000 7f5672600640 1 -- 172.21.15.37:0/3249840153 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f566c1a2050 con 0x7f566c1028b0 2024-03-28T18:07:33.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.949+0000 7f5672600640 1 -- 172.21.15.37:0/3249840153 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f566c1a2280 con 0x7f566c1028b0 2024-03-28T18:07:33.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.951+0000 7f5663e00640 1 -- 172.21.15.37:0/3249840153 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f56580037a0 con 0x7f566c1028b0 2024-03-28T18:07:33.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.953+0000 7f5663e00640 1 --2- 172.21.15.37:0/3249840153 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56500a2800 0x7f56500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:33.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.953+0000 7f5663e00640 1 -- 172.21.15.37:0/3249840153 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f56580ce940 con 0x7f566c1028b0 2024-03-28T18:07:33.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.953+0000 7f566b400640 1 --2- 172.21.15.37:0/3249840153 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56500a2800 0x7f56500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:33.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.954+0000 7f566b400640 1 --2- 172.21.15.37:0/3249840153 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56500a2800 0x7f56500a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f565c02c170 tx=0x7f565c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:33.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:33.957+0000 7f5663e00640 1 -- 172.21.15.37:0/3249840153 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f565809bde0 con 0x7f566c1028b0 2024-03-28T18:07:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:33 smithi170 ceph-mon[36280]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:33 smithi170 ceph-mon[36280]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi037=a;smithi037=d;smithi037=g;smithi037=j;smithi059=b;smithi059=e;smithi059=h;smithi059=k;smithi170=c;smithi170=f;smithi170=i;smithi170=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:07:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:33 smithi170 ceph-mon[36280]: Saving service mds.all spec with placement smithi037=a;smithi037=d;smithi037=g;smithi037=j;smithi059=b;smithi059=e;smithi059=h;smithi059=k;smithi170=c;smithi170=f;smithi170=i;smithi170=l;count:12 2024-03-28T18:07:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:33 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:33 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:07:34.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.162+0000 7f5672600640 1 -- 172.21.15.37:0/3249840153 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f566c1a2450 con 0x7f566c1028b0 2024-03-28T18:07:34.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.164+0000 7f5663e00640 1 -- 172.21.15.37:0/3249840153 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18433 (secure 0 0 0) 0x7f565809b6e0 con 0x7f566c1028b0 2024-03-28T18:07:34.164 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:07:34.164 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":81,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","created":"2024-03-28T17:59:05.949847+0000","modified":"2024-03-28T18:07:22.031148+0000","last_up_change":"2024-03-28T18:07:21.021030+0000","last_in_change":"2024-03-28T18:07:01.416192+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:03:00.517967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db60618d-56f7-4130-8a90-84cbc480965b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6803","nonce":3562602035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6805","nonce":3562602035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6809","nonce":3562602035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6807","nonce":3562602035}]},"public_addr":"172.21.15.37:6803/3562602035","cluster_addr":"172.21.15.37:6805/3562602035","heartbeat_back_addr":"172.21.15.37:6809/3562602035","heartbeat_front_addr":"172.21.15.37:6807/3562602035","state":["exists","up"]},{"osd":1,"uuid":"7a4b7ad6-905e-4258-9574-693d4277dfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6801","nonce":3326537211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6803","nonce":3326537211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6807","nonce":3326537211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6805","nonce":3326537211}]},"public_addr":"172.21.15.59:6801/3326537211","cluster_addr":"172.21.15.59:6803/3326537211","heartbeat_back_addr":"172.21.15.59:6807/3326537211","heartbeat_front_addr":"172.21.15.59:6805/3326537211","state":["exists","up"]},{"osd":2,"uuid":"8b76ec1c-46bf-4336-89b5-12d82906e8b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6801","nonce":1269398427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6803","nonce":1269398427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6807","nonce":1269398427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6805","nonce":1269398427}]},"public_addr":"172.21.15.170:6801/1269398427","cluster_addr":"172.21.15.170:6803/1269398427","heartbeat_back_addr":"172.21.15.170:6807/1269398427","heartbeat_front_addr":"172.21.15.170:6805/1269398427","state":["exists","up"]},{"osd":3,"uuid":"66eca1da-2fbd-44e1-8bbf-5635a2c70ad5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6811","nonce":317805240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6813","nonce":317805240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6817","nonce":317805240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6815","nonce":317805240}]},"public_addr":"172.21.15.37:6811/317805240","cluster_addr":"172.21.15.37:6813/317805240","heartbeat_back_addr":"172.21.15.37:6817/317805240","heartbeat_front_addr":"172.21.15.37:6815/317805240","state":["exists","up"]},{"osd":4,"uuid":"4c877335-e42d-4d8e-9f48-a1d0a89ae438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6809","nonce":2410632623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6811","nonce":2410632623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6815","nonce":2410632623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6813","nonce":2410632623}]},"public_addr":"172.21.15.59:6809/2410632623","cluster_addr":"172.21.15.59:6811/2410632623","heartbeat_back_addr":"172.21.15.59:6815/2410632623","heartbeat_front_addr":"172.21.15.59:6813/2410632623","state":["exists","up"]},{"osd":5,"uuid":"b884842a-86e2-455b-9570-ff163b3bc0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6809","nonce":1218943899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6811","nonce":1218943899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6815","nonce":1218943899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6813","nonce":1218943899}]},"public_addr":"172.21.15.170:6809/1218943899","cluster_addr":"172.21.15.170:6811/1218943899","heartbeat_back_addr":"172.21.15.170:6815/1218943899","heartbeat_front_addr":"172.21.15.170:6813/1218943899","state":["exists","up"]},{"osd":6,"uuid":"249a1200-77f4-44ad-981f-1d352f732b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6819","nonce":1899652046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6821","nonce":1899652046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6825","nonce":1899652046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6823","nonce":1899652046}]},"public_addr":"172.21.15.37:6819/1899652046","cluster_addr":"172.21.15.37:6821/1899652046","heartbeat_back_addr":"172.21.15.37:6825/1899652046","heartbeat_front_addr":"172.21.15.37:6823/1899652046","state":["exists","up"]},{"osd":7,"uuid":"c73338e6-f505-4517-a84b-f35e04750a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6817","nonce":1343675690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6819","nonce":1343675690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6823","nonce":1343675690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6821","nonce":1343675690}]},"public_addr":"172.21.15.59:6817/1343675690","cluster_addr":"172.21.15.59:6819/1343675690","heartbeat_back_addr":"172.21.15.59:6823/1343675690","heartbeat_front_addr":"172.21.15.59:6821/1343675690","state":["exists","up"]},{"osd":8,"uuid":"149b8297-ff2d-4a88-9b1e-0def021cc516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6817","nonce":3235794556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6819","nonce":3235794556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6823","nonce":3235794556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6821","nonce":3235794556}]},"public_addr":"172.21.15.170:6817/3235794556","cluster_addr":"172.21.15.170:6819/3235794556","heartbeat_back_addr":"172.21.15.170:6823/3235794556","heartbeat_front_addr":"172.21.15.170:6821/3235794556","state":["exists","up"]},{"osd":9,"uuid":"55c27cc2-c771-44b7-90fe-8f0feca44162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6827","nonce":2307153008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6829","nonce":2307153008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6833","nonce":2307153008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6831","nonce":2307153008}]},"public_addr":"172.21.15.37:6827/2307153008","cluster_addr":"172.21.15.37:6829/2307153008","heartbeat_back_addr":"172.21.15.37:6833/2307153008","heartbeat_front_addr":"172.21.15.37:6831/2307153008","state":["exists","up"]},{"osd":10,"uuid":"c0595135-ab81-49d3-8c23-19f960c03e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6825","nonce":2113830275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6827","nonce":2113830275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6831","nonce":2113830275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6829","nonce":2113830275}]},"public_addr":"172.21.15.59:6825/2113830275","cluster_addr":"172.21.15.59:6827/2113830275","heartbeat_back_addr":"172.21.15.59:6831/2113830275","heartbeat_front_addr":"172.21.15.59:6829/2113830275","state":["exists","up"]},{"osd":11,"uuid":"5871e177-c17b-4637-a797-4fa5125920dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6825","nonce":3082439360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6827","nonce":3082439360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6831","nonce":3082439360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6829","nonce":3082439360}]},"public_addr":"172.21.15.170:6825/3082439360","cluster_addr":"172.21.15.170:6827/3082439360","heartbeat_back_addr":"172.21.15.170:6831/3082439360","heartbeat_front_addr":"172.21.15.170:6829/3082439360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:07.690545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:33.183396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:57.400263+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:30.597211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:57.259044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:22.627823+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:57.585863+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:25.000476+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:50.876688+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:25.859761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:53.563146+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:07:19.304115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3023379428":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1443710231":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6800/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6801/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6801/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6800/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1007625490":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/2895698584":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/497313034":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/3293943475":"2024-03-29T17:59:39.162038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:07:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.167+0000 7f5672600640 1 -- 172.21.15.37:0/3249840153 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56500a2800 msgr2=0x7f56500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.167+0000 7f5672600640 1 --2- 172.21.15.37:0/3249840153 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56500a2800 0x7f56500a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f565c02c170 tx=0x7f565c02df90 comp rx=0 tx=0).stop 2024-03-28T18:07:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.167+0000 7f5672600640 1 -- 172.21.15.37:0/3249840153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f566c1028b0 msgr2=0x7f566c19a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.167+0000 7f5672600640 1 --2- 172.21.15.37:0/3249840153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f566c1028b0 0x7f566c19a130 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f565800ea60 tx=0x7f565800ef20 comp rx=0 tx=0).stop 2024-03-28T18:07:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.167+0000 7f5672600640 1 -- 172.21.15.37:0/3249840153 shutdown_connections 2024-03-28T18:07:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.167+0000 7f5672600640 1 --2- 172.21.15.37:0/3249840153 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56500a2800 0x7f56500a4cc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.167+0000 7f5672600640 1 --2- 172.21.15.37:0/3249840153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f566c1028b0 0x7f566c19a130 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.167+0000 7f5672600640 1 --2- 172.21.15.37:0/3249840153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f566c103270 0x7f566c1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.167+0000 7f5672600640 1 --2- 172.21.15.37:0/3249840153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f566c1016c0 0x7f566c199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.167+0000 7f5672600640 1 -- 172.21.15.37:0/3249840153 >> 172.21.15.37:0/3249840153 conn(0x7f566c0fcc50 msgr2=0x7f566c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.167+0000 7f5672600640 1 -- 172.21.15.37:0/3249840153 shutdown_connections 2024-03-28T18:07:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.167+0000 7f5672600640 1 -- 172.21.15.37:0/3249840153 wait complete. 2024-03-28T18:07:34.189 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:03:00.517967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:07:34.189 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:07:34.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.403+0000 7fefcf000640 1 -- 172.21.15.37:0/2347463707 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefc8105370 msgr2=0x7fefc810bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:34.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.403+0000 7fefcf000640 1 --2- 172.21.15.37:0/2347463707 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefc8105370 0x7fefc810bc00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fefc0009be0 tx=0x7fefc002bef0 comp rx=0 tx=0).stop 2024-03-28T18:07:34.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.403+0000 7fefcf000640 1 -- 172.21.15.37:0/2347463707 shutdown_connections 2024-03-28T18:07:34.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.403+0000 7fefcf000640 1 --2- 172.21.15.37:0/2347463707 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefc8105370 0x7fefc810bc00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:34.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.403+0000 7fefcf000640 1 --2- 172.21.15.37:0/2347463707 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefc81049b0 0x7fefc8104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:34.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.403+0000 7fefcf000640 1 --2- 172.21.15.37:0/2347463707 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fefc81037c0 0x7fefc8103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:34.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.403+0000 7fefcf000640 1 -- 172.21.15.37:0/2347463707 >> 172.21.15.37:0/2347463707 conn(0x7fefc80fed50 msgr2=0x7fefc8101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:34.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.403+0000 7fefcf000640 1 -- 172.21.15.37:0/2347463707 shutdown_connections 2024-03-28T18:07:34.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.403+0000 7fefcf000640 1 -- 172.21.15.37:0/2347463707 wait complete. 2024-03-28T18:07:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.404+0000 7fefcf000640 1 Processor -- start 2024-03-28T18:07:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.404+0000 7fefcf000640 1 -- start start 2024-03-28T18:07:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.404+0000 7fefcf000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefc81037c0 0x7fefc8199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.405+0000 7fefcf000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefc81049b0 0x7fefc8199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.405+0000 7fefcf000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fefc8105370 0x7fefc81a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.405+0000 7fefcf000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc810f470 con 0x7fefc8105370 2024-03-28T18:07:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.405+0000 7fefcf000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc810f2f0 con 0x7fefc81049b0 2024-03-28T18:07:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.405+0000 7fefcf000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc810f5f0 con 0x7fefc81037c0 2024-03-28T18:07:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.405+0000 7fefcd200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefc81049b0 0x7fefc8199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.405+0000 7fefcd200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefc81049b0 0x7fefc8199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:46474/0 (socket says 172.21.15.37:46474) 2024-03-28T18:07:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.405+0000 7fefcd200640 1 -- 172.21.15.37:0/2866465996 learned_addr learned my addr 172.21.15.37:0/2866465996 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:07:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.405+0000 7fefcdc00640 1 --2- 172.21.15.37:0/2866465996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefc81037c0 0x7fefc8199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.405+0000 7fefce600640 1 --2- 172.21.15.37:0/2866465996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fefc8105370 0x7fefc81a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.405+0000 7fefcd200640 1 -- 172.21.15.37:0/2866465996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefc81037c0 msgr2=0x7fefc8199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.405+0000 7fefcd200640 1 --2- 172.21.15.37:0/2866465996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefc81037c0 0x7fefc8199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.405+0000 7fefcd200640 1 -- 172.21.15.37:0/2866465996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fefc8105370 msgr2=0x7fefc81a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.405+0000 7fefcd200640 1 --2- 172.21.15.37:0/2866465996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fefc8105370 0x7fefc81a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.405+0000 7fefcd200640 1 -- 172.21.15.37:0/2866465996 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc81a1640 con 0x7fefc81049b0 2024-03-28T18:07:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.406+0000 7fefcd200640 1 --2- 172.21.15.37:0/2866465996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefc81049b0 0x7fefc8199f60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fefb400e9f0 tx=0x7fefb400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.406+0000 7fefc6000640 1 -- 172.21.15.37:0/2866465996 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb400ccc0 con 0x7fefc81049b0 2024-03-28T18:07:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.406+0000 7fefc6000640 1 -- 172.21.15.37:0/2866465996 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fefb4010910 con 0x7fefc81049b0 2024-03-28T18:07:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.406+0000 7fefcf000640 1 -- 172.21.15.37:0/2866465996 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc81a1930 con 0x7fefc81049b0 2024-03-28T18:07:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.406+0000 7fefc6000640 1 -- 172.21.15.37:0/2866465996 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb40188a0 con 0x7fefc81049b0 2024-03-28T18:07:34.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.406+0000 7fefcf000640 1 -- 172.21.15.37:0/2866465996 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc81a1ed0 con 0x7fefc81049b0 2024-03-28T18:07:34.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.407+0000 7fefcf000640 1 -- 172.21.15.37:0/2866465996 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc810aa10 con 0x7fefc81049b0 2024-03-28T18:07:34.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.409+0000 7fefc6000640 1 -- 172.21.15.37:0/2866465996 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fefb4010ac0 con 0x7fefc81049b0 2024-03-28T18:07:34.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.410+0000 7fefc6000640 1 --2- 172.21.15.37:0/2866465996 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fefb00a2880 0x7fefb00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:34.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.410+0000 7fefc6000640 1 -- 172.21.15.37:0/2866465996 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fefb40cea00 con 0x7fefc81049b0 2024-03-28T18:07:34.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.411+0000 7fefcdc00640 1 --2- 172.21.15.37:0/2866465996 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fefb00a2880 0x7fefb00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:34.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.411+0000 7fefcdc00640 1 --2- 172.21.15.37:0/2866465996 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fefb00a2880 0x7fefb00a4d40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fefc8104810 tx=0x7fefbc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:34.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.414+0000 7fefc6000640 1 -- 172.21.15.37:0/2866465996 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fefb409bea0 con 0x7fefc81049b0 2024-03-28T18:07:34.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.620+0000 7fefcf000640 1 -- 172.21.15.37:0/2866465996 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fefc80622a0 con 0x7fefc81049b0 2024-03-28T18:07:34.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.621+0000 7fefc6000640 1 -- 172.21.15.37:0/2866465996 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7fefb409b7a0 con 0x7fefc81049b0 2024-03-28T18:07:34.621 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 1 2024-03-28T18:07:34.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.624+0000 7fefcf000640 1 -- 172.21.15.37:0/2866465996 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fefb00a2880 msgr2=0x7fefb00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:34.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.624+0000 7fefcf000640 1 --2- 172.21.15.37:0/2866465996 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fefb00a2880 0x7fefb00a4d40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fefc8104810 tx=0x7fefbc009260 comp rx=0 tx=0).stop 2024-03-28T18:07:34.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.624+0000 7fefcf000640 1 -- 172.21.15.37:0/2866465996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefc81049b0 msgr2=0x7fefc8199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:34.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.624+0000 7fefcf000640 1 --2- 172.21.15.37:0/2866465996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefc81049b0 0x7fefc8199f60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fefb400e9f0 tx=0x7fefb400edf0 comp rx=0 tx=0).stop 2024-03-28T18:07:34.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.624+0000 7fefcf000640 1 -- 172.21.15.37:0/2866465996 shutdown_connections 2024-03-28T18:07:34.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.624+0000 7fefcf000640 1 --2- 172.21.15.37:0/2866465996 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fefb00a2880 0x7fefb00a4d40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:34.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.624+0000 7fefcf000640 1 --2- 172.21.15.37:0/2866465996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefc81049b0 0x7fefc8199f60 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:34.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.624+0000 7fefcf000640 1 --2- 172.21.15.37:0/2866465996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fefc8105370 0x7fefc81a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:34.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.624+0000 7fefcf000640 1 --2- 172.21.15.37:0/2866465996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefc81037c0 0x7fefc8199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:34.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.624+0000 7fefcf000640 1 -- 172.21.15.37:0/2866465996 >> 172.21.15.37:0/2866465996 conn(0x7fefc80fed50 msgr2=0x7fefc8107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:34.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.625+0000 7fefcf000640 1 -- 172.21.15.37:0/2866465996 shutdown_connections 2024-03-28T18:07:34.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.625+0000 7fefcf000640 1 -- 172.21.15.37:0/2866465996 wait complete. 2024-03-28T18:07:34.644 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:07:34.753 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:34 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:34.753 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:34 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:07:34.753 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:34 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:34.753 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:34 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:34.753 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:34 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:34.753 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:34 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:34.753 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:34 smithi170 ceph-mon[36280]: Deploying daemon mds.l on smithi170 2024-03-28T18:07:34.753 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:34 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3249840153' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:07:34.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.868+0000 7efc19400640 1 -- 172.21.15.37:0/168946433 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc14107ce0 msgr2=0x7efc14108100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:34.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.868+0000 7efc19400640 1 --2- 172.21.15.37:0/168946433 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc14107ce0 0x7efc14108100 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efc04009900 tx=0x7efc0402bc30 comp rx=0 tx=0).stop 2024-03-28T18:07:34.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.868+0000 7efc19400640 1 -- 172.21.15.37:0/168946433 shutdown_connections 2024-03-28T18:07:34.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.868+0000 7efc19400640 1 --2- 172.21.15.37:0/168946433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc14109890 0x7efc14110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:34.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.868+0000 7efc19400640 1 --2- 172.21.15.37:0/168946433 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc14108ed0 0x7efc14109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:34.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.868+0000 7efc19400640 1 --2- 172.21.15.37:0/168946433 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc14107ce0 0x7efc14108100 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:34.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.868+0000 7efc19400640 1 -- 172.21.15.37:0/168946433 >> 172.21.15.37:0/168946433 conn(0x7efc14076cb0 msgr2=0x7efc140790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:34.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.868+0000 7efc19400640 1 -- 172.21.15.37:0/168946433 shutdown_connections 2024-03-28T18:07:34.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.868+0000 7efc19400640 1 -- 172.21.15.37:0/168946433 wait complete. 2024-03-28T18:07:34.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.869+0000 7efc19400640 1 Processor -- start 2024-03-28T18:07:34.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.869+0000 7efc19400640 1 -- start start 2024-03-28T18:07:34.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.869+0000 7efc19400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc14107ce0 0x7efc1419e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:34.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.869+0000 7efc19400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc14108ed0 0x7efc1419e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:34.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.869+0000 7efc19400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc14109890 0x7efc141a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:34.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.869+0000 7efc19400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc14113950 con 0x7efc14108ed0 2024-03-28T18:07:34.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.869+0000 7efc19400640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc141137d0 con 0x7efc14109890 2024-03-28T18:07:34.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.869+0000 7efc19400640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc14113ad0 con 0x7efc14107ce0 2024-03-28T18:07:34.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.870+0000 7efc11e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc14108ed0 0x7efc1419e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:34.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.870+0000 7efc12800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc14107ce0 0x7efc1419e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:34.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.870+0000 7efc11e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc14108ed0 0x7efc1419e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51808/0 (socket says 172.21.15.37:51808) 2024-03-28T18:07:34.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.870+0000 7efc11e00640 1 -- 172.21.15.37:0/777801586 learned_addr learned my addr 172.21.15.37:0/777801586 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:07:34.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.870+0000 7efc13200640 1 --2- 172.21.15.37:0/777801586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc14109890 0x7efc141a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:34.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.870+0000 7efc11e00640 1 -- 172.21.15.37:0/777801586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc14107ce0 msgr2=0x7efc1419e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:34.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.870+0000 7efc11e00640 1 --2- 172.21.15.37:0/777801586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc14107ce0 0x7efc1419e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:34.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.870+0000 7efc11e00640 1 -- 172.21.15.37:0/777801586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc14109890 msgr2=0x7efc141a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:34.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.870+0000 7efc11e00640 1 --2- 172.21.15.37:0/777801586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc14109890 0x7efc141a55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:34.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.870+0000 7efc11e00640 1 -- 172.21.15.37:0/777801586 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc141a5c30 con 0x7efc14108ed0 2024-03-28T18:07:34.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.870+0000 7efc12800640 1 --2- 172.21.15.37:0/777801586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc14107ce0 0x7efc1419e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:07:34.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.870+0000 7efc11e00640 1 --2- 172.21.15.37:0/777801586 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc14108ed0 0x7efc1419e550 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7efc0000e9f0 tx=0x7efc0000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:34.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.871+0000 7efc0b400640 1 -- 172.21.15.37:0/777801586 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0000cd80 con 0x7efc14108ed0 2024-03-28T18:07:34.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.871+0000 7efc0b400640 1 -- 172.21.15.37:0/777801586 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc00005500 con 0x7efc14108ed0 2024-03-28T18:07:34.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.871+0000 7efc0b400640 1 -- 172.21.15.37:0/777801586 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc00010690 con 0x7efc14108ed0 2024-03-28T18:07:34.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.871+0000 7efc19400640 1 -- 172.21.15.37:0/777801586 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc141a5f20 con 0x7efc14108ed0 2024-03-28T18:07:34.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.871+0000 7efc19400640 1 -- 172.21.15.37:0/777801586 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc141a63e0 con 0x7efc14108ed0 2024-03-28T18:07:34.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.872+0000 7efc19400640 1 -- 172.21.15.37:0/777801586 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1410f000 con 0x7efc14108ed0 2024-03-28T18:07:34.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:34 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:34.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:34 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:07:34.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:34 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:34.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:34 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:34.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:34 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:34.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:34 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:34.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:34 smithi037 ceph-mon[31460]: Deploying daemon mds.l on smithi170 2024-03-28T18:07:34.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:34 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3249840153' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:07:34.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.875+0000 7efc0b400640 1 -- 172.21.15.37:0/777801586 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7efc000046c0 con 0x7efc14108ed0 2024-03-28T18:07:34.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.876+0000 7efc0b400640 1 --2- 172.21.15.37:0/777801586 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7efbf80a2880 0x7efbf80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:34.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.876+0000 7efc0b400640 1 -- 172.21.15.37:0/777801586 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7efc000ce9d0 con 0x7efc14108ed0 2024-03-28T18:07:34.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.876+0000 7efc12800640 1 --2- 172.21.15.37:0/777801586 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7efbf80a2880 0x7efbf80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:34.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.877+0000 7efc12800640 1 --2- 172.21.15.37:0/777801586 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7efbf80a2880 0x7efbf80a4d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efc04000c00 tx=0x7efc04005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:34.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:34.881+0000 7efc0b400640 1 -- 172.21.15.37:0/777801586 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efc0009be70 con 0x7efc14108ed0 2024-03-28T18:07:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:34 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:34 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:07:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:34 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:34 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:34 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:34 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:34 smithi059 ceph-mon[36190]: Deploying daemon mds.l on smithi170 2024-03-28T18:07:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:34 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3249840153' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:07:35.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:35.128+0000 7efc19400640 1 -- 172.21.15.37:0/777801586 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc1402d070 con 0x7efc14108ed0 2024-03-28T18:07:35.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:35.129+0000 7efc0b400640 1 -- 172.21.15.37:0/777801586 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7efc0009b770 con 0x7efc14108ed0 2024-03-28T18:07:35.130 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:07:35.130 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-28T18:07:35.130 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 1 2024-03-28T18:07:35.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:35.132+0000 7efc19400640 1 -- 172.21.15.37:0/777801586 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7efbf80a2880 msgr2=0x7efbf80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:35.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:35.132+0000 7efc19400640 1 --2- 172.21.15.37:0/777801586 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7efbf80a2880 0x7efbf80a4d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efc04000c00 tx=0x7efc04005fd0 comp rx=0 tx=0).stop 2024-03-28T18:07:35.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:35.132+0000 7efc19400640 1 -- 172.21.15.37:0/777801586 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc14108ed0 msgr2=0x7efc1419e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:35.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:35.132+0000 7efc19400640 1 --2- 172.21.15.37:0/777801586 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc14108ed0 0x7efc1419e550 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7efc0000e9f0 tx=0x7efc0000edf0 comp rx=0 tx=0).stop 2024-03-28T18:07:35.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:35.132+0000 7efc19400640 1 -- 172.21.15.37:0/777801586 shutdown_connections 2024-03-28T18:07:35.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:35.132+0000 7efc19400640 1 --2- 172.21.15.37:0/777801586 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7efbf80a2880 0x7efbf80a4d40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:35.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:35.132+0000 7efc19400640 1 --2- 172.21.15.37:0/777801586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc14109890 0x7efc141a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:35.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:35.132+0000 7efc19400640 1 --2- 172.21.15.37:0/777801586 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc14108ed0 0x7efc1419e550 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:35.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:35.132+0000 7efc19400640 1 --2- 172.21.15.37:0/777801586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc14107ce0 0x7efc1419e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:35.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:35.132+0000 7efc19400640 1 -- 172.21.15.37:0/777801586 >> 172.21.15.37:0/777801586 conn(0x7efc14076cb0 msgr2=0x7efc1410c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:35.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:35.133+0000 7efc19400640 1 -- 172.21.15.37:0/777801586 shutdown_connections 2024-03-28T18:07:35.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:35.133+0000 7efc19400640 1 -- 172.21.15.37:0/777801586 wait complete. 2024-03-28T18:07:35.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:35 smithi170 ceph-mon[36280]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:35.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:35 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2866465996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:07:35.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:35 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/777801586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:35.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:35 smithi037 ceph-mon[31460]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:35.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:35 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2866465996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:07:35.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:35 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/777801586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:35 smithi059 ceph-mon[36190]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:35 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2866465996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:07:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:35 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/777801586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:36.192 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:07:35 smithi170 systemd[1]: Starting Ceph mds.l for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:07:36.463 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:07:36 smithi170 podman[65710]: 2024-03-28 18:07:36.095203469 +0000 UTC m=+0.018177005 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:07:36.463 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:07:36 smithi170 podman[65710]: 2024-03-28 18:07:36.210985726 +0000 UTC m=+0.133959288 container create 8a70934209e5848cca87708a2602393becd1f8addba647cc9bcd632a46ee8c92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:07:36.785 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:07:36 smithi170 podman[65710]: 2024-03-28 18:07:36.463934639 +0000 UTC m=+0.386908182 container init 8a70934209e5848cca87708a2602393becd1f8addba647cc9bcd632a46ee8c92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-28T18:07:36.786 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:07:36 smithi170 podman[65710]: 2024-03-28 18:07:36.466726142 +0000 UTC m=+0.389699686 container start 8a70934209e5848cca87708a2602393becd1f8addba647cc9bcd632a46ee8c92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:07:36.786 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:07:36 smithi170 ceph-mds[65729]: -- 172.21.15.170:0/2338509470 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557c97f97a40 con 0x557c97f6b000 2024-03-28T18:07:36.786 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:07:36 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: starting mds.l at 2024-03-28T18:07:36.786 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:07:36 smithi170 bash[65710]: 8a70934209e5848cca87708a2602393becd1f8addba647cc9bcd632a46ee8c92 2024-03-28T18:07:36.786 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:07:36 smithi170 systemd[1]: Started Ceph mds.l for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:07:37.154 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:07:37.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.397+0000 7fcbea400640 1 -- 172.21.15.37:0/2139637699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcbe41016b0 msgr2=0x7fcbe4101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:37.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.397+0000 7fcbea400640 1 --2- 172.21.15.37:0/2139637699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcbe41016b0 0x7fcbe4101ad0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4009ca0 tx=0x7fcbd402c100 comp rx=0 tx=0).stop 2024-03-28T18:07:37.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.398+0000 7fcbea400640 1 -- 172.21.15.37:0/2139637699 shutdown_connections 2024-03-28T18:07:37.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.398+0000 7fcbea400640 1 --2- 172.21.15.37:0/2139637699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbe41028a0 0x7fcbe4102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:37.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.398+0000 7fcbea400640 1 --2- 172.21.15.37:0/2139637699 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbe4103260 0x7fcbe4109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:37.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.398+0000 7fcbea400640 1 --2- 172.21.15.37:0/2139637699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcbe41016b0 0x7fcbe4101ad0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:37.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.398+0000 7fcbea400640 1 -- 172.21.15.37:0/2139637699 >> 172.21.15.37:0/2139637699 conn(0x7fcbe40fcc80 msgr2=0x7fcbe40ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:37.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.398+0000 7fcbea400640 1 -- 172.21.15.37:0/2139637699 shutdown_connections 2024-03-28T18:07:37.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.398+0000 7fcbea400640 1 -- 172.21.15.37:0/2139637699 wait complete. 2024-03-28T18:07:37.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.399+0000 7fcbea400640 1 Processor -- start 2024-03-28T18:07:37.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.399+0000 7fcbea400640 1 -- start start 2024-03-28T18:07:37.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.399+0000 7fcbea400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcbe41016b0 0x7fcbe4075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:37.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.399+0000 7fcbea400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbe41028a0 0x7fcbe4073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:37.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.399+0000 7fcbea400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbe4103260 0x7fcbe4073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:37.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.399+0000 7fcbea400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe410d330 con 0x7fcbe4103260 2024-03-28T18:07:37.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.399+0000 7fcbea400640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe410d1b0 con 0x7fcbe41028a0 2024-03-28T18:07:37.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.399+0000 7fcbea400640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe410d4b0 con 0x7fcbe41016b0 2024-03-28T18:07:37.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.399+0000 7fcbe2a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbe41028a0 0x7fcbe4073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:37.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.399+0000 7fcbe3400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcbe41016b0 0x7fcbe4075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:37.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.399+0000 7fcbe3e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbe4103260 0x7fcbe4073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:37.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.399+0000 7fcbe2a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbe41028a0 0x7fcbe4073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:46498/0 (socket says 172.21.15.37:46498) 2024-03-28T18:07:37.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.400+0000 7fcbe2a00640 1 -- 172.21.15.37:0/2958118584 learned_addr learned my addr 172.21.15.37:0/2958118584 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:07:37.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.399+0000 7fcbe3400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcbe41016b0 0x7fcbe4075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:43164/0 (socket says 172.21.15.37:43164) 2024-03-28T18:07:37.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.400+0000 7fcbe3400640 1 -- 172.21.15.37:0/2958118584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbe41028a0 msgr2=0x7fcbe4073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:37.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.400+0000 7fcbe3400640 1 --2- 172.21.15.37:0/2958118584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbe41028a0 0x7fcbe4073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:37.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.400+0000 7fcbe3400640 1 -- 172.21.15.37:0/2958118584 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbe4103260 msgr2=0x7fcbe4073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:37.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.400+0000 7fcbe3400640 1 --2- 172.21.15.37:0/2958118584 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbe4103260 0x7fcbe4073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:37.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.400+0000 7fcbe3400640 1 -- 172.21.15.37:0/2958118584 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe4074860 con 0x7fcbe41016b0 2024-03-28T18:07:37.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.400+0000 7fcbe2a00640 1 --2- 172.21.15.37:0/2958118584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbe41028a0 0x7fcbe4073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:07:37.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.400+0000 7fcbe3400640 1 --2- 172.21.15.37:0/2958118584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcbe41016b0 0x7fcbe4075450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4037040 tx=0x7fcbd4035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:37.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.401+0000 7fcbdbe00640 1 -- 172.21.15.37:0/2958118584 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4039040 con 0x7fcbe41016b0 2024-03-28T18:07:37.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.401+0000 7fcbea400640 1 -- 172.21.15.37:0/2958118584 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe41a1af0 con 0x7fcbe41016b0 2024-03-28T18:07:37.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.401+0000 7fcbdbe00640 1 -- 172.21.15.37:0/2958118584 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbd4003970 con 0x7fcbe41016b0 2024-03-28T18:07:37.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.401+0000 7fcbdbe00640 1 -- 172.21.15.37:0/2958118584 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd40042a0 con 0x7fcbe41016b0 2024-03-28T18:07:37.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.401+0000 7fcbea400640 1 -- 172.21.15.37:0/2958118584 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe41a2060 con 0x7fcbe41016b0 2024-03-28T18:07:37.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.402+0000 7fcbea400640 1 -- 172.21.15.37:0/2958118584 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe4108a50 con 0x7fcbe41016b0 2024-03-28T18:07:37.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.404+0000 7fcbdbe00640 1 -- 172.21.15.37:0/2958118584 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcbd4046030 con 0x7fcbe41016b0 2024-03-28T18:07:37.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.405+0000 7fcbdbe00640 1 --2- 172.21.15.37:0/2958118584 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fcbc80a2800 0x7fcbc80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:37.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.405+0000 7fcbdbe00640 1 -- 172.21.15.37:0/2958118584 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcbd40f01a0 con 0x7fcbe41016b0 2024-03-28T18:07:37.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.405+0000 7fcbe2a00640 1 --2- 172.21.15.37:0/2958118584 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fcbc80a2800 0x7fcbc80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:37.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.406+0000 7fcbe2a00640 1 --2- 172.21.15.37:0/2958118584 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fcbc80a2800 0x7fcbc80a4cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcbe40751c0 tx=0x7fcbcc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:37.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.407+0000 7fcbdbe00640 1 -- 172.21.15.37:0/2958118584 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcbd404d060 con 0x7fcbe41016b0 2024-03-28T18:07:37.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:37 smithi037 ceph-mon[31460]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:37.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:37 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/576965351,v1:172.21.15.170:6833/576965351] up:boot 2024-03-28T18:07:37.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:37 smithi037 ceph-mon[31460]: fsmap 1 up:standby 2024-03-28T18:07:37.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:37 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:07:37.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:37 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:37.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:37 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:37.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:37 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:37.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:37 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:37.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:37 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:37.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:37 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:37.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:37 smithi037 ceph-mon[31460]: Deploying daemon mds.k on smithi059 2024-03-28T18:07:37.626 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:37 smithi059 ceph-mon[36190]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:37.626 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:37 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/576965351,v1:172.21.15.170:6833/576965351] up:boot 2024-03-28T18:07:37.626 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:37 smithi059 ceph-mon[36190]: fsmap 1 up:standby 2024-03-28T18:07:37.626 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:37 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:07:37.626 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:37 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:37.626 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:37 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:37.626 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:37 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:37.626 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:37 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:37.626 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:37 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:37.626 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:37 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:37.626 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:37 smithi059 ceph-mon[36190]: Deploying daemon mds.k on smithi059 2024-03-28T18:07:37.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.652+0000 7fcbea400640 1 -- 172.21.15.37:0/2958118584 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbe402d070 con 0x7fcbe41016b0 2024-03-28T18:07:37.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.653+0000 7fcbdbe00640 1 -- 172.21.15.37:0/2958118584 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1325 (secure 0 0 0) 0x7fcbd40bd6d0 con 0x7fcbe41016b0 2024-03-28T18:07:37.653 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:07:37.653 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T18:07:37.653 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 2 2024-03-28T18:07:37.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.655+0000 7fcbea400640 1 -- 172.21.15.37:0/2958118584 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fcbc80a2800 msgr2=0x7fcbc80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:37.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.655+0000 7fcbea400640 1 --2- 172.21.15.37:0/2958118584 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fcbc80a2800 0x7fcbc80a4cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcbe40751c0 tx=0x7fcbcc005f90 comp rx=0 tx=0).stop 2024-03-28T18:07:37.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.655+0000 7fcbea400640 1 -- 172.21.15.37:0/2958118584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcbe41016b0 msgr2=0x7fcbe4075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:37.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.655+0000 7fcbea400640 1 --2- 172.21.15.37:0/2958118584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcbe41016b0 0x7fcbe4075450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4037040 tx=0x7fcbd4035d10 comp rx=0 tx=0).stop 2024-03-28T18:07:37.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.655+0000 7fcbea400640 1 -- 172.21.15.37:0/2958118584 shutdown_connections 2024-03-28T18:07:37.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.655+0000 7fcbea400640 1 --2- 172.21.15.37:0/2958118584 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fcbc80a2800 0x7fcbc80a4cc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:37.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.655+0000 7fcbea400640 1 --2- 172.21.15.37:0/2958118584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbe41028a0 0x7fcbe4073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:37.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.655+0000 7fcbea400640 1 --2- 172.21.15.37:0/2958118584 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbe4103260 0x7fcbe4073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:37.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.655+0000 7fcbea400640 1 --2- 172.21.15.37:0/2958118584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcbe41016b0 0x7fcbe4075450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:37.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.655+0000 7fcbea400640 1 -- 172.21.15.37:0/2958118584 >> 172.21.15.37:0/2958118584 conn(0x7fcbe40fcc80 msgr2=0x7fcbe4105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:37.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.656+0000 7fcbea400640 1 -- 172.21.15.37:0/2958118584 shutdown_connections 2024-03-28T18:07:37.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:37.656+0000 7fcbea400640 1 -- 172.21.15.37:0/2958118584 wait complete. 2024-03-28T18:07:37.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:37 smithi170 ceph-mon[36280]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:37.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:37 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/576965351,v1:172.21.15.170:6833/576965351] up:boot 2024-03-28T18:07:37.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:37 smithi170 ceph-mon[36280]: fsmap 1 up:standby 2024-03-28T18:07:37.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:37 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:07:37.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:37 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:37.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:37 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:37.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:37 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:37.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:37 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:37.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:37 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:37.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:37 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:37.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:37 smithi170 ceph-mon[36280]: Deploying daemon mds.k on smithi059 2024-03-28T18:07:38.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:38 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2958118584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:38.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:38 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2958118584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:38.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:38 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2958118584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:39.309 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:07:39 smithi059 podman[65847]: 2024-03-28 18:07:39.17632201 +0000 UTC m=+0.018254986 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:07:39.309 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:07:39 smithi059 podman[65847]: 2024-03-28 18:07:39.309699576 +0000 UTC m=+0.151632534 container create 9f1727b54bbe9cf3105f7c4b654df6a4a7f05f36861f880fc4158954c628835b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:07:39.603 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:39 smithi059 ceph-mon[36190]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:39.673 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:07:39.768 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:39 smithi037 ceph-mon[31460]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:39.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:39 smithi170 ceph-mon[36280]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.913+0000 7fc966200640 1 -- 172.21.15.37:0/843767196 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9601016c0 msgr2=0x7fc960101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.913+0000 7fc966200640 1 --2- 172.21.15.37:0/843767196 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9601016c0 0x7fc960101ae0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc950009860 tx=0x7fc95002bb70 comp rx=0 tx=0).stop 2024-03-28T18:07:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.913+0000 7fc966200640 1 -- 172.21.15.37:0/843767196 shutdown_connections 2024-03-28T18:07:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.913+0000 7fc966200640 1 --2- 172.21.15.37:0/843767196 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc960103270 0x7fc960109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.913+0000 7fc966200640 1 --2- 172.21.15.37:0/843767196 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9601028b0 0x7fc960102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.913+0000 7fc966200640 1 --2- 172.21.15.37:0/843767196 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9601016c0 0x7fc960101ae0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.913+0000 7fc966200640 1 -- 172.21.15.37:0/843767196 >> 172.21.15.37:0/843767196 conn(0x7fc9600fcc50 msgr2=0x7fc9600ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.913+0000 7fc966200640 1 -- 172.21.15.37:0/843767196 shutdown_connections 2024-03-28T18:07:39.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.913+0000 7fc966200640 1 -- 172.21.15.37:0/843767196 wait complete. 2024-03-28T18:07:39.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.914+0000 7fc966200640 1 Processor -- start 2024-03-28T18:07:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.914+0000 7fc966200640 1 -- start start 2024-03-28T18:07:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.914+0000 7fc966200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9601016c0 0x7fc960199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.914+0000 7fc966200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9601028b0 0x7fc96019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.914+0000 7fc966200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc960103270 0x7fc9601a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.914+0000 7fc966200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96010d310 con 0x7fc9601016c0 2024-03-28T18:07:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.914+0000 7fc966200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96010d190 con 0x7fc960103270 2024-03-28T18:07:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.914+0000 7fc966200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96010d490 con 0x7fc9601028b0 2024-03-28T18:07:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.914+0000 7fc95f400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9601016c0 0x7fc960199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.914+0000 7fc95f400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9601016c0 0x7fc960199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51850/0 (socket says 172.21.15.37:51850) 2024-03-28T18:07:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.914+0000 7fc95f400640 1 -- 172.21.15.37:0/361444190 learned_addr learned my addr 172.21.15.37:0/361444190 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:07:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.915+0000 7fc957e00640 1 --2- 172.21.15.37:0/361444190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9601028b0 0x7fc96019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.915+0000 7fc95fe00640 1 --2- 172.21.15.37:0/361444190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc960103270 0x7fc9601a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.915+0000 7fc95f400640 1 -- 172.21.15.37:0/361444190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9601028b0 msgr2=0x7fc96019a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:39.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.915+0000 7fc95f400640 1 --2- 172.21.15.37:0/361444190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9601028b0 0x7fc96019a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:39.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.915+0000 7fc95f400640 1 -- 172.21.15.37:0/361444190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc960103270 msgr2=0x7fc9601a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:39.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.915+0000 7fc95f400640 1 --2- 172.21.15.37:0/361444190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc960103270 0x7fc9601a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:39.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.915+0000 7fc95f400640 1 -- 172.21.15.37:0/361444190 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9601a1850 con 0x7fc9601016c0 2024-03-28T18:07:39.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.915+0000 7fc957e00640 1 --2- 172.21.15.37:0/361444190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9601028b0 0x7fc96019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:07:39.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.915+0000 7fc95f400640 1 --2- 172.21.15.37:0/361444190 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9601016c0 0x7fc960199c40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc9500349e0 tx=0x7fc950032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:39.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.915+0000 7fc95cc00640 1 -- 172.21.15.37:0/361444190 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950041020 con 0x7fc9601016c0 2024-03-28T18:07:39.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.916+0000 7fc95cc00640 1 -- 172.21.15.37:0/361444190 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc95002c9f0 con 0x7fc9601016c0 2024-03-28T18:07:39.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.916+0000 7fc966200640 1 -- 172.21.15.37:0/361444190 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9601a1ae0 con 0x7fc9601016c0 2024-03-28T18:07:39.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.916+0000 7fc95cc00640 1 -- 172.21.15.37:0/361444190 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950039040 con 0x7fc9601016c0 2024-03-28T18:07:39.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.916+0000 7fc966200640 1 -- 172.21.15.37:0/361444190 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9601a1fd0 con 0x7fc9601016c0 2024-03-28T18:07:39.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.917+0000 7fc966200640 1 -- 172.21.15.37:0/361444190 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9601a21a0 con 0x7fc9601016c0 2024-03-28T18:07:39.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.919+0000 7fc95cc00640 1 -- 172.21.15.37:0/361444190 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc950044020 con 0x7fc9601016c0 2024-03-28T18:07:39.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.920+0000 7fc95cc00640 1 --2- 172.21.15.37:0/361444190 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc9440a2800 0x7fc9440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:39.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.921+0000 7fc95cc00640 1 -- 172.21.15.37:0/361444190 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc9500eff50 con 0x7fc9601016c0 2024-03-28T18:07:39.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.921+0000 7fc957e00640 1 --2- 172.21.15.37:0/361444190 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc9440a2800 0x7fc9440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:39.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.922+0000 7fc957e00640 1 --2- 172.21.15.37:0/361444190 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc9440a2800 0x7fc9440a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc9480096c0 tx=0x7fc948009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:39.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:39.925+0000 7fc95cc00640 1 -- 172.21.15.37:0/361444190 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc95004b020 con 0x7fc9601016c0 2024-03-28T18:07:40.035 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:07:39 smithi059 podman[65847]: 2024-03-28 18:07:39.60417541 +0000 UTC m=+0.446108377 container init 9f1727b54bbe9cf3105f7c4b654df6a4a7f05f36861f880fc4158954c628835b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-28T18:07:40.035 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:07:39 smithi059 podman[65847]: 2024-03-28 18:07:39.607103247 +0000 UTC m=+0.449036207 container start 9f1727b54bbe9cf3105f7c4b654df6a4a7f05f36861f880fc4158954c628835b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:07:40.035 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:07:39 smithi059 ceph-mds[65865]: -- 172.21.15.59:0/196744637 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c6aef77a40 con 0x55c6aef4a800 2024-03-28T18:07:40.035 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:07:39 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: starting mds.k at 2024-03-28T18:07:40.035 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:07:39 smithi059 bash[65847]: 9f1727b54bbe9cf3105f7c4b654df6a4a7f05f36861f880fc4158954c628835b 2024-03-28T18:07:40.035 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:07:39 smithi059 systemd[1]: Started Ceph mds.k for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:07:40.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:40.171+0000 7fc966200640 1 -- 172.21.15.37:0/361444190 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9601a2370 con 0x7fc9601016c0 2024-03-28T18:07:40.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:40.172+0000 7fc95cc00640 1 -- 172.21.15.37:0/361444190 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1325 (secure 0 0 0) 0x7fc9500bd3f0 con 0x7fc9601016c0 2024-03-28T18:07:40.172 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:07:40.172 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T18:07:40.172 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 2 2024-03-28T18:07:40.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:40.174+0000 7fc966200640 1 -- 172.21.15.37:0/361444190 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc9440a2800 msgr2=0x7fc9440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:40.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:40.174+0000 7fc966200640 1 --2- 172.21.15.37:0/361444190 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc9440a2800 0x7fc9440a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc9480096c0 tx=0x7fc948009260 comp rx=0 tx=0).stop 2024-03-28T18:07:40.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:40.174+0000 7fc966200640 1 -- 172.21.15.37:0/361444190 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9601016c0 msgr2=0x7fc960199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:40.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:40.174+0000 7fc966200640 1 --2- 172.21.15.37:0/361444190 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9601016c0 0x7fc960199c40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc9500349e0 tx=0x7fc950032430 comp rx=0 tx=0).stop 2024-03-28T18:07:40.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:40.174+0000 7fc966200640 1 -- 172.21.15.37:0/361444190 shutdown_connections 2024-03-28T18:07:40.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:40.174+0000 7fc966200640 1 --2- 172.21.15.37:0/361444190 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc9440a2800 0x7fc9440a4cc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:40.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:40.174+0000 7fc966200640 1 --2- 172.21.15.37:0/361444190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc960103270 0x7fc9601a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:40.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:40.174+0000 7fc966200640 1 --2- 172.21.15.37:0/361444190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9601028b0 0x7fc96019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:40.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:40.174+0000 7fc966200640 1 --2- 172.21.15.37:0/361444190 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9601016c0 0x7fc960199c40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:40.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:40.174+0000 7fc966200640 1 -- 172.21.15.37:0/361444190 >> 172.21.15.37:0/361444190 conn(0x7fc9600fcc50 msgr2=0x7fc9600738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:40.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:40.175+0000 7fc966200640 1 -- 172.21.15.37:0/361444190 shutdown_connections 2024-03-28T18:07:40.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:40.175+0000 7fc966200640 1 -- 172.21.15.37:0/361444190 wait complete. 2024-03-28T18:07:41.030 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:40 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:41.030 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:40 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:41.030 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:40 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:41.031 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:40 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:41.031 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:40 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:41.031 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:40 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:41.031 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:40 smithi037 ceph-mon[31460]: Deploying daemon mds.j on smithi037 2024-03-28T18:07:41.031 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:40 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/361444190' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:41.031 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:40 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/16451159,v1:172.21.15.59:6833/16451159] up:boot 2024-03-28T18:07:41.031 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:40 smithi037 ceph-mon[31460]: fsmap 2 up:standby 2024-03-28T18:07:41.031 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:40 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:07:41.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:40 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:40 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:40 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:40 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:40 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:40 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:40 smithi059 ceph-mon[36190]: Deploying daemon mds.j on smithi037 2024-03-28T18:07:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:40 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/361444190' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:40 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/16451159,v1:172.21.15.59:6833/16451159] up:boot 2024-03-28T18:07:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:40 smithi059 ceph-mon[36190]: fsmap 2 up:standby 2024-03-28T18:07:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:40 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:07:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:40 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:40 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:40 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:40 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:40 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:40 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:40 smithi170 ceph-mon[36280]: Deploying daemon mds.j on smithi037 2024-03-28T18:07:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:40 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/361444190' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:40 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/16451159,v1:172.21.15.59:6833/16451159] up:boot 2024-03-28T18:07:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:40 smithi170 ceph-mon[36280]: fsmap 2 up:standby 2024-03-28T18:07:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:40 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:07:42.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:41 smithi059 ceph-mon[36190]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:42.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:41 smithi037 ceph-mon[31460]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:42.194 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:07:42.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:41 smithi170 ceph-mon[36280]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.697+0000 7f6ee9800640 1 -- 172.21.15.37:0/3702512422 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ee41038e0 msgr2=0x7f6ee4103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.697+0000 7f6ee9800640 1 --2- 172.21.15.37:0/3702512422 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ee41038e0 0x7f6ee4103d00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4009900 tx=0x7f6ed402bc30 comp rx=0 tx=0).stop 2024-03-28T18:07:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.697+0000 7f6ee9800640 1 -- 172.21.15.37:0/3702512422 shutdown_connections 2024-03-28T18:07:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.697+0000 7f6ee9800640 1 --2- 172.21.15.37:0/3702512422 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ee4105490 0x7f6ee410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.697+0000 7f6ee9800640 1 --2- 172.21.15.37:0/3702512422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ee4104ad0 0x7f6ee4104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.697+0000 7f6ee9800640 1 --2- 172.21.15.37:0/3702512422 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ee41038e0 0x7f6ee4103d00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.697+0000 7f6ee9800640 1 -- 172.21.15.37:0/3702512422 >> 172.21.15.37:0/3702512422 conn(0x7f6ee40fee90 msgr2=0x7f6ee41012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.697+0000 7f6ee9800640 1 -- 172.21.15.37:0/3702512422 shutdown_connections 2024-03-28T18:07:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.697+0000 7f6ee9800640 1 -- 172.21.15.37:0/3702512422 wait complete. 2024-03-28T18:07:42.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.698+0000 7f6ee9800640 1 Processor -- start 2024-03-28T18:07:42.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.698+0000 7f6ee9800640 1 -- start start 2024-03-28T18:07:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.699+0000 7f6ee9800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ee41038e0 0x7f6ee4199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.699+0000 7f6ee9800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ee4104ad0 0x7f6ee419a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.699+0000 7f6ee9800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ee4105490 0x7f6ee41a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.699+0000 7f6ee9800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee410f5a0 con 0x7f6ee4104ad0 2024-03-28T18:07:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.699+0000 7f6ee9800640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee410f420 con 0x7f6ee41038e0 2024-03-28T18:07:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.699+0000 7f6ee9800640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee410f720 con 0x7f6ee4105490 2024-03-28T18:07:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.699+0000 7f6ee2000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ee4104ad0 0x7f6ee419a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.699+0000 7f6ee2a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ee41038e0 0x7f6ee4199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.699+0000 7f6ee2a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ee41038e0 0x7f6ee4199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:46536/0 (socket says 172.21.15.37:46536) 2024-03-28T18:07:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.699+0000 7f6ee2a00640 1 -- 172.21.15.37:0/1002311530 learned_addr learned my addr 172.21.15.37:0/1002311530 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:07:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.699+0000 7f6ee2000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ee4104ad0 0x7f6ee419a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51876/0 (socket says 172.21.15.37:51876) 2024-03-28T18:07:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.699+0000 7f6ee3400640 1 --2- 172.21.15.37:0/1002311530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ee4105490 0x7f6ee41a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.699+0000 7f6ee2a00640 1 -- 172.21.15.37:0/1002311530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ee4105490 msgr2=0x7f6ee41a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.699+0000 7f6ee2a00640 1 --2- 172.21.15.37:0/1002311530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ee4105490 0x7f6ee41a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.699+0000 7f6ee2a00640 1 -- 172.21.15.37:0/1002311530 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ee4104ad0 msgr2=0x7f6ee419a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.699+0000 7f6ee2a00640 1 --2- 172.21.15.37:0/1002311530 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ee4104ad0 0x7f6ee419a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.699+0000 7f6ee2a00640 1 -- 172.21.15.37:0/1002311530 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee41a1850 con 0x7f6ee41038e0 2024-03-28T18:07:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.699+0000 7f6ee3400640 1 --2- 172.21.15.37:0/1002311530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ee4105490 0x7f6ee41a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:07:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.700+0000 7f6ee2a00640 1 --2- 172.21.15.37:0/1002311530 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ee41038e0 0x7f6ee4199c40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4000c00 tx=0x7f6ed402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.700+0000 7f6edb400640 1 -- 172.21.15.37:0/1002311530 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4043020 con 0x7f6ee41038e0 2024-03-28T18:07:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.700+0000 7f6edb400640 1 -- 172.21.15.37:0/1002311530 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6ed40054e0 con 0x7f6ee41038e0 2024-03-28T18:07:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.700+0000 7f6ee9800640 1 -- 172.21.15.37:0/1002311530 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee41a1ae0 con 0x7f6ee41038e0 2024-03-28T18:07:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.700+0000 7f6edb400640 1 -- 172.21.15.37:0/1002311530 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4003c40 con 0x7f6ee41038e0 2024-03-28T18:07:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.700+0000 7f6ee9800640 1 -- 172.21.15.37:0/1002311530 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee41a2030 con 0x7f6ee41038e0 2024-03-28T18:07:42.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.702+0000 7f6ee9800640 1 -- 172.21.15.37:0/1002311530 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee40688e0 con 0x7f6ee41038e0 2024-03-28T18:07:42.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.704+0000 7f6edb400640 1 -- 172.21.15.37:0/1002311530 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6ed4046020 con 0x7f6ee41038e0 2024-03-28T18:07:42.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.705+0000 7f6edb400640 1 --2- 172.21.15.37:0/1002311530 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f6ec80a2800 0x7f6ec80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:42.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.705+0000 7f6edb400640 1 -- 172.21.15.37:0/1002311530 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6ed40f0760 con 0x7f6ee41038e0 2024-03-28T18:07:42.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.705+0000 7f6ee2000640 1 --2- 172.21.15.37:0/1002311530 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f6ec80a2800 0x7f6ec80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:42.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.706+0000 7f6ee2000640 1 --2- 172.21.15.37:0/1002311530 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f6ec80a2800 0x7f6ec80a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0009bd0 tx=0x7f6ed0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:42.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.709+0000 7f6edb400640 1 -- 172.21.15.37:0/1002311530 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6ed404d020 con 0x7f6ee41038e0 2024-03-28T18:07:42.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.959+0000 7f6ee9800640 1 -- 172.21.15.37:0/1002311530 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ee40622a0 con 0x7f6ee41038e0 2024-03-28T18:07:42.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.960+0000 7f6edb400640 1 -- 172.21.15.37:0/1002311530 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2110 (secure 0 0 0) 0x7f6ed40bdc00 con 0x7f6ee41038e0 2024-03-28T18:07:42.960 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:07:42.960 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T18:07:42.960 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 3 2024-03-28T18:07:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.962+0000 7f6ee9800640 1 -- 172.21.15.37:0/1002311530 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f6ec80a2800 msgr2=0x7f6ec80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.963+0000 7f6ee9800640 1 --2- 172.21.15.37:0/1002311530 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f6ec80a2800 0x7f6ec80a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0009bd0 tx=0x7f6ed0005f90 comp rx=0 tx=0).stop 2024-03-28T18:07:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.963+0000 7f6ee9800640 1 -- 172.21.15.37:0/1002311530 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ee41038e0 msgr2=0x7f6ee4199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.963+0000 7f6ee9800640 1 --2- 172.21.15.37:0/1002311530 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ee41038e0 0x7f6ee4199c40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4000c00 tx=0x7f6ed402c650 comp rx=0 tx=0).stop 2024-03-28T18:07:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.963+0000 7f6ee9800640 1 -- 172.21.15.37:0/1002311530 shutdown_connections 2024-03-28T18:07:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.963+0000 7f6ee9800640 1 --2- 172.21.15.37:0/1002311530 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f6ec80a2800 0x7f6ec80a4cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.963+0000 7f6ee9800640 1 --2- 172.21.15.37:0/1002311530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ee4105490 0x7f6ee41a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.963+0000 7f6ee9800640 1 --2- 172.21.15.37:0/1002311530 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ee4104ad0 0x7f6ee419a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.963+0000 7f6ee9800640 1 --2- 172.21.15.37:0/1002311530 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ee41038e0 0x7f6ee4199c40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.963+0000 7f6ee9800640 1 -- 172.21.15.37:0/1002311530 >> 172.21.15.37:0/1002311530 conn(0x7f6ee40fee90 msgr2=0x7f6ee4107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:42.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.963+0000 7f6ee9800640 1 -- 172.21.15.37:0/1002311530 shutdown_connections 2024-03-28T18:07:42.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:42.963+0000 7f6ee9800640 1 -- 172.21.15.37:0/1002311530 wait complete. 2024-03-28T18:07:43.595 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:07:43 smithi037 podman[69720]: 2024-03-28 18:07:43.298768881 +0000 UTC m=+0.155408425 container create feecadfcbd8b35fa662e58858bafb839c34dc4edcfd6840159af18c7c2da48c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, CEPH_POINT_RELEASE=) 2024-03-28T18:07:43.595 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:07:43 smithi037 podman[69720]: 2024-03-28 18:07:43.594836775 +0000 UTC m=+0.451476325 container init feecadfcbd8b35fa662e58858bafb839c34dc4edcfd6840159af18c7c2da48c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-28T18:07:43.848 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:07:43 smithi037 podman[69720]: 2024-03-28 18:07:43.607975674 +0000 UTC m=+0.464615248 container start feecadfcbd8b35fa662e58858bafb839c34dc4edcfd6840159af18c7c2da48c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-28T18:07:43.848 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:07:43 smithi037 ceph-mds[69738]: -- 172.21.15.37:0/390523230 <== mon.0 v2:172.21.15.37:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bdd5249a40 con 0x55bdd521c800 2024-03-28T18:07:43.848 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:07:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: starting mds.j at 2024-03-28T18:07:43.848 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:07:43 smithi037 bash[69720]: feecadfcbd8b35fa662e58858bafb839c34dc4edcfd6840159af18c7c2da48c6 2024-03-28T18:07:43.848 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:07:43 smithi037 systemd[1]: Started Ceph mds.j for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:07:44.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:43 smithi037 ceph-mon[31460]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:44.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:43 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1002311530' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:43 smithi059 ceph-mon[36190]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:43 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1002311530' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:44.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:43 smithi170 ceph-mon[36280]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:43 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1002311530' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:44.983 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:07:45.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:44 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:45.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:44 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:45.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:44 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/967056034,v1:172.21.15.37:6835/967056034] up:boot 2024-03-28T18:07:45.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:44 smithi037 ceph-mon[31460]: fsmap 3 up:standby 2024-03-28T18:07:45.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:44 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:07:45.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:44 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:45.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:44 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:45.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:44 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:45.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:44 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:45.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.223+0000 7fafb1c00640 1 -- 172.21.15.37:0/1157030899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafac1026d0 msgr2=0x7fafac104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:45.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.223+0000 7fafb1c00640 1 --2- 172.21.15.37:0/1157030899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafac1026d0 0x7fafac104b40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7faf94009900 tx=0x7faf9402bc30 comp rx=0 tx=0).stop 2024-03-28T18:07:45.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.223+0000 7fafb1c00640 1 -- 172.21.15.37:0/1157030899 shutdown_connections 2024-03-28T18:07:45.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.223+0000 7fafb1c00640 1 --2- 172.21.15.37:0/1157030899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafac1026d0 0x7fafac104b40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:45.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.223+0000 7fafb1c00640 1 --2- 172.21.15.37:0/1157030899 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafac105080 0x7fafac105500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:45.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.223+0000 7fafb1c00640 1 --2- 172.21.15.37:0/1157030899 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafac0ffd60 0x7fafac102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:45.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.223+0000 7fafb1c00640 1 -- 172.21.15.37:0/1157030899 >> 172.21.15.37:0/1157030899 conn(0x7fafac0fb980 msgr2=0x7fafac0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:45.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.223+0000 7fafb1c00640 1 -- 172.21.15.37:0/1157030899 shutdown_connections 2024-03-28T18:07:45.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.223+0000 7fafb1c00640 1 -- 172.21.15.37:0/1157030899 wait complete. 2024-03-28T18:07:45.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.224+0000 7fafb1c00640 1 Processor -- start 2024-03-28T18:07:45.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.224+0000 7fafb1c00640 1 -- start start 2024-03-28T18:07:45.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.225+0000 7fafb1c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafac0ffd60 0x7fafac19ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:45.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.225+0000 7fafb1c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafac1026d0 0x7fafac19e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:45.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.225+0000 7fafb1c00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafac105080 0x7fafac1a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:45.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.225+0000 7fafb1c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafac1137b0 con 0x7fafac0ffd60 2024-03-28T18:07:45.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.225+0000 7fafb1c00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafac113630 con 0x7fafac1026d0 2024-03-28T18:07:45.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.225+0000 7fafb1c00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafac113930 con 0x7fafac105080 2024-03-28T18:07:45.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.225+0000 7fafabe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafac0ffd60 0x7fafac19ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:45.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.225+0000 7fafa3400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafac1026d0 0x7fafac19e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:45.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.225+0000 7fafabe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafac0ffd60 0x7fafac19ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37862/0 (socket says 172.21.15.37:37862) 2024-03-28T18:07:45.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.225+0000 7fafabe00640 1 -- 172.21.15.37:0/3687859295 learned_addr learned my addr 172.21.15.37:0/3687859295 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:07:45.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.225+0000 7fafb1200640 1 --2- 172.21.15.37:0/3687859295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafac105080 0x7fafac1a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:45.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.225+0000 7fafa3400640 1 -- 172.21.15.37:0/3687859295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafac105080 msgr2=0x7fafac1a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:45.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.225+0000 7fafa3400640 1 --2- 172.21.15.37:0/3687859295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafac105080 0x7fafac1a5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:45.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.225+0000 7fafa3400640 1 -- 172.21.15.37:0/3687859295 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafac0ffd60 msgr2=0x7fafac19ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:45.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.225+0000 7fafa3400640 1 --2- 172.21.15.37:0/3687859295 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafac0ffd60 0x7fafac19ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:45.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.225+0000 7fafa3400640 1 -- 172.21.15.37:0/3687859295 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafac1a59d0 con 0x7fafac1026d0 2024-03-28T18:07:45.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.226+0000 7fafa3400640 1 --2- 172.21.15.37:0/3687859295 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafac1026d0 0x7fafac19e2f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7faf94036040 tx=0x7faf9402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:45.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.226+0000 7fafa9600640 1 -- 172.21.15.37:0/3687859295 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf94043020 con 0x7fafac1026d0 2024-03-28T18:07:45.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.226+0000 7fafa9600640 1 -- 172.21.15.37:0/3687859295 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf940054e0 con 0x7fafac1026d0 2024-03-28T18:07:45.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.226+0000 7fafb1c00640 1 -- 172.21.15.37:0/3687859295 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafac1a5c60 con 0x7fafac1026d0 2024-03-28T18:07:45.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.226+0000 7fafa9600640 1 -- 172.21.15.37:0/3687859295 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf94003c30 con 0x7fafac1026d0 2024-03-28T18:07:45.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.226+0000 7fafb1c00640 1 -- 172.21.15.37:0/3687859295 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafac1a61d0 con 0x7fafac1026d0 2024-03-28T18:07:45.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.227+0000 7fafb1c00640 1 -- 172.21.15.37:0/3687859295 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafac0fd470 con 0x7fafac1026d0 2024-03-28T18:07:45.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.229+0000 7fafa9600640 1 -- 172.21.15.37:0/3687859295 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7faf94046020 con 0x7fafac1026d0 2024-03-28T18:07:45.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.231+0000 7fafa9600640 1 --2- 172.21.15.37:0/3687859295 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7faf900a2880 0x7faf900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:45.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.231+0000 7fafa9600640 1 -- 172.21.15.37:0/3687859295 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7faf940f0550 con 0x7fafac1026d0 2024-03-28T18:07:45.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.231+0000 7fafabe00640 1 --2- 172.21.15.37:0/3687859295 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7faf900a2880 0x7faf900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:45.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.232+0000 7fafabe00640 1 --2- 172.21.15.37:0/3687859295 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7faf900a2880 0x7faf900a4d40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7faf9c009c90 tx=0x7faf9c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:45.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.235+0000 7fafa9600640 1 -- 172.21.15.37:0/3687859295 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faf9404d020 con 0x7fafac1026d0 2024-03-28T18:07:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:44 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:44 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:44 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/967056034,v1:172.21.15.37:6835/967056034] up:boot 2024-03-28T18:07:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:44 smithi059 ceph-mon[36190]: fsmap 3 up:standby 2024-03-28T18:07:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:44 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:07:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:44 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:44 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:44 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:44 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:45.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:44 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:45.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:44 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:45.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:44 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/967056034,v1:172.21.15.37:6835/967056034] up:boot 2024-03-28T18:07:45.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:44 smithi170 ceph-mon[36280]: fsmap 3 up:standby 2024-03-28T18:07:45.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:44 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:07:45.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:44 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:45.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:44 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:45.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:44 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:45.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:44 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:45.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.489+0000 7fafb1c00640 1 -- 172.21.15.37:0/3687859295 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafac0622a0 con 0x7fafac1026d0 2024-03-28T18:07:45.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.490+0000 7fafa9600640 1 -- 172.21.15.37:0/3687859295 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2898 (secure 0 0 0) 0x7faf940bd9f0 con 0x7fafac1026d0 2024-03-28T18:07:45.490 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:07:45.490 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T18:07:45.490 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 4 2024-03-28T18:07:45.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.493+0000 7fafb1c00640 1 -- 172.21.15.37:0/3687859295 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7faf900a2880 msgr2=0x7faf900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:45.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.493+0000 7fafb1c00640 1 --2- 172.21.15.37:0/3687859295 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7faf900a2880 0x7faf900a4d40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7faf9c009c90 tx=0x7faf9c005f90 comp rx=0 tx=0).stop 2024-03-28T18:07:45.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.493+0000 7fafb1c00640 1 -- 172.21.15.37:0/3687859295 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafac1026d0 msgr2=0x7fafac19e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:45.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.493+0000 7fafb1c00640 1 --2- 172.21.15.37:0/3687859295 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafac1026d0 0x7fafac19e2f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7faf94036040 tx=0x7faf9402c650 comp rx=0 tx=0).stop 2024-03-28T18:07:45.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.493+0000 7fafb1c00640 1 -- 172.21.15.37:0/3687859295 shutdown_connections 2024-03-28T18:07:45.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.493+0000 7fafb1c00640 1 --2- 172.21.15.37:0/3687859295 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7faf900a2880 0x7faf900a4d40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:45.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.493+0000 7fafb1c00640 1 --2- 172.21.15.37:0/3687859295 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafac1026d0 0x7fafac19e2f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:45.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.493+0000 7fafb1c00640 1 --2- 172.21.15.37:0/3687859295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafac105080 0x7fafac1a5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:45.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.493+0000 7fafb1c00640 1 --2- 172.21.15.37:0/3687859295 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafac0ffd60 0x7fafac19ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:45.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.493+0000 7fafb1c00640 1 -- 172.21.15.37:0/3687859295 >> 172.21.15.37:0/3687859295 conn(0x7fafac0fb980 msgr2=0x7fafac100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:45.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.493+0000 7fafb1c00640 1 -- 172.21.15.37:0/3687859295 shutdown_connections 2024-03-28T18:07:45.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:45.493+0000 7fafb1c00640 1 -- 172.21.15.37:0/3687859295 wait complete. 2024-03-28T18:07:46.119 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:45 smithi037 ceph-mon[31460]: Deploying daemon mds.d on smithi037 2024-03-28T18:07:46.119 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:45 smithi037 ceph-mon[31460]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:46.119 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:45 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3687859295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:45 smithi059 ceph-mon[36190]: Deploying daemon mds.d on smithi037 2024-03-28T18:07:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:45 smithi059 ceph-mon[36190]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:45 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3687859295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:45 smithi170 ceph-mon[36280]: Deploying daemon mds.d on smithi037 2024-03-28T18:07:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:45 smithi170 ceph-mon[36280]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:45 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3687859295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:47.273 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:07:46 smithi037 systemd[1]: Starting Ceph mds.d for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:07:47.514 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:07:47.592 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:07:47 smithi037 podman[70064]: 2024-03-28 18:07:47.271030804 +0000 UTC m=+0.120538519 container create 4b0671f3d1a0534f9ddce8a0b049c2139432de516dc34edaa49e43157365f884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:07:47.592 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:07:47 smithi037 podman[70064]: 2024-03-28 18:07:47.183316925 +0000 UTC m=+0.032824616 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:07:47.592 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:07:47 smithi037 podman[70064]: 2024-03-28 18:07:47.591744015 +0000 UTC m=+0.441251737 container init 4b0671f3d1a0534f9ddce8a0b049c2139432de516dc34edaa49e43157365f884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:07:47.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.747+0000 7f5e26e00640 1 -- 172.21.15.37:0/1882144982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e20105490 msgr2=0x7f5e2010bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:47.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.747+0000 7f5e26e00640 1 --2- 172.21.15.37:0/1882144982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e20105490 0x7f5e2010bd20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5e18009ca0 tx=0x7f5e1802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:07:47.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.747+0000 7f5e26e00640 1 -- 172.21.15.37:0/1882144982 shutdown_connections 2024-03-28T18:07:47.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.747+0000 7f5e26e00640 1 --2- 172.21.15.37:0/1882144982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e20105490 0x7f5e2010bd20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:47.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.747+0000 7f5e26e00640 1 --2- 172.21.15.37:0/1882144982 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e20104ad0 0x7f5e20104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:47.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.747+0000 7f5e26e00640 1 --2- 172.21.15.37:0/1882144982 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e201038e0 0x7f5e20103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:47.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.747+0000 7f5e26e00640 1 -- 172.21.15.37:0/1882144982 >> 172.21.15.37:0/1882144982 conn(0x7f5e200fee50 msgr2=0x7f5e20101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:47.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.747+0000 7f5e26e00640 1 -- 172.21.15.37:0/1882144982 shutdown_connections 2024-03-28T18:07:47.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.748+0000 7f5e26e00640 1 -- 172.21.15.37:0/1882144982 wait complete. 2024-03-28T18:07:47.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.748+0000 7f5e26e00640 1 Processor -- start 2024-03-28T18:07:47.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.749+0000 7f5e26e00640 1 -- start start 2024-03-28T18:07:47.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.749+0000 7f5e26e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e201038e0 0x7f5e20199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:47.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.749+0000 7f5e26e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e20104ad0 0x7f5e2019a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:47.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.749+0000 7f5e26e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e20105490 0x7f5e201a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:47.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.749+0000 7f5e26e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2010f560 con 0x7f5e201038e0 2024-03-28T18:07:47.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.749+0000 7f5e26e00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2010f3e0 con 0x7f5e20105490 2024-03-28T18:07:47.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.749+0000 7f5e26e00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2010f6e0 con 0x7f5e20104ad0 2024-03-28T18:07:47.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.749+0000 7f5e1fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e201038e0 0x7f5e20199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:47.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.749+0000 7f5e1fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e201038e0 0x7f5e20199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37894/0 (socket says 172.21.15.37:37894) 2024-03-28T18:07:47.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.749+0000 7f5e1fe00640 1 -- 172.21.15.37:0/2165185539 learned_addr learned my addr 172.21.15.37:0/2165185539 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:07:47.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.749+0000 7f5e24e00640 1 --2- 172.21.15.37:0/2165185539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e20105490 0x7f5e201a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:47.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.749+0000 7f5e1f400640 1 --2- 172.21.15.37:0/2165185539 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e20104ad0 0x7f5e2019a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:47.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.750+0000 7f5e1fe00640 1 -- 172.21.15.37:0/2165185539 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e20104ad0 msgr2=0x7f5e2019a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:47.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.750+0000 7f5e1fe00640 1 --2- 172.21.15.37:0/2165185539 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e20104ad0 0x7f5e2019a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:47.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.750+0000 7f5e1fe00640 1 -- 172.21.15.37:0/2165185539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e20105490 msgr2=0x7f5e201a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:47.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.750+0000 7f5e1fe00640 1 --2- 172.21.15.37:0/2165185539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e20105490 0x7f5e201a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:47.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.750+0000 7f5e1fe00640 1 -- 172.21.15.37:0/2165185539 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e201a17c0 con 0x7f5e201038e0 2024-03-28T18:07:47.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.750+0000 7f5e24e00640 1 --2- 172.21.15.37:0/2165185539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e20105490 0x7f5e201a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:07:47.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.750+0000 7f5e1fe00640 1 --2- 172.21.15.37:0/2165185539 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e201038e0 0x7f5e20199ba0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5e1000e9c0 tx=0x7f5e1000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:47.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.750+0000 7f5e1cc00640 1 -- 172.21.15.37:0/2165185539 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1000cd80 con 0x7f5e201038e0 2024-03-28T18:07:47.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.750+0000 7f5e26e00640 1 -- 172.21.15.37:0/2165185539 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e201a1ab0 con 0x7f5e201038e0 2024-03-28T18:07:47.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.751+0000 7f5e1cc00640 1 -- 172.21.15.37:0/2165185539 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e100035f0 con 0x7f5e201038e0 2024-03-28T18:07:47.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.751+0000 7f5e1cc00640 1 -- 172.21.15.37:0/2165185539 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e10010690 con 0x7f5e201038e0 2024-03-28T18:07:47.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.751+0000 7f5e26e00640 1 -- 172.21.15.37:0/2165185539 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e201a2050 con 0x7f5e201038e0 2024-03-28T18:07:47.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.752+0000 7f5e26e00640 1 -- 172.21.15.37:0/2165185539 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e2010ac00 con 0x7f5e201038e0 2024-03-28T18:07:47.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.754+0000 7f5e1cc00640 1 -- 172.21.15.37:0/2165185539 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5e100037a0 con 0x7f5e201038e0 2024-03-28T18:07:47.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.756+0000 7f5e1cc00640 1 --2- 172.21.15.37:0/2165185539 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5e040a2800 0x7f5e040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:47.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.757+0000 7f5e1cc00640 1 -- 172.21.15.37:0/2165185539 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5e100ceaa0 con 0x7f5e201038e0 2024-03-28T18:07:47.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.757+0000 7f5e1f400640 1 --2- 172.21.15.37:0/2165185539 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5e040a2800 0x7f5e040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:47.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.757+0000 7f5e1f400640 1 --2- 172.21.15.37:0/2165185539 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5e040a2800 0x7f5e040a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5e2019b0b0 tx=0x7f5e0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:47.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:47.760+0000 7f5e1cc00640 1 -- 172.21.15.37:0/2165185539 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5e1009bf40 con 0x7f5e201038e0 2024-03-28T18:07:47.866 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:07:47 smithi037 podman[70064]: 2024-03-28 18:07:47.602746755 +0000 UTC m=+0.452254476 container start 4b0671f3d1a0534f9ddce8a0b049c2139432de516dc34edaa49e43157365f884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-28T18:07:47.866 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:07:47 smithi037 ceph-mds[70099]: -- 172.21.15.37:0/2716549042 <== mon.0 v2:172.21.15.37:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5645b8a8da40 con 0x5645b8a60800 2024-03-28T18:07:47.866 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:07:47 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: starting mds.d at 2024-03-28T18:07:47.866 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:07:47 smithi037 bash[70064]: 4b0671f3d1a0534f9ddce8a0b049c2139432de516dc34edaa49e43157365f884 2024-03-28T18:07:47.866 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:07:47 smithi037 systemd[1]: Started Ceph mds.d for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:07:48.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:48.033+0000 7f5e26e00640 1 -- 172.21.15.37:0/2165185539 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e2002d070 con 0x7f5e201038e0 2024-03-28T18:07:48.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:48.034+0000 7f5e1cc00640 1 -- 172.21.15.37:0/2165185539 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3689 (secure 0 0 0) 0x7f5e2002d070 con 0x7f5e201038e0 2024-03-28T18:07:48.034 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:07:48.034 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[]} 2024-03-28T18:07:48.035 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 5 2024-03-28T18:07:48.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:48.037+0000 7f5e26e00640 1 -- 172.21.15.37:0/2165185539 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5e040a2800 msgr2=0x7f5e040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:48.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:48.037+0000 7f5e26e00640 1 --2- 172.21.15.37:0/2165185539 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5e040a2800 0x7f5e040a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5e2019b0b0 tx=0x7f5e0c005f90 comp rx=0 tx=0).stop 2024-03-28T18:07:48.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:48.037+0000 7f5e26e00640 1 -- 172.21.15.37:0/2165185539 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e201038e0 msgr2=0x7f5e20199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:48.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:48.037+0000 7f5e26e00640 1 --2- 172.21.15.37:0/2165185539 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e201038e0 0x7f5e20199ba0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5e1000e9c0 tx=0x7f5e1000ee80 comp rx=0 tx=0).stop 2024-03-28T18:07:48.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:48.037+0000 7f5e26e00640 1 -- 172.21.15.37:0/2165185539 shutdown_connections 2024-03-28T18:07:48.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:48.037+0000 7f5e26e00640 1 --2- 172.21.15.37:0/2165185539 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5e040a2800 0x7f5e040a4cc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:48.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:48.037+0000 7f5e26e00640 1 --2- 172.21.15.37:0/2165185539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e20105490 0x7f5e201a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:48.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:48.037+0000 7f5e26e00640 1 --2- 172.21.15.37:0/2165185539 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e20104ad0 0x7f5e2019a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:48.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:48.037+0000 7f5e26e00640 1 --2- 172.21.15.37:0/2165185539 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e201038e0 0x7f5e20199ba0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:48.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:48.037+0000 7f5e26e00640 1 -- 172.21.15.37:0/2165185539 >> 172.21.15.37:0/2165185539 conn(0x7f5e200fee50 msgr2=0x7f5e20107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:48.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:48.037+0000 7f5e26e00640 1 -- 172.21.15.37:0/2165185539 shutdown_connections 2024-03-28T18:07:48.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:48.037+0000 7f5e26e00640 1 -- 172.21.15.37:0/2165185539 wait complete. 2024-03-28T18:07:48.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:47 smithi037 ceph-mon[31460]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:48.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:47 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:48.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:47 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:48.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:47 smithi059 ceph-mon[36190]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:48.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:47 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:48.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:47 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:48.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:47 smithi170 ceph-mon[36280]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:48.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:47 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:48.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:47 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:49.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:48 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3972375740,v1:172.21.15.37:6837/3972375740] up:boot 2024-03-28T18:07:49.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:48 smithi037 ceph-mon[31460]: fsmap 4 up:standby 2024-03-28T18:07:49.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:48 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:07:49.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:48 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:49.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:48 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:49.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:48 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:49.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:48 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:49.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:48 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2165185539' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:48 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3972375740,v1:172.21.15.37:6837/3972375740] up:boot 2024-03-28T18:07:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:48 smithi059 ceph-mon[36190]: fsmap 4 up:standby 2024-03-28T18:07:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:48 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:07:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:48 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:48 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:48 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:48 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:49.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:48 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2165185539' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:49.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:48 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3972375740,v1:172.21.15.37:6837/3972375740] up:boot 2024-03-28T18:07:49.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:48 smithi170 ceph-mon[36280]: fsmap 4 up:standby 2024-03-28T18:07:49.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:48 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:07:49.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:48 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:49.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:48 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:49.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:48 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:49.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:48 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:49.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:48 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2165185539' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:50.059 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:07:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:49 smithi037 ceph-mon[31460]: Deploying daemon mds.e on smithi059 2024-03-28T18:07:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:49 smithi037 ceph-mon[31460]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:50.263 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:49 smithi059 ceph-mon[36190]: Deploying daemon mds.e on smithi059 2024-03-28T18:07:50.263 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:49 smithi059 ceph-mon[36190]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:50.263 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:07:50 smithi059 systemd[1]: Starting Ceph mds.e for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:07:50.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:49 smithi170 ceph-mon[36280]: Deploying daemon mds.e on smithi059 2024-03-28T18:07:50.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:49 smithi170 ceph-mon[36280]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:50.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.297+0000 7f27a0000640 1 -- 172.21.15.37:0/3380346082 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2798108ed0 msgr2=0x7f2798109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:50.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.297+0000 7f27a0000640 1 --2- 172.21.15.37:0/3380346082 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2798108ed0 0x7f2798109350 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2784009900 tx=0x7f278402bc30 comp rx=0 tx=0).stop 2024-03-28T18:07:50.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.297+0000 7f27a0000640 1 -- 172.21.15.37:0/3380346082 shutdown_connections 2024-03-28T18:07:50.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.297+0000 7f27a0000640 1 --2- 172.21.15.37:0/3380346082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2798109890 0x7f2798110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:50.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.297+0000 7f27a0000640 1 --2- 172.21.15.37:0/3380346082 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2798108ed0 0x7f2798109350 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:50.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.297+0000 7f27a0000640 1 --2- 172.21.15.37:0/3380346082 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2798107ce0 0x7f2798108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:50.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.297+0000 7f27a0000640 1 -- 172.21.15.37:0/3380346082 >> 172.21.15.37:0/3380346082 conn(0x7f2798076cb0 msgr2=0x7f27980790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:50.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.297+0000 7f27a0000640 1 -- 172.21.15.37:0/3380346082 shutdown_connections 2024-03-28T18:07:50.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.297+0000 7f27a0000640 1 -- 172.21.15.37:0/3380346082 wait complete. 2024-03-28T18:07:50.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.298+0000 7f27a0000640 1 Processor -- start 2024-03-28T18:07:50.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.298+0000 7f27a0000640 1 -- start start 2024-03-28T18:07:50.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.298+0000 7f27a0000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2798107ce0 0x7f279819dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:50.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.298+0000 7f27a0000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2798108ed0 0x7f279819e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:50.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.298+0000 7f27a0000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2798109890 0x7f27981a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:50.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.298+0000 7f27a0000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2798113940 con 0x7f2798109890 2024-03-28T18:07:50.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.298+0000 7f27a0000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27981137c0 con 0x7f2798108ed0 2024-03-28T18:07:50.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.298+0000 7f27a0000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2798113ac0 con 0x7f2798107ce0 2024-03-28T18:07:50.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.299+0000 7f279d600640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2798107ce0 0x7f279819dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:50.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.299+0000 7f279d600640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2798107ce0 0x7f279819dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:49078/0 (socket says 172.21.15.37:49078) 2024-03-28T18:07:50.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.299+0000 7f279d600640 1 -- 172.21.15.37:0/2905683553 learned_addr learned my addr 172.21.15.37:0/2905683553 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:07:50.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.299+0000 7f279e000640 1 --2- 172.21.15.37:0/2905683553 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2798109890 0x7f27981a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:50.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.299+0000 7f279cc00640 1 --2- 172.21.15.37:0/2905683553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2798108ed0 0x7f279819e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:50.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.299+0000 7f279d600640 1 -- 172.21.15.37:0/2905683553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2798108ed0 msgr2=0x7f279819e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:50.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.299+0000 7f279d600640 1 --2- 172.21.15.37:0/2905683553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2798108ed0 0x7f279819e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:50.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.299+0000 7f279d600640 1 -- 172.21.15.37:0/2905683553 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2798109890 msgr2=0x7f27981a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:50.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.299+0000 7f279d600640 1 --2- 172.21.15.37:0/2905683553 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2798109890 0x7f27981a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:50.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.299+0000 7f279d600640 1 -- 172.21.15.37:0/2905683553 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27981a5bd0 con 0x7f2798107ce0 2024-03-28T18:07:50.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.299+0000 7f279d600640 1 --2- 172.21.15.37:0/2905683553 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2798107ce0 0x7f279819dfb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f278c00e9c0 tx=0x7f278c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:50.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.300+0000 7f2796000640 1 -- 172.21.15.37:0/2905683553 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278c00cd80 con 0x7f2798107ce0 2024-03-28T18:07:50.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.300+0000 7f2796000640 1 -- 172.21.15.37:0/2905683553 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f278c0035f0 con 0x7f2798107ce0 2024-03-28T18:07:50.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.300+0000 7f27a0000640 1 -- 172.21.15.37:0/2905683553 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27981a5ec0 con 0x7f2798107ce0 2024-03-28T18:07:50.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.300+0000 7f2796000640 1 -- 172.21.15.37:0/2905683553 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278c010690 con 0x7f2798107ce0 2024-03-28T18:07:50.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.300+0000 7f27a0000640 1 -- 172.21.15.37:0/2905683553 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27981a6320 con 0x7f2798107ce0 2024-03-28T18:07:50.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.301+0000 7f27a0000640 1 -- 172.21.15.37:0/2905683553 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f279810f000 con 0x7f2798107ce0 2024-03-28T18:07:50.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.303+0000 7f2796000640 1 -- 172.21.15.37:0/2905683553 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f278c0037a0 con 0x7f2798107ce0 2024-03-28T18:07:50.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.304+0000 7f2796000640 1 --2- 172.21.15.37:0/2905683553 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f27800a2800 0x7f27800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:50.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.304+0000 7f2796000640 1 -- 172.21.15.37:0/2905683553 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f278c0cea40 con 0x7f2798107ce0 2024-03-28T18:07:50.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.305+0000 7f279cc00640 1 --2- 172.21.15.37:0/2905683553 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f27800a2800 0x7f27800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:50.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.305+0000 7f279cc00640 1 --2- 172.21.15.37:0/2905683553 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f27800a2800 0x7f27800a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f278402c7b0 tx=0x7f2784005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:50.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.308+0000 7f2796000640 1 -- 172.21.15.37:0/2905683553 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f278c09bee0 con 0x7f2798107ce0 2024-03-28T18:07:50.535 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:07:50 smithi059 podman[66151]: 2024-03-28 18:07:50.166623774 +0000 UTC m=+0.018285709 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:07:50.535 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:07:50 smithi059 podman[66151]: 2024-03-28 18:07:50.32348189 +0000 UTC m=+0.175143823 container create e836f5f0fe333e76e353802ed768f297f18f5d171dc1c10771ec0729586a0077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:07:50.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.555+0000 7f27a0000640 1 -- 172.21.15.37:0/2905683553 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f279802d070 con 0x7f2798107ce0 2024-03-28T18:07:50.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.556+0000 7f2796000640 1 -- 172.21.15.37:0/2905683553 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3689 (secure 0 0 0) 0x7f278c09b7e0 con 0x7f2798107ce0 2024-03-28T18:07:50.556 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:07:50.556 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[]} 2024-03-28T18:07:50.557 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 5 2024-03-28T18:07:50.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.559+0000 7f27a0000640 1 -- 172.21.15.37:0/2905683553 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f27800a2800 msgr2=0x7f27800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:50.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.559+0000 7f27a0000640 1 --2- 172.21.15.37:0/2905683553 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f27800a2800 0x7f27800a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f278402c7b0 tx=0x7f2784005fd0 comp rx=0 tx=0).stop 2024-03-28T18:07:50.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.559+0000 7f27a0000640 1 -- 172.21.15.37:0/2905683553 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2798107ce0 msgr2=0x7f279819dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:50.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.559+0000 7f27a0000640 1 --2- 172.21.15.37:0/2905683553 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2798107ce0 0x7f279819dfb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f278c00e9c0 tx=0x7f278c00ee80 comp rx=0 tx=0).stop 2024-03-28T18:07:50.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.559+0000 7f27a0000640 1 -- 172.21.15.37:0/2905683553 shutdown_connections 2024-03-28T18:07:50.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.559+0000 7f27a0000640 1 --2- 172.21.15.37:0/2905683553 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f27800a2800 0x7f27800a4cc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:50.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.559+0000 7f27a0000640 1 --2- 172.21.15.37:0/2905683553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2798108ed0 0x7f279819e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:50.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.559+0000 7f27a0000640 1 --2- 172.21.15.37:0/2905683553 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2798109890 0x7f27981a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:50.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.559+0000 7f27a0000640 1 --2- 172.21.15.37:0/2905683553 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2798107ce0 0x7f279819dfb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:50.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.559+0000 7f27a0000640 1 -- 172.21.15.37:0/2905683553 >> 172.21.15.37:0/2905683553 conn(0x7f2798076cb0 msgr2=0x7f279810c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:50.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.559+0000 7f27a0000640 1 -- 172.21.15.37:0/2905683553 shutdown_connections 2024-03-28T18:07:50.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:50.559+0000 7f27a0000640 1 -- 172.21.15.37:0/2905683553 wait complete. 2024-03-28T18:07:50.882 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:07:50 smithi059 podman[66151]: 2024-03-28 18:07:50.593071017 +0000 UTC m=+0.444732972 container init e836f5f0fe333e76e353802ed768f297f18f5d171dc1c10771ec0729586a0077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:07:50.883 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:07:50 smithi059 podman[66151]: 2024-03-28 18:07:50.595850095 +0000 UTC m=+0.447512041 container start e836f5f0fe333e76e353802ed768f297f18f5d171dc1c10771ec0729586a0077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-28T18:07:50.883 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:07:50 smithi059 ceph-mds[66171]: -- 172.21.15.59:0/1374165187 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557f7c0fba40 con 0x557f7c0cf000 2024-03-28T18:07:50.883 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:07:50 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: starting mds.e at 2024-03-28T18:07:50.883 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:07:50 smithi059 bash[66151]: e836f5f0fe333e76e353802ed768f297f18f5d171dc1c10771ec0729586a0077 2024-03-28T18:07:50.883 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:07:50 smithi059 systemd[1]: Started Ceph mds.e for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:07:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:50 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2905683553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:50 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:50 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:50 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:50 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:50 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:50 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:50 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2905683553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:50 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:50 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:50 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:50 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:50 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:50 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:50 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2905683553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:51.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:50 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:51.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:50 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:51.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:50 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:51.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:50 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:51.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:50 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:51.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:50 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:51 smithi170 ceph-mon[36280]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:51 smithi170 ceph-mon[36280]: Deploying daemon mds.f on smithi170 2024-03-28T18:07:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:51 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/2253706559,v1:172.21.15.59:6835/2253706559] up:boot 2024-03-28T18:07:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:51 smithi170 ceph-mon[36280]: fsmap 5 up:standby 2024-03-28T18:07:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:51 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:07:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:51 smithi059 ceph-mon[36190]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:51 smithi059 ceph-mon[36190]: Deploying daemon mds.f on smithi170 2024-03-28T18:07:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:51 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/2253706559,v1:172.21.15.59:6835/2253706559] up:boot 2024-03-28T18:07:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:51 smithi059 ceph-mon[36190]: fsmap 5 up:standby 2024-03-28T18:07:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:51 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:07:52.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:51 smithi037 ceph-mon[31460]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:52.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:51 smithi037 ceph-mon[31460]: Deploying daemon mds.f on smithi170 2024-03-28T18:07:52.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:51 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/2253706559,v1:172.21.15.59:6835/2253706559] up:boot 2024-03-28T18:07:52.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:51 smithi037 ceph-mon[31460]: fsmap 5 up:standby 2024-03-28T18:07:52.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:51 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:07:52.581 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:07:52.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.818+0000 7fdce3000640 1 -- 172.21.15.37:0/2204659401 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcdc104ad0 msgr2=0x7fdcdc104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:52.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.818+0000 7fdce3000640 1 --2- 172.21.15.37:0/2204659401 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcdc104ad0 0x7fdcdc104f50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8009900 tx=0x7fdcc802bc50 comp rx=0 tx=0).stop 2024-03-28T18:07:52.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.818+0000 7fdce3000640 1 -- 172.21.15.37:0/2204659401 shutdown_connections 2024-03-28T18:07:52.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.818+0000 7fdce3000640 1 --2- 172.21.15.37:0/2204659401 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcdc105490 0x7fdcdc10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:52.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.818+0000 7fdce3000640 1 --2- 172.21.15.37:0/2204659401 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcdc104ad0 0x7fdcdc104f50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:52.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.818+0000 7fdce3000640 1 --2- 172.21.15.37:0/2204659401 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcdc1038e0 0x7fdcdc103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:52.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.818+0000 7fdce3000640 1 -- 172.21.15.37:0/2204659401 >> 172.21.15.37:0/2204659401 conn(0x7fdcdc0fee50 msgr2=0x7fdcdc101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:52.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.818+0000 7fdce3000640 1 -- 172.21.15.37:0/2204659401 shutdown_connections 2024-03-28T18:07:52.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.818+0000 7fdce3000640 1 -- 172.21.15.37:0/2204659401 wait complete. 2024-03-28T18:07:52.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.819+0000 7fdce3000640 1 Processor -- start 2024-03-28T18:07:52.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.819+0000 7fdce3000640 1 -- start start 2024-03-28T18:07:52.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.819+0000 7fdce3000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcdc1038e0 0x7fdcdc199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:52.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.819+0000 7fdce3000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcdc104ad0 0x7fdcdc19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:52.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.819+0000 7fdce3000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcdc105490 0x7fdcdc1a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:52.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.819+0000 7fdce3000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc10f5b0 con 0x7fdcdc1038e0 2024-03-28T18:07:52.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.819+0000 7fdce3000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc10f430 con 0x7fdcdc104ad0 2024-03-28T18:07:52.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.819+0000 7fdce3000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc10f730 con 0x7fdcdc105490 2024-03-28T18:07:52.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.819+0000 7fdcdbe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcdc1038e0 0x7fdcdc199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:52.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.819+0000 7fdce1000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcdc105490 0x7fdcdc1a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:52.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.819+0000 7fdcdbe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcdc1038e0 0x7fdcdc199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37910/0 (socket says 172.21.15.37:37910) 2024-03-28T18:07:52.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.819+0000 7fdcdbe00640 1 -- 172.21.15.37:0/3771191726 learned_addr learned my addr 172.21.15.37:0/3771191726 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:07:52.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.820+0000 7fdcdb400640 1 --2- 172.21.15.37:0/3771191726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcdc104ad0 0x7fdcdc19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:52.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.820+0000 7fdcdbe00640 1 -- 172.21.15.37:0/3771191726 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcdc105490 msgr2=0x7fdcdc1a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:52.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.820+0000 7fdcdbe00640 1 --2- 172.21.15.37:0/3771191726 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcdc105490 0x7fdcdc1a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:52.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.820+0000 7fdcdbe00640 1 -- 172.21.15.37:0/3771191726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcdc104ad0 msgr2=0x7fdcdc19a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:52.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.820+0000 7fdcdbe00640 1 --2- 172.21.15.37:0/3771191726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcdc104ad0 0x7fdcdc19a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:52.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.820+0000 7fdcdbe00640 1 -- 172.21.15.37:0/3771191726 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcdc1a1880 con 0x7fdcdc1038e0 2024-03-28T18:07:52.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.820+0000 7fdcdbe00640 1 --2- 172.21.15.37:0/3771191726 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcdc1038e0 0x7fdcdc199c70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdccc00e9c0 tx=0x7fdccc00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:52.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.820+0000 7fdcd8c00640 1 -- 172.21.15.37:0/3771191726 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc00ccc0 con 0x7fdcdc1038e0 2024-03-28T18:07:52.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.820+0000 7fdcd8c00640 1 -- 172.21.15.37:0/3771191726 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdccc0035f0 con 0x7fdcdc1038e0 2024-03-28T18:07:52.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.821+0000 7fdce3000640 1 -- 172.21.15.37:0/3771191726 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcdc1a1b70 con 0x7fdcdc1038e0 2024-03-28T18:07:52.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.821+0000 7fdcd8c00640 1 -- 172.21.15.37:0/3771191726 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc00ce60 con 0x7fdcdc1038e0 2024-03-28T18:07:52.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.821+0000 7fdce3000640 1 -- 172.21.15.37:0/3771191726 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcdc1a20c0 con 0x7fdcdc1038e0 2024-03-28T18:07:52.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.822+0000 7fdce3000640 1 -- 172.21.15.37:0/3771191726 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcdc10ac00 con 0x7fdcdc1038e0 2024-03-28T18:07:52.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.824+0000 7fdcd8c00640 1 -- 172.21.15.37:0/3771191726 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdccc0037a0 con 0x7fdcdc1038e0 2024-03-28T18:07:52.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.825+0000 7fdcd8c00640 1 --2- 172.21.15.37:0/3771191726 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fdcc00a2880 0x7fdcc00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:52.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.825+0000 7fdcd8c00640 1 -- 172.21.15.37:0/3771191726 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdccc0ceb70 con 0x7fdcdc1038e0 2024-03-28T18:07:52.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.826+0000 7fdcdb400640 1 --2- 172.21.15.37:0/3771191726 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fdcc00a2880 0x7fdcc00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:52.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.826+0000 7fdcdb400640 1 --2- 172.21.15.37:0/3771191726 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fdcc00a2880 0x7fdcc00a4d40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdcc802c260 tx=0x7fdcc8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:52.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:52.828+0000 7fdcd8c00640 1 -- 172.21.15.37:0/3771191726 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdccc09c010 con 0x7fdcdc1038e0 2024-03-28T18:07:53.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:53.070+0000 7fdce3000640 1 -- 172.21.15.37:0/3771191726 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcdc02d070 con 0x7fdcdc1038e0 2024-03-28T18:07:53.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:53.072+0000 7fdcd8c00640 1 -- 172.21.15.37:0/3771191726 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4480 (secure 0 0 0) 0x7fdccc09b910 con 0x7fdcdc1038e0 2024-03-28T18:07:53.072 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:07:53.072 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[]} 2024-03-28T18:07:53.072 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 6 2024-03-28T18:07:53.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:53.074+0000 7fdce3000640 1 -- 172.21.15.37:0/3771191726 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fdcc00a2880 msgr2=0x7fdcc00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:53.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:53.074+0000 7fdce3000640 1 --2- 172.21.15.37:0/3771191726 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fdcc00a2880 0x7fdcc00a4d40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdcc802c260 tx=0x7fdcc8005fd0 comp rx=0 tx=0).stop 2024-03-28T18:07:53.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:53.074+0000 7fdce3000640 1 -- 172.21.15.37:0/3771191726 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcdc1038e0 msgr2=0x7fdcdc199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:53.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:53.074+0000 7fdce3000640 1 --2- 172.21.15.37:0/3771191726 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcdc1038e0 0x7fdcdc199c70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdccc00e9c0 tx=0x7fdccc00edc0 comp rx=0 tx=0).stop 2024-03-28T18:07:53.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:53.074+0000 7fdce3000640 1 -- 172.21.15.37:0/3771191726 shutdown_connections 2024-03-28T18:07:53.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:53.074+0000 7fdce3000640 1 --2- 172.21.15.37:0/3771191726 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fdcc00a2880 0x7fdcc00a4d40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:53.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:53.074+0000 7fdce3000640 1 --2- 172.21.15.37:0/3771191726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcdc104ad0 0x7fdcdc19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:53.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:53.074+0000 7fdce3000640 1 --2- 172.21.15.37:0/3771191726 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcdc105490 0x7fdcdc1a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:53.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:53.074+0000 7fdce3000640 1 --2- 172.21.15.37:0/3771191726 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcdc1038e0 0x7fdcdc199c70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:53.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:53.074+0000 7fdce3000640 1 -- 172.21.15.37:0/3771191726 >> 172.21.15.37:0/3771191726 conn(0x7fdcdc0fee50 msgr2=0x7fdcdc107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:53.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:53.074+0000 7fdce3000640 1 -- 172.21.15.37:0/3771191726 shutdown_connections 2024-03-28T18:07:53.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:53.074+0000 7fdce3000640 1 -- 172.21.15.37:0/3771191726 wait complete. 2024-03-28T18:07:53.216 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:07:53 smithi170 systemd[1]: Starting Ceph mds.f for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:07:53.510 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:07:53 smithi170 podman[66020]: 2024-03-28 18:07:53.119585185 +0000 UTC m=+0.017787139 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:07:53.510 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:07:53 smithi170 podman[66020]: 2024-03-28 18:07:53.249975803 +0000 UTC m=+0.148177793 container create ac5c3af1d16c4f15ddb809bdc133f8174963bc5fd5ca9853d18e8a059b4c53b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-28T18:07:53.510 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:07:53 smithi170 podman[66020]: 2024-03-28 18:07:53.510785149 +0000 UTC m=+0.408987097 container init ac5c3af1d16c4f15ddb809bdc133f8174963bc5fd5ca9853d18e8a059b4c53b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-28T18:07:53.785 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:07:53 smithi170 podman[66020]: 2024-03-28 18:07:53.513432056 +0000 UTC m=+0.411634007 container start ac5c3af1d16c4f15ddb809bdc133f8174963bc5fd5ca9853d18e8a059b4c53b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:07:53.785 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:07:53 smithi170 ceph-mds[66039]: -- 172.21.15.170:0/3501484661 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55adfdcd7a40 con 0x55adfdcaa800 2024-03-28T18:07:53.786 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:07:53 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: starting mds.f at 2024-03-28T18:07:53.786 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:07:53 smithi170 bash[66020]: ac5c3af1d16c4f15ddb809bdc133f8174963bc5fd5ca9853d18e8a059b4c53b0 2024-03-28T18:07:53.786 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:07:53 smithi170 systemd[1]: Started Ceph mds.f for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:07:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:53 smithi059 ceph-mon[36190]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:53 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3771191726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:54.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:53 smithi170 ceph-mon[36280]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:53 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3771191726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:54.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:53 smithi037 ceph-mon[31460]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:53 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3771191726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:55.009 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:54 smithi170 ceph-mon[36280]: Deploying daemon mds.c on smithi170 2024-03-28T18:07:55.009 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:54 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/4133178144,v1:172.21.15.170:6835/4133178144] up:boot 2024-03-28T18:07:55.009 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:54 smithi170 ceph-mon[36280]: fsmap 6 up:standby 2024-03-28T18:07:55.009 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:54 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:07:55.009 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:54 smithi170 ceph-mon[36280]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:55.095 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:07:55.186 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:54 smithi037 ceph-mon[31460]: Deploying daemon mds.c on smithi170 2024-03-28T18:07:55.186 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:54 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/4133178144,v1:172.21.15.170:6835/4133178144] up:boot 2024-03-28T18:07:55.186 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:54 smithi037 ceph-mon[31460]: fsmap 6 up:standby 2024-03-28T18:07:55.186 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:54 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:07:55.186 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:54 smithi037 ceph-mon[31460]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:55.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:54 smithi059 ceph-mon[36190]: Deploying daemon mds.c on smithi170 2024-03-28T18:07:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:54 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/4133178144,v1:172.21.15.170:6835/4133178144] up:boot 2024-03-28T18:07:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:54 smithi059 ceph-mon[36190]: fsmap 6 up:standby 2024-03-28T18:07:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:54 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:07:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:54 smithi059 ceph-mon[36190]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:55.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.327+0000 7f13e9c00640 1 -- 172.21.15.37:0/3811699663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13e41048e0 msgr2=0x7f13e4104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:55.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.327+0000 7f13e9c00640 1 --2- 172.21.15.37:0/3811699663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13e41048e0 0x7f13e4104d60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f13cc009900 tx=0x7f13cc02bc30 comp rx=0 tx=0).stop 2024-03-28T18:07:55.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.328+0000 7f13e9c00640 1 -- 172.21.15.37:0/3811699663 shutdown_connections 2024-03-28T18:07:55.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.328+0000 7f13e9c00640 1 --2- 172.21.15.37:0/3811699663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13e41052a0 0x7f13e410bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:55.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.328+0000 7f13e9c00640 1 --2- 172.21.15.37:0/3811699663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13e41048e0 0x7f13e4104d60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:55.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.328+0000 7f13e9c00640 1 --2- 172.21.15.37:0/3811699663 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13e41036f0 0x7f13e4103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:55.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.328+0000 7f13e9c00640 1 -- 172.21.15.37:0/3811699663 >> 172.21.15.37:0/3811699663 conn(0x7f13e40fec80 msgr2=0x7f13e41010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:55.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.328+0000 7f13e9c00640 1 -- 172.21.15.37:0/3811699663 shutdown_connections 2024-03-28T18:07:55.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.328+0000 7f13e9c00640 1 -- 172.21.15.37:0/3811699663 wait complete. 2024-03-28T18:07:55.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.329+0000 7f13e9c00640 1 Processor -- start 2024-03-28T18:07:55.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.329+0000 7f13e9c00640 1 -- start start 2024-03-28T18:07:55.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.329+0000 7f13e9c00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13e41036f0 0x7f13e41999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:55.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.329+0000 7f13e9c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13e41048e0 0x7f13e4199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:55.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.329+0000 7f13e9c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13e41052a0 0x7f13e41a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:55.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.329+0000 7f13e9c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e410f3c0 con 0x7f13e41048e0 2024-03-28T18:07:55.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.329+0000 7f13e9c00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e410f240 con 0x7f13e41052a0 2024-03-28T18:07:55.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.329+0000 7f13e9c00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e410f540 con 0x7f13e41036f0 2024-03-28T18:07:55.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.329+0000 7f13e3400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13e41048e0 0x7f13e4199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:55.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.329+0000 7f13e3400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13e41048e0 0x7f13e4199f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:42470/0 (socket says 172.21.15.37:42470) 2024-03-28T18:07:55.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.329+0000 7f13e3400640 1 -- 172.21.15.37:0/2038806558 learned_addr learned my addr 172.21.15.37:0/2038806558 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:07:55.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.329+0000 7f13e3e00640 1 --2- 172.21.15.37:0/2038806558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13e41036f0 0x7f13e41999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:55.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.330+0000 7f13e9200640 1 --2- 172.21.15.37:0/2038806558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13e41052a0 0x7f13e41a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:55.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.330+0000 7f13e3400640 1 -- 172.21.15.37:0/2038806558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13e41036f0 msgr2=0x7f13e41999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:55.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.330+0000 7f13e3400640 1 --2- 172.21.15.37:0/2038806558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13e41036f0 0x7f13e41999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:55.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.330+0000 7f13e3400640 1 -- 172.21.15.37:0/2038806558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13e41052a0 msgr2=0x7f13e41a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:55.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.330+0000 7f13e3400640 1 --2- 172.21.15.37:0/2038806558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13e41052a0 0x7f13e41a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:55.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.330+0000 7f13e3400640 1 -- 172.21.15.37:0/2038806558 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e41a15f0 con 0x7f13e41048e0 2024-03-28T18:07:55.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.330+0000 7f13e3400640 1 --2- 172.21.15.37:0/2038806558 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13e41048e0 0x7f13e4199f10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f13cc000c00 tx=0x7f13cc02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:55.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.331+0000 7f13e0c00640 1 -- 172.21.15.37:0/2038806558 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13cc043020 con 0x7f13e41048e0 2024-03-28T18:07:55.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.331+0000 7f13e0c00640 1 -- 172.21.15.37:0/2038806558 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f13cc0054e0 con 0x7f13e41048e0 2024-03-28T18:07:55.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.331+0000 7f13e9c00640 1 -- 172.21.15.37:0/2038806558 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e41a1880 con 0x7f13e41048e0 2024-03-28T18:07:55.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.331+0000 7f13e0c00640 1 -- 172.21.15.37:0/2038806558 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13cc003c40 con 0x7f13e41048e0 2024-03-28T18:07:55.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.331+0000 7f13e9c00640 1 -- 172.21.15.37:0/2038806558 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e41a1dd0 con 0x7f13e41048e0 2024-03-28T18:07:55.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.332+0000 7f13e9c00640 1 -- 172.21.15.37:0/2038806558 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e410aa10 con 0x7f13e41048e0 2024-03-28T18:07:55.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.334+0000 7f13e0c00640 1 -- 172.21.15.37:0/2038806558 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f13cc046020 con 0x7f13e41048e0 2024-03-28T18:07:55.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.335+0000 7f13e0c00640 1 --2- 172.21.15.37:0/2038806558 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f13c80a2800 0x7f13c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:55.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.335+0000 7f13e0c00640 1 -- 172.21.15.37:0/2038806558 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f13cc0f06d0 con 0x7f13e41048e0 2024-03-28T18:07:55.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.336+0000 7f13e3e00640 1 --2- 172.21.15.37:0/2038806558 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f13c80a2800 0x7f13c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:55.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.336+0000 7f13e3e00640 1 --2- 172.21.15.37:0/2038806558 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f13c80a2800 0x7f13c80a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f13e4104740 tx=0x7f13d4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:55.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.341+0000 7f13e0c00640 1 -- 172.21.15.37:0/2038806558 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f13cc04d020 con 0x7f13e41048e0 2024-03-28T18:07:55.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.585+0000 7f13e9c00640 1 -- 172.21.15.37:0/2038806558 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13e402d070 con 0x7f13e41048e0 2024-03-28T18:07:55.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.586+0000 7f13e0c00640 1 -- 172.21.15.37:0/2038806558 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5274 (secure 0 0 0) 0x7f13cc0bdb70 con 0x7f13e41048e0 2024-03-28T18:07:55.586 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:07:55.587 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-28T18:07:55.587 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 7 2024-03-28T18:07:55.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.589+0000 7f13e9c00640 1 -- 172.21.15.37:0/2038806558 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f13c80a2800 msgr2=0x7f13c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:55.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.589+0000 7f13e9c00640 1 --2- 172.21.15.37:0/2038806558 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f13c80a2800 0x7f13c80a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f13e4104740 tx=0x7f13d4009500 comp rx=0 tx=0).stop 2024-03-28T18:07:55.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.589+0000 7f13e9c00640 1 -- 172.21.15.37:0/2038806558 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13e41048e0 msgr2=0x7f13e4199f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:55.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.589+0000 7f13e9c00640 1 --2- 172.21.15.37:0/2038806558 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13e41048e0 0x7f13e4199f10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f13cc000c00 tx=0x7f13cc02c650 comp rx=0 tx=0).stop 2024-03-28T18:07:55.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.589+0000 7f13e9c00640 1 -- 172.21.15.37:0/2038806558 shutdown_connections 2024-03-28T18:07:55.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.589+0000 7f13e9c00640 1 --2- 172.21.15.37:0/2038806558 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f13c80a2800 0x7f13c80a4cc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:55.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.589+0000 7f13e9c00640 1 --2- 172.21.15.37:0/2038806558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13e41052a0 0x7f13e41a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:55.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.589+0000 7f13e9c00640 1 --2- 172.21.15.37:0/2038806558 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13e41048e0 0x7f13e4199f10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:55.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.589+0000 7f13e9c00640 1 --2- 172.21.15.37:0/2038806558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13e41036f0 0x7f13e41999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:55.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.589+0000 7f13e9c00640 1 -- 172.21.15.37:0/2038806558 >> 172.21.15.37:0/2038806558 conn(0x7f13e40fec80 msgr2=0x7f13e4107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:55.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.590+0000 7f13e9c00640 1 -- 172.21.15.37:0/2038806558 shutdown_connections 2024-03-28T18:07:55.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:55.590+0000 7f13e9c00640 1 -- 172.21.15.37:0/2038806558 wait complete. 2024-03-28T18:07:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:55 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2038806558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:55 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2038806558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:56.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:55 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2038806558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:56.785 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 28 18:07:56 smithi170 systemd[1]: Started Ceph mds.c for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:07:57.610 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:07:57.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.846+0000 7f4cce000640 1 -- 172.21.15.37:0/1684143371 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cc81038e0 msgr2=0x7f4cc8103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:57.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.846+0000 7f4cce000640 1 --2- 172.21.15.37:0/1684143371 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cc81038e0 0x7f4cc8103d00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8009900 tx=0x7f4cb802bc40 comp rx=0 tx=0).stop 2024-03-28T18:07:57.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.846+0000 7f4cce000640 1 -- 172.21.15.37:0/1684143371 shutdown_connections 2024-03-28T18:07:57.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.846+0000 7f4cce000640 1 --2- 172.21.15.37:0/1684143371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cc8105490 0x7f4cc810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:57.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.846+0000 7f4cce000640 1 --2- 172.21.15.37:0/1684143371 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4cc8104ad0 0x7f4cc8104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:57.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.846+0000 7f4cce000640 1 --2- 172.21.15.37:0/1684143371 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cc81038e0 0x7f4cc8103d00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:57.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.846+0000 7f4cce000640 1 -- 172.21.15.37:0/1684143371 >> 172.21.15.37:0/1684143371 conn(0x7f4cc80fee50 msgr2=0x7f4cc8101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:57.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.846+0000 7f4cce000640 1 -- 172.21.15.37:0/1684143371 shutdown_connections 2024-03-28T18:07:57.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.846+0000 7f4cce000640 1 -- 172.21.15.37:0/1684143371 wait complete. 2024-03-28T18:07:57.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.847+0000 7f4cce000640 1 Processor -- start 2024-03-28T18:07:57.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.847+0000 7f4cce000640 1 -- start start 2024-03-28T18:07:57.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.848+0000 7f4cce000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cc81038e0 0x7f4cc8199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:57.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.848+0000 7f4cce000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cc8104ad0 0x7f4cc819a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:57.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.848+0000 7f4cce000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4cc8105490 0x7f4cc81a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:57.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.848+0000 7f4cce000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc810f580 con 0x7f4cc8105490 2024-03-28T18:07:57.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.848+0000 7f4cce000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc810f400 con 0x7f4cc81038e0 2024-03-28T18:07:57.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.848+0000 7f4cce000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc810f700 con 0x7f4cc8104ad0 2024-03-28T18:07:57.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.848+0000 7f4cc7400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cc81038e0 0x7f4cc8199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:57.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.848+0000 7f4cc7e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4cc8105490 0x7f4cc81a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:57.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.848+0000 7f4cc7400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cc81038e0 0x7f4cc8199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:49910/0 (socket says 172.21.15.37:49910) 2024-03-28T18:07:57.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.848+0000 7f4cc7400640 1 -- 172.21.15.37:0/4055758823 learned_addr learned my addr 172.21.15.37:0/4055758823 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:07:57.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.848+0000 7f4cc6a00640 1 --2- 172.21.15.37:0/4055758823 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cc8104ad0 0x7f4cc819a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:57.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.848+0000 7f4cc7400640 1 -- 172.21.15.37:0/4055758823 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cc8104ad0 msgr2=0x7f4cc819a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:57.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.848+0000 7f4cc7400640 1 --2- 172.21.15.37:0/4055758823 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cc8104ad0 0x7f4cc819a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:57.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.848+0000 7f4cc7400640 1 -- 172.21.15.37:0/4055758823 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4cc8105490 msgr2=0x7f4cc81a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:57.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.848+0000 7f4cc7400640 1 --2- 172.21.15.37:0/4055758823 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4cc8105490 0x7f4cc81a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:57.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.848+0000 7f4cc7400640 1 -- 172.21.15.37:0/4055758823 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc81a1850 con 0x7f4cc81038e0 2024-03-28T18:07:57.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.848+0000 7f4cc7e00640 1 --2- 172.21.15.37:0/4055758823 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4cc8105490 0x7f4cc81a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:07:57.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.849+0000 7f4cc7400640 1 --2- 172.21.15.37:0/4055758823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cc81038e0 0x7f4cc8199c40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4cb802efd0 tx=0x7f4cb802c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:57.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.849+0000 7f4cbfe00640 1 -- 172.21.15.37:0/4055758823 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8041070 con 0x7f4cc81038e0 2024-03-28T18:07:57.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.849+0000 7f4cbfe00640 1 -- 172.21.15.37:0/4055758823 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4cb8004450 con 0x7f4cc81038e0 2024-03-28T18:07:57.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.849+0000 7f4cce000640 1 -- 172.21.15.37:0/4055758823 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc81a1ae0 con 0x7f4cc81038e0 2024-03-28T18:07:57.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.849+0000 7f4cbfe00640 1 -- 172.21.15.37:0/4055758823 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8003d70 con 0x7f4cc81038e0 2024-03-28T18:07:57.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.849+0000 7f4cce000640 1 -- 172.21.15.37:0/4055758823 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc81a1fa0 con 0x7f4cc81038e0 2024-03-28T18:07:57.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.851+0000 7f4cce000640 1 -- 172.21.15.37:0/4055758823 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc810ac00 con 0x7f4cc81038e0 2024-03-28T18:07:57.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.853+0000 7f4cbfe00640 1 -- 172.21.15.37:0/4055758823 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4cb8044070 con 0x7f4cc81038e0 2024-03-28T18:07:57.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.854+0000 7f4cbfe00640 1 --2- 172.21.15.37:0/4055758823 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f4cac0a2800 0x7f4cac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:07:57.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.854+0000 7f4cbfe00640 1 -- 172.21.15.37:0/4055758823 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4cb80f0260 con 0x7f4cc81038e0 2024-03-28T18:07:57.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.854+0000 7f4cc6a00640 1 --2- 172.21.15.37:0/4055758823 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f4cac0a2800 0x7f4cac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:07:57.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.855+0000 7f4cc6a00640 1 --2- 172.21.15.37:0/4055758823 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f4cac0a2800 0x7f4cac0a4cc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009bd0 tx=0x7f4cb4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:07:57.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:57.858+0000 7f4cbfe00640 1 -- 172.21.15.37:0/4055758823 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4cb80bd700 con 0x7f4cc81038e0 2024-03-28T18:07:57.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:57 smithi037 ceph-mon[31460]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:57.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:57.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:57.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:57.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:57.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:57.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:57.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:57 smithi037 ceph-mon[31460]: Deploying daemon mds.h on smithi059 2024-03-28T18:07:57.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:57 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6836/1308157923,v1:172.21.15.170:6837/1308157923] up:boot 2024-03-28T18:07:57.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:57 smithi037 ceph-mon[31460]: fsmap 7 up:standby 2024-03-28T18:07:57.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:57 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:07:58.025 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:57 smithi059 ceph-mon[36190]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:58.025 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:57 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:58.026 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:57 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:58.026 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:57 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:58.026 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:57 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:58.026 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:57 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:58.026 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:57 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:58.026 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:57 smithi059 ceph-mon[36190]: Deploying daemon mds.h on smithi059 2024-03-28T18:07:58.026 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:57 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6836/1308157923,v1:172.21.15.170:6837/1308157923] up:boot 2024-03-28T18:07:58.026 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:57 smithi059 ceph-mon[36190]: fsmap 7 up:standby 2024-03-28T18:07:58.026 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:57 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:07:58.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:57 smithi170 ceph-mon[36280]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:58.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:57 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:58.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:57 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:58.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:57 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:58.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:57 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:58.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:57 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:58.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:57 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:58.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:57 smithi170 ceph-mon[36280]: Deploying daemon mds.h on smithi059 2024-03-28T18:07:58.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:57 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6836/1308157923,v1:172.21.15.170:6837/1308157923] up:boot 2024-03-28T18:07:58.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:57 smithi170 ceph-mon[36280]: fsmap 7 up:standby 2024-03-28T18:07:58.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:57 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:07:58.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:58.104+0000 7f4cce000640 1 -- 172.21.15.37:0/4055758823 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cc802d070 con 0x7f4cc81038e0 2024-03-28T18:07:58.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:58.105+0000 7f4cbfe00640 1 -- 172.21.15.37:0/4055758823 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6068 (secure 0 0 0) 0x7f4cb80bd000 con 0x7f4cc81038e0 2024-03-28T18:07:58.105 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:07:58.106 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-28T18:07:58.106 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 8 2024-03-28T18:07:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:58.109+0000 7f4cce000640 1 -- 172.21.15.37:0/4055758823 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f4cac0a2800 msgr2=0x7f4cac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:58.109+0000 7f4cce000640 1 --2- 172.21.15.37:0/4055758823 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f4cac0a2800 0x7f4cac0a4cc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009bd0 tx=0x7f4cb4005f90 comp rx=0 tx=0).stop 2024-03-28T18:07:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:58.109+0000 7f4cce000640 1 -- 172.21.15.37:0/4055758823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cc81038e0 msgr2=0x7f4cc8199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:07:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:58.109+0000 7f4cce000640 1 --2- 172.21.15.37:0/4055758823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cc81038e0 0x7f4cc8199c40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4cb802efd0 tx=0x7f4cb802c660 comp rx=0 tx=0).stop 2024-03-28T18:07:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:58.109+0000 7f4cce000640 1 -- 172.21.15.37:0/4055758823 shutdown_connections 2024-03-28T18:07:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:58.109+0000 7f4cce000640 1 --2- 172.21.15.37:0/4055758823 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f4cac0a2800 0x7f4cac0a4cc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:58.109+0000 7f4cce000640 1 --2- 172.21.15.37:0/4055758823 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4cc8105490 0x7f4cc81a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:58.109+0000 7f4cce000640 1 --2- 172.21.15.37:0/4055758823 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cc8104ad0 0x7f4cc819a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:58.109+0000 7f4cce000640 1 --2- 172.21.15.37:0/4055758823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cc81038e0 0x7f4cc8199c40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:07:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:58.109+0000 7f4cce000640 1 -- 172.21.15.37:0/4055758823 >> 172.21.15.37:0/4055758823 conn(0x7f4cc80fee50 msgr2=0x7f4cc8107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:07:58.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:58.109+0000 7f4cce000640 1 -- 172.21.15.37:0/4055758823 shutdown_connections 2024-03-28T18:07:58.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:07:58.109+0000 7f4cce000640 1 -- 172.21.15.37:0/4055758823 wait complete. 2024-03-28T18:07:58.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:58 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/4055758823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:58 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/4055758823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:59.035 INFO:journalctl@ceph.mds.h.smithi059.stdout:Mar 28 18:07:58 smithi059 systemd[1]: Starting Ceph mds.h for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:07:59.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:58 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/4055758823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:07:59.298 INFO:journalctl@ceph.mds.h.smithi059.stdout:Mar 28 18:07:59 smithi059 podman[66459]: 2024-03-28 18:07:58.949897253 +0000 UTC m=+0.018065960 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:07:59.298 INFO:journalctl@ceph.mds.h.smithi059.stdout:Mar 28 18:07:59 smithi059 podman[66459]: 2024-03-28 18:07:59.071316597 +0000 UTC m=+0.139485305 container create 1e6766595523a6768d01891d1f7e3eb3a7f9ee953527df8a16a92d15764d1d8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-h, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-28T18:07:59.585 INFO:journalctl@ceph.mds.h.smithi059.stdout:Mar 28 18:07:59 smithi059 podman[66459]: 2024-03-28 18:07:59.299100104 +0000 UTC m=+0.367268801 container init 1e6766595523a6768d01891d1f7e3eb3a7f9ee953527df8a16a92d15764d1d8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-h, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2) 2024-03-28T18:07:59.585 INFO:journalctl@ceph.mds.h.smithi059.stdout:Mar 28 18:07:59 smithi059 podman[66459]: 2024-03-28 18:07:59.30285897 +0000 UTC m=+0.371027659 container start 1e6766595523a6768d01891d1f7e3eb3a7f9ee953527df8a16a92d15764d1d8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-h, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-03-28T18:07:59.585 INFO:journalctl@ceph.mds.h.smithi059.stdout:Mar 28 18:07:59 smithi059 ceph-mds[66477]: -- 172.21.15.59:0/2986427427 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d5667f5a40 con 0x55d5667c9000 2024-03-28T18:07:59.586 INFO:journalctl@ceph.mds.h.smithi059.stdout:Mar 28 18:07:59 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-h[66473]: starting mds.h at 2024-03-28T18:07:59.586 INFO:journalctl@ceph.mds.h.smithi059.stdout:Mar 28 18:07:59 smithi059 bash[66459]: 1e6766595523a6768d01891d1f7e3eb3a7f9ee953527df8a16a92d15764d1d8a 2024-03-28T18:07:59.586 INFO:journalctl@ceph.mds.h.smithi059.stdout:Mar 28 18:07:59 smithi059 systemd[1]: Started Ceph mds.h for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:07:59.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:59 smithi037 ceph-mon[31460]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:59.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:59 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:59.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:59 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:59.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:59 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:59.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:59 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:59.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:59 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:59.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:07:59 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:59.936 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:59 smithi170 ceph-mon[36280]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:07:59.936 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:59 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:59.936 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:59 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:59.936 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:59 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:07:59.937 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:59 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:07:59.937 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:59 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:07:59.937 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:07:59 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:08:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:59 smithi059 ceph-mon[36190]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:59 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:59 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:59 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:59 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:08:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:59 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:08:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:07:59 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:08:00.130 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:08:00.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.365+0000 7f3df9400640 1 -- 172.21.15.37:0/206602759 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df41048e0 msgr2=0x7f3df4104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:00.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.365+0000 7f3df9400640 1 --2- 172.21.15.37:0/206602759 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df41048e0 0x7f3df4104d60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc009900 tx=0x7f3ddc02bc30 comp rx=0 tx=0).stop 2024-03-28T18:08:00.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.365+0000 7f3df9400640 1 -- 172.21.15.37:0/206602759 shutdown_connections 2024-03-28T18:08:00.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.365+0000 7f3df9400640 1 --2- 172.21.15.37:0/206602759 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3df41052a0 0x7f3df410bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:00.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.365+0000 7f3df9400640 1 --2- 172.21.15.37:0/206602759 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df41048e0 0x7f3df4104d60 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:00.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.365+0000 7f3df9400640 1 --2- 172.21.15.37:0/206602759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3df41036f0 0x7f3df4103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:00.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.365+0000 7f3df9400640 1 -- 172.21.15.37:0/206602759 >> 172.21.15.37:0/206602759 conn(0x7f3df40fec80 msgr2=0x7f3df41010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:00.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.365+0000 7f3df9400640 1 -- 172.21.15.37:0/206602759 shutdown_connections 2024-03-28T18:08:00.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.365+0000 7f3df9400640 1 -- 172.21.15.37:0/206602759 wait complete. 2024-03-28T18:08:00.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.366+0000 7f3df9400640 1 Processor -- start 2024-03-28T18:08:00.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.366+0000 7f3df9400640 1 -- start start 2024-03-28T18:08:00.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.367+0000 7f3df9400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3df41036f0 0x7f3df4199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:00.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.367+0000 7f3df9400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3df41048e0 0x7f3df4199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.367+0000 7f3df9400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df41052a0 0x7f3df41a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.367+0000 7f3df9400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df410f3a0 con 0x7f3df41052a0 2024-03-28T18:08:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.367+0000 7f3df9400640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df410f220 con 0x7f3df41036f0 2024-03-28T18:08:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.367+0000 7f3df9400640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df410f520 con 0x7f3df41048e0 2024-03-28T18:08:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.367+0000 7f3debe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3df41048e0 0x7f3df4199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.367+0000 7f3debe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3df41048e0 0x7f3df4199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:50708/0 (socket says 172.21.15.37:50708) 2024-03-28T18:08:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.367+0000 7f3debe00640 1 -- 172.21.15.37:0/3799687038 learned_addr learned my addr 172.21.15.37:0/3799687038 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.367+0000 7f3df8a00640 1 --2- 172.21.15.37:0/3799687038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df41052a0 0x7f3df41a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.367+0000 7f3df3e00640 1 --2- 172.21.15.37:0/3799687038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3df41036f0 0x7f3df4199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.367+0000 7f3debe00640 1 -- 172.21.15.37:0/3799687038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3df41036f0 msgr2=0x7f3df4199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.367+0000 7f3debe00640 1 --2- 172.21.15.37:0/3799687038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3df41036f0 0x7f3df4199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.367+0000 7f3debe00640 1 -- 172.21.15.37:0/3799687038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df41052a0 msgr2=0x7f3df41a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.367+0000 7f3debe00640 1 --2- 172.21.15.37:0/3799687038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df41052a0 0x7f3df41a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.367+0000 7f3debe00640 1 -- 172.21.15.37:0/3799687038 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df41a1630 con 0x7f3df41048e0 2024-03-28T18:08:00.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.368+0000 7f3debe00640 1 --2- 172.21.15.37:0/3799687038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3df41048e0 0x7f3df4199f50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc000c00 tx=0x7f3ddc02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:00.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.368+0000 7f3df1600640 1 -- 172.21.15.37:0/3799687038 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc043020 con 0x7f3df41048e0 2024-03-28T18:08:00.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.368+0000 7f3df1600640 1 -- 172.21.15.37:0/3799687038 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ddc0054e0 con 0x7f3df41048e0 2024-03-28T18:08:00.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.368+0000 7f3df1600640 1 -- 172.21.15.37:0/3799687038 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc003c40 con 0x7f3df41048e0 2024-03-28T18:08:00.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.368+0000 7f3df9400640 1 -- 172.21.15.37:0/3799687038 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df41a18c0 con 0x7f3df41048e0 2024-03-28T18:08:00.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.368+0000 7f3df9400640 1 -- 172.21.15.37:0/3799687038 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df41a1e60 con 0x7f3df41048e0 2024-03-28T18:08:00.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.369+0000 7f3df9400640 1 -- 172.21.15.37:0/3799687038 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df410aa10 con 0x7f3df41048e0 2024-03-28T18:08:00.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.371+0000 7f3df1600640 1 -- 172.21.15.37:0/3799687038 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3ddc046020 con 0x7f3df41048e0 2024-03-28T18:08:00.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.372+0000 7f3df1600640 1 --2- 172.21.15.37:0/3799687038 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f3dd80a2880 0x7f3dd80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:00.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.373+0000 7f3df1600640 1 -- 172.21.15.37:0/3799687038 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3ddc0f0ae0 con 0x7f3df41048e0 2024-03-28T18:08:00.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.373+0000 7f3df3e00640 1 --2- 172.21.15.37:0/3799687038 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f3dd80a2880 0x7f3dd80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:00.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.373+0000 7f3df3e00640 1 --2- 172.21.15.37:0/3799687038 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f3dd80a2880 0x7f3dd80a4d40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3df4104740 tx=0x7f3de4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:00.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.375+0000 7f3df1600640 1 -- 172.21.15.37:0/3799687038 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3ddc04d020 con 0x7f3df41048e0 2024-03-28T18:08:00.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.625+0000 7f3df9400640 1 -- 172.21.15.37:0/3799687038 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3df402d070 con 0x7f3df41048e0 2024-03-28T18:08:00.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.626+0000 7f3df1600640 1 -- 172.21.15.37:0/3799687038 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6859 (secure 0 0 0) 0x7f3ddc0bdf80 con 0x7f3df41048e0 2024-03-28T18:08:00.626 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:08:00.626 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6837/4034607523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":4034607523},{"type":"v1","addr":"172.21.15.59:6837","nonce":4034607523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-28T18:08:00.626 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 9 2024-03-28T18:08:00.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.628+0000 7f3df9400640 1 -- 172.21.15.37:0/3799687038 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f3dd80a2880 msgr2=0x7f3dd80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:00.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.628+0000 7f3df9400640 1 --2- 172.21.15.37:0/3799687038 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f3dd80a2880 0x7f3dd80a4d40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3df4104740 tx=0x7f3de4005f90 comp rx=0 tx=0).stop 2024-03-28T18:08:00.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.628+0000 7f3df9400640 1 -- 172.21.15.37:0/3799687038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3df41048e0 msgr2=0x7f3df4199f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:00.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.628+0000 7f3df9400640 1 --2- 172.21.15.37:0/3799687038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3df41048e0 0x7f3df4199f50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc000c00 tx=0x7f3ddc02c650 comp rx=0 tx=0).stop 2024-03-28T18:08:00.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.628+0000 7f3df9400640 1 -- 172.21.15.37:0/3799687038 shutdown_connections 2024-03-28T18:08:00.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.628+0000 7f3df9400640 1 --2- 172.21.15.37:0/3799687038 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f3dd80a2880 0x7f3dd80a4d40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:00.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.628+0000 7f3df9400640 1 --2- 172.21.15.37:0/3799687038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3df41052a0 0x7f3df41a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:00.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.628+0000 7f3df9400640 1 --2- 172.21.15.37:0/3799687038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3df41048e0 0x7f3df4199f50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:00.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.628+0000 7f3df9400640 1 --2- 172.21.15.37:0/3799687038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3df41036f0 0x7f3df4199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:00.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.628+0000 7f3df9400640 1 -- 172.21.15.37:0/3799687038 >> 172.21.15.37:0/3799687038 conn(0x7f3df40fec80 msgr2=0x7f3df4107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:00.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.629+0000 7f3df9400640 1 -- 172.21.15.37:0/3799687038 shutdown_connections 2024-03-28T18:08:00.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:00.629+0000 7f3df9400640 1 -- 172.21.15.37:0/3799687038 wait complete. 2024-03-28T18:08:00.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:00 smithi037 ceph-mon[31460]: Deploying daemon mds.i on smithi170 2024-03-28T18:08:00.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:00 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6836/4034607523,v1:172.21.15.59:6837/4034607523] up:boot 2024-03-28T18:08:00.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:00 smithi037 ceph-mon[31460]: fsmap 8 up:standby 2024-03-28T18:08:00.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:00 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:08:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:00 smithi059 ceph-mon[36190]: Deploying daemon mds.i on smithi170 2024-03-28T18:08:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:00 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6836/4034607523,v1:172.21.15.59:6837/4034607523] up:boot 2024-03-28T18:08:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:00 smithi059 ceph-mon[36190]: fsmap 8 up:standby 2024-03-28T18:08:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:00 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:08:01.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:00 smithi170 ceph-mon[36280]: Deploying daemon mds.i on smithi170 2024-03-28T18:08:01.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:00 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6836/4034607523,v1:172.21.15.59:6837/4034607523] up:boot 2024-03-28T18:08:01.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:00 smithi170 ceph-mon[36280]: fsmap 8 up:standby 2024-03-28T18:08:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:00 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:08:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:01 smithi170 ceph-mon[36280]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:01.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:01 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3799687038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:01.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:01 smithi037 ceph-mon[31460]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:01.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:01 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3799687038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:01.938 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:01 smithi059 ceph-mon[36190]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:01.938 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:01 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3799687038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:02.535 INFO:journalctl@ceph.mds.i.smithi170.stdout:Mar 28 18:08:02 smithi170 systemd[1]: Started Ceph mds.i for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:08:02.651 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:08:02.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.847+0000 7f45cb000640 1 -- 172.21.15.37:0/727107150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c41028b0 msgr2=0x7f45c4102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:02.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.847+0000 7f45cb000640 1 --2- 172.21.15.37:0/727107150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c41028b0 0x7f45c4102d30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f45ac009900 tx=0x7f45ac02bc30 comp rx=0 tx=0).stop 2024-03-28T18:08:02.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.847+0000 7f45cb000640 1 -- 172.21.15.37:0/727107150 shutdown_connections 2024-03-28T18:08:02.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.847+0000 7f45cb000640 1 --2- 172.21.15.37:0/727107150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c41028b0 0x7f45c4102d30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:02.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.847+0000 7f45cb000640 1 --2- 172.21.15.37:0/727107150 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45c4103270 0x7f45c4109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:02.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.847+0000 7f45cb000640 1 --2- 172.21.15.37:0/727107150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45c41016c0 0x7f45c4101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:02.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.847+0000 7f45cb000640 1 -- 172.21.15.37:0/727107150 >> 172.21.15.37:0/727107150 conn(0x7f45c40fcc50 msgr2=0x7f45c40ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:02.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.847+0000 7f45cb000640 1 -- 172.21.15.37:0/727107150 shutdown_connections 2024-03-28T18:08:02.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.847+0000 7f45cb000640 1 -- 172.21.15.37:0/727107150 wait complete. 2024-03-28T18:08:02.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.848+0000 7f45cb000640 1 Processor -- start 2024-03-28T18:08:02.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.848+0000 7f45cb000640 1 -- start start 2024-03-28T18:08:02.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.849+0000 7f45cb000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c41016c0 0x7f45c41979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:02.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.849+0000 7f45cb000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45c41028b0 0x7f45c4197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:02.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.849+0000 7f45cb000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45c4103270 0x7f45c419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:02.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.849+0000 7f45cb000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c410d320 con 0x7f45c41028b0 2024-03-28T18:08:02.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.849+0000 7f45cb000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c410d1a0 con 0x7f45c41016c0 2024-03-28T18:08:02.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.849+0000 7f45cb000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c410d4a0 con 0x7f45c4103270 2024-03-28T18:08:02.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.849+0000 7f45c3e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c41016c0 0x7f45c41979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:02.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.849+0000 7f45c3e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c41016c0 0x7f45c41979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:49940/0 (socket says 172.21.15.37:49940) 2024-03-28T18:08:02.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.849+0000 7f45bb400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45c41028b0 0x7f45c4197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:02.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.849+0000 7f45c3e00640 1 -- 172.21.15.37:0/3844015349 learned_addr learned my addr 172.21.15.37:0/3844015349 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:02.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.849+0000 7f45c9000640 1 --2- 172.21.15.37:0/3844015349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45c4103270 0x7f45c419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:02.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.849+0000 7f45c3e00640 1 -- 172.21.15.37:0/3844015349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45c4103270 msgr2=0x7f45c419ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:02.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.849+0000 7f45c3e00640 1 --2- 172.21.15.37:0/3844015349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45c4103270 0x7f45c419ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:02.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.849+0000 7f45c3e00640 1 -- 172.21.15.37:0/3844015349 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45c41028b0 msgr2=0x7f45c4197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:02.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.849+0000 7f45c3e00640 1 --2- 172.21.15.37:0/3844015349 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45c41028b0 0x7f45c4197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:02.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.849+0000 7f45c3e00640 1 -- 172.21.15.37:0/3844015349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c419f600 con 0x7f45c41016c0 2024-03-28T18:08:02.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.850+0000 7f45c3e00640 1 --2- 172.21.15.37:0/3844015349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c41016c0 0x7f45c41979e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f45b400e9c0 tx=0x7f45b400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:02.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.850+0000 7f45c1600640 1 -- 172.21.15.37:0/3844015349 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b400cd80 con 0x7f45c41016c0 2024-03-28T18:08:02.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.850+0000 7f45c1600640 1 -- 172.21.15.37:0/3844015349 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45b40035f0 con 0x7f45c41016c0 2024-03-28T18:08:02.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.850+0000 7f45cb000640 1 -- 172.21.15.37:0/3844015349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c419f8f0 con 0x7f45c41016c0 2024-03-28T18:08:02.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.850+0000 7f45c1600640 1 -- 172.21.15.37:0/3844015349 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b4010690 con 0x7f45c41016c0 2024-03-28T18:08:02.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.850+0000 7f45cb000640 1 -- 172.21.15.37:0/3844015349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45c419fe10 con 0x7f45c41016c0 2024-03-28T18:08:02.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.851+0000 7f45cb000640 1 -- 172.21.15.37:0/3844015349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45c4108ab0 con 0x7f45c41016c0 2024-03-28T18:08:02.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.853+0000 7f45c1600640 1 -- 172.21.15.37:0/3844015349 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f45b40037a0 con 0x7f45c41016c0 2024-03-28T18:08:02.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.854+0000 7f45c1600640 1 --2- 172.21.15.37:0/3844015349 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f45a80a2800 0x7f45a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:02.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.854+0000 7f45c1600640 1 -- 172.21.15.37:0/3844015349 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f45b40cea10 con 0x7f45c41016c0 2024-03-28T18:08:02.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.854+0000 7f45bb400640 1 --2- 172.21.15.37:0/3844015349 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f45a80a2800 0x7f45a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:02.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.855+0000 7f45bb400640 1 --2- 172.21.15.37:0/3844015349 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f45a80a2800 0x7f45a80a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f45ac02c7b0 tx=0x7f45ac005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:02.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:02.858+0000 7f45c1600640 1 -- 172.21.15.37:0/3844015349 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f45b409beb0 con 0x7f45c41016c0 2024-03-28T18:08:03.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:03.104+0000 7f45cb000640 1 -- 172.21.15.37:0/3844015349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45c40622a0 con 0x7f45c41016c0 2024-03-28T18:08:03.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:03.105+0000 7f45c1600640 1 -- 172.21.15.37:0/3844015349 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7655 (secure 0 0 0) 0x7f45b409b7b0 con 0x7f45c41016c0 2024-03-28T18:08:03.105 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:08:03.105 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6837/4034607523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":4034607523},{"type":"v1","addr":"172.21.15.59:6837","nonce":4034607523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6839/1575518279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":1575518279},{"type":"v1","addr":"172.21.15.170:6839","nonce":1575518279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T18:08:03.105 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 10 2024-03-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:03.107+0000 7f45cb000640 1 -- 172.21.15.37:0/3844015349 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f45a80a2800 msgr2=0x7f45a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:03.107+0000 7f45cb000640 1 --2- 172.21.15.37:0/3844015349 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f45a80a2800 0x7f45a80a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f45ac02c7b0 tx=0x7f45ac005fd0 comp rx=0 tx=0).stop 2024-03-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:03.107+0000 7f45cb000640 1 -- 172.21.15.37:0/3844015349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c41016c0 msgr2=0x7f45c41979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:03.107+0000 7f45cb000640 1 --2- 172.21.15.37:0/3844015349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c41016c0 0x7f45c41979e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f45b400e9c0 tx=0x7f45b400ee80 comp rx=0 tx=0).stop 2024-03-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:03.108+0000 7f45cb000640 1 -- 172.21.15.37:0/3844015349 shutdown_connections 2024-03-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:03.108+0000 7f45cb000640 1 --2- 172.21.15.37:0/3844015349 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f45a80a2800 0x7f45a80a4cc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:03.108+0000 7f45cb000640 1 --2- 172.21.15.37:0/3844015349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45c4103270 0x7f45c419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:03.108+0000 7f45cb000640 1 --2- 172.21.15.37:0/3844015349 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45c41028b0 0x7f45c4197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:03.108+0000 7f45cb000640 1 --2- 172.21.15.37:0/3844015349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c41016c0 0x7f45c41979e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:03.108+0000 7f45cb000640 1 -- 172.21.15.37:0/3844015349 >> 172.21.15.37:0/3844015349 conn(0x7f45c40fcc50 msgr2=0x7f45c4105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:03.108+0000 7f45cb000640 1 -- 172.21.15.37:0/3844015349 shutdown_connections 2024-03-28T18:08:03.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:03.108+0000 7f45cb000640 1 -- 172.21.15.37:0/3844015349 wait complete. 2024-03-28T18:08:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:03 smithi059 ceph-mon[36190]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:03 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:03 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:03 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:03 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:08:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:03 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:08:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:03 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:08:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:03 smithi059 ceph-mon[36190]: Deploying daemon mds.g on smithi037 2024-03-28T18:08:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:03 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] up:boot 2024-03-28T18:08:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:03 smithi059 ceph-mon[36190]: fsmap 9 up:standby 2024-03-28T18:08:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:03 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:08:03.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:03 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3844015349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:03.750 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:03 smithi037 ceph-mon[31460]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:03.750 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:03.750 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:03.751 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:03.751 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:08:03.751 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:08:03.751 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:08:03.751 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:03 smithi037 ceph-mon[31460]: Deploying daemon mds.g on smithi037 2024-03-28T18:08:03.751 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:03 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] up:boot 2024-03-28T18:08:03.752 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:03 smithi037 ceph-mon[31460]: fsmap 9 up:standby 2024-03-28T18:08:03.752 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:03 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:08:03.752 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:03 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3844015349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:03.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:03 smithi170 ceph-mon[36280]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:03.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:03 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:03.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:03 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:03.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:03 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:03.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:03 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:08:03.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:03 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:08:03.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:03 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:08:03.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:03 smithi170 ceph-mon[36280]: Deploying daemon mds.g on smithi037 2024-03-28T18:08:03.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:03 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] up:boot 2024-03-28T18:08:03.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:03 smithi170 ceph-mon[36280]: fsmap 9 up:standby 2024-03-28T18:08:03.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:03 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:08:03.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:03 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3844015349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:05.127 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:08:05.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.562+0000 7ffaf1400640 1 -- 172.21.15.37:0/1641778758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffaec10d9e0 msgr2=0x7ffaec10de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:05.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.562+0000 7ffaf1400640 1 --2- 172.21.15.37:0/1641778758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffaec10d9e0 0x7ffaec10de90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ffae4009be0 tx=0x7ffae402bf10 comp rx=0 tx=0).stop 2024-03-28T18:08:05.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.563+0000 7ffaf1400640 1 -- 172.21.15.37:0/1641778758 shutdown_connections 2024-03-28T18:08:05.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.563+0000 7ffaf1400640 1 --2- 172.21.15.37:0/1641778758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaec10b010 0x7ffaec10d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:05.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.563+0000 7ffaf1400640 1 --2- 172.21.15.37:0/1641778758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffaec10d9e0 0x7ffaec10de90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:05.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.563+0000 7ffaf1400640 1 --2- 172.21.15.37:0/1641778758 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffaec1086a0 0x7ffaec10aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:05.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.563+0000 7ffaf1400640 1 -- 172.21.15.37:0/1641778758 >> 172.21.15.37:0/1641778758 conn(0x7ffaec06e510 msgr2=0x7ffaec070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:05.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.563+0000 7ffaf1400640 1 -- 172.21.15.37:0/1641778758 shutdown_connections 2024-03-28T18:08:05.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.563+0000 7ffaf1400640 1 -- 172.21.15.37:0/1641778758 wait complete. 2024-03-28T18:08:05.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.564+0000 7ffaf1400640 1 Processor -- start 2024-03-28T18:08:05.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.565+0000 7ffaf1400640 1 -- start start 2024-03-28T18:08:05.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.566+0000 7ffaf1400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffaec1086a0 0x7ffaec1a6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:05.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.566+0000 7ffaf1400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaec10b010 0x7ffaec1a6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:05.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.566+0000 7ffaf1400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffaec10d9e0 0x7ffaec1add40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:05.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.566+0000 7ffaea800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffaec1086a0 0x7ffaec1a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:05.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.566+0000 7ffaea800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffaec1086a0 0x7ffaec1a6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:51560/0 (socket says 172.21.15.37:51560) 2024-03-28T18:08:05.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.566+0000 7ffaea800640 1 -- 172.21.15.37:0/4080950888 learned_addr learned my addr 172.21.15.37:0/4080950888 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:05.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.566+0000 7ffaf1400640 1 -- 172.21.15.37:0/4080950888 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec11c1e0 con 0x7ffaec10d9e0 2024-03-28T18:08:05.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.566+0000 7ffaf1400640 1 -- 172.21.15.37:0/4080950888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec11c060 con 0x7ffaec10b010 2024-03-28T18:08:05.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.566+0000 7ffaf1400640 1 -- 172.21.15.37:0/4080950888 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec11c360 con 0x7ffaec1086a0 2024-03-28T18:08:05.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.567+0000 7ffaeb200640 1 --2- 172.21.15.37:0/4080950888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffaec10d9e0 0x7ffaec1add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:05.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.567+0000 7ffaea800640 1 -- 172.21.15.37:0/4080950888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaec10b010 msgr2=0x7ffaec1a6cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:08:05.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.567+0000 7ffaea800640 1 --2- 172.21.15.37:0/4080950888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaec10b010 0x7ffaec1a6cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:05.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.567+0000 7ffaea800640 1 -- 172.21.15.37:0/4080950888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffaec10d9e0 msgr2=0x7ffaec1add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:05.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.567+0000 7ffaea800640 1 --2- 172.21.15.37:0/4080950888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffaec10d9e0 0x7ffaec1add40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:05.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.567+0000 7ffaea800640 1 -- 172.21.15.37:0/4080950888 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec073950 con 0x7ffaec1086a0 2024-03-28T18:08:05.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.568+0000 7ffaea800640 1 --2- 172.21.15.37:0/4080950888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffaec1086a0 0x7ffaec1a6790 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffad400e9f0 tx=0x7ffad400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:05.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.568+0000 7ffae3400640 1 -- 172.21.15.37:0/4080950888 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad400ccc0 con 0x7ffaec1086a0 2024-03-28T18:08:05.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.568+0000 7ffae3400640 1 -- 172.21.15.37:0/4080950888 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffad40035f0 con 0x7ffaec1086a0 2024-03-28T18:08:05.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.568+0000 7ffae3400640 1 -- 172.21.15.37:0/4080950888 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad400ce60 con 0x7ffaec1086a0 2024-03-28T18:08:05.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.568+0000 7ffaf1400640 1 -- 172.21.15.37:0/4080950888 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaec073c40 con 0x7ffaec1086a0 2024-03-28T18:08:05.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.569+0000 7ffaf1400640 1 -- 172.21.15.37:0/4080950888 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaec074140 con 0x7ffaec1086a0 2024-03-28T18:08:05.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.570+0000 7ffaf1400640 1 -- 172.21.15.37:0/4080950888 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaec071010 con 0x7ffaec1086a0 2024-03-28T18:08:05.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.574+0000 7ffae3400640 1 -- 172.21.15.37:0/4080950888 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ffad40037a0 con 0x7ffaec1086a0 2024-03-28T18:08:05.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.576+0000 7ffae3400640 1 --2- 172.21.15.37:0/4080950888 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ffad00a2900 0x7ffad00a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:05.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.576+0000 7ffae3400640 1 -- 172.21.15.37:0/4080950888 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ffad40ceaa0 con 0x7ffaec1086a0 2024-03-28T18:08:05.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.577+0000 7ffae9e00640 1 --2- 172.21.15.37:0/4080950888 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ffad00a2900 0x7ffad00a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:05.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.577+0000 7ffae9e00640 1 --2- 172.21.15.37:0/4080950888 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ffad00a2900 0x7ffad00a4dc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ffaec1a7ca0 tx=0x7ffadc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:05.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.580+0000 7ffae3400640 1 -- 172.21.15.37:0/4080950888 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffad409bf40 con 0x7ffaec1086a0 2024-03-28T18:08:05.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:05 smithi037 ceph-mon[31460]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:05.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:05 smithi059 ceph-mon[36190]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:05.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:05 smithi170 ceph-mon[36280]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:05.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.835+0000 7ffaf1400640 1 -- 172.21.15.37:0/4080950888 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffaec02d070 con 0x7ffaec1086a0 2024-03-28T18:08:05.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.835+0000 7ffae3400640 1 -- 172.21.15.37:0/4080950888 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7655 (secure 0 0 0) 0x7ffad409b840 con 0x7ffaec1086a0 2024-03-28T18:08:05.836 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:08:05.836 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6837/4034607523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":4034607523},{"type":"v1","addr":"172.21.15.59:6837","nonce":4034607523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6839/1575518279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":1575518279},{"type":"v1","addr":"172.21.15.170:6839","nonce":1575518279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T18:08:05.836 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 10 2024-03-28T18:08:05.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.838+0000 7ffaf1400640 1 -- 172.21.15.37:0/4080950888 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ffad00a2900 msgr2=0x7ffad00a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:05.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.838+0000 7ffaf1400640 1 --2- 172.21.15.37:0/4080950888 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ffad00a2900 0x7ffad00a4dc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ffaec1a7ca0 tx=0x7ffadc005f90 comp rx=0 tx=0).stop 2024-03-28T18:08:05.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.838+0000 7ffaf1400640 1 -- 172.21.15.37:0/4080950888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffaec1086a0 msgr2=0x7ffaec1a6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:05.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.838+0000 7ffaf1400640 1 --2- 172.21.15.37:0/4080950888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffaec1086a0 0x7ffaec1a6790 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffad400e9f0 tx=0x7ffad400eeb0 comp rx=0 tx=0).stop 2024-03-28T18:08:05.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.838+0000 7ffaf1400640 1 -- 172.21.15.37:0/4080950888 shutdown_connections 2024-03-28T18:08:05.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.838+0000 7ffaf1400640 1 --2- 172.21.15.37:0/4080950888 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ffad00a2900 0x7ffad00a4dc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:05.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.838+0000 7ffaf1400640 1 --2- 172.21.15.37:0/4080950888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaec10b010 0x7ffaec1a6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:05.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.838+0000 7ffaf1400640 1 --2- 172.21.15.37:0/4080950888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffaec10d9e0 0x7ffaec1add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:05.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.838+0000 7ffaf1400640 1 --2- 172.21.15.37:0/4080950888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffaec1086a0 0x7ffaec1a6790 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:05.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.838+0000 7ffaf1400640 1 -- 172.21.15.37:0/4080950888 >> 172.21.15.37:0/4080950888 conn(0x7ffaec06e510 msgr2=0x7ffaec10a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:05.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.838+0000 7ffaf1400640 1 -- 172.21.15.37:0/4080950888 shutdown_connections 2024-03-28T18:08:05.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:05.839+0000 7ffaf1400640 1 -- 172.21.15.37:0/4080950888 wait complete. 2024-03-28T18:08:06.358 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:08:06 smithi037 podman[70692]: 2024-03-28 18:08:06.186003851 +0000 UTC m=+0.175583435 container create 5c83e57ee4ad2444c7fd51e10067c69e1d1ab98690c22d751358d64408ce3a90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:08:06.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:06 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/4080950888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:06.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:06 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/4080950888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:06.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:06 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/4080950888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:06.874 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:08:06 smithi037 podman[70692]: 2024-03-28 18:08:06.481065366 +0000 UTC m=+0.470644919 container init 5c83e57ee4ad2444c7fd51e10067c69e1d1ab98690c22d751358d64408ce3a90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-28T18:08:06.874 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:08:06 smithi037 podman[70692]: 2024-03-28 18:08:06.493199305 +0000 UTC m=+0.482778852 container start 5c83e57ee4ad2444c7fd51e10067c69e1d1ab98690c22d751358d64408ce3a90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-28T18:08:06.874 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:08:06 smithi037 ceph-mds[70718]: -- 172.21.15.37:0/3782036512 <== mon.0 v2:172.21.15.37:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558164db1a40 con 0x558164d84c00 2024-03-28T18:08:06.874 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:08:06 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: starting mds.g at 2024-03-28T18:08:06.875 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:08:06 smithi037 bash[70692]: 5c83e57ee4ad2444c7fd51e10067c69e1d1ab98690c22d751358d64408ce3a90 2024-03-28T18:08:06.875 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:08:06 smithi037 systemd[1]: Started Ceph mds.g for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:08:07.860 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:08:07.953 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:07 smithi037 ceph-mon[31460]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:07.953 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:07.953 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:07.953 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:07.953 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:08:07.953 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:08:07.953 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:08:07.953 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:07 smithi037 ceph-mon[31460]: Deploying daemon mds.a on smithi037 2024-03-28T18:08:07.953 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:07 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6838/1926270371,v1:172.21.15.37:6839/1926270371] up:boot 2024-03-28T18:08:07.953 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:07 smithi037 ceph-mon[31460]: fsmap 10 up:standby 2024-03-28T18:08:07.953 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:07 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:08:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:07 smithi059 ceph-mon[36190]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:08:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:08:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:08:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:07 smithi059 ceph-mon[36190]: Deploying daemon mds.a on smithi037 2024-03-28T18:08:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:07 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6838/1926270371,v1:172.21.15.37:6839/1926270371] up:boot 2024-03-28T18:08:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:07 smithi059 ceph-mon[36190]: fsmap 10 up:standby 2024-03-28T18:08:08.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:07 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:08:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:07 smithi170 ceph-mon[36280]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:08:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:08:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:08:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:07 smithi170 ceph-mon[36280]: Deploying daemon mds.a on smithi037 2024-03-28T18:08:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:07 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6838/1926270371,v1:172.21.15.37:6839/1926270371] up:boot 2024-03-28T18:08:08.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:07 smithi170 ceph-mon[36280]: fsmap 10 up:standby 2024-03-28T18:08:08.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:07 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:08:08.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.089+0000 7fdaa3e00640 1 -- 172.21.15.37:0/2298584523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaa4107b00 msgr2=0x7fdaa4107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:08.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.089+0000 7fdaa3e00640 1 --2- 172.21.15.37:0/2298584523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaa4107b00 0x7fdaa4107f20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fda94009900 tx=0x7fda9402bc30 comp rx=0 tx=0).stop 2024-03-28T18:08:08.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.089+0000 7fdaa3e00640 1 -- 172.21.15.37:0/2298584523 shutdown_connections 2024-03-28T18:08:08.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.089+0000 7fdaa3e00640 1 --2- 172.21.15.37:0/2298584523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdaa41096b0 0x7fdaa410ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:08.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.089+0000 7fdaa3e00640 1 --2- 172.21.15.37:0/2298584523 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdaa4108cf0 0x7fdaa4109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:08.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.089+0000 7fdaa3e00640 1 --2- 172.21.15.37:0/2298584523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaa4107b00 0x7fdaa4107f20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:08.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.089+0000 7fdaa3e00640 1 -- 172.21.15.37:0/2298584523 >> 172.21.15.37:0/2298584523 conn(0x7fdaa4076ab0 msgr2=0x7fdaa4078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:08.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.089+0000 7fdaa3e00640 1 -- 172.21.15.37:0/2298584523 shutdown_connections 2024-03-28T18:08:08.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.089+0000 7fdaa3e00640 1 -- 172.21.15.37:0/2298584523 wait complete. 2024-03-28T18:08:08.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.091+0000 7fdaa3e00640 1 Processor -- start 2024-03-28T18:08:08.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.091+0000 7fdaa3e00640 1 -- start start 2024-03-28T18:08:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.091+0000 7fdaa3e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdaa4107b00 0x7fdaa419dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.091+0000 7fdaa3e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdaa4108cf0 0x7fdaa419e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.091+0000 7fdaa3e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaa41096b0 0x7fdaa41a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.091+0000 7fdaa3e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa41137d0 con 0x7fdaa4107b00 2024-03-28T18:08:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.091+0000 7fdaa3e00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa4113650 con 0x7fdaa4108cf0 2024-03-28T18:08:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.091+0000 7fdaa3e00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa4113950 con 0x7fdaa41096b0 2024-03-28T18:08:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.091+0000 7fdaa2a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdaa4107b00 0x7fdaa419dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.091+0000 7fdaa2a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdaa4107b00 0x7fdaa419dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54334/0 (socket says 172.21.15.37:54334) 2024-03-28T18:08:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.091+0000 7fdaa2a00640 1 -- 172.21.15.37:0/1728068307 learned_addr learned my addr 172.21.15.37:0/1728068307 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.091+0000 7fdaa3400640 1 --2- 172.21.15.37:0/1728068307 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaa41096b0 0x7fdaa41a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.091+0000 7fda9be00640 1 --2- 172.21.15.37:0/1728068307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdaa4108cf0 0x7fdaa419e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.092+0000 7fdaa2a00640 1 -- 172.21.15.37:0/1728068307 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaa41096b0 msgr2=0x7fdaa41a5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.092+0000 7fdaa2a00640 1 --2- 172.21.15.37:0/1728068307 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaa41096b0 0x7fdaa41a5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.092+0000 7fdaa2a00640 1 -- 172.21.15.37:0/1728068307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdaa4108cf0 msgr2=0x7fdaa419e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.092+0000 7fdaa2a00640 1 --2- 172.21.15.37:0/1728068307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdaa4108cf0 0x7fdaa419e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.092+0000 7fdaa2a00640 1 -- 172.21.15.37:0/1728068307 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa41a5a00 con 0x7fdaa4107b00 2024-03-28T18:08:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.092+0000 7fdaa3400640 1 --2- 172.21.15.37:0/1728068307 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaa41096b0 0x7fdaa41a5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:08:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.092+0000 7fdaa2a00640 1 --2- 172.21.15.37:0/1728068307 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdaa4107b00 0x7fdaa419dde0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fda94000c00 tx=0x7fda9402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.092+0000 7fda9b400640 1 -- 172.21.15.37:0/1728068307 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda94043020 con 0x7fdaa4107b00 2024-03-28T18:08:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.093+0000 7fda9b400640 1 -- 172.21.15.37:0/1728068307 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda940054e0 con 0x7fdaa4107b00 2024-03-28T18:08:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.093+0000 7fda9b400640 1 -- 172.21.15.37:0/1728068307 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda94003c40 con 0x7fdaa4107b00 2024-03-28T18:08:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.093+0000 7fdaa3e00640 1 -- 172.21.15.37:0/1728068307 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa41a5c90 con 0x7fdaa4107b00 2024-03-28T18:08:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.093+0000 7fdaa3e00640 1 -- 172.21.15.37:0/1728068307 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa41a6190 con 0x7fdaa4107b00 2024-03-28T18:08:08.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.094+0000 7fdaa3e00640 1 -- 172.21.15.37:0/1728068307 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa410ee20 con 0x7fdaa4107b00 2024-03-28T18:08:08.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.096+0000 7fda9b400640 1 -- 172.21.15.37:0/1728068307 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fda94046020 con 0x7fdaa4107b00 2024-03-28T18:08:08.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.098+0000 7fda9b400640 1 --2- 172.21.15.37:0/1728068307 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fda880a2800 0x7fda880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:08.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.099+0000 7fda9b400640 1 -- 172.21.15.37:0/1728068307 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fda940f0a60 con 0x7fdaa4107b00 2024-03-28T18:08:08.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.099+0000 7fda9be00640 1 --2- 172.21.15.37:0/1728068307 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fda880a2800 0x7fda880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.100+0000 7fda9be00640 1 --2- 172.21.15.37:0/1728068307 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fda880a2800 0x7fda880a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdaa419f2f0 tx=0x7fda8c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.101+0000 7fda9b400640 1 -- 172.21.15.37:0/1728068307 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fda9404d020 con 0x7fdaa4107b00 2024-03-28T18:08:08.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.348+0000 7fdaa3e00640 1 -- 172.21.15.37:0/1728068307 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdaa402d230 con 0x7fdaa4107b00 2024-03-28T18:08:08.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.350+0000 7fda9b400640 1 -- 172.21.15.37:0/1728068307 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8447 (secure 0 0 0) 0x7fda940bdf00 con 0x7fdaa4107b00 2024-03-28T18:08:08.350 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:08:08.350 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1926270371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6838","nonce":1926270371},{"type":"v1","addr":"172.21.15.37:6839","nonce":1926270371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6837/4034607523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":4034607523},{"type":"v1","addr":"172.21.15.59:6837","nonce":4034607523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6839/1575518279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":1575518279},{"type":"v1","addr":"172.21.15.170:6839","nonce":1575518279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T18:08:08.351 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 11 2024-03-28T18:08:08.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.352+0000 7fdaa3e00640 1 -- 172.21.15.37:0/1728068307 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fda880a2800 msgr2=0x7fda880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:08.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.352+0000 7fdaa3e00640 1 --2- 172.21.15.37:0/1728068307 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fda880a2800 0x7fda880a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdaa419f2f0 tx=0x7fda8c005f90 comp rx=0 tx=0).stop 2024-03-28T18:08:08.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.352+0000 7fdaa3e00640 1 -- 172.21.15.37:0/1728068307 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdaa4107b00 msgr2=0x7fdaa419dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:08.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.352+0000 7fdaa3e00640 1 --2- 172.21.15.37:0/1728068307 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdaa4107b00 0x7fdaa419dde0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fda94000c00 tx=0x7fda9402c650 comp rx=0 tx=0).stop 2024-03-28T18:08:08.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.353+0000 7fdaa3e00640 1 -- 172.21.15.37:0/1728068307 shutdown_connections 2024-03-28T18:08:08.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.353+0000 7fdaa3e00640 1 --2- 172.21.15.37:0/1728068307 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fda880a2800 0x7fda880a4cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:08.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.353+0000 7fdaa3e00640 1 --2- 172.21.15.37:0/1728068307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdaa4108cf0 0x7fdaa419e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:08.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.353+0000 7fdaa3e00640 1 --2- 172.21.15.37:0/1728068307 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaa41096b0 0x7fdaa41a5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:08.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.353+0000 7fdaa3e00640 1 --2- 172.21.15.37:0/1728068307 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdaa4107b00 0x7fdaa419dde0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:08.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.353+0000 7fdaa3e00640 1 -- 172.21.15.37:0/1728068307 >> 172.21.15.37:0/1728068307 conn(0x7fdaa4076ab0 msgr2=0x7fdaa410bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:08.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.353+0000 7fdaa3e00640 1 -- 172.21.15.37:0/1728068307 shutdown_connections 2024-03-28T18:08:08.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:08.353+0000 7fdaa3e00640 1 -- 172.21.15.37:0/1728068307 wait complete. 2024-03-28T18:08:08.998 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:08 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1728068307' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:09.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:08 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1728068307' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:09.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:08 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1728068307' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:09.830 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:09 smithi037 ceph-mon[31460]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:09.831 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:08:09 smithi037 systemd[1]: Starting Ceph mds.a for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:08:10.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:09 smithi059 ceph-mon[36190]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:10.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:09 smithi170 ceph-mon[36280]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:10.373 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:08:10.457 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:08:10 smithi037 podman[71045]: 2024-03-28 18:08:10.102054595 +0000 UTC m=+0.036684318 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:08:10.457 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:08:10 smithi037 podman[71045]: 2024-03-28 18:08:10.250772097 +0000 UTC m=+0.185401850 container create 38fcb06288bf118ffa1097e123eb85ba5096999c2137001043b346c8d8089dce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-28T18:08:10.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.595+0000 7f2289000640 1 -- 172.21.15.37:0/2196915714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22841038e0 msgr2=0x7f2284103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:10.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.595+0000 7f2289000640 1 --2- 172.21.15.37:0/2196915714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22841038e0 0x7f2284103d00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2274009860 tx=0x7f227402bb70 comp rx=0 tx=0).stop 2024-03-28T18:08:10.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.595+0000 7f2289000640 1 -- 172.21.15.37:0/2196915714 shutdown_connections 2024-03-28T18:08:10.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.595+0000 7f2289000640 1 --2- 172.21.15.37:0/2196915714 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2284105490 0x7f228410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:10.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.595+0000 7f2289000640 1 --2- 172.21.15.37:0/2196915714 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2284104ad0 0x7f2284104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:10.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.595+0000 7f2289000640 1 --2- 172.21.15.37:0/2196915714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22841038e0 0x7f2284103d00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:10.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.595+0000 7f2289000640 1 -- 172.21.15.37:0/2196915714 >> 172.21.15.37:0/2196915714 conn(0x7f22840fee50 msgr2=0x7f2284101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:10.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.595+0000 7f2289000640 1 -- 172.21.15.37:0/2196915714 shutdown_connections 2024-03-28T18:08:10.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.595+0000 7f2289000640 1 -- 172.21.15.37:0/2196915714 wait complete. 2024-03-28T18:08:10.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.596+0000 7f2289000640 1 Processor -- start 2024-03-28T18:08:10.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.597+0000 7f2289000640 1 -- start start 2024-03-28T18:08:10.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.597+0000 7f2289000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22841038e0 0x7f2284199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:10.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.597+0000 7f2289000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2284104ad0 0x7f228419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:10.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.597+0000 7f2289000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2284105490 0x7f22841a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:10.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.597+0000 7f2289000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228410f530 con 0x7f22841038e0 2024-03-28T18:08:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.597+0000 7f2289000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228410f3b0 con 0x7f2284104ad0 2024-03-28T18:08:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.597+0000 7f2289000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228410f6b0 con 0x7f2284105490 2024-03-28T18:08:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.597+0000 7f2281e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22841038e0 0x7f2284199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.597+0000 7f2281e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22841038e0 0x7f2284199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54366/0 (socket says 172.21.15.37:54366) 2024-03-28T18:08:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.597+0000 7f2281e00640 1 -- 172.21.15.37:0/765037291 learned_addr learned my addr 172.21.15.37:0/765037291 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.597+0000 7f227be00640 1 --2- 172.21.15.37:0/765037291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2284104ad0 0x7f228419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.597+0000 7f2282800640 1 --2- 172.21.15.37:0/765037291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2284105490 0x7f22841a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.598+0000 7f2281e00640 1 -- 172.21.15.37:0/765037291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2284105490 msgr2=0x7f22841a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.598+0000 7f2281e00640 1 --2- 172.21.15.37:0/765037291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2284105490 0x7f22841a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.598+0000 7f2281e00640 1 -- 172.21.15.37:0/765037291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2284104ad0 msgr2=0x7f228419a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.598+0000 7f2281e00640 1 --2- 172.21.15.37:0/765037291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2284104ad0 0x7f228419a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.598+0000 7f2281e00640 1 -- 172.21.15.37:0/765037291 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22841a1830 con 0x7f22841038e0 2024-03-28T18:08:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.598+0000 7f2281e00640 1 --2- 172.21.15.37:0/765037291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22841038e0 0x7f2284199c20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f22740349e0 tx=0x7f2274032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.598+0000 7f227aa00640 1 -- 172.21.15.37:0/765037291 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2274041020 con 0x7f22841038e0 2024-03-28T18:08:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.598+0000 7f2289000640 1 -- 172.21.15.37:0/765037291 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22841a1ac0 con 0x7f22841038e0 2024-03-28T18:08:10.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.598+0000 7f227aa00640 1 -- 172.21.15.37:0/765037291 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f227402c9f0 con 0x7f22841038e0 2024-03-28T18:08:10.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.598+0000 7f227aa00640 1 -- 172.21.15.37:0/765037291 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2274039040 con 0x7f22841038e0 2024-03-28T18:08:10.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.598+0000 7f2289000640 1 -- 172.21.15.37:0/765037291 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22841a2000 con 0x7f22841038e0 2024-03-28T18:08:10.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.600+0000 7f2289000640 1 -- 172.21.15.37:0/765037291 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f228410ac80 con 0x7f22841038e0 2024-03-28T18:08:10.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.601+0000 7f227aa00640 1 -- 172.21.15.37:0/765037291 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2274044020 con 0x7f22841038e0 2024-03-28T18:08:10.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.603+0000 7f227aa00640 1 --2- 172.21.15.37:0/765037291 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f22680a2880 0x7f22680a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:10.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.603+0000 7f227aa00640 1 -- 172.21.15.37:0/765037291 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f22740efe90 con 0x7f22841038e0 2024-03-28T18:08:10.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.603+0000 7f227be00640 1 --2- 172.21.15.37:0/765037291 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f22680a2880 0x7f22680a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:10.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.604+0000 7f227be00640 1 --2- 172.21.15.37:0/765037291 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f22680a2880 0x7f22680a4d40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f226c005fd0 tx=0x7f226c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:10.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.607+0000 7f227aa00640 1 -- 172.21.15.37:0/765037291 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f227404b020 con 0x7f22841038e0 2024-03-28T18:08:10.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.867+0000 7f2289000640 1 -- 172.21.15.37:0/765037291 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f228402d070 con 0x7f22841038e0 2024-03-28T18:08:10.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.868+0000 7f227aa00640 1 -- 172.21.15.37:0/765037291 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9239 (secure 0 0 0) 0x7f22740ef090 con 0x7f22841038e0 2024-03-28T18:08:10.868 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:08:10.869 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1926270371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6838","nonce":1926270371},{"type":"v1","addr":"172.21.15.37:6839","nonce":1926270371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6837/4034607523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":4034607523},{"type":"v1","addr":"172.21.15.59:6837","nonce":4034607523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6839/1575518279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":1575518279},{"type":"v1","addr":"172.21.15.170:6839","nonce":1575518279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24580,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6841/1923263622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6840","nonce":1923263622},{"type":"v1","addr":"172.21.15.37:6841","nonce":1923263622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[]} 2024-03-28T18:08:10.869 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 12 2024-03-28T18:08:10.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.871+0000 7f2289000640 1 -- 172.21.15.37:0/765037291 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f22680a2880 msgr2=0x7f22680a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:10.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.871+0000 7f2289000640 1 --2- 172.21.15.37:0/765037291 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f22680a2880 0x7f22680a4d40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f226c005fd0 tx=0x7f226c009260 comp rx=0 tx=0).stop 2024-03-28T18:08:10.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.871+0000 7f2289000640 1 -- 172.21.15.37:0/765037291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22841038e0 msgr2=0x7f2284199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:10.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.871+0000 7f2289000640 1 --2- 172.21.15.37:0/765037291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22841038e0 0x7f2284199c20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f22740349e0 tx=0x7f2274032430 comp rx=0 tx=0).stop 2024-03-28T18:08:10.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.871+0000 7f2289000640 1 -- 172.21.15.37:0/765037291 shutdown_connections 2024-03-28T18:08:10.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.871+0000 7f2289000640 1 --2- 172.21.15.37:0/765037291 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f22680a2880 0x7f22680a4d40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:10.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.871+0000 7f2289000640 1 --2- 172.21.15.37:0/765037291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2284104ad0 0x7f228419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:10.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.871+0000 7f2289000640 1 --2- 172.21.15.37:0/765037291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2284105490 0x7f22841a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:10.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.871+0000 7f2289000640 1 --2- 172.21.15.37:0/765037291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22841038e0 0x7f2284199c20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:10.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.871+0000 7f2289000640 1 -- 172.21.15.37:0/765037291 >> 172.21.15.37:0/765037291 conn(0x7f22840fee50 msgr2=0x7f2284107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:10.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.871+0000 7f2289000640 1 -- 172.21.15.37:0/765037291 shutdown_connections 2024-03-28T18:08:10.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:10.871+0000 7f2289000640 1 -- 172.21.15.37:0/765037291 wait complete. 2024-03-28T18:08:10.874 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:08:10 smithi037 podman[71045]: 2024-03-28 18:08:10.521632236 +0000 UTC m=+0.456262012 container init 38fcb06288bf118ffa1097e123eb85ba5096999c2137001043b346c8d8089dce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-28T18:08:10.874 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:08:10 smithi037 podman[71045]: 2024-03-28 18:08:10.529590543 +0000 UTC m=+0.464220270 container start 38fcb06288bf118ffa1097e123eb85ba5096999c2137001043b346c8d8089dce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:08:10.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:08:10 smithi037 ceph-mds[71084]: -- 172.21.15.37:0/751209306 <== mon.0 v2:172.21.15.37:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55907245da40 con 0x559072431000 2024-03-28T18:08:10.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:08:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: starting mds.a at 2024-03-28T18:08:10.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:08:10 smithi037 bash[71045]: 38fcb06288bf118ffa1097e123eb85ba5096999c2137001043b346c8d8089dce 2024-03-28T18:08:10.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:08:10 smithi037 systemd[1]: Started Ceph mds.a for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:08:11.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:11 smithi059 ceph-mon[36190]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:11.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:11 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] up:boot 2024-03-28T18:08:11.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:11 smithi059 ceph-mon[36190]: fsmap 11 up:standby 2024-03-28T18:08:11.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:11 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:08:11.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:11 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:11.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:11 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:11.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:11 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:11.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:11 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:08:11.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:11 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:08:11.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:11 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:08:11.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:11 smithi059 ceph-mon[36190]: Deploying daemon mds.b on smithi059 2024-03-28T18:08:11.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:11 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/765037291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:11 smithi170 ceph-mon[36280]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:11 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] up:boot 2024-03-28T18:08:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:11 smithi170 ceph-mon[36280]: fsmap 11 up:standby 2024-03-28T18:08:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:11 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:08:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:11 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:11 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:11 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:11 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:08:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:11 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:08:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:11 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:08:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:11 smithi170 ceph-mon[36280]: Deploying daemon mds.b on smithi059 2024-03-28T18:08:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:11 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/765037291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:11 smithi037 ceph-mon[31460]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:11 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] up:boot 2024-03-28T18:08:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:11 smithi037 ceph-mon[31460]: fsmap 11 up:standby 2024-03-28T18:08:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:11 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:08:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:11 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:11 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:11 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:11 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:08:12.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:11 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:08:12.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:11 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:08:12.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:11 smithi037 ceph-mon[31460]: Deploying daemon mds.b on smithi059 2024-03-28T18:08:12.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:11 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/765037291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:12.896 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:08:13.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.132+0000 7fb5cee00640 1 -- 172.21.15.37:0/2545120605 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c8105490 msgr2=0x7fb5c810bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.132+0000 7fb5cee00640 1 --2- 172.21.15.37:0/2545120605 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c8105490 0x7fb5c810bd20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0009be0 tx=0x7fb5c002bef0 comp rx=0 tx=0).stop 2024-03-28T18:08:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.132+0000 7fb5cee00640 1 -- 172.21.15.37:0/2545120605 shutdown_connections 2024-03-28T18:08:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.132+0000 7fb5cee00640 1 --2- 172.21.15.37:0/2545120605 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c8105490 0x7fb5c810bd20 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.132+0000 7fb5cee00640 1 --2- 172.21.15.37:0/2545120605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5c8104ad0 0x7fb5c8104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.132+0000 7fb5cee00640 1 --2- 172.21.15.37:0/2545120605 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5c81038e0 0x7fb5c8103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.132+0000 7fb5cee00640 1 -- 172.21.15.37:0/2545120605 >> 172.21.15.37:0/2545120605 conn(0x7fb5c80fee90 msgr2=0x7fb5c81012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.132+0000 7fb5cee00640 1 -- 172.21.15.37:0/2545120605 shutdown_connections 2024-03-28T18:08:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.132+0000 7fb5cee00640 1 -- 172.21.15.37:0/2545120605 wait complete. 2024-03-28T18:08:13.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.133+0000 7fb5cee00640 1 Processor -- start 2024-03-28T18:08:13.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.134+0000 7fb5cee00640 1 -- start start 2024-03-28T18:08:13.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.134+0000 7fb5cee00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5c81038e0 0x7fb5c8199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:13.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.134+0000 7fb5cee00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5c8104ad0 0x7fb5c819a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:13.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.134+0000 7fb5cee00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c8105490 0x7fb5c81a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:13.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.134+0000 7fb5cee00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c810f5b0 con 0x7fb5c8105490 2024-03-28T18:08:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.134+0000 7fb5cee00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c810f430 con 0x7fb5c81038e0 2024-03-28T18:08:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.134+0000 7fb5cee00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c810f730 con 0x7fb5c8104ad0 2024-03-28T18:08:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.134+0000 7fb5c7e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5c81038e0 0x7fb5c8199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.134+0000 7fb5c7e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5c81038e0 0x7fb5c8199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:33124/0 (socket says 172.21.15.37:33124) 2024-03-28T18:08:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.134+0000 7fb5c7400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5c8104ad0 0x7fb5c819a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.134+0000 7fb5c7e00640 1 -- 172.21.15.37:0/2955230626 learned_addr learned my addr 172.21.15.37:0/2955230626 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.134+0000 7fb5cce00640 1 --2- 172.21.15.37:0/2955230626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c8105490 0x7fb5c81a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.135+0000 7fb5c7e00640 1 -- 172.21.15.37:0/2955230626 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5c8104ad0 msgr2=0x7fb5c819a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.135+0000 7fb5c7e00640 1 --2- 172.21.15.37:0/2955230626 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5c8104ad0 0x7fb5c819a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.135+0000 7fb5c7e00640 1 -- 172.21.15.37:0/2955230626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c8105490 msgr2=0x7fb5c81a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.135+0000 7fb5c7e00640 1 --2- 172.21.15.37:0/2955230626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c8105490 0x7fb5c81a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.135+0000 7fb5c7e00640 1 -- 172.21.15.37:0/2955230626 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c81a17c0 con 0x7fb5c81038e0 2024-03-28T18:08:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.135+0000 7fb5c7e00640 1 --2- 172.21.15.37:0/2955230626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5c81038e0 0x7fb5c8199ba0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800e9a0 tx=0x7fb5b800ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.135+0000 7fb5c4c00640 1 -- 172.21.15.37:0/2955230626 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b800cd80 con 0x7fb5c81038e0 2024-03-28T18:08:13.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.135+0000 7fb5c4c00640 1 -- 172.21.15.37:0/2955230626 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5b8010910 con 0x7fb5c81038e0 2024-03-28T18:08:13.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.135+0000 7fb5cee00640 1 -- 172.21.15.37:0/2955230626 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c81a1ab0 con 0x7fb5c81038e0 2024-03-28T18:08:13.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.135+0000 7fb5c4c00640 1 -- 172.21.15.37:0/2955230626 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8018a30 con 0x7fb5c81038e0 2024-03-28T18:08:13.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.135+0000 7fb5cee00640 1 -- 172.21.15.37:0/2955230626 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c81a2000 con 0x7fb5c81038e0 2024-03-28T18:08:13.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.136+0000 7fb5cee00640 1 -- 172.21.15.37:0/2955230626 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c810aba0 con 0x7fb5c81038e0 2024-03-28T18:08:13.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.139+0000 7fb5c4c00640 1 -- 172.21.15.37:0/2955230626 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb5b8010ac0 con 0x7fb5c81038e0 2024-03-28T18:08:13.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.140+0000 7fb5c4c00640 1 --2- 172.21.15.37:0/2955230626 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb5ac0a2800 0x7fb5ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:13.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.140+0000 7fb5c4c00640 1 -- 172.21.15.37:0/2955230626 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb5b80cf0b0 con 0x7fb5c81038e0 2024-03-28T18:08:13.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.140+0000 7fb5c7400640 1 --2- 172.21.15.37:0/2955230626 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb5ac0a2800 0x7fb5ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:13.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.141+0000 7fb5c7400640 1 --2- 172.21.15.37:0/2955230626 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb5ac0a2800 0x7fb5ac0a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb5c819b0b0 tx=0x7fb5b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:13.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.146+0000 7fb5c4c00640 1 -- 172.21.15.37:0/2955230626 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb5b8026070 con 0x7fb5c81038e0 2024-03-28T18:08:13.249 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:08:13 smithi059 systemd[1]: Starting Ceph mds.b for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:08:13.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.394+0000 7fb5cee00640 1 -- 172.21.15.37:0/2955230626 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5c802d190 con 0x7fb5c81038e0 2024-03-28T18:08:13.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.395+0000 7fb5c4c00640 1 -- 172.21.15.37:0/2955230626 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9239 (secure 0 0 0) 0x7fb5b809c550 con 0x7fb5c81038e0 2024-03-28T18:08:13.396 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:08:13.396 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1926270371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6838","nonce":1926270371},{"type":"v1","addr":"172.21.15.37:6839","nonce":1926270371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6837/4034607523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":4034607523},{"type":"v1","addr":"172.21.15.59:6837","nonce":4034607523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6839/1575518279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":1575518279},{"type":"v1","addr":"172.21.15.170:6839","nonce":1575518279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24580,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6841/1923263622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6840","nonce":1923263622},{"type":"v1","addr":"172.21.15.37:6841","nonce":1923263622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[]} 2024-03-28T18:08:13.396 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 12 2024-03-28T18:08:13.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.398+0000 7fb5cee00640 1 -- 172.21.15.37:0/2955230626 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb5ac0a2800 msgr2=0x7fb5ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:13.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.398+0000 7fb5cee00640 1 --2- 172.21.15.37:0/2955230626 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb5ac0a2800 0x7fb5ac0a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb5c819b0b0 tx=0x7fb5b4005f90 comp rx=0 tx=0).stop 2024-03-28T18:08:13.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.398+0000 7fb5cee00640 1 -- 172.21.15.37:0/2955230626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5c81038e0 msgr2=0x7fb5c8199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:13.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.398+0000 7fb5cee00640 1 --2- 172.21.15.37:0/2955230626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5c81038e0 0x7fb5c8199ba0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800e9a0 tx=0x7fb5b800ee60 comp rx=0 tx=0).stop 2024-03-28T18:08:13.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.398+0000 7fb5cee00640 1 -- 172.21.15.37:0/2955230626 shutdown_connections 2024-03-28T18:08:13.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.398+0000 7fb5cee00640 1 --2- 172.21.15.37:0/2955230626 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb5ac0a2800 0x7fb5ac0a4cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:13.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.398+0000 7fb5cee00640 1 --2- 172.21.15.37:0/2955230626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c8105490 0x7fb5c81a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:13.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.398+0000 7fb5cee00640 1 --2- 172.21.15.37:0/2955230626 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5c8104ad0 0x7fb5c819a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:13.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.398+0000 7fb5cee00640 1 --2- 172.21.15.37:0/2955230626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5c81038e0 0x7fb5c8199ba0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:13.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.398+0000 7fb5cee00640 1 -- 172.21.15.37:0/2955230626 >> 172.21.15.37:0/2955230626 conn(0x7fb5c80fee90 msgr2=0x7fb5c8107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:13.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.398+0000 7fb5cee00640 1 -- 172.21.15.37:0/2955230626 shutdown_connections 2024-03-28T18:08:13.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:13.398+0000 7fb5cee00640 1 -- 172.21.15.37:0/2955230626 wait complete. 2024-03-28T18:08:13.506 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:08:13 smithi059 podman[66775]: 2024-03-28 18:08:13.152866512 +0000 UTC m=+0.021416484 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:08:13.506 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:08:13 smithi059 podman[66775]: 2024-03-28 18:08:13.259672862 +0000 UTC m=+0.128222821 container create 014effd69bd278b531ebf2a39e7be743f1fe0b1449dd873da7ebea38f558f280 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0) 2024-03-28T18:08:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:13 smithi059 ceph-mon[36190]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:13 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2955230626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:13.785 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:08:13 smithi059 podman[66775]: 2024-03-28 18:08:13.61290787 +0000 UTC m=+0.481457833 container init 014effd69bd278b531ebf2a39e7be743f1fe0b1449dd873da7ebea38f558f280 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-28T18:08:13.785 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:08:13 smithi059 podman[66775]: 2024-03-28 18:08:13.61564318 +0000 UTC m=+0.484193139 container start 014effd69bd278b531ebf2a39e7be743f1fe0b1449dd873da7ebea38f558f280 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:08:13.785 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:08:13 smithi059 ceph-mds[66795]: -- 172.21.15.59:0/938997995 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5580f250da40 con 0x5580f24e0c00 2024-03-28T18:08:13.785 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:08:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: starting mds.b at 2024-03-28T18:08:13.785 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:08:13 smithi059 bash[66775]: 014effd69bd278b531ebf2a39e7be743f1fe0b1449dd873da7ebea38f558f280 2024-03-28T18:08:13.786 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:08:13 smithi059 systemd[1]: Started Ceph mds.b for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:08:13.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:13 smithi170 ceph-mon[36280]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:13.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:13 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2955230626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:13 smithi037 ceph-mon[31460]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:13 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2955230626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:14 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] up:boot 2024-03-28T18:08:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:14 smithi059 ceph-mon[36190]: fsmap 12 up:standby 2024-03-28T18:08:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:08:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:08:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:14 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] up:boot 2024-03-28T18:08:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:14 smithi170 ceph-mon[36280]: fsmap 12 up:standby 2024-03-28T18:08:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:08:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:08:15.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:14 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] up:boot 2024-03-28T18:08:15.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:14 smithi037 ceph-mon[31460]: fsmap 12 up:standby 2024-03-28T18:08:15.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:08:15.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:15.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:15.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:15.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:15.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:15.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:15.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:08:15.421 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:08:15.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.641+0000 7f26fce00640 1 -- 172.21.15.37:0/1346240938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26f81030c0 msgr2=0x7f26f8109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:15.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.641+0000 7f26fce00640 1 --2- 172.21.15.37:0/1346240938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26f81030c0 0x7f26f8109950 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f26f0009ca0 tx=0x7f26f002c0f0 comp rx=0 tx=0).stop 2024-03-28T18:08:15.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.641+0000 7f26fce00640 1 -- 172.21.15.37:0/1346240938 shutdown_connections 2024-03-28T18:08:15.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.641+0000 7f26fce00640 1 --2- 172.21.15.37:0/1346240938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26f81030c0 0x7f26f8109950 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:15.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.641+0000 7f26fce00640 1 --2- 172.21.15.37:0/1346240938 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26f8102700 0x7f26f8102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:15.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.641+0000 7f26fce00640 1 --2- 172.21.15.37:0/1346240938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26f8101510 0x7f26f8101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:15.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.641+0000 7f26fce00640 1 -- 172.21.15.37:0/1346240938 >> 172.21.15.37:0/1346240938 conn(0x7f26f80fca80 msgr2=0x7f26f80feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:15.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.642+0000 7f26fce00640 1 -- 172.21.15.37:0/1346240938 shutdown_connections 2024-03-28T18:08:15.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.642+0000 7f26fce00640 1 -- 172.21.15.37:0/1346240938 wait complete. 2024-03-28T18:08:15.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.643+0000 7f26fce00640 1 Processor -- start 2024-03-28T18:08:15.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.643+0000 7f26fce00640 1 -- start start 2024-03-28T18:08:15.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.643+0000 7f26fce00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26f8101510 0x7f26f8075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:15.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.643+0000 7f26fce00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26f8102700 0x7f26f8073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:15.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.643+0000 7f26fce00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26f81030c0 0x7f26f8073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:15.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.643+0000 7f26fce00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f810d1a0 con 0x7f26f8102700 2024-03-28T18:08:15.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.643+0000 7f26fce00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f810d020 con 0x7f26f8101510 2024-03-28T18:08:15.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.643+0000 7f26fce00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f810d320 con 0x7f26f81030c0 2024-03-28T18:08:15.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.643+0000 7f26f7400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26f8101510 0x7f26f8075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:15.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.643+0000 7f26f7400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26f8101510 0x7f26f8075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:48180/0 (socket says 172.21.15.37:48180) 2024-03-28T18:08:15.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.644+0000 7f26f6a00640 1 --2- 172.21.15.37:0/1644094579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26f8102700 0x7f26f8073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:15.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.643+0000 7f26f7400640 1 -- 172.21.15.37:0/1644094579 learned_addr learned my addr 172.21.15.37:0/1644094579 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:15.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.644+0000 7f26f7e00640 1 --2- 172.21.15.37:0/1644094579 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26f81030c0 0x7f26f8073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:15.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.644+0000 7f26f7400640 1 -- 172.21.15.37:0/1644094579 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26f81030c0 msgr2=0x7f26f8073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:15.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.644+0000 7f26f7400640 1 --2- 172.21.15.37:0/1644094579 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26f81030c0 0x7f26f8073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:15.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.644+0000 7f26f7400640 1 -- 172.21.15.37:0/1644094579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26f8102700 msgr2=0x7f26f8073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:15.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.644+0000 7f26f7400640 1 --2- 172.21.15.37:0/1644094579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26f8102700 0x7f26f8073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:15.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.644+0000 7f26f7400640 1 -- 172.21.15.37:0/1644094579 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f8074860 con 0x7f26f8101510 2024-03-28T18:08:15.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.644+0000 7f26f6a00640 1 --2- 172.21.15.37:0/1644094579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26f8102700 0x7f26f8073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:08:15.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.645+0000 7f26f7400640 1 --2- 172.21.15.37:0/1644094579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26f8101510 0x7f26f8075450 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f26e800e9c0 tx=0x7f26e800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:15.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.645+0000 7f26efe00640 1 -- 172.21.15.37:0/1644094579 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e800cd80 con 0x7f26f8101510 2024-03-28T18:08:15.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.645+0000 7f26efe00640 1 -- 172.21.15.37:0/1644094579 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26e80035f0 con 0x7f26f8101510 2024-03-28T18:08:15.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.645+0000 7f26fce00640 1 -- 172.21.15.37:0/1644094579 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f81a18f0 con 0x7f26f8101510 2024-03-28T18:08:15.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.645+0000 7f26efe00640 1 -- 172.21.15.37:0/1644094579 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e8010690 con 0x7f26f8101510 2024-03-28T18:08:15.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.645+0000 7f26fce00640 1 -- 172.21.15.37:0/1644094579 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f81a1e40 con 0x7f26f8101510 2024-03-28T18:08:15.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.647+0000 7f26fce00640 1 -- 172.21.15.37:0/1644094579 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f81088d0 con 0x7f26f8101510 2024-03-28T18:08:15.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.648+0000 7f26efe00640 1 -- 172.21.15.37:0/1644094579 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f26e80037a0 con 0x7f26f8101510 2024-03-28T18:08:15.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.650+0000 7f26efe00640 1 --2- 172.21.15.37:0/1644094579 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f26dc0a28d0 0x7f26dc0a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:15.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.650+0000 7f26efe00640 1 -- 172.21.15.37:0/1644094579 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f26e80cea20 con 0x7f26f8101510 2024-03-28T18:08:15.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.650+0000 7f26f6a00640 1 --2- 172.21.15.37:0/1644094579 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f26dc0a28d0 0x7f26dc0a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:15.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.651+0000 7f26f6a00640 1 --2- 172.21.15.37:0/1644094579 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f26dc0a28d0 0x7f26dc0a4d90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f26f80751c0 tx=0x7f26e400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:15.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.656+0000 7f26efe00640 1 -- 172.21.15.37:0/1644094579 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f26e809bec0 con 0x7f26f8101510 2024-03-28T18:08:15.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.904+0000 7f26fce00640 1 -- 172.21.15.37:0/1644094579 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26f802d430 con 0x7f26f8101510 2024-03-28T18:08:15.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.905+0000 7f26efe00640 1 -- 172.21.15.37:0/1644094579 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10031 (secure 0 0 0) 0x7f26e809b7c0 con 0x7f26f8101510 2024-03-28T18:08:15.905 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:08:15.905 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1926270371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6838","nonce":1926270371},{"type":"v1","addr":"172.21.15.37:6839","nonce":1926270371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6837/4034607523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":4034607523},{"type":"v1","addr":"172.21.15.59:6837","nonce":4034607523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6839/1575518279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":1575518279},{"type":"v1","addr":"172.21.15.170:6839","nonce":1575518279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24569,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6839/2091629078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":2091629078},{"type":"v1","addr":"172.21.15.59:6839","nonce":2091629078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":24580,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6841/1923263622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6840","nonce":1923263622},{"type":"v1","addr":"172.21.15.37:6841","nonce":1923263622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[]} 2024-03-28T18:08:15.905 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 13 2024-03-28T18:08:15.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.908+0000 7f26fce00640 1 -- 172.21.15.37:0/1644094579 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f26dc0a28d0 msgr2=0x7f26dc0a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:15.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.908+0000 7f26fce00640 1 --2- 172.21.15.37:0/1644094579 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f26dc0a28d0 0x7f26dc0a4d90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f26f80751c0 tx=0x7f26e400b410 comp rx=0 tx=0).stop 2024-03-28T18:08:15.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.908+0000 7f26fce00640 1 -- 172.21.15.37:0/1644094579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26f8101510 msgr2=0x7f26f8075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:15.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.908+0000 7f26fce00640 1 --2- 172.21.15.37:0/1644094579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26f8101510 0x7f26f8075450 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f26e800e9c0 tx=0x7f26e800ee80 comp rx=0 tx=0).stop 2024-03-28T18:08:15.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.908+0000 7f26fce00640 1 -- 172.21.15.37:0/1644094579 shutdown_connections 2024-03-28T18:08:15.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.908+0000 7f26fce00640 1 --2- 172.21.15.37:0/1644094579 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f26dc0a28d0 0x7f26dc0a4d90 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:15.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.908+0000 7f26fce00640 1 --2- 172.21.15.37:0/1644094579 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26f81030c0 0x7f26f8073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:15.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.908+0000 7f26fce00640 1 --2- 172.21.15.37:0/1644094579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26f8102700 0x7f26f8073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:15.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.908+0000 7f26fce00640 1 --2- 172.21.15.37:0/1644094579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26f8101510 0x7f26f8075450 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:15.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.908+0000 7f26fce00640 1 -- 172.21.15.37:0/1644094579 >> 172.21.15.37:0/1644094579 conn(0x7f26f80fca80 msgr2=0x7f26f8105aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:15.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.908+0000 7f26fce00640 1 -- 172.21.15.37:0/1644094579 shutdown_connections 2024-03-28T18:08:15.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:15.908+0000 7f26fce00640 1 -- 172.21.15.37:0/1644094579 wait complete. 2024-03-28T18:08:15.928 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:08:15.997 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:15 smithi037 ceph-mon[31460]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:16.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:15 smithi059 ceph-mon[36190]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:16.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:15 smithi170 ceph-mon[36280]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:16.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.133+0000 7fd286200640 1 -- 172.21.15.37:0/4291391754 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd280075660 msgr2=0x7fd280073ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:16.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.133+0000 7fd286200640 1 --2- 172.21.15.37:0/4291391754 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd280075660 0x7fd280073ab0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd274009900 tx=0x7fd27402bcd0 comp rx=0 tx=0).stop 2024-03-28T18:08:16.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.133+0000 7fd286200640 1 -- 172.21.15.37:0/4291391754 shutdown_connections 2024-03-28T18:08:16.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.133+0000 7fd286200640 1 --2- 172.21.15.37:0/4291391754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2801093b0 0x7fd28010b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:16.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.133+0000 7fd286200640 1 --2- 172.21.15.37:0/4291391754 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd280074080 0x7fd2800744e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:16.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.133+0000 7fd286200640 1 --2- 172.21.15.37:0/4291391754 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd280075660 0x7fd280073ab0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:16.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.133+0000 7fd286200640 1 -- 172.21.15.37:0/4291391754 >> 172.21.15.37:0/4291391754 conn(0x7fd2800fca70 msgr2=0x7fd2800feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:16.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.133+0000 7fd286200640 1 -- 172.21.15.37:0/4291391754 shutdown_connections 2024-03-28T18:08:16.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.133+0000 7fd286200640 1 -- 172.21.15.37:0/4291391754 wait complete. 2024-03-28T18:08:16.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.135+0000 7fd286200640 1 Processor -- start 2024-03-28T18:08:16.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.135+0000 7fd286200640 1 -- start start 2024-03-28T18:08:16.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.135+0000 7fd286200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd280074080 0x7fd2801999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:16.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.135+0000 7fd286200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd280075660 0x7fd280199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:16.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.136+0000 7fd286200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2801093b0 0x7fd2801a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:16.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.136+0000 7fd286200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28010f3d0 con 0x7fd2801093b0 2024-03-28T18:08:16.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.136+0000 7fd286200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28010f250 con 0x7fd280074080 2024-03-28T18:08:16.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.136+0000 7fd286200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28010f550 con 0x7fd280075660 2024-03-28T18:08:16.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.136+0000 7fd284e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd280074080 0x7fd2801999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:16.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.136+0000 7fd284e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd280074080 0x7fd2801999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:48192/0 (socket says 172.21.15.37:48192) 2024-03-28T18:08:16.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.136+0000 7fd284e00640 1 -- 172.21.15.37:0/4131452485 learned_addr learned my addr 172.21.15.37:0/4131452485 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:16.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.136+0000 7fd285800640 1 --2- 172.21.15.37:0/4131452485 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2801093b0 0x7fd2801a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:16.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.136+0000 7fd27fe00640 1 --2- 172.21.15.37:0/4131452485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd280075660 0x7fd280199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:16.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.136+0000 7fd284e00640 1 -- 172.21.15.37:0/4131452485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd280075660 msgr2=0x7fd280199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:16.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.136+0000 7fd284e00640 1 --2- 172.21.15.37:0/4131452485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd280075660 0x7fd280199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:16.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.136+0000 7fd284e00640 1 -- 172.21.15.37:0/4131452485 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2801093b0 msgr2=0x7fd2801a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:16.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.136+0000 7fd284e00640 1 --2- 172.21.15.37:0/4131452485 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2801093b0 0x7fd2801a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:16.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.136+0000 7fd284e00640 1 -- 172.21.15.37:0/4131452485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2801a1600 con 0x7fd280074080 2024-03-28T18:08:16.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.137+0000 7fd284e00640 1 --2- 172.21.15.37:0/4131452485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd280074080 0x7fd2801999e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd274005ab0 tx=0x7fd27402c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:16.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.137+0000 7fd27d600640 1 -- 172.21.15.37:0/4131452485 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd274043020 con 0x7fd280074080 2024-03-28T18:08:16.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.137+0000 7fd27d600640 1 -- 172.21.15.37:0/4131452485 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2740054e0 con 0x7fd280074080 2024-03-28T18:08:16.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.137+0000 7fd286200640 1 -- 172.21.15.37:0/4131452485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2801a1890 con 0x7fd280074080 2024-03-28T18:08:16.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.137+0000 7fd27d600640 1 -- 172.21.15.37:0/4131452485 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd274003ca0 con 0x7fd280074080 2024-03-28T18:08:16.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.137+0000 7fd286200640 1 -- 172.21.15.37:0/4131452485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2801a1de0 con 0x7fd280074080 2024-03-28T18:08:16.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.138+0000 7fd286200640 1 -- 172.21.15.37:0/4131452485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd280108170 con 0x7fd280074080 2024-03-28T18:08:16.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.140+0000 7fd27d600640 1 -- 172.21.15.37:0/4131452485 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd274046020 con 0x7fd280074080 2024-03-28T18:08:16.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.141+0000 7fd27d600640 1 --2- 172.21.15.37:0/4131452485 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd2680a2800 0x7fd2680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:16.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.141+0000 7fd27d600640 1 -- 172.21.15.37:0/4131452485 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd2740f0870 con 0x7fd280074080 2024-03-28T18:08:16.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.142+0000 7fd27fe00640 1 --2- 172.21.15.37:0/4131452485 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd2680a2800 0x7fd2680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:16.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.142+0000 7fd27fe00640 1 --2- 172.21.15.37:0/4131452485 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd2680a2800 0x7fd2680a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd26c005fd0 tx=0x7fd26c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:16.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.149+0000 7fd27d600640 1 -- 172.21.15.37:0/4131452485 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd27404d020 con 0x7fd280074080 2024-03-28T18:08:16.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.366+0000 7fd286200640 1 -- 172.21.15.37:0/4131452485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2800622a0 con 0x7fd280074080 2024-03-28T18:08:16.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.368+0000 7fd27d600640 1 -- 172.21.15.37:0/4131452485 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18433 (secure 0 0 0) 0x7fd2740bdd10 con 0x7fd280074080 2024-03-28T18:08:16.368 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:08:16.368 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":81,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","created":"2024-03-28T17:59:05.949847+0000","modified":"2024-03-28T18:07:22.031148+0000","last_up_change":"2024-03-28T18:07:21.021030+0000","last_in_change":"2024-03-28T18:07:01.416192+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:03:00.517967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db60618d-56f7-4130-8a90-84cbc480965b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6803","nonce":3562602035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6805","nonce":3562602035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6809","nonce":3562602035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6807","nonce":3562602035}]},"public_addr":"172.21.15.37:6803/3562602035","cluster_addr":"172.21.15.37:6805/3562602035","heartbeat_back_addr":"172.21.15.37:6809/3562602035","heartbeat_front_addr":"172.21.15.37:6807/3562602035","state":["exists","up"]},{"osd":1,"uuid":"7a4b7ad6-905e-4258-9574-693d4277dfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6801","nonce":3326537211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6803","nonce":3326537211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6807","nonce":3326537211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6805","nonce":3326537211}]},"public_addr":"172.21.15.59:6801/3326537211","cluster_addr":"172.21.15.59:6803/3326537211","heartbeat_back_addr":"172.21.15.59:6807/3326537211","heartbeat_front_addr":"172.21.15.59:6805/3326537211","state":["exists","up"]},{"osd":2,"uuid":"8b76ec1c-46bf-4336-89b5-12d82906e8b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6801","nonce":1269398427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6803","nonce":1269398427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6807","nonce":1269398427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6805","nonce":1269398427}]},"public_addr":"172.21.15.170:6801/1269398427","cluster_addr":"172.21.15.170:6803/1269398427","heartbeat_back_addr":"172.21.15.170:6807/1269398427","heartbeat_front_addr":"172.21.15.170:6805/1269398427","state":["exists","up"]},{"osd":3,"uuid":"66eca1da-2fbd-44e1-8bbf-5635a2c70ad5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6811","nonce":317805240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6813","nonce":317805240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6817","nonce":317805240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6815","nonce":317805240}]},"public_addr":"172.21.15.37:6811/317805240","cluster_addr":"172.21.15.37:6813/317805240","heartbeat_back_addr":"172.21.15.37:6817/317805240","heartbeat_front_addr":"172.21.15.37:6815/317805240","state":["exists","up"]},{"osd":4,"uuid":"4c877335-e42d-4d8e-9f48-a1d0a89ae438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6809","nonce":2410632623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6811","nonce":2410632623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6815","nonce":2410632623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6813","nonce":2410632623}]},"public_addr":"172.21.15.59:6809/2410632623","cluster_addr":"172.21.15.59:6811/2410632623","heartbeat_back_addr":"172.21.15.59:6815/2410632623","heartbeat_front_addr":"172.21.15.59:6813/2410632623","state":["exists","up"]},{"osd":5,"uuid":"b884842a-86e2-455b-9570-ff163b3bc0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6809","nonce":1218943899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6811","nonce":1218943899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6815","nonce":1218943899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6813","nonce":1218943899}]},"public_addr":"172.21.15.170:6809/1218943899","cluster_addr":"172.21.15.170:6811/1218943899","heartbeat_back_addr":"172.21.15.170:6815/1218943899","heartbeat_front_addr":"172.21.15.170:6813/1218943899","state":["exists","up"]},{"osd":6,"uuid":"249a1200-77f4-44ad-981f-1d352f732b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6819","nonce":1899652046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6821","nonce":1899652046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6825","nonce":1899652046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6823","nonce":1899652046}]},"public_addr":"172.21.15.37:6819/1899652046","cluster_addr":"172.21.15.37:6821/1899652046","heartbeat_back_addr":"172.21.15.37:6825/1899652046","heartbeat_front_addr":"172.21.15.37:6823/1899652046","state":["exists","up"]},{"osd":7,"uuid":"c73338e6-f505-4517-a84b-f35e04750a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6817","nonce":1343675690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6819","nonce":1343675690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6823","nonce":1343675690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6821","nonce":1343675690}]},"public_addr":"172.21.15.59:6817/1343675690","cluster_addr":"172.21.15.59:6819/1343675690","heartbeat_back_addr":"172.21.15.59:6823/1343675690","heartbeat_front_addr":"172.21.15.59:6821/1343675690","state":["exists","up"]},{"osd":8,"uuid":"149b8297-ff2d-4a88-9b1e-0def021cc516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6817","nonce":3235794556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6819","nonce":3235794556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6823","nonce":3235794556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6821","nonce":3235794556}]},"public_addr":"172.21.15.170:6817/3235794556","cluster_addr":"172.21.15.170:6819/3235794556","heartbeat_back_addr":"172.21.15.170:6823/3235794556","heartbeat_front_addr":"172.21.15.170:6821/3235794556","state":["exists","up"]},{"osd":9,"uuid":"55c27cc2-c771-44b7-90fe-8f0feca44162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6827","nonce":2307153008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6829","nonce":2307153008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6833","nonce":2307153008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6831","nonce":2307153008}]},"public_addr":"172.21.15.37:6827/2307153008","cluster_addr":"172.21.15.37:6829/2307153008","heartbeat_back_addr":"172.21.15.37:6833/2307153008","heartbeat_front_addr":"172.21.15.37:6831/2307153008","state":["exists","up"]},{"osd":10,"uuid":"c0595135-ab81-49d3-8c23-19f960c03e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6825","nonce":2113830275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6827","nonce":2113830275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6831","nonce":2113830275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6829","nonce":2113830275}]},"public_addr":"172.21.15.59:6825/2113830275","cluster_addr":"172.21.15.59:6827/2113830275","heartbeat_back_addr":"172.21.15.59:6831/2113830275","heartbeat_front_addr":"172.21.15.59:6829/2113830275","state":["exists","up"]},{"osd":11,"uuid":"5871e177-c17b-4637-a797-4fa5125920dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6825","nonce":3082439360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6827","nonce":3082439360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6831","nonce":3082439360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6829","nonce":3082439360}]},"public_addr":"172.21.15.170:6825/3082439360","cluster_addr":"172.21.15.170:6827/3082439360","heartbeat_back_addr":"172.21.15.170:6831/3082439360","heartbeat_front_addr":"172.21.15.170:6829/3082439360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:07.690545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:33.183396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:57.400263+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:30.597211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:57.259044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:22.627823+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:57.585863+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:25.000476+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:50.876688+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:25.859761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:53.563146+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:07:19.304115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3023379428":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1443710231":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6800/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6801/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6801/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6800/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1007625490":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/2895698584":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/497313034":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/3293943475":"2024-03-29T17:59:39.162038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:08:16.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.370+0000 7fd286200640 1 -- 172.21.15.37:0/4131452485 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd2680a2800 msgr2=0x7fd2680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:16.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.370+0000 7fd286200640 1 --2- 172.21.15.37:0/4131452485 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd2680a2800 0x7fd2680a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd26c005fd0 tx=0x7fd26c005850 comp rx=0 tx=0).stop 2024-03-28T18:08:16.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.370+0000 7fd286200640 1 -- 172.21.15.37:0/4131452485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd280074080 msgr2=0x7fd2801999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:16.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.370+0000 7fd286200640 1 --2- 172.21.15.37:0/4131452485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd280074080 0x7fd2801999e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd274005ab0 tx=0x7fd27402c760 comp rx=0 tx=0).stop 2024-03-28T18:08:16.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.370+0000 7fd286200640 1 -- 172.21.15.37:0/4131452485 shutdown_connections 2024-03-28T18:08:16.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.370+0000 7fd286200640 1 --2- 172.21.15.37:0/4131452485 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd2680a2800 0x7fd2680a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:16.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.370+0000 7fd286200640 1 --2- 172.21.15.37:0/4131452485 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2801093b0 0x7fd2801a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:16.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.370+0000 7fd286200640 1 --2- 172.21.15.37:0/4131452485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd280075660 0x7fd280199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:16.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.370+0000 7fd286200640 1 --2- 172.21.15.37:0/4131452485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd280074080 0x7fd2801999e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:16.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.370+0000 7fd286200640 1 -- 172.21.15.37:0/4131452485 >> 172.21.15.37:0/4131452485 conn(0x7fd2800fca70 msgr2=0x7fd280105340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:16.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.370+0000 7fd286200640 1 -- 172.21.15.37:0/4131452485 shutdown_connections 2024-03-28T18:08:16.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.370+0000 7fd286200640 1 -- 172.21.15.37:0/4131452485 wait complete. 2024-03-28T18:08:16.389 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:03:00.517967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:08:16.390 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:08:16.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.588+0000 7fc738200640 1 -- 172.21.15.37:0/1227734435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc730102120 msgr2=0x7fc730104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:16.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.588+0000 7fc738200640 1 --2- 172.21.15.37:0/1227734435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc730102120 0x7fc730104510 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc724009860 tx=0x7fc72402bb70 comp rx=0 tx=0).stop 2024-03-28T18:08:16.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.589+0000 7fc738200640 1 -- 172.21.15.37:0/1227734435 shutdown_connections 2024-03-28T18:08:16.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.589+0000 7fc738200640 1 --2- 172.21.15.37:0/1227734435 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc730107380 0x7fc7301097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:16.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.589+0000 7fc738200640 1 --2- 172.21.15.37:0/1227734435 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc730104a50 0x7fc730106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:16.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.589+0000 7fc738200640 1 --2- 172.21.15.37:0/1227734435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc730102120 0x7fc730104510 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:16.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.589+0000 7fc738200640 1 -- 172.21.15.37:0/1227734435 >> 172.21.15.37:0/1227734435 conn(0x7fc7300fbb10 msgr2=0x7fc7300fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:16.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.589+0000 7fc738200640 1 -- 172.21.15.37:0/1227734435 shutdown_connections 2024-03-28T18:08:16.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.589+0000 7fc738200640 1 -- 172.21.15.37:0/1227734435 wait complete. 2024-03-28T18:08:16.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.590+0000 7fc738200640 1 Processor -- start 2024-03-28T18:08:16.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.590+0000 7fc738200640 1 -- start start 2024-03-28T18:08:16.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.590+0000 7fc738200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc730102120 0x7fc730195820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:16.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.590+0000 7fc738200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc730104a50 0x7fc730195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:16.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.590+0000 7fc738200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc730107380 0x7fc7301a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:16.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.590+0000 7fc738200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73010d330 con 0x7fc730102120 2024-03-28T18:08:16.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.590+0000 7fc738200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73010d1b0 con 0x7fc730104a50 2024-03-28T18:08:16.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.590+0000 7fc738200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73010d4b0 con 0x7fc730107380 2024-03-28T18:08:16.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.590+0000 7fc735800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc730102120 0x7fc730195820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:16.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.591+0000 7fc735800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc730102120 0x7fc730195820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56988/0 (socket says 172.21.15.37:56988) 2024-03-28T18:08:16.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.591+0000 7fc735800640 1 -- 172.21.15.37:0/4282151655 learned_addr learned my addr 172.21.15.37:0/4282151655 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:16.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.591+0000 7fc734e00640 1 --2- 172.21.15.37:0/4282151655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc730104a50 0x7fc730195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:16.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.591+0000 7fc736200640 1 --2- 172.21.15.37:0/4282151655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc730107380 0x7fc7301a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:16.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.591+0000 7fc734e00640 1 -- 172.21.15.37:0/4282151655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc730107380 msgr2=0x7fc7301a5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:16.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.591+0000 7fc734e00640 1 --2- 172.21.15.37:0/4282151655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc730107380 0x7fc7301a5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:16.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.591+0000 7fc734e00640 1 -- 172.21.15.37:0/4282151655 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc730102120 msgr2=0x7fc730195820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:16.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.591+0000 7fc734e00640 1 --2- 172.21.15.37:0/4282151655 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc730102120 0x7fc730195820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:16.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.591+0000 7fc734e00640 1 -- 172.21.15.37:0/4282151655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7301a5c50 con 0x7fc730104a50 2024-03-28T18:08:16.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.591+0000 7fc734e00640 1 --2- 172.21.15.37:0/4282151655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc730104a50 0x7fc730195d60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc73006a7d0 tx=0x7fc71c00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:16.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.591+0000 7fc72e000640 1 -- 172.21.15.37:0/4282151655 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c00ccd0 con 0x7fc730104a50 2024-03-28T18:08:16.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.592+0000 7fc72e000640 1 -- 172.21.15.37:0/4282151655 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc71c0035f0 con 0x7fc730104a50 2024-03-28T18:08:16.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.592+0000 7fc738200640 1 -- 172.21.15.37:0/4282151655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7301a5f40 con 0x7fc730104a50 2024-03-28T18:08:16.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.592+0000 7fc72e000640 1 -- 172.21.15.37:0/4282151655 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c010690 con 0x7fc730104a50 2024-03-28T18:08:16.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.592+0000 7fc738200640 1 -- 172.21.15.37:0/4282151655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7301a6450 con 0x7fc730104a50 2024-03-28T18:08:16.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.593+0000 7fc738200640 1 -- 172.21.15.37:0/4282151655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc730108ed0 con 0x7fc730104a50 2024-03-28T18:08:16.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.595+0000 7fc72e000640 1 -- 172.21.15.37:0/4282151655 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc71c0037a0 con 0x7fc730104a50 2024-03-28T18:08:16.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.597+0000 7fc72e000640 1 --2- 172.21.15.37:0/4282151655 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc7180a2880 0x7fc7180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:16.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.597+0000 7fc72e000640 1 -- 172.21.15.37:0/4282151655 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc71c0ce7f0 con 0x7fc730104a50 2024-03-28T18:08:16.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.597+0000 7fc735800640 1 --2- 172.21.15.37:0/4282151655 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc7180a2880 0x7fc7180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:16.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.598+0000 7fc735800640 1 --2- 172.21.15.37:0/4282151655 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc7180a2880 0x7fc7180a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc7240095f0 tx=0x7fc724034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:16.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.601+0000 7fc72e000640 1 -- 172.21.15.37:0/4282151655 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc71c09bc90 con 0x7fc730104a50 2024-03-28T18:08:16.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.824+0000 7fc738200640 1 -- 172.21.15.37:0/4282151655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc7300622a0 con 0x7fc730104a50 2024-03-28T18:08:16.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.826+0000 7fc72e000640 1 -- 172.21.15.37:0/4282151655 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc71c09b590 con 0x7fc730104a50 2024-03-28T18:08:16.826 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 1 2024-03-28T18:08:16.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.829+0000 7fc738200640 1 -- 172.21.15.37:0/4282151655 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc7180a2880 msgr2=0x7fc7180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:16.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.829+0000 7fc738200640 1 --2- 172.21.15.37:0/4282151655 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc7180a2880 0x7fc7180a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc7240095f0 tx=0x7fc724034ab0 comp rx=0 tx=0).stop 2024-03-28T18:08:16.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.829+0000 7fc738200640 1 -- 172.21.15.37:0/4282151655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc730104a50 msgr2=0x7fc730195d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:16.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.829+0000 7fc738200640 1 --2- 172.21.15.37:0/4282151655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc730104a50 0x7fc730195d60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc73006a7d0 tx=0x7fc71c00ee50 comp rx=0 tx=0).stop 2024-03-28T18:08:16.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.829+0000 7fc738200640 1 -- 172.21.15.37:0/4282151655 shutdown_connections 2024-03-28T18:08:16.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.829+0000 7fc738200640 1 --2- 172.21.15.37:0/4282151655 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc7180a2880 0x7fc7180a4d40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:16.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.829+0000 7fc738200640 1 --2- 172.21.15.37:0/4282151655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc730104a50 0x7fc730195d60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:16.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.829+0000 7fc738200640 1 --2- 172.21.15.37:0/4282151655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc730107380 0x7fc7301a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:16.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.829+0000 7fc738200640 1 --2- 172.21.15.37:0/4282151655 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc730102120 0x7fc730195820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:16.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.829+0000 7fc738200640 1 -- 172.21.15.37:0/4282151655 >> 172.21.15.37:0/4282151655 conn(0x7fc7300fbb10 msgr2=0x7fc7301018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:16.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.829+0000 7fc738200640 1 -- 172.21.15.37:0/4282151655 shutdown_connections 2024-03-28T18:08:16.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:16.829+0000 7fc738200640 1 -- 172.21.15.37:0/4282151655 wait complete. 2024-03-28T18:08:16.847 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T18:08:17.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.046+0000 7f2050200640 1 -- 172.21.15.37:0/3555827714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20481073c0 msgr2=0x7f2048109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:17.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.046+0000 7f2050200640 1 --2- 172.21.15.37:0/3555827714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20481073c0 0x7f2048109810 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2040009be0 tx=0x7f204002bef0 comp rx=0 tx=0).stop 2024-03-28T18:08:17.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.046+0000 7f2050200640 1 -- 172.21.15.37:0/3555827714 shutdown_connections 2024-03-28T18:08:17.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.046+0000 7f2050200640 1 --2- 172.21.15.37:0/3555827714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20481073c0 0x7f2048109810 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:17.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.046+0000 7f2050200640 1 --2- 172.21.15.37:0/3555827714 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2048104a90 0x7f2048106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:17.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.046+0000 7f2050200640 1 --2- 172.21.15.37:0/3555827714 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2048102160 0x7f2048104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:17.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.046+0000 7f2050200640 1 -- 172.21.15.37:0/3555827714 >> 172.21.15.37:0/3555827714 conn(0x7f20480fbb50 msgr2=0x7f20480fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:17.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.047+0000 7f2050200640 1 -- 172.21.15.37:0/3555827714 shutdown_connections 2024-03-28T18:08:17.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.047+0000 7f2050200640 1 -- 172.21.15.37:0/3555827714 wait complete. 2024-03-28T18:08:17.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.048+0000 7f2050200640 1 Processor -- start 2024-03-28T18:08:17.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.048+0000 7f2050200640 1 -- start start 2024-03-28T18:08:17.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.048+0000 7f2050200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2048102160 0x7f2048199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:17.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.048+0000 7f2050200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2048104a90 0x7f204819a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:17.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.048+0000 7f2050200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20481073c0 0x7f20481a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:17.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.048+0000 7f2050200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204810d3d0 con 0x7f2048104a90 2024-03-28T18:08:17.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.048+0000 7f2050200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204810d250 con 0x7f20481073c0 2024-03-28T18:08:17.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.048+0000 7f2050200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204810d550 con 0x7f2048102160 2024-03-28T18:08:17.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.048+0000 7f204ce00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2048104a90 0x7f204819a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:17.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.048+0000 7f204ce00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2048104a90 0x7f204819a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57004/0 (socket says 172.21.15.37:57004) 2024-03-28T18:08:17.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.048+0000 7f204ce00640 1 -- 172.21.15.37:0/2090288706 learned_addr learned my addr 172.21.15.37:0/2090288706 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:17.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.048+0000 7f204e200640 1 --2- 172.21.15.37:0/2090288706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20481073c0 0x7f20481a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:17.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.048+0000 7f204d800640 1 --2- 172.21.15.37:0/2090288706 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2048102160 0x7f2048199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:17.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.049+0000 7f204ce00640 1 -- 172.21.15.37:0/2090288706 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2048102160 msgr2=0x7f2048199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:17.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.049+0000 7f204ce00640 1 --2- 172.21.15.37:0/2090288706 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2048102160 0x7f2048199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:17.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.049+0000 7f204ce00640 1 -- 172.21.15.37:0/2090288706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20481073c0 msgr2=0x7f20481a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:17.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.049+0000 7f204ce00640 1 --2- 172.21.15.37:0/2090288706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20481073c0 0x7f20481a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:17.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.049+0000 7f204ce00640 1 -- 172.21.15.37:0/2090288706 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20481a17c0 con 0x7f2048104a90 2024-03-28T18:08:17.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.049+0000 7f204ce00640 1 --2- 172.21.15.37:0/2090288706 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2048104a90 0x7f204819a0e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f203400e9f0 tx=0x7f203400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:17.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.049+0000 7f2046000640 1 -- 172.21.15.37:0/2090288706 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203400ccc0 con 0x7f2048104a90 2024-03-28T18:08:17.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.049+0000 7f2046000640 1 -- 172.21.15.37:0/2090288706 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2034010910 con 0x7f2048104a90 2024-03-28T18:08:17.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.049+0000 7f2050200640 1 -- 172.21.15.37:0/2090288706 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20481a1ab0 con 0x7f2048104a90 2024-03-28T18:08:17.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.049+0000 7f2046000640 1 -- 172.21.15.37:0/2090288706 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2034018960 con 0x7f2048104a90 2024-03-28T18:08:17.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.050+0000 7f2050200640 1 -- 172.21.15.37:0/2090288706 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20481a2000 con 0x7f2048104a90 2024-03-28T18:08:17.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.051+0000 7f2050200640 1 -- 172.21.15.37:0/2090288706 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2048108eb0 con 0x7f2048104a90 2024-03-28T18:08:17.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.053+0000 7f2046000640 1 -- 172.21.15.37:0/2090288706 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2034010ac0 con 0x7f2048104a90 2024-03-28T18:08:17.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.055+0000 7f2046000640 1 --2- 172.21.15.37:0/2090288706 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f20300a2800 0x7f20300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:17.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.055+0000 7f2046000640 1 -- 172.21.15.37:0/2090288706 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f20340ce950 con 0x7f2048104a90 2024-03-28T18:08:17.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.055+0000 7f204d800640 1 --2- 172.21.15.37:0/2090288706 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f20300a2800 0x7f20300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:17.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.056+0000 7f204d800640 1 --2- 172.21.15.37:0/2090288706 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f20300a2800 0x7f20300a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f203c009bd0 tx=0x7f203c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:17.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.059+0000 7f2046000640 1 -- 172.21.15.37:0/2090288706 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f203409b7a0 con 0x7f2048104a90 2024-03-28T18:08:17.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:16 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:17.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:16 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:17.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:16 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1644094579' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:17.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:16 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:17.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:16 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:17.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:16 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/4131452485' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:08:17.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.274+0000 7f2050200640 1 -- 172.21.15.37:0/2090288706 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f20480622a0 con 0x7f2048104a90 2024-03-28T18:08:17.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.274+0000 7f2046000640 1 -- 172.21.15.37:0/2090288706 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+65 (secure 0 0 0) 0x7f203409f840 con 0x7f2048104a90 2024-03-28T18:08:17.274 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:08:17.274 INFO:teuthology.orchestra.run.smithi037.stdout:[ 2024-03-28T18:08:17.274 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-03-28T18:08:17.274 INFO:teuthology.orchestra.run.smithi037.stdout: "poolnum": 1, 2024-03-28T18:08:17.275 INFO:teuthology.orchestra.run.smithi037.stdout: "poolname": ".mgr" 2024-03-28T18:08:17.275 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-03-28T18:08:17.275 INFO:teuthology.orchestra.run.smithi037.stdout:] 2024-03-28T18:08:17.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.277+0000 7f2050200640 1 -- 172.21.15.37:0/2090288706 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f20300a2800 msgr2=0x7f20300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:17.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.277+0000 7f2050200640 1 --2- 172.21.15.37:0/2090288706 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f20300a2800 0x7f20300a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f203c009bd0 tx=0x7f203c005f90 comp rx=0 tx=0).stop 2024-03-28T18:08:17.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.277+0000 7f2050200640 1 -- 172.21.15.37:0/2090288706 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2048104a90 msgr2=0x7f204819a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:17.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.277+0000 7f2050200640 1 --2- 172.21.15.37:0/2090288706 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2048104a90 0x7f204819a0e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f203400e9f0 tx=0x7f203400eeb0 comp rx=0 tx=0).stop 2024-03-28T18:08:17.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.277+0000 7f2050200640 1 -- 172.21.15.37:0/2090288706 shutdown_connections 2024-03-28T18:08:17.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.277+0000 7f2050200640 1 --2- 172.21.15.37:0/2090288706 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f20300a2800 0x7f20300a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:17.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.277+0000 7f2050200640 1 --2- 172.21.15.37:0/2090288706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20481073c0 0x7f20481a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:17.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.277+0000 7f2050200640 1 --2- 172.21.15.37:0/2090288706 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2048104a90 0x7f204819a0e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:17.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.277+0000 7f2050200640 1 --2- 172.21.15.37:0/2090288706 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2048102160 0x7f2048199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:17.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.277+0000 7f2050200640 1 -- 172.21.15.37:0/2090288706 >> 172.21.15.37:0/2090288706 conn(0x7f20480fbb50 msgr2=0x7f2048101810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:17.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.277+0000 7f2050200640 1 -- 172.21.15.37:0/2090288706 shutdown_connections 2024-03-28T18:08:17.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.277+0000 7f2050200640 1 -- 172.21.15.37:0/2090288706 wait complete. 2024-03-28T18:08:17.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:16 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:17.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:16 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:17.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:16 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1644094579' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:17.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:16 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:17.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:16 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:17.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:16 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/4131452485' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:08:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:16 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:16 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:16 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1644094579' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:16 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:16 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:16 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/4131452485' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:08:17.299 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T18:08:17.299 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T18:08:17.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.496+0000 7f8ce6000640 1 -- 172.21.15.37:0/1892786967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ce01016b0 msgr2=0x7f8ce0101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:17.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.496+0000 7f8ce6000640 1 --2- 172.21.15.37:0/1892786967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ce01016b0 0x7f8ce0101ad0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8cd0009900 tx=0x7f8cd002bc50 comp rx=0 tx=0).stop 2024-03-28T18:08:17.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.496+0000 7f8ce6000640 1 -- 172.21.15.37:0/1892786967 shutdown_connections 2024-03-28T18:08:17.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.496+0000 7f8ce6000640 1 --2- 172.21.15.37:0/1892786967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce0103260 0x7f8ce0109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:17.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.496+0000 7f8ce6000640 1 --2- 172.21.15.37:0/1892786967 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8ce01028a0 0x7f8ce0102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:17.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.496+0000 7f8ce6000640 1 --2- 172.21.15.37:0/1892786967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ce01016b0 0x7f8ce0101ad0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:17.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.496+0000 7f8ce6000640 1 -- 172.21.15.37:0/1892786967 >> 172.21.15.37:0/1892786967 conn(0x7f8ce00fcc20 msgr2=0x7f8ce00ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:17.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.496+0000 7f8ce6000640 1 -- 172.21.15.37:0/1892786967 shutdown_connections 2024-03-28T18:08:17.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.496+0000 7f8ce6000640 1 -- 172.21.15.37:0/1892786967 wait complete. 2024-03-28T18:08:17.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.498+0000 7f8ce6000640 1 Processor -- start 2024-03-28T18:08:17.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.498+0000 7f8ce6000640 1 -- start start 2024-03-28T18:08:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.498+0000 7f8ce6000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce01016b0 0x7f8ce0075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.498+0000 7f8ce6000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ce01028a0 0x7f8ce0073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.498+0000 7f8ce6000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8ce0103260 0x7f8ce0073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.498+0000 7f8ce6000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce010d380 con 0x7f8ce0103260 2024-03-28T18:08:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.498+0000 7f8ce6000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce010d200 con 0x7f8ce01016b0 2024-03-28T18:08:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.498+0000 7f8ce6000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce010d500 con 0x7f8ce01028a0 2024-03-28T18:08:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.499+0000 7f8cdf400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce01016b0 0x7f8ce0075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.499+0000 7f8cdf400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce01016b0 0x7f8ce0075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:48238/0 (socket says 172.21.15.37:48238) 2024-03-28T18:08:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.499+0000 7f8cdf400640 1 -- 172.21.15.37:0/1101579216 learned_addr learned my addr 172.21.15.37:0/1101579216 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.499+0000 7f8cdea00640 1 --2- 172.21.15.37:0/1101579216 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ce01028a0 0x7f8ce0073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.499+0000 7f8cdf400640 1 -- 172.21.15.37:0/1101579216 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ce01028a0 msgr2=0x7f8ce0073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.499+0000 7f8cdf400640 1 --2- 172.21.15.37:0/1101579216 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ce01028a0 0x7f8ce0073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.499+0000 7f8cdf400640 1 -- 172.21.15.37:0/1101579216 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8ce0103260 msgr2=0x7f8ce0073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:17.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.499+0000 7f8cdf400640 1 --2- 172.21.15.37:0/1101579216 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8ce0103260 0x7f8ce0073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:17.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.499+0000 7f8cdf400640 1 -- 172.21.15.37:0/1101579216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce0074860 con 0x7f8ce01016b0 2024-03-28T18:08:17.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.499+0000 7f8cdea00640 1 --2- 172.21.15.37:0/1101579216 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ce01028a0 0x7f8ce0073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:08:17.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.499+0000 7f8cdf400640 1 --2- 172.21.15.37:0/1101579216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce01016b0 0x7f8ce0075450 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8cd0005ab0 tx=0x7f8cd002c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:17.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.500+0000 7f8cd7e00640 1 -- 172.21.15.37:0/1101579216 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0041070 con 0x7f8ce01016b0 2024-03-28T18:08:17.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.500+0000 7f8cd7e00640 1 -- 172.21.15.37:0/1101579216 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8cd0004450 con 0x7f8ce01016b0 2024-03-28T18:08:17.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.500+0000 7f8cd7e00640 1 -- 172.21.15.37:0/1101579216 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0034440 con 0x7f8ce01016b0 2024-03-28T18:08:17.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.500+0000 7f8ce6000640 1 -- 172.21.15.37:0/1101579216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce01a1b10 con 0x7f8ce01016b0 2024-03-28T18:08:17.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.500+0000 7f8ce6000640 1 -- 172.21.15.37:0/1101579216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce01a2010 con 0x7f8ce01016b0 2024-03-28T18:08:17.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.501+0000 7f8ce6000640 1 -- 172.21.15.37:0/1101579216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce01089d0 con 0x7f8ce01016b0 2024-03-28T18:08:17.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.503+0000 7f8cd7e00640 1 -- 172.21.15.37:0/1101579216 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8cd0044070 con 0x7f8ce01016b0 2024-03-28T18:08:17.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.504+0000 7f8cd7e00640 1 --2- 172.21.15.37:0/1101579216 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8cc40a2880 0x7f8cc40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:17.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.504+0000 7f8cd7e00640 1 -- 172.21.15.37:0/1101579216 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8cd00f02a0 con 0x7f8ce01016b0 2024-03-28T18:08:17.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.504+0000 7f8cdea00640 1 --2- 172.21.15.37:0/1101579216 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8cc40a2880 0x7f8cc40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:17.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.505+0000 7f8cdea00640 1 --2- 172.21.15.37:0/1101579216 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8cc40a2880 0x7f8cc40a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc005d00 tx=0x7f8ccc005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:17.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.508+0000 7f8cd7e00640 1 -- 172.21.15.37:0/1101579216 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8cd00bd740 con 0x7f8ce01016b0 2024-03-28T18:08:17.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:17.741+0000 7f8ce6000640 1 -- 172.21.15.37:0/1101579216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8ce002d070 con 0x7f8ce01016b0 2024-03-28T18:08:18.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.422+0000 7f8cd7e00640 1 -- 172.21.15.37:0/1101579216 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8cd00bd040 con 0x7f8ce01016b0 2024-03-28T18:08:18.422 INFO:teuthology.orchestra.run.smithi037.stderr:pool 'cephfs_metadata' created 2024-03-28T18:08:18.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.427+0000 7f8ce6000640 1 -- 172.21.15.37:0/1101579216 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8cc40a2880 msgr2=0x7f8cc40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:18.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.427+0000 7f8ce6000640 1 --2- 172.21.15.37:0/1101579216 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8cc40a2880 0x7f8cc40a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc005d00 tx=0x7f8ccc005ed0 comp rx=0 tx=0).stop 2024-03-28T18:08:18.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.427+0000 7f8ce6000640 1 -- 172.21.15.37:0/1101579216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce01016b0 msgr2=0x7f8ce0075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:18.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.427+0000 7f8ce6000640 1 --2- 172.21.15.37:0/1101579216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce01016b0 0x7f8ce0075450 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8cd0005ab0 tx=0x7f8cd002c6e0 comp rx=0 tx=0).stop 2024-03-28T18:08:18.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.428+0000 7f8ce6000640 1 -- 172.21.15.37:0/1101579216 shutdown_connections 2024-03-28T18:08:18.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.428+0000 7f8ce6000640 1 --2- 172.21.15.37:0/1101579216 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8cc40a2880 0x7f8cc40a4d40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:18.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.428+0000 7f8ce6000640 1 --2- 172.21.15.37:0/1101579216 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8ce0103260 0x7f8ce0073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:18.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.428+0000 7f8ce6000640 1 --2- 172.21.15.37:0/1101579216 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ce01028a0 0x7f8ce0073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:18.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.428+0000 7f8ce6000640 1 --2- 172.21.15.37:0/1101579216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce01016b0 0x7f8ce0075450 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:18.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.428+0000 7f8ce6000640 1 -- 172.21.15.37:0/1101579216 >> 172.21.15.37:0/1101579216 conn(0x7f8ce00fcc20 msgr2=0x7f8ce0105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:18.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.430+0000 7f8ce6000640 1 -- 172.21.15.37:0/1101579216 shutdown_connections 2024-03-28T18:08:18.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.432+0000 7f8ce6000640 1 -- 172.21.15.37:0/1101579216 wait complete. 2024-03-28T18:08:18.452 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T18:08:18.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:18 smithi037 ceph-mon[31460]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:18.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:18 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/4282151655' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:08:18.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:18 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:18.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:18 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:18.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:18 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:08:18.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:18 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:08:18.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:18 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:18.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:18 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:08:18.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:18 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2090288706' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T18:08:18.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.654+0000 7f75bd000640 1 -- 172.21.15.37:0/2776432735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75b8105490 msgr2=0x7f75b810bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:18.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.654+0000 7f75bd000640 1 --2- 172.21.15.37:0/2776432735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75b8105490 0x7f75b810bd20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f75b0009ca0 tx=0x7f75b002c100 comp rx=0 tx=0).stop 2024-03-28T18:08:18.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.654+0000 7f75bd000640 1 -- 172.21.15.37:0/2776432735 shutdown_connections 2024-03-28T18:08:18.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.654+0000 7f75bd000640 1 --2- 172.21.15.37:0/2776432735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75b8105490 0x7f75b810bd20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:18.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.654+0000 7f75bd000640 1 --2- 172.21.15.37:0/2776432735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75b8104ad0 0x7f75b8104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:18.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.654+0000 7f75bd000640 1 --2- 172.21.15.37:0/2776432735 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75b81038e0 0x7f75b8103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:18.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.654+0000 7f75bd000640 1 -- 172.21.15.37:0/2776432735 >> 172.21.15.37:0/2776432735 conn(0x7f75b80fee50 msgr2=0x7f75b8101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:18.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.655+0000 7f75bd000640 1 -- 172.21.15.37:0/2776432735 shutdown_connections 2024-03-28T18:08:18.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.655+0000 7f75bd000640 1 -- 172.21.15.37:0/2776432735 wait complete. 2024-03-28T18:08:18.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.655+0000 7f75bd000640 1 Processor -- start 2024-03-28T18:08:18.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.656+0000 7f75bd000640 1 -- start start 2024-03-28T18:08:18.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.656+0000 7f75bd000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75b81038e0 0x7f75b8199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:18.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.656+0000 7f75bd000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75b8104ad0 0x7f75b819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:18.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.656+0000 7f75bd000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75b8105490 0x7f75b81a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:18.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.656+0000 7f75bd000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b810f540 con 0x7f75b81038e0 2024-03-28T18:08:18.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.656+0000 7f75bd000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b810f3c0 con 0x7f75b8105490 2024-03-28T18:08:18.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.656+0000 7f75bd000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b810f6c0 con 0x7f75b8104ad0 2024-03-28T18:08:18.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.656+0000 7f75b5e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75b81038e0 0x7f75b8199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:18.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.656+0000 7f75b5400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75b8104ad0 0x7f75b819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:18.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.656+0000 7f75b6800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75b8105490 0x7f75b81a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:18.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.656+0000 7f75b5400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75b8104ad0 0x7f75b819a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:51294/0 (socket says 172.21.15.37:51294) 2024-03-28T18:08:18.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.656+0000 7f75b5e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75b81038e0 0x7f75b8199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57036/0 (socket says 172.21.15.37:57036) 2024-03-28T18:08:18.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.656+0000 7f75b5400640 1 -- 172.21.15.37:0/3918138494 learned_addr learned my addr 172.21.15.37:0/3918138494 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:18.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.656+0000 7f75b6800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75b8105490 0x7f75b81a1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:48254/0 (socket says 172.21.15.37:48254) 2024-03-28T18:08:18.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.657+0000 7f75b5400640 1 -- 172.21.15.37:0/3918138494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75b8105490 msgr2=0x7f75b81a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:18.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.657+0000 7f75b5400640 1 --2- 172.21.15.37:0/3918138494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75b8105490 0x7f75b81a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:18.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.657+0000 7f75b5400640 1 -- 172.21.15.37:0/3918138494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75b81038e0 msgr2=0x7f75b8199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:18.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.657+0000 7f75b5400640 1 --2- 172.21.15.37:0/3918138494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75b81038e0 0x7f75b8199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:18.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.657+0000 7f75b5400640 1 -- 172.21.15.37:0/3918138494 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b81a17a0 con 0x7f75b8104ad0 2024-03-28T18:08:18.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.657+0000 7f75b6800640 1 --2- 172.21.15.37:0/3918138494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75b8105490 0x7f75b81a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:08:18.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.657+0000 7f75b5400640 1 --2- 172.21.15.37:0/3918138494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75b8104ad0 0x7f75b819a0c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f75a000e9f0 tx=0x7f75a000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:18.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.657+0000 7f75aea00640 1 -- 172.21.15.37:0/3918138494 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a000cd80 con 0x7f75b8104ad0 2024-03-28T18:08:18.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.657+0000 7f75aea00640 1 -- 172.21.15.37:0/3918138494 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75a0010910 con 0x7f75b8104ad0 2024-03-28T18:08:18.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.657+0000 7f75bd000640 1 -- 172.21.15.37:0/3918138494 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b81a1a90 con 0x7f75b8104ad0 2024-03-28T18:08:18.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.657+0000 7f75aea00640 1 -- 172.21.15.37:0/3918138494 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a0018990 con 0x7f75b8104ad0 2024-03-28T18:08:18.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.658+0000 7f75bd000640 1 -- 172.21.15.37:0/3918138494 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b81a1fe0 con 0x7f75b8104ad0 2024-03-28T18:08:18.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.659+0000 7f75bd000640 1 -- 172.21.15.37:0/3918138494 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b810ac00 con 0x7f75b8104ad0 2024-03-28T18:08:18.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.661+0000 7f75aea00640 1 -- 172.21.15.37:0/3918138494 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f75a0010ac0 con 0x7f75b8104ad0 2024-03-28T18:08:18.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.663+0000 7f75aea00640 1 --2- 172.21.15.37:0/3918138494 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f759c0a2800 0x7f759c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:18.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.663+0000 7f75aea00640 1 -- 172.21.15.37:0/3918138494 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f75a00ceb20 con 0x7f75b8104ad0 2024-03-28T18:08:18.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.663+0000 7f75b5e00640 1 --2- 172.21.15.37:0/3918138494 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f759c0a2800 0x7f759c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:18.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.664+0000 7f75b5e00640 1 --2- 172.21.15.37:0/3918138494 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f759c0a2800 0x7f759c0a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f75b8104930 tx=0x7f75a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:18.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.668+0000 7f75aea00640 1 -- 172.21.15.37:0/3918138494 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f75a009b800 con 0x7f75b8104ad0 2024-03-28T18:08:18.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:18 smithi059 ceph-mon[36190]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:18.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:18 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/4282151655' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:08:18.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:18 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:18.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:18 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:18.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:18 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:08:18.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:18 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:08:18.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:18 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:18.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:18 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:08:18.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:18 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2090288706' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T18:08:18.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:18 smithi170 ceph-mon[36280]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:18.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:18 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/4282151655' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:08:18.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:18 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:18.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:18 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:18.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:18 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:08:18.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:18 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:08:18.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:18 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:18.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:18 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:08:18.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:18 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2090288706' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T18:08:18.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:18.870+0000 7f75bd000640 1 -- 172.21.15.37:0/3918138494 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f75b802d070 con 0x7f75b8104ad0 2024-03-28T18:08:19.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.463+0000 7f75aea00640 1 -- 172.21.15.37:0/3918138494 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) v1 ==== 170+0+0 (secure 0 0 0) 0x7f75a009f8a0 con 0x7f75b8104ad0 2024-03-28T18:08:19.464 INFO:teuthology.orchestra.run.smithi037.stderr:pool 'cephfs_data' created 2024-03-28T18:08:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.476+0000 7f75bd000640 1 -- 172.21.15.37:0/3918138494 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f759c0a2800 msgr2=0x7f759c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.476+0000 7f75bd000640 1 --2- 172.21.15.37:0/3918138494 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f759c0a2800 0x7f759c0a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f75b8104930 tx=0x7f75a8005f90 comp rx=0 tx=0).stop 2024-03-28T18:08:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.476+0000 7f75bd000640 1 -- 172.21.15.37:0/3918138494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75b8104ad0 msgr2=0x7f75b819a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.476+0000 7f75bd000640 1 --2- 172.21.15.37:0/3918138494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75b8104ad0 0x7f75b819a0c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f75a000e9f0 tx=0x7f75a000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:08:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.476+0000 7f75bd000640 1 -- 172.21.15.37:0/3918138494 shutdown_connections 2024-03-28T18:08:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.476+0000 7f75bd000640 1 --2- 172.21.15.37:0/3918138494 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f759c0a2800 0x7f759c0a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.476+0000 7f75bd000640 1 --2- 172.21.15.37:0/3918138494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75b8105490 0x7f75b81a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.476+0000 7f75bd000640 1 --2- 172.21.15.37:0/3918138494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75b8104ad0 0x7f75b819a0c0 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f75a000e9f0 tx=0x7f75a000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:08:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.476+0000 7f75bd000640 1 --2- 172.21.15.37:0/3918138494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75b81038e0 0x7f75b8199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.476+0000 7f75bd000640 1 -- 172.21.15.37:0/3918138494 >> 172.21.15.37:0/3918138494 conn(0x7f75b80fee50 msgr2=0x7f75b8107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.478+0000 7f75bd000640 1 -- 172.21.15.37:0/3918138494 shutdown_connections 2024-03-28T18:08:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.478+0000 7f75bd000640 1 -- 172.21.15.37:0/3918138494 wait complete. 2024-03-28T18:08:19.514 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T18:08:19.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.705+0000 7f2ea2200640 1 -- 172.21.15.37:0/2122718889 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e9c1016b0 msgr2=0x7f2e9c101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:19.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.705+0000 7f2ea2200640 1 --2- 172.21.15.37:0/2122718889 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e9c1016b0 0x7f2e9c101ad0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c009900 tx=0x7f2e8c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:08:19.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.705+0000 7f2ea2200640 1 -- 172.21.15.37:0/2122718889 shutdown_connections 2024-03-28T18:08:19.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.705+0000 7f2ea2200640 1 --2- 172.21.15.37:0/2122718889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e9c1028a0 0x7f2e9c102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:19.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.705+0000 7f2ea2200640 1 --2- 172.21.15.37:0/2122718889 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e9c103260 0x7f2e9c109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:19.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.705+0000 7f2ea2200640 1 --2- 172.21.15.37:0/2122718889 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e9c1016b0 0x7f2e9c101ad0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:19.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.705+0000 7f2ea2200640 1 -- 172.21.15.37:0/2122718889 >> 172.21.15.37:0/2122718889 conn(0x7f2e9c0fcc20 msgr2=0x7f2e9c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:19.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.705+0000 7f2ea2200640 1 -- 172.21.15.37:0/2122718889 shutdown_connections 2024-03-28T18:08:19.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.705+0000 7f2ea2200640 1 -- 172.21.15.37:0/2122718889 wait complete. 2024-03-28T18:08:19.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.706+0000 7f2ea2200640 1 Processor -- start 2024-03-28T18:08:19.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.706+0000 7f2ea2200640 1 -- start start 2024-03-28T18:08:19.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.707+0000 7f2ea2200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e9c1016b0 0x7f2e9c199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:19.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.707+0000 7f2ea2200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e9c1028a0 0x7f2e9c19a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:19.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.707+0000 7f2ea2200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e9c103260 0x7f2e9c1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:19.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.707+0000 7f2ea2200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9c10d380 con 0x7f2e9c1028a0 2024-03-28T18:08:19.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.707+0000 7f2ea2200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9c10d200 con 0x7f2e9c103260 2024-03-28T18:08:19.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.707+0000 7f2ea2200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9c10d500 con 0x7f2e9c1016b0 2024-03-28T18:08:19.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.707+0000 7f2e9aa00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e9c1028a0 0x7f2e9c19a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:19.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.707+0000 7f2e9aa00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e9c1028a0 0x7f2e9c19a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57046/0 (socket says 172.21.15.37:57046) 2024-03-28T18:08:19.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.707+0000 7f2e9aa00640 1 -- 172.21.15.37:0/1542426149 learned_addr learned my addr 172.21.15.37:0/1542426149 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:19.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.707+0000 7f2e9be00640 1 --2- 172.21.15.37:0/1542426149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e9c103260 0x7f2e9c1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:19.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.707+0000 7f2e9b400640 1 --2- 172.21.15.37:0/1542426149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e9c1016b0 0x7f2e9c199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:19.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.707+0000 7f2e9b400640 1 -- 172.21.15.37:0/1542426149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e9c103260 msgr2=0x7f2e9c1a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:19.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.707+0000 7f2e9b400640 1 --2- 172.21.15.37:0/1542426149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e9c103260 0x7f2e9c1a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:19.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.707+0000 7f2e9b400640 1 -- 172.21.15.37:0/1542426149 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e9c1028a0 msgr2=0x7f2e9c19a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:19.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.707+0000 7f2e9b400640 1 --2- 172.21.15.37:0/1542426149 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e9c1028a0 0x7f2e9c19a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:19.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.707+0000 7f2e9b400640 1 -- 172.21.15.37:0/1542426149 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9c1a1840 con 0x7f2e9c1016b0 2024-03-28T18:08:19.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.707+0000 7f2e9aa00640 1 --2- 172.21.15.37:0/1542426149 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e9c1028a0 0x7f2e9c19a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:08:19.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.708+0000 7f2e9b400640 1 --2- 172.21.15.37:0/1542426149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e9c1016b0 0x7f2e9c199c30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c009230 tx=0x7f2e8c02c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:19.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.708+0000 7f2e93e00640 1 -- 172.21.15.37:0/1542426149 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c043020 con 0x7f2e9c1016b0 2024-03-28T18:08:19.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.708+0000 7f2ea2200640 1 -- 172.21.15.37:0/1542426149 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e9c1a1ad0 con 0x7f2e9c1016b0 2024-03-28T18:08:19.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.708+0000 7f2e93e00640 1 -- 172.21.15.37:0/1542426149 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e8c0054e0 con 0x7f2e9c1016b0 2024-03-28T18:08:19.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.708+0000 7f2e93e00640 1 -- 172.21.15.37:0/1542426149 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c003b60 con 0x7f2e9c1016b0 2024-03-28T18:08:19.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.708+0000 7f2ea2200640 1 -- 172.21.15.37:0/1542426149 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e9c1a1ff0 con 0x7f2e9c1016b0 2024-03-28T18:08:19.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.709+0000 7f2ea2200640 1 -- 172.21.15.37:0/1542426149 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e9c1a21c0 con 0x7f2e9c1016b0 2024-03-28T18:08:19.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.711+0000 7f2e93e00640 1 -- 172.21.15.37:0/1542426149 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2e8c046020 con 0x7f2e9c1016b0 2024-03-28T18:08:19.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.712+0000 7f2e93e00640 1 --2- 172.21.15.37:0/1542426149 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f2e800a2800 0x7f2e800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:19.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.712+0000 7f2e93e00640 1 -- 172.21.15.37:0/1542426149 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f2e8c0f09a0 con 0x7f2e9c1016b0 2024-03-28T18:08:19.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.712+0000 7f2e9aa00640 1 --2- 172.21.15.37:0/1542426149 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f2e800a2800 0x7f2e800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:19.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.713+0000 7f2e9aa00640 1 --2- 172.21.15.37:0/1542426149 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f2e800a2800 0x7f2e800a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2e88011550 tx=0x7f2e88005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:19.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.716+0000 7f2e93e00640 1 -- 172.21.15.37:0/1542426149 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2e8c04d020 con 0x7f2e9c1016b0 2024-03-28T18:08:19.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:19 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1101579216' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T18:08:19.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:19 smithi059 ceph-mon[36190]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T18:08:19.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:19 smithi059 ceph-mon[36190]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T18:08:19.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:19 smithi059 ceph-mon[36190]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T18:08:19.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:19 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1101579216' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T18:08:19.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:19 smithi170 ceph-mon[36280]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T18:08:19.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:19 smithi170 ceph-mon[36280]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T18:08:19.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:19 smithi170 ceph-mon[36280]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T18:08:19.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:19 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1101579216' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T18:08:19.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:19 smithi037 ceph-mon[31460]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T18:08:19.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:19 smithi037 ceph-mon[31460]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T18:08:19.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:19 smithi037 ceph-mon[31460]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T18:08:19.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:19.966+0000 7f2ea2200640 1 -- 172.21.15.37:0/1542426149 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2e9c1a2840 con 0x7f2e9c1016b0 2024-03-28T18:08:20.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.513+0000 7f2e93e00640 1 -- 172.21.15.37:0/1542426149 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f2e8c0bdb50 con 0x7f2e9c1016b0 2024-03-28T18:08:20.513 INFO:teuthology.orchestra.run.smithi037.stderr:new fs with metadata pool 2 and data pool 3 2024-03-28T18:08:20.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.517+0000 7f2ea2200640 1 -- 172.21.15.37:0/1542426149 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f2e800a2800 msgr2=0x7f2e800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:20.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.517+0000 7f2ea2200640 1 --2- 172.21.15.37:0/1542426149 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f2e800a2800 0x7f2e800a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2e88011550 tx=0x7f2e88005f90 comp rx=0 tx=0).stop 2024-03-28T18:08:20.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.517+0000 7f2ea2200640 1 -- 172.21.15.37:0/1542426149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e9c1016b0 msgr2=0x7f2e9c199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:20.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.517+0000 7f2ea2200640 1 --2- 172.21.15.37:0/1542426149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e9c1016b0 0x7f2e9c199c30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c009230 tx=0x7f2e8c02c5d0 comp rx=0 tx=0).stop 2024-03-28T18:08:20.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.517+0000 7f2ea2200640 1 -- 172.21.15.37:0/1542426149 shutdown_connections 2024-03-28T18:08:20.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.517+0000 7f2ea2200640 1 --2- 172.21.15.37:0/1542426149 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f2e800a2800 0x7f2e800a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:20.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.517+0000 7f2ea2200640 1 --2- 172.21.15.37:0/1542426149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e9c103260 0x7f2e9c1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:20.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.517+0000 7f2ea2200640 1 --2- 172.21.15.37:0/1542426149 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e9c1028a0 0x7f2e9c19a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:20.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.517+0000 7f2ea2200640 1 --2- 172.21.15.37:0/1542426149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e9c1016b0 0x7f2e9c199c30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:20.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.517+0000 7f2ea2200640 1 -- 172.21.15.37:0/1542426149 >> 172.21.15.37:0/1542426149 conn(0x7f2e9c0fcc20 msgr2=0x7f2e9c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:20.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.517+0000 7f2ea2200640 1 -- 172.21.15.37:0/1542426149 shutdown_connections 2024-03-28T18:08:20.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.517+0000 7f2ea2200640 1 -- 172.21.15.37:0/1542426149 wait complete. 2024-03-28T18:08:20.542 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:08:20.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.735+0000 7f62d9e00640 1 -- 172.21.15.37:0/4206420145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62d41016b0 msgr2=0x7f62d4101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:20.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.735+0000 7f62d9e00640 1 --2- 172.21.15.37:0/4206420145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62d41016b0 0x7f62d4101ad0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f62c4009900 tx=0x7f62c402bc50 comp rx=0 tx=0).stop 2024-03-28T18:08:20.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.735+0000 7f62d9e00640 1 -- 172.21.15.37:0/4206420145 shutdown_connections 2024-03-28T18:08:20.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.735+0000 7f62d9e00640 1 --2- 172.21.15.37:0/4206420145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d4103260 0x7f62d4109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:20.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.735+0000 7f62d9e00640 1 --2- 172.21.15.37:0/4206420145 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62d41028a0 0x7f62d4102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:20.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.735+0000 7f62d9e00640 1 --2- 172.21.15.37:0/4206420145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62d41016b0 0x7f62d4101ad0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:20.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.735+0000 7f62d9e00640 1 -- 172.21.15.37:0/4206420145 >> 172.21.15.37:0/4206420145 conn(0x7f62d40fcc80 msgr2=0x7f62d40ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:20.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.735+0000 7f62d9e00640 1 -- 172.21.15.37:0/4206420145 shutdown_connections 2024-03-28T18:08:20.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.735+0000 7f62d9e00640 1 -- 172.21.15.37:0/4206420145 wait complete. 2024-03-28T18:08:20.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.736+0000 7f62d9e00640 1 Processor -- start 2024-03-28T18:08:20.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.736+0000 7f62d9e00640 1 -- start start 2024-03-28T18:08:20.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.736+0000 7f62d9e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62d41016b0 0x7f62d4199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:20.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.736+0000 7f62d9e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62d41028a0 0x7f62d419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:20.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.736+0000 7f62d9e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d4103260 0x7f62d41a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:20.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.736+0000 7f62d9e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d410d350 con 0x7f62d41016b0 2024-03-28T18:08:20.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.736+0000 7f62d9e00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d410d1d0 con 0x7f62d41028a0 2024-03-28T18:08:20.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.736+0000 7f62d9e00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d410d4d0 con 0x7f62d4103260 2024-03-28T18:08:20.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.737+0000 7f62d2a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62d41016b0 0x7f62d4199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:20.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.737+0000 7f62d2a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62d41016b0 0x7f62d4199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57050/0 (socket says 172.21.15.37:57050) 2024-03-28T18:08:20.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.737+0000 7f62d2a00640 1 -- 172.21.15.37:0/2108938803 learned_addr learned my addr 172.21.15.37:0/2108938803 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:20.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.737+0000 7f62d2000640 1 --2- 172.21.15.37:0/2108938803 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62d41028a0 0x7f62d419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:20.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.737+0000 7f62d3400640 1 --2- 172.21.15.37:0/2108938803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d4103260 0x7f62d41a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:20.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.737+0000 7f62d2a00640 1 -- 172.21.15.37:0/2108938803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d4103260 msgr2=0x7f62d41a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:20.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.737+0000 7f62d2a00640 1 --2- 172.21.15.37:0/2108938803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d4103260 0x7f62d41a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:20.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.737+0000 7f62d2a00640 1 -- 172.21.15.37:0/2108938803 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62d41028a0 msgr2=0x7f62d419a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:20.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.737+0000 7f62d2a00640 1 --2- 172.21.15.37:0/2108938803 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62d41028a0 0x7f62d419a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:20.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.737+0000 7f62d2a00640 1 -- 172.21.15.37:0/2108938803 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d41a1810 con 0x7f62d41016b0 2024-03-28T18:08:20.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.738+0000 7f62d2a00640 1 --2- 172.21.15.37:0/2108938803 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62d41016b0 0x7f62d4199bf0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f62c402c260 tx=0x7f62c402c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:20.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.738+0000 7f62d8a00640 1 -- 172.21.15.37:0/2108938803 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c4041070 con 0x7f62d41016b0 2024-03-28T18:08:20.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.738+0000 7f62d8a00640 1 -- 172.21.15.37:0/2108938803 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62c4004450 con 0x7f62d41016b0 2024-03-28T18:08:20.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.738+0000 7f62d9e00640 1 -- 172.21.15.37:0/2108938803 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d41a1aa0 con 0x7f62d41016b0 2024-03-28T18:08:20.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.738+0000 7f62d8a00640 1 -- 172.21.15.37:0/2108938803 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c4034440 con 0x7f62d41016b0 2024-03-28T18:08:20.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.738+0000 7f62d9e00640 1 -- 172.21.15.37:0/2108938803 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d41a1ff0 con 0x7f62d41016b0 2024-03-28T18:08:20.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.739+0000 7f62d9e00640 1 -- 172.21.15.37:0/2108938803 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d41a2220 con 0x7f62d41016b0 2024-03-28T18:08:20.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.743+0000 7f62d8a00640 1 -- 172.21.15.37:0/2108938803 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f62c4044070 con 0x7f62d41016b0 2024-03-28T18:08:20.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.744+0000 7f62d8a00640 1 --2- 172.21.15.37:0/2108938803 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f62b80a2800 0x7f62b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:20.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.745+0000 7f62d8a00640 1 -- 172.21.15.37:0/2108938803 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f62c40f0830 con 0x7f62d41016b0 2024-03-28T18:08:20.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.745+0000 7f62d2000640 1 --2- 172.21.15.37:0/2108938803 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f62b80a2800 0x7f62b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:20.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.745+0000 7f62d2000640 1 --2- 172.21.15.37:0/2108938803 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f62b80a2800 0x7f62b80a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f62d419b100 tx=0x7f62c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:20.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.747+0000 7f62d8a00640 1 -- 172.21.15.37:0/2108938803 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f62c40bd330 con 0x7f62d41016b0 2024-03-28T18:08:20.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:20 smithi059 ceph-mon[36190]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:20.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:20 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3918138494' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T18:08:20.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:20 smithi059 ceph-mon[36190]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T18:08:20.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:20 smithi059 ceph-mon[36190]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T18:08:20.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:20 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:20 smithi059 ceph-mon[36190]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T18:08:20.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:20 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:20 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:20 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:20 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1542426149' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T18:08:20.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:20 smithi059 ceph-mon[36190]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T18:08:20.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:20 smithi059 ceph-mon[36190]: daemon mds.a assigned to filesystem cephfs as rank 0 2024-03-28T18:08:20.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:20 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:20 smithi059 ceph-mon[36190]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T18:08:20.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:20 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:20 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:08:20.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:20 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:08:20.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:20 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:20 smithi170 ceph-mon[36280]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:20.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:20 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3918138494' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T18:08:20.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:20 smithi170 ceph-mon[36280]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T18:08:20.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:20 smithi170 ceph-mon[36280]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T18:08:20.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:20 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:20 smithi170 ceph-mon[36280]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T18:08:20.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:20 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:20 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:20 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:20 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1542426149' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T18:08:20.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:20 smithi170 ceph-mon[36280]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T18:08:20.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:20 smithi170 ceph-mon[36280]: daemon mds.a assigned to filesystem cephfs as rank 0 2024-03-28T18:08:20.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:20 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:20 smithi170 ceph-mon[36280]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T18:08:20.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:20 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:20 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:08:20.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:20 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:08:20.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:20 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:20 smithi037 ceph-mon[31460]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:20.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:20 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3918138494' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T18:08:20.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:20 smithi037 ceph-mon[31460]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T18:08:20.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:20 smithi037 ceph-mon[31460]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T18:08:20.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:20 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:20 smithi037 ceph-mon[31460]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T18:08:20.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:20 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:20 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:20 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:20 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1542426149' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T18:08:20.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:20 smithi037 ceph-mon[31460]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T18:08:20.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:20 smithi037 ceph-mon[31460]: daemon mds.a assigned to filesystem cephfs as rank 0 2024-03-28T18:08:20.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:20 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:20 smithi037 ceph-mon[31460]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T18:08:20.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:20 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:20 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:08:20.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:20 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:08:20.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:20 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:08:20.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.945+0000 7f62d9e00640 1 -- 172.21.15.37:0/2108938803 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62d41a23f0 con 0x7f62d41016b0 2024-03-28T18:08:20.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.947+0000 7f62d8a00640 1 -- 172.21.15.37:0/2108938803 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+22209 (secure 0 0 0) 0x7f62c40c13d0 con 0x7f62d41016b0 2024-03-28T18:08:20.948 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:08:20.948 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":84,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","created":"2024-03-28T17:59:05.949847+0000","modified":"2024-03-28T18:08:19.968500+0000","last_up_change":"2024-03-28T18:07:21.021030+0000","last_in_change":"2024-03-28T18:07:01.416192+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:03:00.517967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:08:18.365827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:08:19.446282+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db60618d-56f7-4130-8a90-84cbc480965b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6803","nonce":3562602035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6805","nonce":3562602035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6809","nonce":3562602035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6807","nonce":3562602035}]},"public_addr":"172.21.15.37:6803/3562602035","cluster_addr":"172.21.15.37:6805/3562602035","heartbeat_back_addr":"172.21.15.37:6809/3562602035","heartbeat_front_addr":"172.21.15.37:6807/3562602035","state":["exists","up"]},{"osd":1,"uuid":"7a4b7ad6-905e-4258-9574-693d4277dfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6801","nonce":3326537211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6803","nonce":3326537211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6807","nonce":3326537211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6805","nonce":3326537211}]},"public_addr":"172.21.15.59:6801/3326537211","cluster_addr":"172.21.15.59:6803/3326537211","heartbeat_back_addr":"172.21.15.59:6807/3326537211","heartbeat_front_addr":"172.21.15.59:6805/3326537211","state":["exists","up"]},{"osd":2,"uuid":"8b76ec1c-46bf-4336-89b5-12d82906e8b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6801","nonce":1269398427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6803","nonce":1269398427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6807","nonce":1269398427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6805","nonce":1269398427}]},"public_addr":"172.21.15.170:6801/1269398427","cluster_addr":"172.21.15.170:6803/1269398427","heartbeat_back_addr":"172.21.15.170:6807/1269398427","heartbeat_front_addr":"172.21.15.170:6805/1269398427","state":["exists","up"]},{"osd":3,"uuid":"66eca1da-2fbd-44e1-8bbf-5635a2c70ad5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6811","nonce":317805240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6813","nonce":317805240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6817","nonce":317805240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6815","nonce":317805240}]},"public_addr":"172.21.15.37:6811/317805240","cluster_addr":"172.21.15.37:6813/317805240","heartbeat_back_addr":"172.21.15.37:6817/317805240","heartbeat_front_addr":"172.21.15.37:6815/317805240","state":["exists","up"]},{"osd":4,"uuid":"4c877335-e42d-4d8e-9f48-a1d0a89ae438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6809","nonce":2410632623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6811","nonce":2410632623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6815","nonce":2410632623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6813","nonce":2410632623}]},"public_addr":"172.21.15.59:6809/2410632623","cluster_addr":"172.21.15.59:6811/2410632623","heartbeat_back_addr":"172.21.15.59:6815/2410632623","heartbeat_front_addr":"172.21.15.59:6813/2410632623","state":["exists","up"]},{"osd":5,"uuid":"b884842a-86e2-455b-9570-ff163b3bc0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6809","nonce":1218943899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6811","nonce":1218943899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6815","nonce":1218943899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6813","nonce":1218943899}]},"public_addr":"172.21.15.170:6809/1218943899","cluster_addr":"172.21.15.170:6811/1218943899","heartbeat_back_addr":"172.21.15.170:6815/1218943899","heartbeat_front_addr":"172.21.15.170:6813/1218943899","state":["exists","up"]},{"osd":6,"uuid":"249a1200-77f4-44ad-981f-1d352f732b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6819","nonce":1899652046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6821","nonce":1899652046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6825","nonce":1899652046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6823","nonce":1899652046}]},"public_addr":"172.21.15.37:6819/1899652046","cluster_addr":"172.21.15.37:6821/1899652046","heartbeat_back_addr":"172.21.15.37:6825/1899652046","heartbeat_front_addr":"172.21.15.37:6823/1899652046","state":["exists","up"]},{"osd":7,"uuid":"c73338e6-f505-4517-a84b-f35e04750a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6817","nonce":1343675690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6819","nonce":1343675690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6823","nonce":1343675690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6821","nonce":1343675690}]},"public_addr":"172.21.15.59:6817/1343675690","cluster_addr":"172.21.15.59:6819/1343675690","heartbeat_back_addr":"172.21.15.59:6823/1343675690","heartbeat_front_addr":"172.21.15.59:6821/1343675690","state":["exists","up"]},{"osd":8,"uuid":"149b8297-ff2d-4a88-9b1e-0def021cc516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6817","nonce":3235794556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6819","nonce":3235794556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6823","nonce":3235794556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6821","nonce":3235794556}]},"public_addr":"172.21.15.170:6817/3235794556","cluster_addr":"172.21.15.170:6819/3235794556","heartbeat_back_addr":"172.21.15.170:6823/3235794556","heartbeat_front_addr":"172.21.15.170:6821/3235794556","state":["exists","up"]},{"osd":9,"uuid":"55c27cc2-c771-44b7-90fe-8f0feca44162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6827","nonce":2307153008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6829","nonce":2307153008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6833","nonce":2307153008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6831","nonce":2307153008}]},"public_addr":"172.21.15.37:6827/2307153008","cluster_addr":"172.21.15.37:6829/2307153008","heartbeat_back_addr":"172.21.15.37:6833/2307153008","heartbeat_front_addr":"172.21.15.37:6831/2307153008","state":["exists","up"]},{"osd":10,"uuid":"c0595135-ab81-49d3-8c23-19f960c03e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6825","nonce":2113830275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6827","nonce":2113830275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6831","nonce":2113830275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6829","nonce":2113830275}]},"public_addr":"172.21.15.59:6825/2113830275","cluster_addr":"172.21.15.59:6827/2113830275","heartbeat_back_addr":"172.21.15.59:6831/2113830275","heartbeat_front_addr":"172.21.15.59:6829/2113830275","state":["exists","up"]},{"osd":11,"uuid":"5871e177-c17b-4637-a797-4fa5125920dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6825","nonce":3082439360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6827","nonce":3082439360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6831","nonce":3082439360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6829","nonce":3082439360}]},"public_addr":"172.21.15.170:6825/3082439360","cluster_addr":"172.21.15.170:6827/3082439360","heartbeat_back_addr":"172.21.15.170:6831/3082439360","heartbeat_front_addr":"172.21.15.170:6829/3082439360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:07.690545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:33.183396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:57.400263+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:30.597211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:57.259044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:22.627823+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:57.585863+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:25.000476+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:50.876688+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:25.859761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:53.563146+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:07:19.304115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3023379428":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1443710231":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6800/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6801/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6801/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6800/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1007625490":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/2895698584":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/497313034":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/3293943475":"2024-03-29T17:59:39.162038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:08:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.950+0000 7f62d9e00640 1 -- 172.21.15.37:0/2108938803 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f62b80a2800 msgr2=0x7f62b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.950+0000 7f62d9e00640 1 --2- 172.21.15.37:0/2108938803 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f62b80a2800 0x7f62b80a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f62d419b100 tx=0x7f62c0005f90 comp rx=0 tx=0).stop 2024-03-28T18:08:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.950+0000 7f62d9e00640 1 -- 172.21.15.37:0/2108938803 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62d41016b0 msgr2=0x7f62d4199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.950+0000 7f62d9e00640 1 --2- 172.21.15.37:0/2108938803 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62d41016b0 0x7f62d4199bf0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f62c402c260 tx=0x7f62c402c670 comp rx=0 tx=0).stop 2024-03-28T18:08:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.950+0000 7f62d9e00640 1 -- 172.21.15.37:0/2108938803 shutdown_connections 2024-03-28T18:08:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.950+0000 7f62d9e00640 1 --2- 172.21.15.37:0/2108938803 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f62b80a2800 0x7f62b80a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.950+0000 7f62d9e00640 1 --2- 172.21.15.37:0/2108938803 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62d41028a0 0x7f62d419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.950+0000 7f62d9e00640 1 --2- 172.21.15.37:0/2108938803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d4103260 0x7f62d41a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:20.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.950+0000 7f62d9e00640 1 --2- 172.21.15.37:0/2108938803 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62d41016b0 0x7f62d4199bf0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:20.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.950+0000 7f62d9e00640 1 -- 172.21.15.37:0/2108938803 >> 172.21.15.37:0/2108938803 conn(0x7f62d40fcc80 msgr2=0x7f62d40738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:20.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.951+0000 7f62d9e00640 1 -- 172.21.15.37:0/2108938803 shutdown_connections 2024-03-28T18:08:20.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:20.951+0000 7f62d9e00640 1 -- 172.21.15.37:0/2108938803 wait complete. 2024-03-28T18:08:20.969 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:08:21.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.178+0000 7fa12f200640 1 -- 172.21.15.37:0/1981175533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa130074100 msgr2=0x7fa130074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:21.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.178+0000 7fa12f200640 1 --2- 172.21.15.37:0/1981175533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa130074100 0x7fa130074580 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa118009cd0 tx=0x7fa11802c0e0 comp rx=0 tx=0).stop 2024-03-28T18:08:21.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.179+0000 7fa12f200640 1 -- 172.21.15.37:0/1981175533 shutdown_connections 2024-03-28T18:08:21.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.179+0000 7fa12f200640 1 --2- 172.21.15.37:0/1981175533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa130074100 0x7fa130074580 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:21.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.179+0000 7fa12f200640 1 --2- 172.21.15.37:0/1981175533 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa13010dae0 0x7fa13010fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:21.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.179+0000 7fa12f200640 1 --2- 172.21.15.37:0/1981175533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1300756e0 0x7fa130073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:21.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.179+0000 7fa12f200640 1 -- 172.21.15.37:0/1981175533 >> 172.21.15.37:0/1981175533 conn(0x7fa1300fca80 msgr2=0x7fa1300feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:21.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.179+0000 7fa12f200640 1 -- 172.21.15.37:0/1981175533 shutdown_connections 2024-03-28T18:08:21.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.179+0000 7fa12f200640 1 -- 172.21.15.37:0/1981175533 wait complete. 2024-03-28T18:08:21.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.180+0000 7fa12f200640 1 Processor -- start 2024-03-28T18:08:21.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.180+0000 7fa12f200640 1 -- start start 2024-03-28T18:08:21.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.180+0000 7fa12f200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa130074100 0x7fa13019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:21.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.180+0000 7fa12f200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1300756e0 0x7fa13019e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:21.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.180+0000 7fa12f200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa13010dae0 0x7fa1301a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:21.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.180+0000 7fa12f200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa130113760 con 0x7fa130074100 2024-03-28T18:08:21.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.180+0000 7fa12f200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1301135e0 con 0x7fa1300756e0 2024-03-28T18:08:21.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.180+0000 7fa12f200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1301138e0 con 0x7fa13010dae0 2024-03-28T18:08:21.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.181+0000 7fa12de00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa130074100 0x7fa13019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:21.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.181+0000 7fa12de00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa130074100 0x7fa13019de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57064/0 (socket says 172.21.15.37:57064) 2024-03-28T18:08:21.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.181+0000 7fa12de00640 1 -- 172.21.15.37:0/4180485347 learned_addr learned my addr 172.21.15.37:0/4180485347 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:21.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.181+0000 7fa12d400640 1 --2- 172.21.15.37:0/4180485347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1300756e0 0x7fa13019e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:21.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.181+0000 7fa12e800640 1 --2- 172.21.15.37:0/4180485347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa13010dae0 0x7fa1301a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:21.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.181+0000 7fa12de00640 1 -- 172.21.15.37:0/4180485347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa13010dae0 msgr2=0x7fa1301a53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:21.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.181+0000 7fa12de00640 1 --2- 172.21.15.37:0/4180485347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa13010dae0 0x7fa1301a53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:21.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.181+0000 7fa12de00640 1 -- 172.21.15.37:0/4180485347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1300756e0 msgr2=0x7fa13019e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:21.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.181+0000 7fa12de00640 1 --2- 172.21.15.37:0/4180485347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1300756e0 0x7fa13019e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:21.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.181+0000 7fa12de00640 1 -- 172.21.15.37:0/4180485347 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1301a5a40 con 0x7fa130074100 2024-03-28T18:08:21.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.181+0000 7fa12d400640 1 --2- 172.21.15.37:0/4180485347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1300756e0 0x7fa13019e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:08:21.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.181+0000 7fa12de00640 1 --2- 172.21.15.37:0/4180485347 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa130074100 0x7fa13019de20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa12000ea60 tx=0x7fa12000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:21.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.182+0000 7fa126a00640 1 -- 172.21.15.37:0/4180485347 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12000cd20 con 0x7fa130074100 2024-03-28T18:08:21.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.182+0000 7fa126a00640 1 -- 172.21.15.37:0/4180485347 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1200035f0 con 0x7fa130074100 2024-03-28T18:08:21.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.182+0000 7fa126a00640 1 -- 172.21.15.37:0/4180485347 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa120010690 con 0x7fa130074100 2024-03-28T18:08:21.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.182+0000 7fa12f200640 1 -- 172.21.15.37:0/4180485347 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1301a5d30 con 0x7fa130074100 2024-03-28T18:08:21.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.182+0000 7fa12f200640 1 -- 172.21.15.37:0/4180485347 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1301a61a0 con 0x7fa130074100 2024-03-28T18:08:21.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.183+0000 7fa12f200640 1 -- 172.21.15.37:0/4180485347 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa130103cb0 con 0x7fa130074100 2024-03-28T18:08:21.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.185+0000 7fa126a00640 1 -- 172.21.15.37:0/4180485347 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa1200037a0 con 0x7fa130074100 2024-03-28T18:08:21.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.187+0000 7fa126a00640 1 --2- 172.21.15.37:0/4180485347 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa1100a2880 0x7fa1100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:21.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.188+0000 7fa126a00640 1 -- 172.21.15.37:0/4180485347 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa1200ceca0 con 0x7fa130074100 2024-03-28T18:08:21.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.188+0000 7fa12d400640 1 --2- 172.21.15.37:0/4180485347 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa1100a2880 0x7fa1100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:21.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.188+0000 7fa12d400640 1 --2- 172.21.15.37:0/4180485347 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa1100a2880 0x7fa1100a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa11802c620 tx=0x7fa118032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:21.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.190+0000 7fa126a00640 1 -- 172.21.15.37:0/4180485347 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa12009be80 con 0x7fa130074100 2024-03-28T18:08:21.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.394+0000 7fa12f200640 1 -- 172.21.15.37:0/4180485347 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa130062420 con 0x7fa130074100 2024-03-28T18:08:21.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.398+0000 7fa126a00640 1 -- 172.21.15.37:0/4180485347 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22196 (secure 0 0 0) 0x7fa12009b780 con 0x7fa130074100 2024-03-28T18:08:21.399 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:08:21.399 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":85,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","created":"2024-03-28T17:59:05.949847+0000","modified":"2024-03-28T18:08:20.991162+0000","last_up_change":"2024-03-28T18:07:21.021030+0000","last_in_change":"2024-03-28T18:07:01.416192+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:03:00.517967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:08:18.365827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:08:19.446282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db60618d-56f7-4130-8a90-84cbc480965b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6803","nonce":3562602035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6805","nonce":3562602035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6809","nonce":3562602035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6807","nonce":3562602035}]},"public_addr":"172.21.15.37:6803/3562602035","cluster_addr":"172.21.15.37:6805/3562602035","heartbeat_back_addr":"172.21.15.37:6809/3562602035","heartbeat_front_addr":"172.21.15.37:6807/3562602035","state":["exists","up"]},{"osd":1,"uuid":"7a4b7ad6-905e-4258-9574-693d4277dfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6801","nonce":3326537211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6803","nonce":3326537211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6807","nonce":3326537211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6805","nonce":3326537211}]},"public_addr":"172.21.15.59:6801/3326537211","cluster_addr":"172.21.15.59:6803/3326537211","heartbeat_back_addr":"172.21.15.59:6807/3326537211","heartbeat_front_addr":"172.21.15.59:6805/3326537211","state":["exists","up"]},{"osd":2,"uuid":"8b76ec1c-46bf-4336-89b5-12d82906e8b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6801","nonce":1269398427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6803","nonce":1269398427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6807","nonce":1269398427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6805","nonce":1269398427}]},"public_addr":"172.21.15.170:6801/1269398427","cluster_addr":"172.21.15.170:6803/1269398427","heartbeat_back_addr":"172.21.15.170:6807/1269398427","heartbeat_front_addr":"172.21.15.170:6805/1269398427","state":["exists","up"]},{"osd":3,"uuid":"66eca1da-2fbd-44e1-8bbf-5635a2c70ad5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6811","nonce":317805240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6813","nonce":317805240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6817","nonce":317805240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6815","nonce":317805240}]},"public_addr":"172.21.15.37:6811/317805240","cluster_addr":"172.21.15.37:6813/317805240","heartbeat_back_addr":"172.21.15.37:6817/317805240","heartbeat_front_addr":"172.21.15.37:6815/317805240","state":["exists","up"]},{"osd":4,"uuid":"4c877335-e42d-4d8e-9f48-a1d0a89ae438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6809","nonce":2410632623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6811","nonce":2410632623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6815","nonce":2410632623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6813","nonce":2410632623}]},"public_addr":"172.21.15.59:6809/2410632623","cluster_addr":"172.21.15.59:6811/2410632623","heartbeat_back_addr":"172.21.15.59:6815/2410632623","heartbeat_front_addr":"172.21.15.59:6813/2410632623","state":["exists","up"]},{"osd":5,"uuid":"b884842a-86e2-455b-9570-ff163b3bc0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6809","nonce":1218943899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6811","nonce":1218943899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6815","nonce":1218943899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6813","nonce":1218943899}]},"public_addr":"172.21.15.170:6809/1218943899","cluster_addr":"172.21.15.170:6811/1218943899","heartbeat_back_addr":"172.21.15.170:6815/1218943899","heartbeat_front_addr":"172.21.15.170:6813/1218943899","state":["exists","up"]},{"osd":6,"uuid":"249a1200-77f4-44ad-981f-1d352f732b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6819","nonce":1899652046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6821","nonce":1899652046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6825","nonce":1899652046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6823","nonce":1899652046}]},"public_addr":"172.21.15.37:6819/1899652046","cluster_addr":"172.21.15.37:6821/1899652046","heartbeat_back_addr":"172.21.15.37:6825/1899652046","heartbeat_front_addr":"172.21.15.37:6823/1899652046","state":["exists","up"]},{"osd":7,"uuid":"c73338e6-f505-4517-a84b-f35e04750a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6817","nonce":1343675690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6819","nonce":1343675690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6823","nonce":1343675690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6821","nonce":1343675690}]},"public_addr":"172.21.15.59:6817/1343675690","cluster_addr":"172.21.15.59:6819/1343675690","heartbeat_back_addr":"172.21.15.59:6823/1343675690","heartbeat_front_addr":"172.21.15.59:6821/1343675690","state":["exists","up"]},{"osd":8,"uuid":"149b8297-ff2d-4a88-9b1e-0def021cc516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6817","nonce":3235794556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6819","nonce":3235794556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6823","nonce":3235794556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6821","nonce":3235794556}]},"public_addr":"172.21.15.170:6817/3235794556","cluster_addr":"172.21.15.170:6819/3235794556","heartbeat_back_addr":"172.21.15.170:6823/3235794556","heartbeat_front_addr":"172.21.15.170:6821/3235794556","state":["exists","up"]},{"osd":9,"uuid":"55c27cc2-c771-44b7-90fe-8f0feca44162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6827","nonce":2307153008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6829","nonce":2307153008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6833","nonce":2307153008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6831","nonce":2307153008}]},"public_addr":"172.21.15.37:6827/2307153008","cluster_addr":"172.21.15.37:6829/2307153008","heartbeat_back_addr":"172.21.15.37:6833/2307153008","heartbeat_front_addr":"172.21.15.37:6831/2307153008","state":["exists","up"]},{"osd":10,"uuid":"c0595135-ab81-49d3-8c23-19f960c03e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6825","nonce":2113830275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6827","nonce":2113830275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6831","nonce":2113830275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6829","nonce":2113830275}]},"public_addr":"172.21.15.59:6825/2113830275","cluster_addr":"172.21.15.59:6827/2113830275","heartbeat_back_addr":"172.21.15.59:6831/2113830275","heartbeat_front_addr":"172.21.15.59:6829/2113830275","state":["exists","up"]},{"osd":11,"uuid":"5871e177-c17b-4637-a797-4fa5125920dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6825","nonce":3082439360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6827","nonce":3082439360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6831","nonce":3082439360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6829","nonce":3082439360}]},"public_addr":"172.21.15.170:6825/3082439360","cluster_addr":"172.21.15.170:6827/3082439360","heartbeat_back_addr":"172.21.15.170:6831/3082439360","heartbeat_front_addr":"172.21.15.170:6829/3082439360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:07.690545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:33.183396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:57.400263+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:30.597211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:57.259044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:22.627823+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:57.585863+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:25.000476+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:50.876688+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:25.859761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:53.563146+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:07:19.304115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3023379428":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1443710231":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6800/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6801/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6801/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6800/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1007625490":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/2895698584":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/497313034":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/3293943475":"2024-03-29T17:59:39.162038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:08:21.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.401+0000 7fa12f200640 1 -- 172.21.15.37:0/4180485347 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa1100a2880 msgr2=0x7fa1100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:21.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.401+0000 7fa12f200640 1 --2- 172.21.15.37:0/4180485347 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa1100a2880 0x7fa1100a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa11802c620 tx=0x7fa118032590 comp rx=0 tx=0).stop 2024-03-28T18:08:21.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.401+0000 7fa12f200640 1 -- 172.21.15.37:0/4180485347 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa130074100 msgr2=0x7fa13019de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:21.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.401+0000 7fa12f200640 1 --2- 172.21.15.37:0/4180485347 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa130074100 0x7fa13019de20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa12000ea60 tx=0x7fa12000ee60 comp rx=0 tx=0).stop 2024-03-28T18:08:21.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.401+0000 7fa12f200640 1 -- 172.21.15.37:0/4180485347 shutdown_connections 2024-03-28T18:08:21.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.401+0000 7fa12f200640 1 --2- 172.21.15.37:0/4180485347 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa1100a2880 0x7fa1100a4d40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:21.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.401+0000 7fa12f200640 1 --2- 172.21.15.37:0/4180485347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1300756e0 0x7fa13019e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:21.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.401+0000 7fa12f200640 1 --2- 172.21.15.37:0/4180485347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa13010dae0 0x7fa1301a53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:21.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.401+0000 7fa12f200640 1 --2- 172.21.15.37:0/4180485347 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa130074100 0x7fa13019de20 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:21.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.401+0000 7fa12f200640 1 -- 172.21.15.37:0/4180485347 >> 172.21.15.37:0/4180485347 conn(0x7fa1300fca80 msgr2=0x7fa130100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:21.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.401+0000 7fa12f200640 1 -- 172.21.15.37:0/4180485347 shutdown_connections 2024-03-28T18:08:21.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.402+0000 7fa12f200640 1 -- 172.21.15.37:0/4180485347 wait complete. 2024-03-28T18:08:21.422 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T18:08:21.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.645+0000 7f4f77e00640 1 -- 172.21.15.37:0/1512490177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f781015a0 msgr2=0x7f4f781019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:21.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.645+0000 7f4f77e00640 1 --2- 172.21.15.37:0/1512490177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f781015a0 0x7f4f781019c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4f68009ca0 tx=0x7f4f6802c100 comp rx=0 tx=0).stop 2024-03-28T18:08:21.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.645+0000 7f4f77e00640 1 -- 172.21.15.37:0/1512490177 shutdown_connections 2024-03-28T18:08:21.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.645+0000 7f4f77e00640 1 --2- 172.21.15.37:0/1512490177 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f78103150 0x7f4f781099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:21.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.645+0000 7f4f77e00640 1 --2- 172.21.15.37:0/1512490177 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f78102790 0x7f4f78102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:21.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.645+0000 7f4f77e00640 1 --2- 172.21.15.37:0/1512490177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f781015a0 0x7f4f781019c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:21.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.645+0000 7f4f77e00640 1 -- 172.21.15.37:0/1512490177 >> 172.21.15.37:0/1512490177 conn(0x7f4f780fcb10 msgr2=0x7f4f780fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:21.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.645+0000 7f4f77e00640 1 -- 172.21.15.37:0/1512490177 shutdown_connections 2024-03-28T18:08:21.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.645+0000 7f4f77e00640 1 -- 172.21.15.37:0/1512490177 wait complete. 2024-03-28T18:08:21.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.646+0000 7f4f77e00640 1 Processor -- start 2024-03-28T18:08:21.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.646+0000 7f4f77e00640 1 -- start start 2024-03-28T18:08:21.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.646+0000 7f4f77e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f781015a0 0x7f4f781978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:21.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.646+0000 7f4f77e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f78102790 0x7f4f78197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:21.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.646+0000 7f4f77e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f78103150 0x7f4f7819ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:21.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.646+0000 7f4f77e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7810d1f0 con 0x7f4f78103150 2024-03-28T18:08:21.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.646+0000 7f4f77e00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7810d070 con 0x7f4f78102790 2024-03-28T18:08:21.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.646+0000 7f4f77e00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7810d370 con 0x7f4f781015a0 2024-03-28T18:08:21.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.647+0000 7f4f77400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f78103150 0x7f4f7819ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:21.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.647+0000 7f4f77400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f78103150 0x7f4f7819ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57084/0 (socket says 172.21.15.37:57084) 2024-03-28T18:08:21.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.647+0000 7f4f77400640 1 -- 172.21.15.37:0/4193723360 learned_addr learned my addr 172.21.15.37:0/4193723360 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:21.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.647+0000 7f4f76a00640 1 --2- 172.21.15.37:0/4193723360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f781015a0 0x7f4f781978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:21.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.647+0000 7f4f6e000640 1 --2- 172.21.15.37:0/4193723360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f78102790 0x7f4f78197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:21.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.647+0000 7f4f76a00640 1 -- 172.21.15.37:0/4193723360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f78102790 msgr2=0x7f4f78197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:21.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.647+0000 7f4f76a00640 1 --2- 172.21.15.37:0/4193723360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f78102790 0x7f4f78197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:21.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.647+0000 7f4f76a00640 1 -- 172.21.15.37:0/4193723360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f78103150 msgr2=0x7f4f7819ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:21.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.647+0000 7f4f76a00640 1 --2- 172.21.15.37:0/4193723360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f78103150 0x7f4f7819ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:21.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.647+0000 7f4f76a00640 1 -- 172.21.15.37:0/4193723360 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7819f4e0 con 0x7f4f781015a0 2024-03-28T18:08:21.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.647+0000 7f4f6e000640 1 --2- 172.21.15.37:0/4193723360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f78102790 0x7f4f78197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:08:21.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.648+0000 7f4f76a00640 1 --2- 172.21.15.37:0/4193723360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f781015a0 0x7f4f781978c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4f68037040 tx=0x7f4f68035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:21.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.648+0000 7f4f6fe00640 1 -- 172.21.15.37:0/4193723360 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f68039040 con 0x7f4f781015a0 2024-03-28T18:08:21.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.648+0000 7f4f6fe00640 1 -- 172.21.15.37:0/4193723360 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f68003970 con 0x7f4f781015a0 2024-03-28T18:08:21.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.648+0000 7f4f77e00640 1 -- 172.21.15.37:0/4193723360 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f7819f770 con 0x7f4f781015a0 2024-03-28T18:08:21.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.648+0000 7f4f6fe00640 1 -- 172.21.15.37:0/4193723360 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f680045a0 con 0x7f4f781015a0 2024-03-28T18:08:21.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.648+0000 7f4f77e00640 1 -- 172.21.15.37:0/4193723360 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7819fc30 con 0x7f4f781015a0 2024-03-28T18:08:21.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.649+0000 7f4f77e00640 1 -- 172.21.15.37:0/4193723360 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f78108960 con 0x7f4f781015a0 2024-03-28T18:08:21.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.651+0000 7f4f6fe00640 1 -- 172.21.15.37:0/4193723360 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4f68046030 con 0x7f4f781015a0 2024-03-28T18:08:21.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.653+0000 7f4f6fe00640 1 --2- 172.21.15.37:0/4193723360 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f4f5c0a2800 0x7f4f5c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:21.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.653+0000 7f4f6fe00640 1 -- 172.21.15.37:0/4193723360 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f4f680f06c0 con 0x7f4f781015a0 2024-03-28T18:08:21.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.653+0000 7f4f6e000640 1 --2- 172.21.15.37:0/4193723360 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f4f5c0a2800 0x7f4f5c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:21.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.653+0000 7f4f6e000640 1 --2- 172.21.15.37:0/4193723360 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f4f5c0a2800 0x7f4f5c0a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4f78198dd0 tx=0x7f4f6000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:21.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.655+0000 7f4f6fe00640 1 -- 172.21.15.37:0/4193723360 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4f6804d060 con 0x7f4f781015a0 2024-03-28T18:08:21.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:21 smithi059 ceph-mon[36190]: pgmap v316: 97 pgs: 9 creating+peering, 7 active+clean, 81 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:21.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:21 smithi059 ceph-mon[36190]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T18:08:21.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:21 smithi059 ceph-mon[36190]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T18:08:21.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:21 smithi059 ceph-mon[36190]: fsmap cephfs:1 {0=a=up:creating} 11 up:standby 2024-03-28T18:08:21.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:21 smithi059 ceph-mon[36190]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-03-28T18:08:21.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:21 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2108938803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:08:21.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:21 smithi059 ceph-mon[36190]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T18:08:21.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:21 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/4180485347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:08:21.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:21 smithi170 ceph-mon[36280]: pgmap v316: 97 pgs: 9 creating+peering, 7 active+clean, 81 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:21.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:21 smithi170 ceph-mon[36280]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T18:08:21.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:21 smithi170 ceph-mon[36280]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T18:08:21.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:21 smithi170 ceph-mon[36280]: fsmap cephfs:1 {0=a=up:creating} 11 up:standby 2024-03-28T18:08:21.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:21 smithi170 ceph-mon[36280]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-03-28T18:08:21.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:21 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2108938803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:08:21.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:21 smithi170 ceph-mon[36280]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T18:08:21.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:21 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/4180485347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:08:21.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:21 smithi037 ceph-mon[31460]: pgmap v316: 97 pgs: 9 creating+peering, 7 active+clean, 81 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:08:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:21 smithi037 ceph-mon[31460]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T18:08:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:21 smithi037 ceph-mon[31460]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T18:08:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:21 smithi037 ceph-mon[31460]: fsmap cephfs:1 {0=a=up:creating} 11 up:standby 2024-03-28T18:08:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:21 smithi037 ceph-mon[31460]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-03-28T18:08:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:21 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2108938803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:08:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:21 smithi037 ceph-mon[31460]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T18:08:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:21 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/4180485347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:08:21.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:21.904+0000 7f4f77e00640 1 -- 172.21.15.37:0/4193723360 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f4f7802d070 con 0x7f4f781015a0 2024-03-28T18:08:22.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.545+0000 7f4f6fe00640 1 -- 172.21.15.37:0/4193723360 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4f680bd8a0 con 0x7f4f781015a0 2024-03-28T18:08:22.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.550+0000 7f4f77e00640 1 -- 172.21.15.37:0/4193723360 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f4f5c0a2800 msgr2=0x7f4f5c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:22.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.550+0000 7f4f77e00640 1 --2- 172.21.15.37:0/4193723360 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f4f5c0a2800 0x7f4f5c0a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4f78198dd0 tx=0x7f4f6000b390 comp rx=0 tx=0).stop 2024-03-28T18:08:22.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.550+0000 7f4f77e00640 1 -- 172.21.15.37:0/4193723360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f781015a0 msgr2=0x7f4f781978c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:22.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.550+0000 7f4f77e00640 1 --2- 172.21.15.37:0/4193723360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f781015a0 0x7f4f781978c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4f68037040 tx=0x7f4f68035d10 comp rx=0 tx=0).stop 2024-03-28T18:08:22.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.550+0000 7f4f77e00640 1 -- 172.21.15.37:0/4193723360 shutdown_connections 2024-03-28T18:08:22.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.550+0000 7f4f77e00640 1 --2- 172.21.15.37:0/4193723360 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f4f5c0a2800 0x7f4f5c0a4cc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:22.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.550+0000 7f4f77e00640 1 --2- 172.21.15.37:0/4193723360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f78102790 0x7f4f78197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:22.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.550+0000 7f4f77e00640 1 --2- 172.21.15.37:0/4193723360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f78103150 0x7f4f7819ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:22.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.550+0000 7f4f77e00640 1 --2- 172.21.15.37:0/4193723360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f781015a0 0x7f4f781978c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:22.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.550+0000 7f4f77e00640 1 -- 172.21.15.37:0/4193723360 >> 172.21.15.37:0/4193723360 conn(0x7f4f780fcb10 msgr2=0x7f4f78105b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:22.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.550+0000 7f4f77e00640 1 -- 172.21.15.37:0/4193723360 shutdown_connections 2024-03-28T18:08:22.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.550+0000 7f4f77e00640 1 -- 172.21.15.37:0/4193723360 wait complete. 2024-03-28T18:08:22.575 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-28T18:08:22.576 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-28T18:08:22.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:22 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] up:active 2024-03-28T18:08:22.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:22 smithi059 ceph-mon[36190]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-28T18:08:22.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:22 smithi059 ceph-mon[36190]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-28T18:08:22.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:22 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/4193723360' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T18:08:22.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:22 smithi059 ceph-mon[36190]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T18:08:22.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:22 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] up:active 2024-03-28T18:08:22.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:22 smithi170 ceph-mon[36280]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-28T18:08:22.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:22 smithi170 ceph-mon[36280]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-28T18:08:22.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:22 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/4193723360' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T18:08:22.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:22 smithi170 ceph-mon[36280]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T18:08:22.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.793+0000 7fba86200640 1 -- 172.21.15.37:0/1834556659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba80075660 msgr2=0x7fba80073ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:22.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.793+0000 7fba86200640 1 --2- 172.21.15.37:0/1834556659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba80075660 0x7fba80073ab0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fba70009900 tx=0x7fba7002bcd0 comp rx=0 tx=0).stop 2024-03-28T18:08:22.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.793+0000 7fba86200640 1 -- 172.21.15.37:0/1834556659 shutdown_connections 2024-03-28T18:08:22.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.793+0000 7fba86200640 1 --2- 172.21.15.37:0/1834556659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba80109450 0x7fba8010b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:22.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.793+0000 7fba86200640 1 --2- 172.21.15.37:0/1834556659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba80074080 0x7fba800744e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:22.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.793+0000 7fba86200640 1 --2- 172.21.15.37:0/1834556659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba80075660 0x7fba80073ab0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:22.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.793+0000 7fba86200640 1 -- 172.21.15.37:0/1834556659 >> 172.21.15.37:0/1834556659 conn(0x7fba800fcb10 msgr2=0x7fba800fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:22.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.793+0000 7fba86200640 1 -- 172.21.15.37:0/1834556659 shutdown_connections 2024-03-28T18:08:22.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.793+0000 7fba86200640 1 -- 172.21.15.37:0/1834556659 wait complete. 2024-03-28T18:08:22.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.794+0000 7fba86200640 1 Processor -- start 2024-03-28T18:08:22.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.795+0000 7fba86200640 1 -- start start 2024-03-28T18:08:22.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.795+0000 7fba86200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba80074080 0x7fba80199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:22.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.795+0000 7fba86200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba80075660 0x7fba80199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:22.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.795+0000 7fba86200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba80109450 0x7fba801a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:22.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.795+0000 7fba86200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8010f470 con 0x7fba80074080 2024-03-28T18:08:22.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.795+0000 7fba86200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8010f2f0 con 0x7fba80075660 2024-03-28T18:08:22.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.795+0000 7fba86200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8010f5f0 con 0x7fba80109450 2024-03-28T18:08:22.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.795+0000 7fba84e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba80074080 0x7fba80199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:22.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.795+0000 7fba77e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba80075660 0x7fba80199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:22.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.795+0000 7fba84e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba80074080 0x7fba80199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57102/0 (socket says 172.21.15.37:57102) 2024-03-28T18:08:22.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.795+0000 7fba77e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba80075660 0x7fba80199fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:48344/0 (socket says 172.21.15.37:48344) 2024-03-28T18:08:22.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.795+0000 7fba84e00640 1 -- 172.21.15.37:0/4026761129 learned_addr learned my addr 172.21.15.37:0/4026761129 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:22.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.796+0000 7fba85800640 1 --2- 172.21.15.37:0/4026761129 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba80109450 0x7fba801a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:22.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.796+0000 7fba84e00640 1 -- 172.21.15.37:0/4026761129 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba80109450 msgr2=0x7fba801a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:22.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.796+0000 7fba84e00640 1 --2- 172.21.15.37:0/4026761129 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba80109450 0x7fba801a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:22.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.796+0000 7fba84e00640 1 -- 172.21.15.37:0/4026761129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba80075660 msgr2=0x7fba80199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:22.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.796+0000 7fba84e00640 1 --2- 172.21.15.37:0/4026761129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba80075660 0x7fba80199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:22.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.796+0000 7fba84e00640 1 -- 172.21.15.37:0/4026761129 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba801a16a0 con 0x7fba80074080 2024-03-28T18:08:22.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.796+0000 7fba77e00640 1 --2- 172.21.15.37:0/4026761129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba80075660 0x7fba80199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:08:22.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.796+0000 7fba84e00640 1 --2- 172.21.15.37:0/4026761129 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba80074080 0x7fba80199a80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fba70000c00 tx=0x7fba7002c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:22.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.797+0000 7fba7e000640 1 -- 172.21.15.37:0/4026761129 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba70043020 con 0x7fba80074080 2024-03-28T18:08:22.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.797+0000 7fba86200640 1 -- 172.21.15.37:0/4026761129 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba801a1930 con 0x7fba80074080 2024-03-28T18:08:22.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.797+0000 7fba7e000640 1 -- 172.21.15.37:0/4026761129 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba700054e0 con 0x7fba80074080 2024-03-28T18:08:22.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.797+0000 7fba7e000640 1 -- 172.21.15.37:0/4026761129 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba70003c20 con 0x7fba80074080 2024-03-28T18:08:22.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.797+0000 7fba86200640 1 -- 172.21.15.37:0/4026761129 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba801a1e80 con 0x7fba80074080 2024-03-28T18:08:22.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.798+0000 7fba86200640 1 -- 172.21.15.37:0/4026761129 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba80108210 con 0x7fba80074080 2024-03-28T18:08:22.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.800+0000 7fba7e000640 1 -- 172.21.15.37:0/4026761129 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fba70046020 con 0x7fba80074080 2024-03-28T18:08:22.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.801+0000 7fba7e000640 1 --2- 172.21.15.37:0/4026761129 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fba640a2800 0x7fba640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:22.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.801+0000 7fba7e000640 1 -- 172.21.15.37:0/4026761129 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fba700f0960 con 0x7fba80074080 2024-03-28T18:08:22.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.802+0000 7fba77e00640 1 --2- 172.21.15.37:0/4026761129 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fba640a2800 0x7fba640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:22.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.802+0000 7fba77e00640 1 --2- 172.21.15.37:0/4026761129 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fba640a2800 0x7fba640a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fba8019af90 tx=0x7fba68005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:22.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:22.806+0000 7fba7e000640 1 -- 172.21.15.37:0/4026761129 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fba7004d020 con 0x7fba80074080 2024-03-28T18:08:22.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:22 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] up:active 2024-03-28T18:08:22.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:22 smithi037 ceph-mon[31460]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-28T18:08:22.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:22 smithi037 ceph-mon[31460]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-28T18:08:22.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:22 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/4193723360' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T18:08:22.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:22 smithi037 ceph-mon[31460]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T18:08:23.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.086+0000 7fba86200640 1 -- 172.21.15.37:0/4026761129 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7fba800622a0 con 0x7fba80074080 2024-03-28T18:08:23.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.378+0000 7fba7e000640 1 -- 172.21.15.37:0/4026761129 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7fba700bdb50 con 0x7fba80074080 2024-03-28T18:08:23.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.381+0000 7fba86200640 1 -- 172.21.15.37:0/4026761129 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fba640a2800 msgr2=0x7fba640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:23.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.381+0000 7fba86200640 1 --2- 172.21.15.37:0/4026761129 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fba640a2800 0x7fba640a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fba8019af90 tx=0x7fba68005f90 comp rx=0 tx=0).stop 2024-03-28T18:08:23.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.381+0000 7fba86200640 1 -- 172.21.15.37:0/4026761129 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba80074080 msgr2=0x7fba80199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:23.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.381+0000 7fba86200640 1 --2- 172.21.15.37:0/4026761129 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba80074080 0x7fba80199a80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fba70000c00 tx=0x7fba7002c6f0 comp rx=0 tx=0).stop 2024-03-28T18:08:23.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.382+0000 7fba86200640 1 -- 172.21.15.37:0/4026761129 shutdown_connections 2024-03-28T18:08:23.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.382+0000 7fba86200640 1 --2- 172.21.15.37:0/4026761129 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fba640a2800 0x7fba640a4cc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:23.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.382+0000 7fba86200640 1 --2- 172.21.15.37:0/4026761129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba80075660 0x7fba80199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:23.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.382+0000 7fba86200640 1 --2- 172.21.15.37:0/4026761129 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba80109450 0x7fba801a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:23.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.382+0000 7fba86200640 1 --2- 172.21.15.37:0/4026761129 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba80074080 0x7fba80199a80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:23.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.382+0000 7fba86200640 1 -- 172.21.15.37:0/4026761129 >> 172.21.15.37:0/4026761129 conn(0x7fba800fcb10 msgr2=0x7fba801053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:23.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.382+0000 7fba86200640 1 -- 172.21.15.37:0/4026761129 shutdown_connections 2024-03-28T18:08:23.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.382+0000 7fba86200640 1 -- 172.21.15.37:0/4026761129 wait complete. 2024-03-28T18:08:23.410 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-28T18:08:23.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.629+0000 7f5698000640 1 -- 172.21.15.37:0/1803526548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56901016b0 msgr2=0x7f5690101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:23.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.629+0000 7f5698000640 1 --2- 172.21.15.37:0/1803526548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56901016b0 0x7f5690101ad0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5684009900 tx=0x7f568402bc30 comp rx=0 tx=0).stop 2024-03-28T18:08:23.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.629+0000 7f5698000640 1 -- 172.21.15.37:0/1803526548 shutdown_connections 2024-03-28T18:08:23.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.629+0000 7f5698000640 1 --2- 172.21.15.37:0/1803526548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5690103260 0x7f5690109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:23.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.629+0000 7f5698000640 1 --2- 172.21.15.37:0/1803526548 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56901028a0 0x7f5690102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:23.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.629+0000 7f5698000640 1 --2- 172.21.15.37:0/1803526548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56901016b0 0x7f5690101ad0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:23.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.629+0000 7f5698000640 1 -- 172.21.15.37:0/1803526548 >> 172.21.15.37:0/1803526548 conn(0x7f56900fcc40 msgr2=0x7f56900ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:23.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.629+0000 7f5698000640 1 -- 172.21.15.37:0/1803526548 shutdown_connections 2024-03-28T18:08:23.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.629+0000 7f5698000640 1 -- 172.21.15.37:0/1803526548 wait complete. 2024-03-28T18:08:23.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.630+0000 7f5698000640 1 Processor -- start 2024-03-28T18:08:23.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.631+0000 7f5698000640 1 -- start start 2024-03-28T18:08:23.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.631+0000 7f5698000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56901016b0 0x7f5690075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:23.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.631+0000 7f5698000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56901028a0 0x7f5690073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:23.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.631+0000 7f5698000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5690103260 0x7f5690073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:23.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.631+0000 7f5698000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569010d380 con 0x7f56901016b0 2024-03-28T18:08:23.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.631+0000 7f5698000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569010d200 con 0x7f5690103260 2024-03-28T18:08:23.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.631+0000 7f5698000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569010d500 con 0x7f56901028a0 2024-03-28T18:08:23.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.631+0000 7f5695600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56901016b0 0x7f5690075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:23.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.631+0000 7f5695600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56901016b0 0x7f5690075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58860/0 (socket says 172.21.15.37:58860) 2024-03-28T18:08:23.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.631+0000 7f5695600640 1 -- 172.21.15.37:0/1740238187 learned_addr learned my addr 172.21.15.37:0/1740238187 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:23.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.631+0000 7f5696000640 1 --2- 172.21.15.37:0/1740238187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5690103260 0x7f5690073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:23.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.631+0000 7f5694c00640 1 --2- 172.21.15.37:0/1740238187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56901028a0 0x7f5690073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:23.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.632+0000 7f5695600640 1 -- 172.21.15.37:0/1740238187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56901028a0 msgr2=0x7f5690073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:23.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.632+0000 7f5695600640 1 --2- 172.21.15.37:0/1740238187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56901028a0 0x7f5690073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:23.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.632+0000 7f5695600640 1 -- 172.21.15.37:0/1740238187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5690103260 msgr2=0x7f5690073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:23.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.632+0000 7f5695600640 1 --2- 172.21.15.37:0/1740238187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5690103260 0x7f5690073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:23.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.632+0000 7f5695600640 1 -- 172.21.15.37:0/1740238187 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5690074860 con 0x7f56901016b0 2024-03-28T18:08:23.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.632+0000 7f5694c00640 1 --2- 172.21.15.37:0/1740238187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56901028a0 0x7f5690073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:08:23.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.632+0000 7f5695600640 1 --2- 172.21.15.37:0/1740238187 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56901016b0 0x7f5690075450 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5684000c00 tx=0x7f568402c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:23.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.632+0000 7f568e000640 1 -- 172.21.15.37:0/1740238187 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684043020 con 0x7f56901016b0 2024-03-28T18:08:23.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.632+0000 7f5698000640 1 -- 172.21.15.37:0/1740238187 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56901a1b40 con 0x7f56901016b0 2024-03-28T18:08:23.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.632+0000 7f568e000640 1 -- 172.21.15.37:0/1740238187 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56840054e0 con 0x7f56901016b0 2024-03-28T18:08:23.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.632+0000 7f568e000640 1 -- 172.21.15.37:0/1740238187 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684003ce0 con 0x7f56901016b0 2024-03-28T18:08:23.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.633+0000 7f5698000640 1 -- 172.21.15.37:0/1740238187 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56901a20b0 con 0x7f56901016b0 2024-03-28T18:08:23.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.634+0000 7f5698000640 1 -- 172.21.15.37:0/1740238187 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56901089f0 con 0x7f56901016b0 2024-03-28T18:08:23.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.636+0000 7f568e000640 1 -- 172.21.15.37:0/1740238187 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5684046020 con 0x7f56901016b0 2024-03-28T18:08:23.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.637+0000 7f568e000640 1 --2- 172.21.15.37:0/1740238187 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56780a2880 0x7f56780a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:23.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.637+0000 7f568e000640 1 -- 172.21.15.37:0/1740238187 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f56840f0980 con 0x7f56901016b0 2024-03-28T18:08:23.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.637+0000 7f5694c00640 1 --2- 172.21.15.37:0/1740238187 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56780a2880 0x7f56780a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:23.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.638+0000 7f5694c00640 1 --2- 172.21.15.37:0/1740238187 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56780a2880 0x7f56780a4d40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f56900751c0 tx=0x7f567c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:23.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:23.644+0000 7f568e000640 1 -- 172.21.15.37:0/1740238187 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f568404d020 con 0x7f56901016b0 2024-03-28T18:08:23.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:23 smithi059 ceph-mon[36190]: pgmap v319: 97 pgs: 9 creating+peering, 59 active+clean, 29 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-28T18:08:23.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:23 smithi059 ceph-mon[36190]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-28T18:08:23.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:23 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:08:23.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:23 smithi059 ceph-mon[36190]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T18:08:23.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:23 smithi059 ceph-mon[36190]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-28T18:08:23.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:23 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/4026761129' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-28T18:08:23.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:23 smithi059 ceph-mon[36190]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T18:08:23.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:23 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T18:08:23.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:23 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/4026761129' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-28T18:08:23.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:23 smithi059 ceph-mon[36190]: fsmap cephfs:2 {0=a=up:active,1=b=up:creating} 10 up:standby 2024-03-28T18:08:23.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:23 smithi059 ceph-mon[36190]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-28T18:08:23.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:23 smithi170 ceph-mon[36280]: pgmap v319: 97 pgs: 9 creating+peering, 59 active+clean, 29 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-28T18:08:23.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:23 smithi170 ceph-mon[36280]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-28T18:08:23.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:23 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:08:23.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:23 smithi170 ceph-mon[36280]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T18:08:23.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:23 smithi170 ceph-mon[36280]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-28T18:08:23.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:23 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/4026761129' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-28T18:08:23.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:23 smithi170 ceph-mon[36280]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T18:08:23.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:23 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T18:08:23.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:23 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/4026761129' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-28T18:08:23.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:23 smithi170 ceph-mon[36280]: fsmap cephfs:2 {0=a=up:active,1=b=up:creating} 10 up:standby 2024-03-28T18:08:23.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:23 smithi170 ceph-mon[36280]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-28T18:08:23.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:23 smithi037 ceph-mon[31460]: pgmap v319: 97 pgs: 9 creating+peering, 59 active+clean, 29 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-28T18:08:23.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:23 smithi037 ceph-mon[31460]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-28T18:08:23.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:23 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:08:23.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:23 smithi037 ceph-mon[31460]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T18:08:23.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:23 smithi037 ceph-mon[31460]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-28T18:08:23.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:23 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/4026761129' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-28T18:08:23.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:23 smithi037 ceph-mon[31460]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T18:08:23.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:23 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T18:08:23.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:23 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/4026761129' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-28T18:08:23.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:23 smithi037 ceph-mon[31460]: fsmap cephfs:2 {0=a=up:active,1=b=up:creating} 10 up:standby 2024-03-28T18:08:23.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:23 smithi037 ceph-mon[31460]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-28T18:08:24.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.018+0000 7f5698000640 1 -- 172.21.15.37:0/1740238187 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f569002d070 con 0x7f56901016b0 2024-03-28T18:08:24.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.396+0000 7f568e000640 1 -- 172.21.15.37:0/1740238187 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) v1 ==== 121+0+0 (secure 0 0 0) 0x7f56840bdb70 con 0x7f56901016b0 2024-03-28T18:08:24.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.401+0000 7f5698000640 1 -- 172.21.15.37:0/1740238187 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56780a2880 msgr2=0x7f56780a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:24.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.401+0000 7f5698000640 1 --2- 172.21.15.37:0/1740238187 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56780a2880 0x7f56780a4d40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f56900751c0 tx=0x7f567c00b390 comp rx=0 tx=0).stop 2024-03-28T18:08:24.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.401+0000 7f5698000640 1 -- 172.21.15.37:0/1740238187 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56901016b0 msgr2=0x7f5690075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:24.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.401+0000 7f5698000640 1 --2- 172.21.15.37:0/1740238187 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56901016b0 0x7f5690075450 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5684000c00 tx=0x7f568402c6c0 comp rx=0 tx=0).stop 2024-03-28T18:08:24.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.401+0000 7f5698000640 1 -- 172.21.15.37:0/1740238187 shutdown_connections 2024-03-28T18:08:24.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.401+0000 7f5698000640 1 --2- 172.21.15.37:0/1740238187 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56780a2880 0x7f56780a4d40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:24.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.401+0000 7f5698000640 1 --2- 172.21.15.37:0/1740238187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5690103260 0x7f5690073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:24.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.401+0000 7f5698000640 1 --2- 172.21.15.37:0/1740238187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56901028a0 0x7f5690073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:24.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.401+0000 7f5698000640 1 --2- 172.21.15.37:0/1740238187 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56901016b0 0x7f5690075450 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:24.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.401+0000 7f5698000640 1 -- 172.21.15.37:0/1740238187 >> 172.21.15.37:0/1740238187 conn(0x7f56900fcc40 msgr2=0x7f5690105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:24.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.401+0000 7f5698000640 1 -- 172.21.15.37:0/1740238187 shutdown_connections 2024-03-28T18:08:24.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.401+0000 7f5698000640 1 -- 172.21.15.37:0/1740238187 wait complete. 2024-03-28T18:08:24.432 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-28T18:08:24.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.633+0000 7f746b000640 1 -- 172.21.15.37:0/54898686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7464102030 msgr2=0x7f74641088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:24.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.633+0000 7f746b000640 1 --2- 172.21.15.37:0/54898686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7464102030 0x7f74641088c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f745c009ca0 tx=0x7f745c02c0f0 comp rx=0 tx=0).stop 2024-03-28T18:08:24.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.634+0000 7f746b000640 1 -- 172.21.15.37:0/54898686 shutdown_connections 2024-03-28T18:08:24.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.634+0000 7f746b000640 1 --2- 172.21.15.37:0/54898686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7464102030 0x7f74641088c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:24.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.634+0000 7f746b000640 1 --2- 172.21.15.37:0/54898686 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7464101670 0x7f7464101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:24.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.634+0000 7f746b000640 1 --2- 172.21.15.37:0/54898686 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7464100480 0x7f74641008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:24.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.634+0000 7f746b000640 1 -- 172.21.15.37:0/54898686 >> 172.21.15.37:0/54898686 conn(0x7f74640fba10 msgr2=0x7f74640fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:24.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.634+0000 7f746b000640 1 -- 172.21.15.37:0/54898686 shutdown_connections 2024-03-28T18:08:24.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.634+0000 7f746b000640 1 -- 172.21.15.37:0/54898686 wait complete. 2024-03-28T18:08:24.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.635+0000 7f746b000640 1 Processor -- start 2024-03-28T18:08:24.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.635+0000 7f746b000640 1 -- start start 2024-03-28T18:08:24.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.635+0000 7f746b000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7464100480 0x7f7464199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:24.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.635+0000 7f746b000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7464101670 0x7f7464199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:24.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.635+0000 7f746b000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7464102030 0x7f74641a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:24.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.635+0000 7f746b000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746410c110 con 0x7f7464102030 2024-03-28T18:08:24.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.635+0000 7f746b000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746410bf90 con 0x7f7464101670 2024-03-28T18:08:24.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.635+0000 7f746b000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746410c290 con 0x7f7464100480 2024-03-28T18:08:24.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.636+0000 7f7469200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7464101670 0x7f7464199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:24.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.636+0000 7f746a600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7464102030 0x7f74641a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:24.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.636+0000 7f7469200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7464101670 0x7f7464199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:57810/0 (socket says 172.21.15.37:57810) 2024-03-28T18:08:24.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.636+0000 7f7469c00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7464100480 0x7f7464199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:24.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.636+0000 7f7469200640 1 -- 172.21.15.37:0/3221751120 learned_addr learned my addr 172.21.15.37:0/3221751120 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:24.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.636+0000 7f7469200640 1 -- 172.21.15.37:0/3221751120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7464100480 msgr2=0x7f7464199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:24.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.636+0000 7f7469200640 1 --2- 172.21.15.37:0/3221751120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7464100480 0x7f7464199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:24.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.636+0000 7f7469200640 1 -- 172.21.15.37:0/3221751120 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7464102030 msgr2=0x7f74641a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:24.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.636+0000 7f7469200640 1 --2- 172.21.15.37:0/3221751120 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7464102030 0x7f74641a0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:24.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.636+0000 7f7469200640 1 -- 172.21.15.37:0/3221751120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74641a1650 con 0x7f7464101670 2024-03-28T18:08:24.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.636+0000 7f7469200640 1 --2- 172.21.15.37:0/3221751120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7464101670 0x7f7464199f70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f745000e9f0 tx=0x7f745000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:24.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.636+0000 7f7462000640 1 -- 172.21.15.37:0/3221751120 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745000cd80 con 0x7f7464101670 2024-03-28T18:08:24.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.637+0000 7f7462000640 1 -- 172.21.15.37:0/3221751120 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7450005500 con 0x7f7464101670 2024-03-28T18:08:24.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.637+0000 7f7462000640 1 -- 172.21.15.37:0/3221751120 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7450010690 con 0x7f7464101670 2024-03-28T18:08:24.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.637+0000 7f746b000640 1 -- 172.21.15.37:0/3221751120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74641a1940 con 0x7f7464101670 2024-03-28T18:08:24.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.637+0000 7f746b000640 1 -- 172.21.15.37:0/3221751120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74641a1e40 con 0x7f7464101670 2024-03-28T18:08:24.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.638+0000 7f746b000640 1 -- 172.21.15.37:0/3221751120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74641a20d0 con 0x7f7464101670 2024-03-28T18:08:24.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.640+0000 7f7462000640 1 -- 172.21.15.37:0/3221751120 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f74500046c0 con 0x7f7464101670 2024-03-28T18:08:24.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.641+0000 7f7462000640 1 --2- 172.21.15.37:0/3221751120 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f744c0a2880 0x7f744c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:24.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.641+0000 7f7462000640 1 -- 172.21.15.37:0/3221751120 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f74500ced10 con 0x7f7464101670 2024-03-28T18:08:24.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.642+0000 7f7469c00640 1 --2- 172.21.15.37:0/3221751120 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f744c0a2880 0x7f744c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:24.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.642+0000 7f7469c00640 1 --2- 172.21.15.37:0/3221751120 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f744c0a2880 0x7f744c0a4d40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f74641014d0 tx=0x7f7458005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:24.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.650+0000 7f7462000640 1 -- 172.21.15.37:0/3221751120 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f74500d4020 con 0x7f7464101670 2024-03-28T18:08:24.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:24 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1740238187' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T18:08:24.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:24 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1740238187' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T18:08:24.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:24 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] up:active 2024-03-28T18:08:24.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:24 smithi059 ceph-mon[36190]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T18:08:24.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:24 smithi059 ceph-mon[36190]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T18:08:24.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:24 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:08:24.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:24 smithi059 ceph-mon[36190]: fsmap cephfs:2 {0=a=up:active,1=b=up:active} 10 up:standby 2024-03-28T18:08:24.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:24 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2024-03-28T18:08:24.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:24 smithi059 ceph-mon[36190]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-28T18:08:24.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:24 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1740238187' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T18:08:24.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:24 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1740238187' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T18:08:24.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:24 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] up:active 2024-03-28T18:08:24.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:24 smithi170 ceph-mon[36280]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T18:08:24.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:24 smithi170 ceph-mon[36280]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T18:08:24.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:24 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:08:24.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:24 smithi170 ceph-mon[36280]: fsmap cephfs:2 {0=a=up:active,1=b=up:active} 10 up:standby 2024-03-28T18:08:24.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:24 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2024-03-28T18:08:24.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:24 smithi170 ceph-mon[36280]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-28T18:08:24.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:24 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1740238187' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T18:08:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:24 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1740238187' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T18:08:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:24 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] up:active 2024-03-28T18:08:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:24 smithi037 ceph-mon[31460]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T18:08:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:24 smithi037 ceph-mon[31460]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T18:08:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:24 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:08:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:24 smithi037 ceph-mon[31460]: fsmap cephfs:2 {0=a=up:active,1=b=up:active} 10 up:standby 2024-03-28T18:08:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:24 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2024-03-28T18:08:24.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:24 smithi037 ceph-mon[31460]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-28T18:08:24.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:24.896+0000 7f746b000640 1 -- 172.21.15.37:0/3221751120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f74641a22a0 con 0x7f7464101670 2024-03-28T18:08:25.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.410+0000 7f7462000640 1 -- 172.21.15.37:0/3221751120 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) v1 ==== 115+0+0 (secure 0 0 0) 0x7f745009be60 con 0x7f7464101670 2024-03-28T18:08:25.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.413+0000 7f746b000640 1 -- 172.21.15.37:0/3221751120 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f744c0a2880 msgr2=0x7f744c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:25.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.413+0000 7f746b000640 1 --2- 172.21.15.37:0/3221751120 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f744c0a2880 0x7f744c0a4d40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f74641014d0 tx=0x7f7458005f90 comp rx=0 tx=0).stop 2024-03-28T18:08:25.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.413+0000 7f746b000640 1 -- 172.21.15.37:0/3221751120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7464101670 msgr2=0x7f7464199f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:25.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.413+0000 7f746b000640 1 --2- 172.21.15.37:0/3221751120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7464101670 0x7f7464199f70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f745000e9f0 tx=0x7f745000edf0 comp rx=0 tx=0).stop 2024-03-28T18:08:25.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.413+0000 7f746b000640 1 -- 172.21.15.37:0/3221751120 shutdown_connections 2024-03-28T18:08:25.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.413+0000 7f746b000640 1 --2- 172.21.15.37:0/3221751120 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f744c0a2880 0x7f744c0a4d40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:25.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.413+0000 7f746b000640 1 --2- 172.21.15.37:0/3221751120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7464101670 0x7f7464199f70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:25.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.413+0000 7f746b000640 1 --2- 172.21.15.37:0/3221751120 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7464102030 0x7f74641a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:25.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.413+0000 7f746b000640 1 --2- 172.21.15.37:0/3221751120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7464100480 0x7f7464199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:25.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.413+0000 7f746b000640 1 -- 172.21.15.37:0/3221751120 >> 172.21.15.37:0/3221751120 conn(0x7f74640fba10 msgr2=0x7f74640fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:25.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.413+0000 7f746b000640 1 -- 172.21.15.37:0/3221751120 shutdown_connections 2024-03-28T18:08:25.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.413+0000 7f746b000640 1 -- 172.21.15.37:0/3221751120 wait complete. 2024-03-28T18:08:25.435 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-03-28T18:08:25.435 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-28T18:08:25.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.639+0000 7f531de00640 1 -- 172.21.15.37:0/1243755794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5318104a50 msgr2=0x7f5318106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:25.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.639+0000 7f531de00640 1 --2- 172.21.15.37:0/1243755794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5318104a50 0x7f5318106e40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5300009900 tx=0x7f530002bc30 comp rx=0 tx=0).stop 2024-03-28T18:08:25.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.639+0000 7f531de00640 1 -- 172.21.15.37:0/1243755794 shutdown_connections 2024-03-28T18:08:25.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.639+0000 7f531de00640 1 --2- 172.21.15.37:0/1243755794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5318104a50 0x7f5318106e40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:25.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.639+0000 7f531de00640 1 --2- 172.21.15.37:0/1243755794 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5318107380 0x7f53181097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:25.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.639+0000 7f531de00640 1 --2- 172.21.15.37:0/1243755794 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5318102120 0x7f5318104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:25.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.639+0000 7f531de00640 1 -- 172.21.15.37:0/1243755794 >> 172.21.15.37:0/1243755794 conn(0x7f53180fbb10 msgr2=0x7f53180fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:25.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.639+0000 7f531de00640 1 -- 172.21.15.37:0/1243755794 shutdown_connections 2024-03-28T18:08:25.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.639+0000 7f531de00640 1 -- 172.21.15.37:0/1243755794 wait complete. 2024-03-28T18:08:25.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.640+0000 7f531de00640 1 Processor -- start 2024-03-28T18:08:25.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.640+0000 7f531de00640 1 -- start start 2024-03-28T18:08:25.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.640+0000 7f531de00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5318102120 0x7f5318199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:25.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.640+0000 7f531de00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5318104a50 0x7f531819a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:25.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.640+0000 7f531de00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5318107380 0x7f53181a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:25.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.640+0000 7f531de00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531810d380 con 0x7f5318104a50 2024-03-28T18:08:25.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.640+0000 7f531de00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531810d200 con 0x7f5318107380 2024-03-28T18:08:25.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.640+0000 7f531de00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531810d500 con 0x7f5318102120 2024-03-28T18:08:25.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.640+0000 7f530fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5318104a50 0x7f531819a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:25.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.640+0000 7f5316a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5318102120 0x7f5318199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:25.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.640+0000 7f5317400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5318107380 0x7f53181a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:25.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.640+0000 7f530fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5318104a50 0x7f531819a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58902/0 (socket says 172.21.15.37:58902) 2024-03-28T18:08:25.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.640+0000 7f5317400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5318107380 0x7f53181a1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:57828/0 (socket says 172.21.15.37:57828) 2024-03-28T18:08:25.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.640+0000 7f5317400640 1 -- 172.21.15.37:0/360932136 learned_addr learned my addr 172.21.15.37:0/360932136 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:25.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.640+0000 7f5316a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5318102120 0x7f5318199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:32984/0 (socket says 172.21.15.37:32984) 2024-03-28T18:08:25.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.641+0000 7f5317400640 1 -- 172.21.15.37:0/360932136 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5318102120 msgr2=0x7f5318199bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:25.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.641+0000 7f5317400640 1 --2- 172.21.15.37:0/360932136 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5318102120 0x7f5318199bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:25.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.641+0000 7f5317400640 1 -- 172.21.15.37:0/360932136 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5318104a50 msgr2=0x7f531819a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:25.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.641+0000 7f5317400640 1 --2- 172.21.15.37:0/360932136 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5318104a50 0x7f531819a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:25.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.641+0000 7f5317400640 1 -- 172.21.15.37:0/360932136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53181a17d0 con 0x7f5318107380 2024-03-28T18:08:25.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.641+0000 7f5317400640 1 --2- 172.21.15.37:0/360932136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5318107380 0x7f53181a1160 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f531000eaa0 tx=0x7f531000ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:25.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.641+0000 7f531ca00640 1 -- 172.21.15.37:0/360932136 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531000cdb0 con 0x7f5318107380 2024-03-28T18:08:25.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.641+0000 7f531ca00640 1 -- 172.21.15.37:0/360932136 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5310005020 con 0x7f5318107380 2024-03-28T18:08:25.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.641+0000 7f531ca00640 1 -- 172.21.15.37:0/360932136 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53100116b0 con 0x7f5318107380 2024-03-28T18:08:25.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.641+0000 7f531de00640 1 -- 172.21.15.37:0/360932136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53181a1ac0 con 0x7f5318107380 2024-03-28T18:08:25.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.642+0000 7f531de00640 1 -- 172.21.15.37:0/360932136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53181a1f90 con 0x7f5318107380 2024-03-28T18:08:25.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.643+0000 7f531de00640 1 -- 172.21.15.37:0/360932136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5318108ed0 con 0x7f5318107380 2024-03-28T18:08:25.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.644+0000 7f531ca00640 1 -- 172.21.15.37:0/360932136 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5310005560 con 0x7f5318107380 2024-03-28T18:08:25.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.645+0000 7f531ca00640 1 --2- 172.21.15.37:0/360932136 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f52fc0a2880 0x7f52fc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:25.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.646+0000 7f531ca00640 1 -- 172.21.15.37:0/360932136 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f53100ceaf0 con 0x7f5318107380 2024-03-28T18:08:25.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.646+0000 7f5316a00640 1 --2- 172.21.15.37:0/360932136 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f52fc0a2880 0x7f52fc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:25.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.646+0000 7f5316a00640 1 --2- 172.21.15.37:0/360932136 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f52fc0a2880 0x7f52fc0a4d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5308009bd0 tx=0x7f5308005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:25.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.649+0000 7f531ca00640 1 -- 172.21.15.37:0/360932136 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5310016030 con 0x7f5318107380 2024-03-28T18:08:25.735 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:25 smithi037 ceph-mon[31460]: pgmap v320: 97 pgs: 6 creating+peering, 91 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 740 B/s wr, 5 op/s 2024-03-28T18:08:25.735 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:25 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3221751120' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T18:08:25.735 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:25 smithi037 ceph-mon[31460]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T18:08:25.735 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:25 smithi037 ceph-mon[31460]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T18:08:25.735 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:25 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] up:active 2024-03-28T18:08:25.735 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:25 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2024-03-28T18:08:25.735 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:25 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:08:25.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:25 smithi059 ceph-mon[36190]: pgmap v320: 97 pgs: 6 creating+peering, 91 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 740 B/s wr, 5 op/s 2024-03-28T18:08:25.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:25 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3221751120' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T18:08:25.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:25 smithi059 ceph-mon[36190]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T18:08:25.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:25 smithi059 ceph-mon[36190]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T18:08:25.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:25 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] up:active 2024-03-28T18:08:25.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:25 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2024-03-28T18:08:25.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:25 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:08:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:25 smithi170 ceph-mon[36280]: pgmap v320: 97 pgs: 6 creating+peering, 91 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 740 B/s wr, 5 op/s 2024-03-28T18:08:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:25 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3221751120' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T18:08:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:25 smithi170 ceph-mon[36280]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T18:08:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:25 smithi170 ceph-mon[36280]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T18:08:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:25 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] up:active 2024-03-28T18:08:25.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:25 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2024-03-28T18:08:25.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:25 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:08:25.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.877+0000 7f531de00640 1 -- 172.21.15.37:0/360932136 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f5318061ae0 con 0x7f52fc0a2880 2024-03-28T18:08:25.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.906+0000 7f531ca00640 1 -- 172.21.15.37:0/360932136 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5318061ae0 con 0x7f52fc0a2880 2024-03-28T18:08:25.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.909+0000 7f531de00640 1 -- 172.21.15.37:0/360932136 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f52fc0a2880 msgr2=0x7f52fc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:25.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.909+0000 7f531de00640 1 --2- 172.21.15.37:0/360932136 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f52fc0a2880 0x7f52fc0a4d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5308009bd0 tx=0x7f5308005f90 comp rx=0 tx=0).stop 2024-03-28T18:08:25.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.909+0000 7f531de00640 1 -- 172.21.15.37:0/360932136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5318107380 msgr2=0x7f53181a1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:25.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.909+0000 7f531de00640 1 --2- 172.21.15.37:0/360932136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5318107380 0x7f53181a1160 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f531000eaa0 tx=0x7f531000ef60 comp rx=0 tx=0).stop 2024-03-28T18:08:25.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.909+0000 7f531de00640 1 -- 172.21.15.37:0/360932136 shutdown_connections 2024-03-28T18:08:25.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.909+0000 7f531de00640 1 --2- 172.21.15.37:0/360932136 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f52fc0a2880 0x7f52fc0a4d40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:25.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.909+0000 7f531de00640 1 --2- 172.21.15.37:0/360932136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5318107380 0x7f53181a1160 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:25.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.909+0000 7f531de00640 1 --2- 172.21.15.37:0/360932136 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5318104a50 0x7f531819a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:25.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.909+0000 7f531de00640 1 --2- 172.21.15.37:0/360932136 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5318102120 0x7f5318199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:25.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.909+0000 7f531de00640 1 -- 172.21.15.37:0/360932136 >> 172.21.15.37:0/360932136 conn(0x7f53180fbb10 msgr2=0x7f53181018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:25.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.909+0000 7f531de00640 1 -- 172.21.15.37:0/360932136 shutdown_connections 2024-03-28T18:08:25.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:25.909+0000 7f531de00640 1 -- 172.21.15.37:0/360932136 wait complete. 2024-03-28T18:08:25.925 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-28T18:08:25.985 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:08:25 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:08:25.886+0000 7f8fb7c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:08:25.986 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:08:25 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:08:25.886+0000 7f8fb7c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:08:25.986 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:08:25 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:08:25.886+0000 7f8fb7c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:08:25.986 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:08:25 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:08:25.886+0000 7f8fb7c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:08:25.986 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:08:25 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:08:25.886+0000 7f8fb7c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:08:25.986 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:08:25 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:08:25.733+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T18:08:25.986 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:08:25 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:08:25.733+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:08:25.986 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:08:25 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:08:25.733+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-28T18:08:26.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.119+0000 7fcee7800640 1 -- 172.21.15.37:0/1332557190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcee01028a0 msgr2=0x7fcee0102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:26.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.119+0000 7fcee7800640 1 --2- 172.21.15.37:0/1332557190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcee01028a0 0x7fcee0102d20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcecc009860 tx=0x7fcecc02bb70 comp rx=0 tx=0).stop 2024-03-28T18:08:26.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.119+0000 7fcee7800640 1 -- 172.21.15.37:0/1332557190 shutdown_connections 2024-03-28T18:08:26.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.119+0000 7fcee7800640 1 --2- 172.21.15.37:0/1332557190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcee01028a0 0x7fcee0102d20 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:26.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.119+0000 7fcee7800640 1 --2- 172.21.15.37:0/1332557190 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcee0103260 0x7fcee0109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:26.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.119+0000 7fcee7800640 1 --2- 172.21.15.37:0/1332557190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcee01016b0 0x7fcee0101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:26.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.119+0000 7fcee7800640 1 -- 172.21.15.37:0/1332557190 >> 172.21.15.37:0/1332557190 conn(0x7fcee00fcc20 msgr2=0x7fcee00ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:26.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.119+0000 7fcee7800640 1 -- 172.21.15.37:0/1332557190 shutdown_connections 2024-03-28T18:08:26.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.119+0000 7fcee7800640 1 -- 172.21.15.37:0/1332557190 wait complete. 2024-03-28T18:08:26.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.120+0000 7fcee7800640 1 Processor -- start 2024-03-28T18:08:26.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.120+0000 7fcee7800640 1 -- start start 2024-03-28T18:08:26.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.120+0000 7fcee7800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcee01016b0 0x7fcee0199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:26.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.120+0000 7fcee7800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcee01028a0 0x7fcee019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:26.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.120+0000 7fcee7800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcee0103260 0x7fcee01a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:26.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.120+0000 7fcee7800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee010d310 con 0x7fcee01028a0 2024-03-28T18:08:26.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.120+0000 7fcee7800640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee010d190 con 0x7fcee01016b0 2024-03-28T18:08:26.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.120+0000 7fcee7800640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee010d490 con 0x7fcee0103260 2024-03-28T18:08:26.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.121+0000 7fcee4e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcee01016b0 0x7fcee0199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:26.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.121+0000 7fcedfe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcee01028a0 0x7fcee019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:26.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.121+0000 7fcee5800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcee0103260 0x7fcee01a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:26.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.121+0000 7fcedfe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcee01028a0 0x7fcee019a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58930/0 (socket says 172.21.15.37:58930) 2024-03-28T18:08:26.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.121+0000 7fcedfe00640 1 -- 172.21.15.37:0/313933982 learned_addr learned my addr 172.21.15.37:0/313933982 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:26.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.121+0000 7fcee5800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcee0103260 0x7fcee01a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:33034/0 (socket says 172.21.15.37:33034) 2024-03-28T18:08:26.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.121+0000 7fcee5800640 1 -- 172.21.15.37:0/313933982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcee01016b0 msgr2=0x7fcee0199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:26.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.121+0000 7fcee5800640 1 --2- 172.21.15.37:0/313933982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcee01016b0 0x7fcee0199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:26.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.121+0000 7fcee5800640 1 -- 172.21.15.37:0/313933982 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcee01028a0 msgr2=0x7fcee019a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:26.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.121+0000 7fcee5800640 1 --2- 172.21.15.37:0/313933982 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcee01028a0 0x7fcee019a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:26.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.121+0000 7fcee5800640 1 -- 172.21.15.37:0/313933982 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee01a1810 con 0x7fcee0103260 2024-03-28T18:08:26.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.121+0000 7fcedfe00640 1 --2- 172.21.15.37:0/313933982 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcee01028a0 0x7fcee019a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:08:26.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.121+0000 7fcee5800640 1 --2- 172.21.15.37:0/313933982 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcee0103260 0x7fcee01a11a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fced8009fd0 tx=0x7fced800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:26.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.122+0000 7fcedd600640 1 -- 172.21.15.37:0/313933982 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced800ce20 con 0x7fcee0103260 2024-03-28T18:08:26.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.122+0000 7fcedd600640 1 -- 172.21.15.37:0/313933982 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fced8005020 con 0x7fcee0103260 2024-03-28T18:08:26.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.122+0000 7fcee7800640 1 -- 172.21.15.37:0/313933982 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee01a1b00 con 0x7fcee0103260 2024-03-28T18:08:26.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.122+0000 7fcedd600640 1 -- 172.21.15.37:0/313933982 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced8010720 con 0x7fcee0103260 2024-03-28T18:08:26.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.122+0000 7fcee7800640 1 -- 172.21.15.37:0/313933982 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee01a2050 con 0x7fcee0103260 2024-03-28T18:08:26.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.123+0000 7fcee7800640 1 -- 172.21.15.37:0/313933982 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee01a2280 con 0x7fcee0103260 2024-03-28T18:08:26.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.125+0000 7fcedd600640 1 -- 172.21.15.37:0/313933982 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fced8005540 con 0x7fcee0103260 2024-03-28T18:08:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.126+0000 7fcedd600640 1 --2- 172.21.15.37:0/313933982 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fcec80a2880 0x7fcec80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.126+0000 7fcedd600640 1 -- 172.21.15.37:0/313933982 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fced80ceff0 con 0x7fcee0103260 2024-03-28T18:08:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.126+0000 7fcee4e00640 1 --2- 172.21.15.37:0/313933982 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fcec80a2880 0x7fcec80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.127+0000 7fcee4e00640 1 --2- 172.21.15.37:0/313933982 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fcec80a2880 0x7fcec80a4d40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fced4009530 tx=0x7fced4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:26.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.130+0000 7fcedd600640 1 -- 172.21.15.37:0/313933982 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fced809c140 con 0x7fcee0103260 2024-03-28T18:08:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.355+0000 7fcee7800640 1 -- 172.21.15.37:0/313933982 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fcee0074ff0 con 0x7fcec80a2880 2024-03-28T18:08:26.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.395+0000 7fcedd600640 1 -- 172.21.15.37:0/313933982 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcee0074ff0 con 0x7fcec80a2880 2024-03-28T18:08:26.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.398+0000 7fcee7800640 1 -- 172.21.15.37:0/313933982 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fcec80a2880 msgr2=0x7fcec80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:26.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.398+0000 7fcee7800640 1 --2- 172.21.15.37:0/313933982 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fcec80a2880 0x7fcec80a4d40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fced4009530 tx=0x7fced4009260 comp rx=0 tx=0).stop 2024-03-28T18:08:26.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.398+0000 7fcee7800640 1 -- 172.21.15.37:0/313933982 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcee0103260 msgr2=0x7fcee01a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:26.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.398+0000 7fcee7800640 1 --2- 172.21.15.37:0/313933982 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcee0103260 0x7fcee01a11a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fced8009fd0 tx=0x7fced800ef10 comp rx=0 tx=0).stop 2024-03-28T18:08:26.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.398+0000 7fcee7800640 1 -- 172.21.15.37:0/313933982 shutdown_connections 2024-03-28T18:08:26.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.398+0000 7fcee7800640 1 --2- 172.21.15.37:0/313933982 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fcec80a2880 0x7fcec80a4d40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:26.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.398+0000 7fcee7800640 1 --2- 172.21.15.37:0/313933982 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcee0103260 0x7fcee01a11a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:26.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.398+0000 7fcee7800640 1 --2- 172.21.15.37:0/313933982 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcee01028a0 0x7fcee019a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:26.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.398+0000 7fcee7800640 1 --2- 172.21.15.37:0/313933982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcee01016b0 0x7fcee0199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:26.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.398+0000 7fcee7800640 1 -- 172.21.15.37:0/313933982 >> 172.21.15.37:0/313933982 conn(0x7fcee00fcc20 msgr2=0x7fcee00738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:26.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.398+0000 7fcee7800640 1 -- 172.21.15.37:0/313933982 shutdown_connections 2024-03-28T18:08:26.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.398+0000 7fcee7800640 1 -- 172.21.15.37:0/313933982 wait complete. 2024-03-28T18:08:26.415 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-28T18:08:26.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.629+0000 7fa3dca00640 1 -- 172.21.15.37:0/2609893703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3d81015a0 msgr2=0x7fa3d81019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:26.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.629+0000 7fa3dca00640 1 --2- 172.21.15.37:0/2609893703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3d81015a0 0x7fa3d81019c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009900 tx=0x7fa3c802bc40 comp rx=0 tx=0).stop 2024-03-28T18:08:26.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.629+0000 7fa3dca00640 1 -- 172.21.15.37:0/2609893703 shutdown_connections 2024-03-28T18:08:26.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.630+0000 7fa3dca00640 1 --2- 172.21.15.37:0/2609893703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3d8102790 0x7fa3d8102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:26.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.630+0000 7fa3dca00640 1 --2- 172.21.15.37:0/2609893703 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3d8103150 0x7fa3d81099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:26.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.630+0000 7fa3dca00640 1 --2- 172.21.15.37:0/2609893703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3d81015a0 0x7fa3d81019c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:26.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.630+0000 7fa3dca00640 1 -- 172.21.15.37:0/2609893703 >> 172.21.15.37:0/2609893703 conn(0x7fa3d80fcb10 msgr2=0x7fa3d80fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:26.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.630+0000 7fa3dca00640 1 -- 172.21.15.37:0/2609893703 shutdown_connections 2024-03-28T18:08:26.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.630+0000 7fa3dca00640 1 -- 172.21.15.37:0/2609893703 wait complete. 2024-03-28T18:08:26.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.631+0000 7fa3dca00640 1 Processor -- start 2024-03-28T18:08:26.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.631+0000 7fa3dca00640 1 -- start start 2024-03-28T18:08:26.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.631+0000 7fa3dca00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3d81015a0 0x7fa3d8075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:26.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.631+0000 7fa3dca00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3d8102790 0x7fa3d8073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:26.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.631+0000 7fa3dca00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3d8103150 0x7fa3d8073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:26.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.631+0000 7fa3dca00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d810d260 con 0x7fa3d81015a0 2024-03-28T18:08:26.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.631+0000 7fa3dca00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d810d0e0 con 0x7fa3d8102790 2024-03-28T18:08:26.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.631+0000 7fa3dca00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d810d3e0 con 0x7fa3d8103150 2024-03-28T18:08:26.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.632+0000 7fa3d6800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3d81015a0 0x7fa3d8075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:26.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.632+0000 7fa3d5e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3d8102790 0x7fa3d8073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:26.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.632+0000 7fa3d6800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3d81015a0 0x7fa3d8075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58950/0 (socket says 172.21.15.37:58950) 2024-03-28T18:08:26.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.632+0000 7fa3d6800640 1 -- 172.21.15.37:0/34410258 learned_addr learned my addr 172.21.15.37:0/34410258 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:26.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.632+0000 7fa3d7200640 1 --2- 172.21.15.37:0/34410258 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3d8103150 0x7fa3d8073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:26.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.632+0000 7fa3d5e00640 1 -- 172.21.15.37:0/34410258 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3d8103150 msgr2=0x7fa3d8073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:26.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.632+0000 7fa3d5e00640 1 --2- 172.21.15.37:0/34410258 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3d8103150 0x7fa3d8073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:26.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.632+0000 7fa3d5e00640 1 -- 172.21.15.37:0/34410258 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3d81015a0 msgr2=0x7fa3d8075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:26.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.632+0000 7fa3d5e00640 1 --2- 172.21.15.37:0/34410258 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3d81015a0 0x7fa3d8075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:26.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.632+0000 7fa3d5e00640 1 -- 172.21.15.37:0/34410258 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d8074860 con 0x7fa3d8102790 2024-03-28T18:08:26.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.632+0000 7fa3d6800640 1 --2- 172.21.15.37:0/34410258 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3d81015a0 0x7fa3d8075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:08:26.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.632+0000 7fa3d5e00640 1 --2- 172.21.15.37:0/34410258 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3d8102790 0x7fa3d8073a90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400e9f0 tx=0x7fa3c400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:26.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.633+0000 7fa3cf400640 1 -- 172.21.15.37:0/34410258 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c400ccc0 con 0x7fa3d8102790 2024-03-28T18:08:26.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.633+0000 7fa3cf400640 1 -- 172.21.15.37:0/34410258 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3c4005020 con 0x7fa3d8102790 2024-03-28T18:08:26.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.633+0000 7fa3cf400640 1 -- 172.21.15.37:0/34410258 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c40049e0 con 0x7fa3d8102790 2024-03-28T18:08:26.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.633+0000 7fa3dca00640 1 -- 172.21.15.37:0/34410258 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d81a19c0 con 0x7fa3d8102790 2024-03-28T18:08:26.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.633+0000 7fa3dca00640 1 -- 172.21.15.37:0/34410258 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d81a1ec0 con 0x7fa3d8102790 2024-03-28T18:08:26.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.634+0000 7fa3dca00640 1 -- 172.21.15.37:0/34410258 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d8108940 con 0x7fa3d8102790 2024-03-28T18:08:26.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.636+0000 7fa3cf400640 1 -- 172.21.15.37:0/34410258 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fa3c4005560 con 0x7fa3d8102790 2024-03-28T18:08:26.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.638+0000 7fa3cf400640 1 --2- 172.21.15.37:0/34410258 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa3bc0a2910 0x7fa3bc0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:26.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.638+0000 7fa3cf400640 1 -- 172.21.15.37:0/34410258 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa3c40cead0 con 0x7fa3d8102790 2024-03-28T18:08:26.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.638+0000 7fa3d6800640 1 --2- 172.21.15.37:0/34410258 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa3bc0a2910 0x7fa3bc0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:26.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.639+0000 7fa3d6800640 1 --2- 172.21.15.37:0/34410258 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa3bc0a2910 0x7fa3bc0a4dd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8005ab0 tx=0x7fa3c802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:26.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.642+0000 7fa3cf400640 1 -- 172.21.15.37:0/34410258 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa3c40051d0 con 0x7fa3d8102790 2024-03-28T18:08:26.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:26 smithi037 ceph-mon[31460]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:08:26.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.878+0000 7fa3dca00640 1 -- 172.21.15.37:0/34410258 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3d8061ae0 con 0x7fa3bc0a2910 2024-03-28T18:08:26.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.888+0000 7fa3cf400640 1 -- 172.21.15.37:0/34410258 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fa3d8061ae0 con 0x7fa3bc0a2910 2024-03-28T18:08:26.888 INFO:teuthology.orchestra.run.smithi037.stdout:/volumes/qa/sv_0/1d78198f-2ae2-4a2d-90e1-aa543b21a61d 2024-03-28T18:08:26.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.892+0000 7fa3dca00640 1 -- 172.21.15.37:0/34410258 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa3bc0a2910 msgr2=0x7fa3bc0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:26.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.892+0000 7fa3dca00640 1 --2- 172.21.15.37:0/34410258 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa3bc0a2910 0x7fa3bc0a4dd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8005ab0 tx=0x7fa3c802cf90 comp rx=0 tx=0).stop 2024-03-28T18:08:26.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.892+0000 7fa3dca00640 1 -- 172.21.15.37:0/34410258 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3d8102790 msgr2=0x7fa3d8073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:26.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.892+0000 7fa3dca00640 1 --2- 172.21.15.37:0/34410258 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3d8102790 0x7fa3d8073a90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400e9f0 tx=0x7fa3c400eeb0 comp rx=0 tx=0).stop 2024-03-28T18:08:26.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.892+0000 7fa3dca00640 1 -- 172.21.15.37:0/34410258 shutdown_connections 2024-03-28T18:08:26.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.892+0000 7fa3dca00640 1 --2- 172.21.15.37:0/34410258 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa3bc0a2910 0x7fa3bc0a4dd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:26.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.892+0000 7fa3dca00640 1 --2- 172.21.15.37:0/34410258 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3d8102790 0x7fa3d8073a90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:26.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.892+0000 7fa3dca00640 1 --2- 172.21.15.37:0/34410258 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3d8103150 0x7fa3d8073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.892+0000 7fa3dca00640 1 --2- 172.21.15.37:0/34410258 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3d81015a0 0x7fa3d8075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.892+0000 7fa3dca00640 1 -- 172.21.15.37:0/34410258 >> 172.21.15.37:0/34410258 conn(0x7fa3d80fcb10 msgr2=0x7fa3d8105b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.892+0000 7fa3dca00640 1 -- 172.21.15.37:0/34410258 shutdown_connections 2024-03-28T18:08:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:26.892+0000 7fa3dca00640 1 -- 172.21.15.37:0/34410258 wait complete. 2024-03-28T18:08:26.916 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-28T18:08:27.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:26 smithi059 ceph-mon[36190]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:08:27.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:26 smithi170 ceph-mon[36280]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:08:27.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.163+0000 7f94a4a00640 1 -- 172.21.15.37:0/2144493593 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949402cc80 con 0x7f94a8102790 2024-03-28T18:08:27.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.164+0000 7f94a7200640 1 -- 172.21.15.37:0/2144493593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94a8102790 msgr2=0x7f94a8102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:27.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.164+0000 7f94a7200640 1 --2- 172.21.15.37:0/2144493593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94a8102790 0x7f94a8102c10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9494009900 tx=0x7f949402bc30 comp rx=0 tx=0).stop 2024-03-28T18:08:27.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.164+0000 7f94a7200640 1 -- 172.21.15.37:0/2144493593 shutdown_connections 2024-03-28T18:08:27.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.164+0000 7f94a7200640 1 --2- 172.21.15.37:0/2144493593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94a8103150 0x7f94a81099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:27.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.164+0000 7f94a7200640 1 --2- 172.21.15.37:0/2144493593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94a8102790 0x7f94a8102c10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:27.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.164+0000 7f94a7200640 1 --2- 172.21.15.37:0/2144493593 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94a81015a0 0x7f94a81019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:27.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.164+0000 7f94a7200640 1 -- 172.21.15.37:0/2144493593 >> 172.21.15.37:0/2144493593 conn(0x7f94a80fcb10 msgr2=0x7f94a80fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:27.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.164+0000 7f94a7200640 1 -- 172.21.15.37:0/2144493593 shutdown_connections 2024-03-28T18:08:27.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.164+0000 7f94a7200640 1 -- 172.21.15.37:0/2144493593 wait complete. 2024-03-28T18:08:27.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.166+0000 7f94a7200640 1 Processor -- start 2024-03-28T18:08:27.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.166+0000 7f94a7200640 1 -- start start 2024-03-28T18:08:27.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.167+0000 7f94a7200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94a81015a0 0x7f94a8075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:27.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.167+0000 7f94a7200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94a8102790 0x7f94a8073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:27.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.167+0000 7f94a7200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94a8103150 0x7f94a8073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:27.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.167+0000 7f94a7200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a810d200 con 0x7f94a8103150 2024-03-28T18:08:27.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.167+0000 7f94a7200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a810d080 con 0x7f94a81015a0 2024-03-28T18:08:27.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.167+0000 7f94a7200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a810d380 con 0x7f94a8102790 2024-03-28T18:08:27.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.167+0000 7f94a5e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94a81015a0 0x7f94a8075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:27.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.167+0000 7f94a5e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94a81015a0 0x7f94a8075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:57904/0 (socket says 172.21.15.37:57904) 2024-03-28T18:08:27.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.167+0000 7f94a5e00640 1 -- 172.21.15.37:0/2184277533 learned_addr learned my addr 172.21.15.37:0/2184277533 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:27.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.167+0000 7f94a5400640 1 --2- 172.21.15.37:0/2184277533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94a8102790 0x7f94a8073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:27.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.167+0000 7f94a6800640 1 --2- 172.21.15.37:0/2184277533 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94a8103150 0x7f94a8073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:27.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.167+0000 7f94a5e00640 1 -- 172.21.15.37:0/2184277533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94a8102790 msgr2=0x7f94a8073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:27.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.167+0000 7f94a5e00640 1 --2- 172.21.15.37:0/2184277533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94a8102790 0x7f94a8073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:27.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.167+0000 7f94a5e00640 1 -- 172.21.15.37:0/2184277533 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94a8103150 msgr2=0x7f94a8073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:27.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.167+0000 7f94a5e00640 1 --2- 172.21.15.37:0/2184277533 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94a8103150 0x7f94a8073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:27.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.167+0000 7f94a5e00640 1 -- 172.21.15.37:0/2184277533 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a8074860 con 0x7f94a81015a0 2024-03-28T18:08:27.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.167+0000 7f94a6800640 1 --2- 172.21.15.37:0/2184277533 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94a8103150 0x7f94a8073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:08:27.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.168+0000 7f94a5e00640 1 --2- 172.21.15.37:0/2184277533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94a81015a0 0x7f94a8075450 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f949800e9c0 tx=0x7f949800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:27.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.168+0000 7f949ea00640 1 -- 172.21.15.37:0/2184277533 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949800cd80 con 0x7f94a81015a0 2024-03-28T18:08:27.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.168+0000 7f949ea00640 1 -- 172.21.15.37:0/2184277533 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94980035f0 con 0x7f94a81015a0 2024-03-28T18:08:27.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.168+0000 7f949ea00640 1 -- 172.21.15.37:0/2184277533 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9498010690 con 0x7f94a81015a0 2024-03-28T18:08:27.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.168+0000 7f94a7200640 1 -- 172.21.15.37:0/2184277533 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a81a1a10 con 0x7f94a81015a0 2024-03-28T18:08:27.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.168+0000 7f94a7200640 1 -- 172.21.15.37:0/2184277533 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a81a1ee0 con 0x7f94a81015a0 2024-03-28T18:08:27.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.169+0000 7f94a7200640 1 -- 172.21.15.37:0/2184277533 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a80683b0 con 0x7f94a81015a0 2024-03-28T18:08:27.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.171+0000 7f949ea00640 1 -- 172.21.15.37:0/2184277533 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f94980037a0 con 0x7f94a81015a0 2024-03-28T18:08:27.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.172+0000 7f949ea00640 1 --2- 172.21.15.37:0/2184277533 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f948c0a2910 0x7f948c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:27.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.172+0000 7f949ea00640 1 -- 172.21.15.37:0/2184277533 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f94980cef80 con 0x7f94a81015a0 2024-03-28T18:08:27.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.173+0000 7f94a5400640 1 --2- 172.21.15.37:0/2184277533 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f948c0a2910 0x7f948c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:27.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.173+0000 7f94a5400640 1 --2- 172.21.15.37:0/2184277533 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f948c0a2910 0x7f948c0a4dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f949402c7b0 tx=0x7f9494005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:27.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.177+0000 7f949ea00640 1 -- 172.21.15.37:0/2184277533 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f949809bf70 con 0x7f94a81015a0 2024-03-28T18:08:27.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.526+0000 7f94a7200640 1 -- 172.21.15.37:0/2184277533 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f94a8061c30 con 0x7f948c0a2910 2024-03-28T18:08:27.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.573+0000 7f949ea00640 1 -- 172.21.15.37:0/2184277533 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f94a8061c30 con 0x7f948c0a2910 2024-03-28T18:08:27.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.575+0000 7f94a7200640 1 -- 172.21.15.37:0/2184277533 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f948c0a2910 msgr2=0x7f948c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:27.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.575+0000 7f94a7200640 1 --2- 172.21.15.37:0/2184277533 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f948c0a2910 0x7f948c0a4dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f949402c7b0 tx=0x7f9494005fd0 comp rx=0 tx=0).stop 2024-03-28T18:08:27.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.575+0000 7f94a7200640 1 -- 172.21.15.37:0/2184277533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94a81015a0 msgr2=0x7f94a8075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:27.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.575+0000 7f94a7200640 1 --2- 172.21.15.37:0/2184277533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94a81015a0 0x7f94a8075450 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f949800e9c0 tx=0x7f949800ee80 comp rx=0 tx=0).stop 2024-03-28T18:08:27.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.576+0000 7f94a7200640 1 -- 172.21.15.37:0/2184277533 shutdown_connections 2024-03-28T18:08:27.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.576+0000 7f94a7200640 1 --2- 172.21.15.37:0/2184277533 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f948c0a2910 0x7f948c0a4dd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:27.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.576+0000 7f94a7200640 1 --2- 172.21.15.37:0/2184277533 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94a8103150 0x7f94a8073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:27.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.576+0000 7f94a7200640 1 --2- 172.21.15.37:0/2184277533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94a8102790 0x7f94a8073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:27.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.576+0000 7f94a7200640 1 --2- 172.21.15.37:0/2184277533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94a81015a0 0x7f94a8075450 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:27.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.576+0000 7f94a7200640 1 -- 172.21.15.37:0/2184277533 >> 172.21.15.37:0/2184277533 conn(0x7f94a80fcb10 msgr2=0x7f94a8105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:27.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.576+0000 7f94a7200640 1 -- 172.21.15.37:0/2184277533 shutdown_connections 2024-03-28T18:08:27.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.576+0000 7f94a7200640 1 -- 172.21.15.37:0/2184277533 wait complete. 2024-03-28T18:08:27.598 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-28T18:08:27.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.819+0000 7fab89200640 1 -- 172.21.15.37:0/1701845452 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab84102120 msgr2=0x7fab84104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:27.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.819+0000 7fab89200640 1 --2- 172.21.15.37:0/1701845452 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab84102120 0x7fab84104510 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fab74009900 tx=0x7fab7402bc30 comp rx=0 tx=0).stop 2024-03-28T18:08:27.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.819+0000 7fab89200640 1 -- 172.21.15.37:0/1701845452 shutdown_connections 2024-03-28T18:08:27.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.819+0000 7fab89200640 1 --2- 172.21.15.37:0/1701845452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab84107380 0x7fab841097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:27.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.819+0000 7fab89200640 1 --2- 172.21.15.37:0/1701845452 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab84104a50 0x7fab84106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:27.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.819+0000 7fab89200640 1 --2- 172.21.15.37:0/1701845452 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab84102120 0x7fab84104510 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:27.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.819+0000 7fab89200640 1 -- 172.21.15.37:0/1701845452 >> 172.21.15.37:0/1701845452 conn(0x7fab840fbb30 msgr2=0x7fab840fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:27.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.819+0000 7fab89200640 1 -- 172.21.15.37:0/1701845452 shutdown_connections 2024-03-28T18:08:27.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.819+0000 7fab89200640 1 -- 172.21.15.37:0/1701845452 wait complete. 2024-03-28T18:08:27.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.820+0000 7fab89200640 1 Processor -- start 2024-03-28T18:08:27.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.820+0000 7fab89200640 1 -- start start 2024-03-28T18:08:27.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.820+0000 7fab89200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab84102120 0x7fab84199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:27.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.820+0000 7fab89200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab84104a50 0x7fab8419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:27.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.820+0000 7fab89200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab84107380 0x7fab841a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:27.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.820+0000 7fab89200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8410d380 con 0x7fab84104a50 2024-03-28T18:08:27.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.820+0000 7fab89200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8410d200 con 0x7fab84107380 2024-03-28T18:08:27.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.820+0000 7fab89200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8410d500 con 0x7fab84102120 2024-03-28T18:08:27.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.820+0000 7fab81e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab84102120 0x7fab84199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:27.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.821+0000 7fab81e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab84102120 0x7fab84199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:33074/0 (socket says 172.21.15.37:33074) 2024-03-28T18:08:27.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.821+0000 7fab81e00640 1 -- 172.21.15.37:0/3807388471 learned_addr learned my addr 172.21.15.37:0/3807388471 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:27.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.821+0000 7fab82800640 1 --2- 172.21.15.37:0/3807388471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab84107380 0x7fab841a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:27.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.821+0000 7fab81400640 1 --2- 172.21.15.37:0/3807388471 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab84104a50 0x7fab8419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:27.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.821+0000 7fab81e00640 1 -- 172.21.15.37:0/3807388471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab84107380 msgr2=0x7fab841a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:27.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.821+0000 7fab81e00640 1 --2- 172.21.15.37:0/3807388471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab84107380 0x7fab841a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:27.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.821+0000 7fab81e00640 1 -- 172.21.15.37:0/3807388471 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab84104a50 msgr2=0x7fab8419a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:27.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.821+0000 7fab81e00640 1 --2- 172.21.15.37:0/3807388471 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab84104a50 0x7fab8419a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:27.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.821+0000 7fab81e00640 1 -- 172.21.15.37:0/3807388471 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab841a1880 con 0x7fab84102120 2024-03-28T18:08:27.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.821+0000 7fab81e00640 1 --2- 172.21.15.37:0/3807388471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab84102120 0x7fab84199c70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fab74005ab0 tx=0x7fab7402c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:27.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.822+0000 7fab7aa00640 1 -- 172.21.15.37:0/3807388471 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab74043020 con 0x7fab84102120 2024-03-28T18:08:27.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.822+0000 7fab7aa00640 1 -- 172.21.15.37:0/3807388471 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab740054e0 con 0x7fab84102120 2024-03-28T18:08:27.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.822+0000 7fab89200640 1 -- 172.21.15.37:0/3807388471 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab841a1b10 con 0x7fab84102120 2024-03-28T18:08:27.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.822+0000 7fab7aa00640 1 -- 172.21.15.37:0/3807388471 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab74003ce0 con 0x7fab84102120 2024-03-28T18:08:27.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.822+0000 7fab89200640 1 -- 172.21.15.37:0/3807388471 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab841a20b0 con 0x7fab84102120 2024-03-28T18:08:27.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.823+0000 7fab89200640 1 -- 172.21.15.37:0/3807388471 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab84108ed0 con 0x7fab84102120 2024-03-28T18:08:27.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.824+0000 7fab7aa00640 1 -- 172.21.15.37:0/3807388471 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fab74046020 con 0x7fab84102120 2024-03-28T18:08:27.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.826+0000 7fab7aa00640 1 --2- 172.21.15.37:0/3807388471 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fab680a2910 0x7fab680a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:27.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.826+0000 7fab7aa00640 1 -- 172.21.15.37:0/3807388471 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fab740f0b60 con 0x7fab84102120 2024-03-28T18:08:27.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.826+0000 7fab81400640 1 --2- 172.21.15.37:0/3807388471 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fab680a2910 0x7fab680a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:27.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.827+0000 7fab81400640 1 --2- 172.21.15.37:0/3807388471 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fab680a2910 0x7fab680a4dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fab84069360 tx=0x7fab70005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:27.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:27.830+0000 7fab7aa00640 1 -- 172.21.15.37:0/3807388471 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fab7404d020 con 0x7fab84102120 2024-03-28T18:08:27.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:27 smithi037 ceph-mon[31460]: pgmap v321: 97 pgs: 97 active+clean; 581 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 631 B/s rd, 4.3 KiB/s wr, 14 op/s 2024-03-28T18:08:27.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:27 smithi037 ceph-mon[31460]: from='client.24691 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:08:27.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:27 smithi037 ceph-mon[31460]: mgrmap e17: x(active, since 8m), standbys: y, z 2024-03-28T18:08:27.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:27 smithi037 ceph-mon[31460]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:08:28.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:27 smithi059 ceph-mon[36190]: pgmap v321: 97 pgs: 97 active+clean; 581 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 631 B/s rd, 4.3 KiB/s wr, 14 op/s 2024-03-28T18:08:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:27 smithi059 ceph-mon[36190]: from='client.24691 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:08:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:27 smithi059 ceph-mon[36190]: mgrmap e17: x(active, since 8m), standbys: y, z 2024-03-28T18:08:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:27 smithi059 ceph-mon[36190]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:08:28.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:27 smithi170 ceph-mon[36280]: pgmap v321: 97 pgs: 97 active+clean; 581 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 631 B/s rd, 4.3 KiB/s wr, 14 op/s 2024-03-28T18:08:28.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:27 smithi170 ceph-mon[36280]: from='client.24691 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:08:28.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:27 smithi170 ceph-mon[36280]: mgrmap e17: x(active, since 8m), standbys: y, z 2024-03-28T18:08:28.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:27 smithi170 ceph-mon[36280]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:08:28.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.103+0000 7fab89200640 1 -- 172.21.15.37:0/3807388471 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fab84061b70 con 0x7fab680a2910 2024-03-28T18:08:28.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.113+0000 7fab7aa00640 1 -- 172.21.15.37:0/3807388471 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fab84061b70 con 0x7fab680a2910 2024-03-28T18:08:28.112 INFO:teuthology.orchestra.run.smithi037.stdout:/volumes/qa/sv_1/6fbe70b0-f754-450d-9fbd-670541c15017 2024-03-28T18:08:28.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.115+0000 7fab89200640 1 -- 172.21.15.37:0/3807388471 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fab680a2910 msgr2=0x7fab680a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:28.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.115+0000 7fab89200640 1 --2- 172.21.15.37:0/3807388471 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fab680a2910 0x7fab680a4dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fab84069360 tx=0x7fab70005f90 comp rx=0 tx=0).stop 2024-03-28T18:08:28.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.115+0000 7fab89200640 1 -- 172.21.15.37:0/3807388471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab84102120 msgr2=0x7fab84199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:28.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.115+0000 7fab89200640 1 --2- 172.21.15.37:0/3807388471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab84102120 0x7fab84199c70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fab74005ab0 tx=0x7fab7402c6c0 comp rx=0 tx=0).stop 2024-03-28T18:08:28.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.116+0000 7fab89200640 1 -- 172.21.15.37:0/3807388471 shutdown_connections 2024-03-28T18:08:28.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.116+0000 7fab89200640 1 --2- 172.21.15.37:0/3807388471 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fab680a2910 0x7fab680a4dd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:28.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.116+0000 7fab89200640 1 --2- 172.21.15.37:0/3807388471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab84107380 0x7fab841a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:28.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.116+0000 7fab89200640 1 --2- 172.21.15.37:0/3807388471 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab84104a50 0x7fab8419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:28.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.116+0000 7fab89200640 1 --2- 172.21.15.37:0/3807388471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab84102120 0x7fab84199c70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:28.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.116+0000 7fab89200640 1 -- 172.21.15.37:0/3807388471 >> 172.21.15.37:0/3807388471 conn(0x7fab840fbb30 msgr2=0x7fab84101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:28.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.116+0000 7fab89200640 1 -- 172.21.15.37:0/3807388471 shutdown_connections 2024-03-28T18:08:28.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.116+0000 7fab89200640 1 -- 172.21.15.37:0/3807388471 wait complete. 2024-03-28T18:08:28.136 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:08:28.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.353+0000 7f6606200640 1 -- 172.21.15.37:0/1619483561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66001038e0 msgr2=0x7f6600103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:28.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.353+0000 7f6606200640 1 --2- 172.21.15.37:0/1619483561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66001038e0 0x7f6600103d00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f65f0009ca0 tx=0x7f65f002c100 comp rx=0 tx=0).stop 2024-03-28T18:08:28.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.353+0000 7f6606200640 1 -- 172.21.15.37:0/1619483561 shutdown_connections 2024-03-28T18:08:28.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.353+0000 7f6606200640 1 --2- 172.21.15.37:0/1619483561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6600105490 0x7f660010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:28.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.353+0000 7f6606200640 1 --2- 172.21.15.37:0/1619483561 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6600104ad0 0x7f6600104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:28.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.353+0000 7f6606200640 1 --2- 172.21.15.37:0/1619483561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66001038e0 0x7f6600103d00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:28.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.353+0000 7f6606200640 1 -- 172.21.15.37:0/1619483561 >> 172.21.15.37:0/1619483561 conn(0x7f66000fee90 msgr2=0x7f66001012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:28.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.353+0000 7f6606200640 1 -- 172.21.15.37:0/1619483561 shutdown_connections 2024-03-28T18:08:28.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.353+0000 7f6606200640 1 -- 172.21.15.37:0/1619483561 wait complete. 2024-03-28T18:08:28.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.354+0000 7f6606200640 1 Processor -- start 2024-03-28T18:08:28.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.354+0000 7f6606200640 1 -- start start 2024-03-28T18:08:28.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.355+0000 7f6606200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66001038e0 0x7f6600199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:28.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.355+0000 7f6606200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6600104ad0 0x7f660019a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:28.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.355+0000 7f6606200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6600105490 0x7f66001a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:28.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.355+0000 7f6606200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660010f530 con 0x7f66001038e0 2024-03-28T18:08:28.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.355+0000 7f6606200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660010f3b0 con 0x7f6600105490 2024-03-28T18:08:28.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.355+0000 7f6606200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660010f6b0 con 0x7f6600104ad0 2024-03-28T18:08:28.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.355+0000 7f65ff400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66001038e0 0x7f6600199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:28.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.355+0000 7f65f6a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6600104ad0 0x7f660019a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:28.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.355+0000 7f65ff400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66001038e0 0x7f6600199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:59026/0 (socket says 172.21.15.37:59026) 2024-03-28T18:08:28.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.355+0000 7f65ff400640 1 -- 172.21.15.37:0/163413978 learned_addr learned my addr 172.21.15.37:0/163413978 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:28.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.355+0000 7f65ffe00640 1 --2- 172.21.15.37:0/163413978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6600105490 0x7f66001a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:28.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.356+0000 7f65ffe00640 1 -- 172.21.15.37:0/163413978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6600104ad0 msgr2=0x7f660019a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:28.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.356+0000 7f65ffe00640 1 --2- 172.21.15.37:0/163413978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6600104ad0 0x7f660019a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:28.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.356+0000 7f65ffe00640 1 -- 172.21.15.37:0/163413978 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66001038e0 msgr2=0x7f6600199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:28.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.356+0000 7f65ffe00640 1 --2- 172.21.15.37:0/163413978 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66001038e0 0x7f6600199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:28.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.356+0000 7f65ffe00640 1 -- 172.21.15.37:0/163413978 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66001a1800 con 0x7f6600105490 2024-03-28T18:08:28.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.356+0000 7f65ff400640 1 --2- 172.21.15.37:0/163413978 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66001038e0 0x7f6600199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:08:28.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.356+0000 7f65ffe00640 1 --2- 172.21.15.37:0/163413978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6600105490 0x7f66001a1190 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f65f800e970 tx=0x7f65f800ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:28.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.356+0000 7f65fcc00640 1 -- 172.21.15.37:0/163413978 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f800ccc0 con 0x7f6600105490 2024-03-28T18:08:28.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.356+0000 7f65fcc00640 1 -- 172.21.15.37:0/163413978 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65f8005020 con 0x7f6600105490 2024-03-28T18:08:28.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.356+0000 7f6606200640 1 -- 172.21.15.37:0/163413978 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66001a1af0 con 0x7f6600105490 2024-03-28T18:08:28.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.356+0000 7f65fcc00640 1 -- 172.21.15.37:0/163413978 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f8004a00 con 0x7f6600105490 2024-03-28T18:08:28.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.356+0000 7f6606200640 1 -- 172.21.15.37:0/163413978 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66001a1ff0 con 0x7f6600105490 2024-03-28T18:08:28.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.357+0000 7f6606200640 1 -- 172.21.15.37:0/163413978 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f660010ac80 con 0x7f6600105490 2024-03-28T18:08:28.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.359+0000 7f65fcc00640 1 -- 172.21.15.37:0/163413978 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f65f80051d0 con 0x7f6600105490 2024-03-28T18:08:28.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.361+0000 7f65fcc00640 1 --2- 172.21.15.37:0/163413978 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f65e40a2890 0x7f65e40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:28.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.361+0000 7f65fcc00640 1 -- 172.21.15.37:0/163413978 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f65f80ceab0 con 0x7f6600105490 2024-03-28T18:08:28.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.361+0000 7f65ff400640 1 --2- 172.21.15.37:0/163413978 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f65e40a2890 0x7f65e40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:28.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.362+0000 7f65ff400640 1 --2- 172.21.15.37:0/163413978 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f65e40a2890 0x7f65e40a4d50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f65f0005e30 tx=0x7f65f0004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:28.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.370+0000 7f65fcc00640 1 -- 172.21.15.37:0/163413978 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f65f809baa0 con 0x7f6600105490 2024-03-28T18:08:28.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.626+0000 7f6606200640 1 -- 172.21.15.37:0/163413978 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f660002d070 con 0x7f6600105490 2024-03-28T18:08:28.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.628+0000 7f65fcc00640 1 -- 172.21.15.37:0/163413978 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11383 (secure 0 0 0) 0x7f65f809b3a0 con 0x7f6600105490 2024-03-28T18:08:28.628 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:08:28.628 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1926270371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6838","nonce":1926270371},{"type":"v1","addr":"172.21.15.37:6839","nonce":1926270371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:08:19.968894+0000","modified":"2024-03-28T18:08:25.415653+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24580,"mds_1":24569,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6837/4034607523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":4034607523},{"type":"v1","addr":"172.21.15.59:6837","nonce":4034607523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6839/1575518279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":1575518279},{"type":"v1","addr":"172.21.15.170:6839","nonce":1575518279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6839/2091629078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":2091629078},{"type":"v1","addr":"172.21.15.59:6839","nonce":2091629078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24580":{"gid":24580,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.37:6841/1923263622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6840","nonce":1923263622},{"type":"v1","addr":"172.21.15.37:6841","nonce":1923263622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24580,"qdb_cluster":[24550,24569,24580]},"id":1}]} 2024-03-28T18:08:28.628 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 22 2024-03-28T18:08:28.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.630+0000 7f6606200640 1 -- 172.21.15.37:0/163413978 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f65e40a2890 msgr2=0x7f65e40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:28.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.630+0000 7f6606200640 1 --2- 172.21.15.37:0/163413978 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f65e40a2890 0x7f65e40a4d50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f65f0005e30 tx=0x7f65f0004a40 comp rx=0 tx=0).stop 2024-03-28T18:08:28.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.630+0000 7f6606200640 1 -- 172.21.15.37:0/163413978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6600105490 msgr2=0x7f66001a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:28.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.630+0000 7f6606200640 1 --2- 172.21.15.37:0/163413978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6600105490 0x7f66001a1190 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f65f800e970 tx=0x7f65f800ee30 comp rx=0 tx=0).stop 2024-03-28T18:08:28.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.631+0000 7f6606200640 1 -- 172.21.15.37:0/163413978 shutdown_connections 2024-03-28T18:08:28.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.631+0000 7f6606200640 1 --2- 172.21.15.37:0/163413978 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f65e40a2890 0x7f65e40a4d50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:28.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.631+0000 7f6606200640 1 --2- 172.21.15.37:0/163413978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6600105490 0x7f66001a1190 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:28.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.631+0000 7f6606200640 1 --2- 172.21.15.37:0/163413978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6600104ad0 0x7f660019a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:28.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.631+0000 7f6606200640 1 --2- 172.21.15.37:0/163413978 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66001038e0 0x7f6600199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:28.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.631+0000 7f6606200640 1 -- 172.21.15.37:0/163413978 >> 172.21.15.37:0/163413978 conn(0x7f66000fee90 msgr2=0x7f6600107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:28.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.631+0000 7f6606200640 1 -- 172.21.15.37:0/163413978 shutdown_connections 2024-03-28T18:08:28.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.631+0000 7f6606200640 1 -- 172.21.15.37:0/163413978 wait complete. 2024-03-28T18:08:28.650 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:08:28.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:28 smithi037 ceph-mon[31460]: from='client.24677 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:08:28.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.882+0000 7fa2eb800640 1 -- 172.21.15.37:0/1061814907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2e4104ad0 msgr2=0x7fa2e4104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:28.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.882+0000 7fa2eb800640 1 --2- 172.21.15.37:0/1061814907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2e4104ad0 0x7fa2e4104f50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0009900 tx=0x7fa2d002bc50 comp rx=0 tx=0).stop 2024-03-28T18:08:28.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.882+0000 7fa2eb800640 1 -- 172.21.15.37:0/1061814907 shutdown_connections 2024-03-28T18:08:28.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.882+0000 7fa2eb800640 1 --2- 172.21.15.37:0/1061814907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e4105490 0x7fa2e410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:28.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.882+0000 7fa2eb800640 1 --2- 172.21.15.37:0/1061814907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2e4104ad0 0x7fa2e4104f50 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:28.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.882+0000 7fa2eb800640 1 --2- 172.21.15.37:0/1061814907 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2e41038e0 0x7fa2e4103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:28.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.882+0000 7fa2eb800640 1 -- 172.21.15.37:0/1061814907 >> 172.21.15.37:0/1061814907 conn(0x7fa2e40fee90 msgr2=0x7fa2e41012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:28.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.882+0000 7fa2eb800640 1 -- 172.21.15.37:0/1061814907 shutdown_connections 2024-03-28T18:08:28.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.882+0000 7fa2eb800640 1 -- 172.21.15.37:0/1061814907 wait complete. 2024-03-28T18:08:28.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.883+0000 7fa2eb800640 1 Processor -- start 2024-03-28T18:08:28.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.883+0000 7fa2eb800640 1 -- start start 2024-03-28T18:08:28.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.883+0000 7fa2eb800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2e41038e0 0x7fa2e4199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:28.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.883+0000 7fa2eb800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e4104ad0 0x7fa2e419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:28.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.883+0000 7fa2eb800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2e4105490 0x7fa2e41a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:28.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.883+0000 7fa2eb800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e410f590 con 0x7fa2e4105490 2024-03-28T18:08:28.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.883+0000 7fa2eb800640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e410f410 con 0x7fa2e4104ad0 2024-03-28T18:08:28.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.883+0000 7fa2eb800640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e410f710 con 0x7fa2e41038e0 2024-03-28T18:08:28.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.884+0000 7fa2e8e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2e41038e0 0x7fa2e4199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:28.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.884+0000 7fa2e3e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e4104ad0 0x7fa2e419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:28.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.884+0000 7fa2e9800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2e4105490 0x7fa2e41a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:28.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.884+0000 7fa2e8e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2e41038e0 0x7fa2e4199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:33118/0 (socket says 172.21.15.37:33118) 2024-03-28T18:08:28.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.884+0000 7fa2e8e00640 1 -- 172.21.15.37:0/1826529871 learned_addr learned my addr 172.21.15.37:0/1826529871 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:28.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.884+0000 7fa2e3e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e4104ad0 0x7fa2e419a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:57946/0 (socket says 172.21.15.37:57946) 2024-03-28T18:08:28.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.884+0000 7fa2e8e00640 1 -- 172.21.15.37:0/1826529871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e4104ad0 msgr2=0x7fa2e419a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:28.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.884+0000 7fa2e8e00640 1 --2- 172.21.15.37:0/1826529871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e4104ad0 0x7fa2e419a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:28.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.884+0000 7fa2e8e00640 1 -- 172.21.15.37:0/1826529871 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2e4105490 msgr2=0x7fa2e41a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:28.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.884+0000 7fa2e8e00640 1 --2- 172.21.15.37:0/1826529871 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2e4105490 0x7fa2e41a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:28.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.884+0000 7fa2e8e00640 1 -- 172.21.15.37:0/1826529871 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e41a1800 con 0x7fa2e41038e0 2024-03-28T18:08:28.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.884+0000 7fa2e3e00640 1 --2- 172.21.15.37:0/1826529871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e4104ad0 0x7fa2e419a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:08:28.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.884+0000 7fa2e9800640 1 --2- 172.21.15.37:0/1826529871 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2e4105490 0x7fa2e41a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:08:28.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.885+0000 7fa2e8e00640 1 --2- 172.21.15.37:0/1826529871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2e41038e0 0x7fa2e4199be0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa2d800e9c0 tx=0x7fa2d800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:28.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.885+0000 7fa2e1600640 1 -- 172.21.15.37:0/1826529871 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d800cde0 con 0x7fa2e41038e0 2024-03-28T18:08:28.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.885+0000 7fa2e1600640 1 -- 172.21.15.37:0/1826529871 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2d8010910 con 0x7fa2e41038e0 2024-03-28T18:08:28.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.885+0000 7fa2e1600640 1 -- 172.21.15.37:0/1826529871 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8018a00 con 0x7fa2e41038e0 2024-03-28T18:08:28.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.885+0000 7fa2eb800640 1 -- 172.21.15.37:0/1826529871 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e41a1af0 con 0x7fa2e41038e0 2024-03-28T18:08:28.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.885+0000 7fa2eb800640 1 -- 172.21.15.37:0/1826529871 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e41a2060 con 0x7fa2e41038e0 2024-03-28T18:08:28.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.886+0000 7fa2eb800640 1 -- 172.21.15.37:0/1826529871 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e40688e0 con 0x7fa2e41038e0 2024-03-28T18:08:28.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.889+0000 7fa2e1600640 1 -- 172.21.15.37:0/1826529871 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fa2d801f020 con 0x7fa2e41038e0 2024-03-28T18:08:28.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.890+0000 7fa2e1600640 1 --2- 172.21.15.37:0/1826529871 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa2cc0a2910 0x7fa2cc0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:28.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.891+0000 7fa2e1600640 1 -- 172.21.15.37:0/1826529871 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa2d80cec40 con 0x7fa2e41038e0 2024-03-28T18:08:28.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.891+0000 7fa2e3e00640 1 --2- 172.21.15.37:0/1826529871 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa2cc0a2910 0x7fa2cc0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:28.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.891+0000 7fa2e3e00640 1 --2- 172.21.15.37:0/1826529871 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa2cc0a2910 0x7fa2cc0a4dd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa2d002c260 tx=0x7fa2d0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:28.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:28.894+0000 7fa2e1600640 1 -- 172.21.15.37:0/1826529871 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa2d809bc30 con 0x7fa2e41038e0 2024-03-28T18:08:29.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:28 smithi059 ceph-mon[36190]: from='client.24677 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:08:29.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:28 smithi170 ceph-mon[36280]: from='client.24677 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:08:29.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.134+0000 7fa2eb800640 1 -- 172.21.15.37:0/1826529871 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2e40622a0 con 0x7fa2e41038e0 2024-03-28T18:08:29.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.135+0000 7fa2e1600640 1 -- 172.21.15.37:0/1826529871 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22196 (secure 0 0 0) 0x7fa2d809b530 con 0x7fa2e41038e0 2024-03-28T18:08:29.136 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:08:29.136 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":85,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","created":"2024-03-28T17:59:05.949847+0000","modified":"2024-03-28T18:08:20.991162+0000","last_up_change":"2024-03-28T18:07:21.021030+0000","last_in_change":"2024-03-28T18:07:01.416192+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:03:00.517967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:08:18.365827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:08:19.446282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db60618d-56f7-4130-8a90-84cbc480965b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6803","nonce":3562602035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6805","nonce":3562602035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6809","nonce":3562602035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6807","nonce":3562602035}]},"public_addr":"172.21.15.37:6803/3562602035","cluster_addr":"172.21.15.37:6805/3562602035","heartbeat_back_addr":"172.21.15.37:6809/3562602035","heartbeat_front_addr":"172.21.15.37:6807/3562602035","state":["exists","up"]},{"osd":1,"uuid":"7a4b7ad6-905e-4258-9574-693d4277dfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6801","nonce":3326537211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6803","nonce":3326537211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6807","nonce":3326537211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6805","nonce":3326537211}]},"public_addr":"172.21.15.59:6801/3326537211","cluster_addr":"172.21.15.59:6803/3326537211","heartbeat_back_addr":"172.21.15.59:6807/3326537211","heartbeat_front_addr":"172.21.15.59:6805/3326537211","state":["exists","up"]},{"osd":2,"uuid":"8b76ec1c-46bf-4336-89b5-12d82906e8b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6801","nonce":1269398427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6803","nonce":1269398427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6807","nonce":1269398427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6805","nonce":1269398427}]},"public_addr":"172.21.15.170:6801/1269398427","cluster_addr":"172.21.15.170:6803/1269398427","heartbeat_back_addr":"172.21.15.170:6807/1269398427","heartbeat_front_addr":"172.21.15.170:6805/1269398427","state":["exists","up"]},{"osd":3,"uuid":"66eca1da-2fbd-44e1-8bbf-5635a2c70ad5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6811","nonce":317805240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6813","nonce":317805240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6817","nonce":317805240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6815","nonce":317805240}]},"public_addr":"172.21.15.37:6811/317805240","cluster_addr":"172.21.15.37:6813/317805240","heartbeat_back_addr":"172.21.15.37:6817/317805240","heartbeat_front_addr":"172.21.15.37:6815/317805240","state":["exists","up"]},{"osd":4,"uuid":"4c877335-e42d-4d8e-9f48-a1d0a89ae438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6809","nonce":2410632623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6811","nonce":2410632623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6815","nonce":2410632623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6813","nonce":2410632623}]},"public_addr":"172.21.15.59:6809/2410632623","cluster_addr":"172.21.15.59:6811/2410632623","heartbeat_back_addr":"172.21.15.59:6815/2410632623","heartbeat_front_addr":"172.21.15.59:6813/2410632623","state":["exists","up"]},{"osd":5,"uuid":"b884842a-86e2-455b-9570-ff163b3bc0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6809","nonce":1218943899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6811","nonce":1218943899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6815","nonce":1218943899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6813","nonce":1218943899}]},"public_addr":"172.21.15.170:6809/1218943899","cluster_addr":"172.21.15.170:6811/1218943899","heartbeat_back_addr":"172.21.15.170:6815/1218943899","heartbeat_front_addr":"172.21.15.170:6813/1218943899","state":["exists","up"]},{"osd":6,"uuid":"249a1200-77f4-44ad-981f-1d352f732b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6819","nonce":1899652046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6821","nonce":1899652046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6825","nonce":1899652046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6823","nonce":1899652046}]},"public_addr":"172.21.15.37:6819/1899652046","cluster_addr":"172.21.15.37:6821/1899652046","heartbeat_back_addr":"172.21.15.37:6825/1899652046","heartbeat_front_addr":"172.21.15.37:6823/1899652046","state":["exists","up"]},{"osd":7,"uuid":"c73338e6-f505-4517-a84b-f35e04750a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6817","nonce":1343675690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6819","nonce":1343675690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6823","nonce":1343675690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6821","nonce":1343675690}]},"public_addr":"172.21.15.59:6817/1343675690","cluster_addr":"172.21.15.59:6819/1343675690","heartbeat_back_addr":"172.21.15.59:6823/1343675690","heartbeat_front_addr":"172.21.15.59:6821/1343675690","state":["exists","up"]},{"osd":8,"uuid":"149b8297-ff2d-4a88-9b1e-0def021cc516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6817","nonce":3235794556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6819","nonce":3235794556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6823","nonce":3235794556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6821","nonce":3235794556}]},"public_addr":"172.21.15.170:6817/3235794556","cluster_addr":"172.21.15.170:6819/3235794556","heartbeat_back_addr":"172.21.15.170:6823/3235794556","heartbeat_front_addr":"172.21.15.170:6821/3235794556","state":["exists","up"]},{"osd":9,"uuid":"55c27cc2-c771-44b7-90fe-8f0feca44162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6827","nonce":2307153008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6829","nonce":2307153008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6833","nonce":2307153008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6831","nonce":2307153008}]},"public_addr":"172.21.15.37:6827/2307153008","cluster_addr":"172.21.15.37:6829/2307153008","heartbeat_back_addr":"172.21.15.37:6833/2307153008","heartbeat_front_addr":"172.21.15.37:6831/2307153008","state":["exists","up"]},{"osd":10,"uuid":"c0595135-ab81-49d3-8c23-19f960c03e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6825","nonce":2113830275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6827","nonce":2113830275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6831","nonce":2113830275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6829","nonce":2113830275}]},"public_addr":"172.21.15.59:6825/2113830275","cluster_addr":"172.21.15.59:6827/2113830275","heartbeat_back_addr":"172.21.15.59:6831/2113830275","heartbeat_front_addr":"172.21.15.59:6829/2113830275","state":["exists","up"]},{"osd":11,"uuid":"5871e177-c17b-4637-a797-4fa5125920dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6825","nonce":3082439360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6827","nonce":3082439360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6831","nonce":3082439360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6829","nonce":3082439360}]},"public_addr":"172.21.15.170:6825/3082439360","cluster_addr":"172.21.15.170:6827/3082439360","heartbeat_back_addr":"172.21.15.170:6831/3082439360","heartbeat_front_addr":"172.21.15.170:6829/3082439360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:07.690545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:33.183396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:57.400263+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:30.597211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:57.259044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:22.627823+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:57.585863+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:25.000476+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:50.876688+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:25.859761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:53.563146+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:07:19.304115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3023379428":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1443710231":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6800/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6801/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6801/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6800/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1007625490":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/2895698584":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/497313034":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/3293943475":"2024-03-29T17:59:39.162038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:08:29.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.138+0000 7fa2eb800640 1 -- 172.21.15.37:0/1826529871 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa2cc0a2910 msgr2=0x7fa2cc0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:29.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.138+0000 7fa2eb800640 1 --2- 172.21.15.37:0/1826529871 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa2cc0a2910 0x7fa2cc0a4dd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa2d002c260 tx=0x7fa2d0000bc0 comp rx=0 tx=0).stop 2024-03-28T18:08:29.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.138+0000 7fa2eb800640 1 -- 172.21.15.37:0/1826529871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2e41038e0 msgr2=0x7fa2e4199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:29.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.138+0000 7fa2eb800640 1 --2- 172.21.15.37:0/1826529871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2e41038e0 0x7fa2e4199be0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa2d800e9c0 tx=0x7fa2d800ee80 comp rx=0 tx=0).stop 2024-03-28T18:08:29.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.138+0000 7fa2eb800640 1 -- 172.21.15.37:0/1826529871 shutdown_connections 2024-03-28T18:08:29.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.138+0000 7fa2eb800640 1 --2- 172.21.15.37:0/1826529871 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa2cc0a2910 0x7fa2cc0a4dd0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:29.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.138+0000 7fa2eb800640 1 --2- 172.21.15.37:0/1826529871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e4104ad0 0x7fa2e419a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:29.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.138+0000 7fa2eb800640 1 --2- 172.21.15.37:0/1826529871 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2e4105490 0x7fa2e41a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:29.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.138+0000 7fa2eb800640 1 --2- 172.21.15.37:0/1826529871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2e41038e0 0x7fa2e4199be0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:29.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.138+0000 7fa2eb800640 1 -- 172.21.15.37:0/1826529871 >> 172.21.15.37:0/1826529871 conn(0x7fa2e40fee90 msgr2=0x7fa2e4107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:29.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.139+0000 7fa2eb800640 1 -- 172.21.15.37:0/1826529871 shutdown_connections 2024-03-28T18:08:29.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.139+0000 7fa2eb800640 1 -- 172.21.15.37:0/1826529871 wait complete. 2024-03-28T18:08:29.157 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T18:08:29.157 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:08:29.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.396+0000 7f705de00640 1 -- 172.21.15.37:0/3596732660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7058101fd0 msgr2=0x7f70581043c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:29.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.396+0000 7f705de00640 1 --2- 172.21.15.37:0/3596732660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7058101fd0 0x7f70581043c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f704c009900 tx=0x7f704c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:08:29.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.396+0000 7f705de00640 1 -- 172.21.15.37:0/3596732660 shutdown_connections 2024-03-28T18:08:29.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.396+0000 7f705de00640 1 --2- 172.21.15.37:0/3596732660 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7058107230 0x7f7058109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:29.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.396+0000 7f705de00640 1 --2- 172.21.15.37:0/3596732660 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7058104900 0x7f7058106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:29.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.396+0000 7f705de00640 1 --2- 172.21.15.37:0/3596732660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7058101fd0 0x7f70581043c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.396+0000 7f705de00640 1 -- 172.21.15.37:0/3596732660 >> 172.21.15.37:0/3596732660 conn(0x7f70580fb980 msgr2=0x7f70580fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.396+0000 7f705de00640 1 -- 172.21.15.37:0/3596732660 shutdown_connections 2024-03-28T18:08:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.396+0000 7f705de00640 1 -- 172.21.15.37:0/3596732660 wait complete. 2024-03-28T18:08:29.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.397+0000 7f705de00640 1 Processor -- start 2024-03-28T18:08:29.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.397+0000 7f705de00640 1 -- start start 2024-03-28T18:08:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.398+0000 7f705de00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7058101fd0 0x7f7058199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.398+0000 7f705de00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7058104900 0x7f7058199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.398+0000 7f705de00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7058107230 0x7f70581a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.398+0000 7f705de00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705810d1d0 con 0x7f7058107230 2024-03-28T18:08:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.398+0000 7f705de00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705810d050 con 0x7f7058101fd0 2024-03-28T18:08:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.398+0000 7f705de00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705810d350 con 0x7f7058104900 2024-03-28T18:08:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.398+0000 7f705ca00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7058101fd0 0x7f7058199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.398+0000 7f705ca00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7058101fd0 0x7f7058199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:57960/0 (socket says 172.21.15.37:57960) 2024-03-28T18:08:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.398+0000 7f7057e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7058104900 0x7f7058199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.398+0000 7f705ca00640 1 -- 172.21.15.37:0/208392582 learned_addr learned my addr 172.21.15.37:0/208392582 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.398+0000 7f705d400640 1 --2- 172.21.15.37:0/208392582 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7058107230 0x7f70581a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.398+0000 7f705ca00640 1 -- 172.21.15.37:0/208392582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7058104900 msgr2=0x7f7058199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.398+0000 7f705ca00640 1 --2- 172.21.15.37:0/208392582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7058104900 0x7f7058199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.398+0000 7f705ca00640 1 -- 172.21.15.37:0/208392582 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7058107230 msgr2=0x7f70581a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.398+0000 7f705ca00640 1 --2- 172.21.15.37:0/208392582 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7058107230 0x7f70581a0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.398+0000 7f705ca00640 1 -- 172.21.15.37:0/208392582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70581a1620 con 0x7f7058101fd0 2024-03-28T18:08:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.398+0000 7f705d400640 1 --2- 172.21.15.37:0/208392582 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7058107230 0x7f70581a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:08:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.398+0000 7f7057e00640 1 --2- 172.21.15.37:0/208392582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7058104900 0x7f7058199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:08:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.399+0000 7f705ca00640 1 --2- 172.21.15.37:0/208392582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7058101fd0 0x7f7058199a00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f704c005ab0 tx=0x7f704c02c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:29.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.399+0000 7f7055600640 1 -- 172.21.15.37:0/208392582 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c043020 con 0x7f7058101fd0 2024-03-28T18:08:29.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.399+0000 7f7055600640 1 -- 172.21.15.37:0/208392582 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f704c0054e0 con 0x7f7058101fd0 2024-03-28T18:08:29.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.399+0000 7f7055600640 1 -- 172.21.15.37:0/208392582 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c003ce0 con 0x7f7058101fd0 2024-03-28T18:08:29.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.399+0000 7f705de00640 1 -- 172.21.15.37:0/208392582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70581a18b0 con 0x7f7058101fd0 2024-03-28T18:08:29.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.399+0000 7f705de00640 1 -- 172.21.15.37:0/208392582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70581a1db0 con 0x7f7058101fd0 2024-03-28T18:08:29.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.400+0000 7f705de00640 1 -- 172.21.15.37:0/208392582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7058108d20 con 0x7f7058101fd0 2024-03-28T18:08:29.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.402+0000 7f7055600640 1 -- 172.21.15.37:0/208392582 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f704c046020 con 0x7f7058101fd0 2024-03-28T18:08:29.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.404+0000 7f7055600640 1 --2- 172.21.15.37:0/208392582 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f70400a2890 0x7f70400a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:29.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.404+0000 7f7055600640 1 -- 172.21.15.37:0/208392582 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f704c0f0bb0 con 0x7f7058101fd0 2024-03-28T18:08:29.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.404+0000 7f7057e00640 1 --2- 172.21.15.37:0/208392582 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f70400a2890 0x7f70400a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:29.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.405+0000 7f7057e00640 1 --2- 172.21.15.37:0/208392582 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f70400a2890 0x7f70400a4d50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f705819af10 tx=0x7f704400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:29.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.412+0000 7f7055600640 1 -- 172.21.15.37:0/208392582 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f704c04d020 con 0x7f7058101fd0 2024-03-28T18:08:29.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.619+0000 7f705de00640 1 -- 172.21.15.37:0/208392582 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7058061ae0 con 0x7f70400a2890 2024-03-28T18:08:29.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.633+0000 7f7055600640 1 -- 172.21.15.37:0/208392582 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258874 (secure 0 0 0) 0x7f7058061ae0 con 0x7f70400a2890 2024-03-28T18:08:29.637 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:08:29.639 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-03-28T18:08:29.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.638+0000 7f705de00640 1 -- 172.21.15.37:0/208392582 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f70400a2890 msgr2=0x7f70400a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:29.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.638+0000 7f705de00640 1 --2- 172.21.15.37:0/208392582 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f70400a2890 0x7f70400a4d50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f705819af10 tx=0x7f704400b410 comp rx=0 tx=0).stop 2024-03-28T18:08:29.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.638+0000 7f705de00640 1 -- 172.21.15.37:0/208392582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7058101fd0 msgr2=0x7f7058199a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:29.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.638+0000 7f705de00640 1 --2- 172.21.15.37:0/208392582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7058101fd0 0x7f7058199a00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f704c005ab0 tx=0x7f704c02c6c0 comp rx=0 tx=0).stop 2024-03-28T18:08:29.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.638+0000 7f705de00640 1 -- 172.21.15.37:0/208392582 shutdown_connections 2024-03-28T18:08:29.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.638+0000 7f705de00640 1 --2- 172.21.15.37:0/208392582 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f70400a2890 0x7f70400a4d50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:29.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.638+0000 7f705de00640 1 --2- 172.21.15.37:0/208392582 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7058107230 0x7f70581a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:29.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.638+0000 7f705de00640 1 --2- 172.21.15.37:0/208392582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7058104900 0x7f7058199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:29.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.638+0000 7f705de00640 1 --2- 172.21.15.37:0/208392582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7058101fd0 0x7f7058199a00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:29.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.638+0000 7f705de00640 1 -- 172.21.15.37:0/208392582 >> 172.21.15.37:0/208392582 conn(0x7f70580fb980 msgr2=0x7f7058101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:29.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.639+0000 7f705de00640 1 -- 172.21.15.37:0/208392582 shutdown_connections 2024-03-28T18:08:29.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.639+0000 7f705de00640 1 -- 172.21.15.37:0/208392582 wait complete. 2024-03-28T18:08:29.659 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-03-28T18:08:27.951111+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595590,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":146,"num_read_kb":128,"num_write":296,"num_write_kb":2976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":238,"ondisk_log_size":238,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376876,"kb_used_data":11080,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564948,"statfs":{"total":1151940427776,"available":1151554506752,"internally_reserved":0,"allocated":11345920,"data_stored":7048874,"data_compressed":108117,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5222,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":115,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002768"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539441+0000","last_change":"2024-03-28T18:08:20.025576+0000","last_active":"2024-03-28T18:08:21.539441+0000","last_peered":"2024-03-28T18:08:21.539441+0000","last_clean":"2024-03-28T18:08:21.539441+0000","last_became_active":"2024-03-28T18:08:20.025177+0000","last_became_peered":"2024-03-28T18:08:20.025177+0000","last_unstale":"2024-03-28T18:08:21.539441+0000","last_undegraded":"2024-03-28T18:08:21.539441+0000","last_fullsized":"2024-03-28T18:08:21.539441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:14.521170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539526+0000","last_change":"2024-03-28T18:08:20.027418+0000","last_active":"2024-03-28T18:08:21.539526+0000","last_peered":"2024-03-28T18:08:21.539526+0000","last_clean":"2024-03-28T18:08:21.539526+0000","last_became_active":"2024-03-28T18:08:20.026731+0000","last_became_peered":"2024-03-28T18:08:20.026731+0000","last_unstale":"2024-03-28T18:08:21.539526+0000","last_undegraded":"2024-03-28T18:08:21.539526+0000","last_fullsized":"2024-03-28T18:08:21.539526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:52:58.328738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539345+0000","last_change":"2024-03-28T18:08:20.029821+0000","last_active":"2024-03-28T18:08:21.539345+0000","last_peered":"2024-03-28T18:08:21.539345+0000","last_clean":"2024-03-28T18:08:21.539345+0000","last_became_active":"2024-03-28T18:08:20.028245+0000","last_became_peered":"2024-03-28T18:08:20.028245+0000","last_unstale":"2024-03-28T18:08:21.539345+0000","last_undegraded":"2024-03-28T18:08:21.539345+0000","last_fullsized":"2024-03-28T18:08:21.539345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:44.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.510241+0000","last_change":"2024-03-28T18:08:20.015682+0000","last_active":"2024-03-28T18:08:21.510241+0000","last_peered":"2024-03-28T18:08:21.510241+0000","last_clean":"2024-03-28T18:08:21.510241+0000","last_became_active":"2024-03-28T18:08:20.015303+0000","last_became_peered":"2024-03-28T18:08:20.015303+0000","last_unstale":"2024-03-28T18:08:21.510241+0000","last_undegraded":"2024-03-28T18:08:21.510241+0000","last_fullsized":"2024-03-28T18:08:21.510241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:57.456052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.269696+0000","last_change":"2024-03-28T18:08:20.020970+0000","last_active":"2024-03-28T18:08:21.269696+0000","last_peered":"2024-03-28T18:08:21.269696+0000","last_clean":"2024-03-28T18:08:21.269696+0000","last_became_active":"2024-03-28T18:08:20.020646+0000","last_became_peered":"2024-03-28T18:08:20.020646+0000","last_unstale":"2024-03-28T18:08:21.269696+0000","last_undegraded":"2024-03-28T18:08:21.269696+0000","last_fullsized":"2024-03-28T18:08:21.269696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:57:20.422546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539686+0000","last_change":"2024-03-28T18:08:20.022395+0000","last_active":"2024-03-28T18:08:21.539686+0000","last_peered":"2024-03-28T18:08:21.539686+0000","last_clean":"2024-03-28T18:08:21.539686+0000","last_became_active":"2024-03-28T18:08:20.022022+0000","last_became_peered":"2024-03-28T18:08:20.022022+0000","last_unstale":"2024-03-28T18:08:21.539686+0000","last_undegraded":"2024-03-28T18:08:21.539686+0000","last_fullsized":"2024-03-28T18:08:21.539686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:32.590751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539423+0000","last_change":"2024-03-28T18:08:20.019990+0000","last_active":"2024-03-28T18:08:21.539423+0000","last_peered":"2024-03-28T18:08:21.539423+0000","last_clean":"2024-03-28T18:08:21.539423+0000","last_became_active":"2024-03-28T18:08:20.019618+0000","last_became_peered":"2024-03-28T18:08:20.019618+0000","last_unstale":"2024-03-28T18:08:21.539423+0000","last_undegraded":"2024-03-28T18:08:21.539423+0000","last_fullsized":"2024-03-28T18:08:21.539423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:48.442579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.525075+0000","last_change":"2024-03-28T18:08:20.029701+0000","last_active":"2024-03-28T18:08:21.525075+0000","last_peered":"2024-03-28T18:08:21.525075+0000","last_clean":"2024-03-28T18:08:21.525075+0000","last_became_active":"2024-03-28T18:08:20.029181+0000","last_became_peered":"2024-03-28T18:08:20.029181+0000","last_unstale":"2024-03-28T18:08:21.525075+0000","last_undegraded":"2024-03-28T18:08:21.525075+0000","last_fullsized":"2024-03-28T18:08:21.525075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:00:41.213388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996886+0000","last_change":"2024-03-28T18:08:20.009732+0000","last_active":"2024-03-28T18:08:20.996886+0000","last_peered":"2024-03-28T18:08:20.996886+0000","last_clean":"2024-03-28T18:08:20.996886+0000","last_became_active":"2024-03-28T18:08:20.009288+0000","last_became_peered":"2024-03-28T18:08:20.009288+0000","last_unstale":"2024-03-28T18:08:20.996886+0000","last_undegraded":"2024-03-28T18:08:20.996886+0000","last_fullsized":"2024-03-28T18:08:20.996886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:20:54.917964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.540205+0000","last_change":"2024-03-28T18:08:20.009687+0000","last_active":"2024-03-28T18:08:21.540205+0000","last_peered":"2024-03-28T18:08:21.540205+0000","last_clean":"2024-03-28T18:08:21.540205+0000","last_became_active":"2024-03-28T18:08:20.008364+0000","last_became_peered":"2024-03-28T18:08:20.008364+0000","last_unstale":"2024-03-28T18:08:21.540205+0000","last_undegraded":"2024-03-28T18:08:21.540205+0000","last_fullsized":"2024-03-28T18:08:21.540205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:08.419202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539987+0000","last_change":"2024-03-28T18:08:20.011508+0000","last_active":"2024-03-28T18:08:21.539987+0000","last_peered":"2024-03-28T18:08:21.539987+0000","last_clean":"2024-03-28T18:08:21.539987+0000","last_became_active":"2024-03-28T18:08:20.011117+0000","last_became_peered":"2024-03-28T18:08:20.011117+0000","last_unstale":"2024-03-28T18:08:21.539987+0000","last_undegraded":"2024-03-28T18:08:21.539987+0000","last_fullsized":"2024-03-28T18:08:21.539987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:06:19.278931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.510180+0000","last_change":"2024-03-28T18:08:20.011403+0000","last_active":"2024-03-28T18:08:21.510180+0000","last_peered":"2024-03-28T18:08:21.510180+0000","last_clean":"2024-03-28T18:08:21.510180+0000","last_became_active":"2024-03-28T18:08:20.011025+0000","last_became_peered":"2024-03-28T18:08:20.011025+0000","last_unstale":"2024-03-28T18:08:21.510180+0000","last_undegraded":"2024-03-28T18:08:21.510180+0000","last_fullsized":"2024-03-28T18:08:21.510180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:13.599432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996683+0000","last_change":"2024-03-28T18:08:20.001664+0000","last_active":"2024-03-28T18:08:20.996683+0000","last_peered":"2024-03-28T18:08:20.996683+0000","last_clean":"2024-03-28T18:08:20.996683+0000","last_became_active":"2024-03-28T18:08:20.000910+0000","last_became_peered":"2024-03-28T18:08:20.000910+0000","last_unstale":"2024-03-28T18:08:20.996683+0000","last_undegraded":"2024-03-28T18:08:20.996683+0000","last_fullsized":"2024-03-28T18:08:20.996683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:00:23.796302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539047+0000","last_change":"2024-03-28T18:08:20.027174+0000","last_active":"2024-03-28T18:08:21.539047+0000","last_peered":"2024-03-28T18:08:21.539047+0000","last_clean":"2024-03-28T18:08:21.539047+0000","last_became_active":"2024-03-28T18:08:20.026113+0000","last_became_peered":"2024-03-28T18:08:20.026113+0000","last_unstale":"2024-03-28T18:08:21.539047+0000","last_undegraded":"2024-03-28T18:08:21.539047+0000","last_fullsized":"2024-03-28T18:08:21.539047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:55:00.061696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.538870+0000","last_change":"2024-03-28T18:08:20.020060+0000","last_active":"2024-03-28T18:08:21.538870+0000","last_peered":"2024-03-28T18:08:21.538870+0000","last_clean":"2024-03-28T18:08:21.538870+0000","last_became_active":"2024-03-28T18:08:20.019691+0000","last_became_peered":"2024-03-28T18:08:20.019691+0000","last_unstale":"2024-03-28T18:08:21.538870+0000","last_undegraded":"2024-03-28T18:08:21.538870+0000","last_fullsized":"2024-03-28T18:08:21.538870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:27.889737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996294+0000","last_change":"2024-03-28T18:08:20.024239+0000","last_active":"2024-03-28T18:08:20.996294+0000","last_peered":"2024-03-28T18:08:20.996294+0000","last_clean":"2024-03-28T18:08:20.996294+0000","last_became_active":"2024-03-28T18:08:20.023874+0000","last_became_peered":"2024-03-28T18:08:20.023874+0000","last_unstale":"2024-03-28T18:08:20.996294+0000","last_undegraded":"2024-03-28T18:08:20.996294+0000","last_fullsized":"2024-03-28T18:08:20.996294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:05:32.903195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539571+0000","last_change":"2024-03-28T18:08:20.033322+0000","last_active":"2024-03-28T18:08:21.539571+0000","last_peered":"2024-03-28T18:08:21.539571+0000","last_clean":"2024-03-28T18:08:21.539571+0000","last_became_active":"2024-03-28T18:08:20.033048+0000","last_became_peered":"2024-03-28T18:08:20.033048+0000","last_unstale":"2024-03-28T18:08:21.539571+0000","last_undegraded":"2024-03-28T18:08:21.539571+0000","last_fullsized":"2024-03-28T18:08:21.539571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:18:59.234141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.522842+0000","last_change":"2024-03-28T18:08:20.022093+0000","last_active":"2024-03-28T18:08:21.522842+0000","last_peered":"2024-03-28T18:08:21.522842+0000","last_clean":"2024-03-28T18:08:21.522842+0000","last_became_active":"2024-03-28T18:08:20.021411+0000","last_became_peered":"2024-03-28T18:08:20.021411+0000","last_unstale":"2024-03-28T18:08:21.522842+0000","last_undegraded":"2024-03-28T18:08:21.522842+0000","last_fullsized":"2024-03-28T18:08:21.522842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:09:40.366628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.995920+0000","last_change":"2024-03-28T18:08:20.028893+0000","last_active":"2024-03-28T18:08:20.995920+0000","last_peered":"2024-03-28T18:08:20.995920+0000","last_clean":"2024-03-28T18:08:20.995920+0000","last_became_active":"2024-03-28T18:08:20.028634+0000","last_became_peered":"2024-03-28T18:08:20.028634+0000","last_unstale":"2024-03-28T18:08:20.995920+0000","last_undegraded":"2024-03-28T18:08:20.995920+0000","last_fullsized":"2024-03-28T18:08:20.995920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:32:27.454406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.522350+0000","last_change":"2024-03-28T18:08:19.481368+0000","last_active":"2024-03-28T18:08:21.522350+0000","last_peered":"2024-03-28T18:08:21.522350+0000","last_clean":"2024-03-28T18:08:21.522350+0000","last_became_active":"2024-03-28T18:08:19.480980+0000","last_became_peered":"2024-03-28T18:08:19.480980+0000","last_unstale":"2024-03-28T18:08:21.522350+0000","last_undegraded":"2024-03-28T18:08:21.522350+0000","last_fullsized":"2024-03-28T18:08:21.522350+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:57.387755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.526039+0000","last_change":"2024-03-28T18:08:20.027855+0000","last_active":"2024-03-28T18:08:21.526039+0000","last_peered":"2024-03-28T18:08:21.526039+0000","last_clean":"2024-03-28T18:08:21.526039+0000","last_became_active":"2024-03-28T18:08:20.027565+0000","last_became_peered":"2024-03-28T18:08:20.027565+0000","last_unstale":"2024-03-28T18:08:21.526039+0000","last_undegraded":"2024-03-28T18:08:21.526039+0000","last_fullsized":"2024-03-28T18:08:21.526039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:36:37.596573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:22.540745+0000","last_change":"2024-03-28T18:08:19.499055+0000","last_active":"2024-03-28T18:08:22.540745+0000","last_peered":"2024-03-28T18:08:22.540745+0000","last_clean":"2024-03-28T18:08:22.540745+0000","last_became_active":"2024-03-28T18:08:19.497964+0000","last_became_peered":"2024-03-28T18:08:19.497964+0000","last_unstale":"2024-03-28T18:08:22.540745+0000","last_undegraded":"2024-03-28T18:08:22.540745+0000","last_fullsized":"2024-03-28T18:08:22.540745+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:38:45.620406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539645+0000","last_change":"2024-03-28T18:08:20.028967+0000","last_active":"2024-03-28T18:08:21.539645+0000","last_peered":"2024-03-28T18:08:21.539645+0000","last_clean":"2024-03-28T18:08:21.539645+0000","last_became_active":"2024-03-28T18:08:20.028607+0000","last_became_peered":"2024-03-28T18:08:20.028607+0000","last_unstale":"2024-03-28T18:08:21.539645+0000","last_undegraded":"2024-03-28T18:08:21.539645+0000","last_fullsized":"2024-03-28T18:08:21.539645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:21:23.994077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:24.415509+0000","last_change":"2024-03-28T18:08:19.481998+0000","last_active":"2024-03-28T18:08:24.415509+0000","last_peered":"2024-03-28T18:08:24.415509+0000","last_clean":"2024-03-28T18:08:24.415509+0000","last_became_active":"2024-03-28T18:08:19.481135+0000","last_became_peered":"2024-03-28T18:08:19.481135+0000","last_unstale":"2024-03-28T18:08:24.415509+0000","last_undegraded":"2024-03-28T18:08:24.415509+0000","last_fullsized":"2024-03-28T18:08:24.415509+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:37:26.171036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996593+0000","last_change":"2024-03-28T18:08:20.009878+0000","last_active":"2024-03-28T18:08:20.996593+0000","last_peered":"2024-03-28T18:08:20.996593+0000","last_clean":"2024-03-28T18:08:20.996593+0000","last_became_active":"2024-03-28T18:08:20.009534+0000","last_became_peered":"2024-03-28T18:08:20.009534+0000","last_unstale":"2024-03-28T18:08:20.996593+0000","last_undegraded":"2024-03-28T18:08:20.996593+0000","last_fullsized":"2024-03-28T18:08:20.996593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:58.975854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.269908+0000","last_change":"2024-03-28T18:08:19.472613+0000","last_active":"2024-03-28T18:08:21.269908+0000","last_peered":"2024-03-28T18:08:21.269908+0000","last_clean":"2024-03-28T18:08:21.269908+0000","last_became_active":"2024-03-28T18:08:19.472186+0000","last_became_peered":"2024-03-28T18:08:19.472186+0000","last_unstale":"2024-03-28T18:08:21.269908+0000","last_undegraded":"2024-03-28T18:08:21.269908+0000","last_fullsized":"2024-03-28T18:08:21.269908+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:50.807609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.326663+0000","last_change":"2024-03-28T18:08:20.008242+0000","last_active":"2024-03-28T18:08:21.326663+0000","last_peered":"2024-03-28T18:08:21.326663+0000","last_clean":"2024-03-28T18:08:21.326663+0000","last_became_active":"2024-03-28T18:08:20.007942+0000","last_became_peered":"2024-03-28T18:08:20.007942+0000","last_unstale":"2024-03-28T18:08:21.326663+0000","last_undegraded":"2024-03-28T18:08:21.326663+0000","last_fullsized":"2024-03-28T18:08:21.326663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:23.288978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:25.425258+0000","last_change":"2024-03-28T18:08:19.480869+0000","last_active":"2024-03-28T18:08:25.425258+0000","last_peered":"2024-03-28T18:08:25.425258+0000","last_clean":"2024-03-28T18:08:25.425258+0000","last_became_active":"2024-03-28T18:08:19.480441+0000","last_became_peered":"2024-03-28T18:08:19.480441+0000","last_unstale":"2024-03-28T18:08:25.425258+0000","last_undegraded":"2024-03-28T18:08:25.425258+0000","last_fullsized":"2024-03-28T18:08:25.425258+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:26.640605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.269760+0000","last_change":"2024-03-28T18:08:20.021530+0000","last_active":"2024-03-28T18:08:21.269760+0000","last_peered":"2024-03-28T18:08:21.269760+0000","last_clean":"2024-03-28T18:08:21.269760+0000","last_became_active":"2024-03-28T18:08:20.021179+0000","last_became_peered":"2024-03-28T18:08:20.021179+0000","last_unstale":"2024-03-28T18:08:21.269760+0000","last_undegraded":"2024-03-28T18:08:21.269760+0000","last_fullsized":"2024-03-28T18:08:21.269760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:43:07.300300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:24.417719+0000","last_change":"2024-03-28T18:08:19.494597+0000","last_active":"2024-03-28T18:08:24.417719+0000","last_peered":"2024-03-28T18:08:24.417719+0000","last_clean":"2024-03-28T18:08:24.417719+0000","last_became_active":"2024-03-28T18:08:19.494178+0000","last_became_peered":"2024-03-28T18:08:19.494178+0000","last_unstale":"2024-03-28T18:08:24.417719+0000","last_undegraded":"2024-03-28T18:08:24.417719+0000","last_fullsized":"2024-03-28T18:08:24.417719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:08:10.257768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.326878+0000","last_change":"2024-03-28T18:08:20.021062+0000","last_active":"2024-03-28T18:08:21.326878+0000","last_peered":"2024-03-28T18:08:21.326878+0000","last_clean":"2024-03-28T18:08:21.326878+0000","last_became_active":"2024-03-28T18:08:20.020758+0000","last_became_peered":"2024-03-28T18:08:20.020758+0000","last_unstale":"2024-03-28T18:08:21.326878+0000","last_undegraded":"2024-03-28T18:08:21.326878+0000","last_fullsized":"2024-03-28T18:08:21.326878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:47:39.656513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:24.422435+0000","last_change":"2024-03-28T18:08:19.480189+0000","last_active":"2024-03-28T18:08:24.422435+0000","last_peered":"2024-03-28T18:08:24.422435+0000","last_clean":"2024-03-28T18:08:24.422435+0000","last_became_active":"2024-03-28T18:08:19.479812+0000","last_became_peered":"2024-03-28T18:08:19.479812+0000","last_unstale":"2024-03-28T18:08:24.422435+0000","last_undegraded":"2024-03-28T18:08:24.422435+0000","last_fullsized":"2024-03-28T18:08:24.422435+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:46.408565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.525379+0000","last_change":"2024-03-28T18:08:20.023635+0000","last_active":"2024-03-28T18:08:21.525379+0000","last_peered":"2024-03-28T18:08:21.525379+0000","last_clean":"2024-03-28T18:08:21.525379+0000","last_became_active":"2024-03-28T18:08:20.023199+0000","last_became_peered":"2024-03-28T18:08:20.023199+0000","last_unstale":"2024-03-28T18:08:21.525379+0000","last_undegraded":"2024-03-28T18:08:21.525379+0000","last_fullsized":"2024-03-28T18:08:21.525379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:12.477899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.326295+0000","last_change":"2024-03-28T18:08:19.480858+0000","last_active":"2024-03-28T18:08:21.326295+0000","last_peered":"2024-03-28T18:08:21.326295+0000","last_clean":"2024-03-28T18:08:21.326295+0000","last_became_active":"2024-03-28T18:08:19.480418+0000","last_became_peered":"2024-03-28T18:08:19.480418+0000","last_unstale":"2024-03-28T18:08:21.326295+0000","last_undegraded":"2024-03-28T18:08:21.326295+0000","last_fullsized":"2024-03-28T18:08:21.326295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:59:57.788915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.522608+0000","last_change":"2024-03-28T18:08:20.031719+0000","last_active":"2024-03-28T18:08:21.522608+0000","last_peered":"2024-03-28T18:08:21.522608+0000","last_clean":"2024-03-28T18:08:21.522608+0000","last_became_active":"2024-03-28T18:08:20.029482+0000","last_became_peered":"2024-03-28T18:08:20.029482+0000","last_unstale":"2024-03-28T18:08:21.522608+0000","last_undegraded":"2024-03-28T18:08:21.522608+0000","last_fullsized":"2024-03-28T18:08:21.522608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:13.226596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:23.396141+0000","last_change":"2024-03-28T18:08:19.483346+0000","last_active":"2024-03-28T18:08:23.396141+0000","last_peered":"2024-03-28T18:08:23.396141+0000","last_clean":"2024-03-28T18:08:23.396141+0000","last_became_active":"2024-03-28T18:08:19.483016+0000","last_became_peered":"2024-03-28T18:08:19.483016+0000","last_unstale":"2024-03-28T18:08:23.396141+0000","last_undegraded":"2024-03-28T18:08:23.396141+0000","last_fullsized":"2024-03-28T18:08:23.396141+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:34:33.008477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996738+0000","last_change":"2024-03-28T18:08:20.020478+0000","last_active":"2024-03-28T18:08:20.996738+0000","last_peered":"2024-03-28T18:08:20.996738+0000","last_clean":"2024-03-28T18:08:20.996738+0000","last_became_active":"2024-03-28T18:08:20.020054+0000","last_became_peered":"2024-03-28T18:08:20.020054+0000","last_unstale":"2024-03-28T18:08:20.996738+0000","last_undegraded":"2024-03-28T18:08:20.996738+0000","last_fullsized":"2024-03-28T18:08:20.996738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:13:43.655549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:23.390504+0000","last_change":"2024-03-28T18:08:19.479386+0000","last_active":"2024-03-28T18:08:23.390504+0000","last_peered":"2024-03-28T18:08:23.390504+0000","last_clean":"2024-03-28T18:08:23.390504+0000","last_became_active":"2024-03-28T18:08:19.478985+0000","last_became_peered":"2024-03-28T18:08:19.478985+0000","last_unstale":"2024-03-28T18:08:23.390504+0000","last_undegraded":"2024-03-28T18:08:23.390504+0000","last_fullsized":"2024-03-28T18:08:23.390504+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:37:22.359475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539639+0000","last_change":"2024-03-28T18:08:20.021790+0000","last_active":"2024-03-28T18:08:21.539639+0000","last_peered":"2024-03-28T18:08:21.539639+0000","last_clean":"2024-03-28T18:08:21.539639+0000","last_became_active":"2024-03-28T18:08:20.021031+0000","last_became_peered":"2024-03-28T18:08:20.021031+0000","last_unstale":"2024-03-28T18:08:21.539639+0000","last_undegraded":"2024-03-28T18:08:21.539639+0000","last_fullsized":"2024-03-28T18:08:21.539639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:59:50.897692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:23.389970+0000","last_change":"2024-03-28T18:08:19.481576+0000","last_active":"2024-03-28T18:08:23.389970+0000","last_peered":"2024-03-28T18:08:23.389970+0000","last_clean":"2024-03-28T18:08:23.389970+0000","last_became_active":"2024-03-28T18:08:19.481226+0000","last_became_peered":"2024-03-28T18:08:19.481226+0000","last_unstale":"2024-03-28T18:08:23.389970+0000","last_undegraded":"2024-03-28T18:08:23.389970+0000","last_fullsized":"2024-03-28T18:08:23.389970+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:54:44.677343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996555+0000","last_change":"2024-03-28T18:08:20.013231+0000","last_active":"2024-03-28T18:08:20.996555+0000","last_peered":"2024-03-28T18:08:20.996555+0000","last_clean":"2024-03-28T18:08:20.996555+0000","last_became_active":"2024-03-28T18:08:20.012267+0000","last_became_peered":"2024-03-28T18:08:20.012267+0000","last_unstale":"2024-03-28T18:08:20.996555+0000","last_undegraded":"2024-03-28T18:08:20.996555+0000","last_fullsized":"2024-03-28T18:08:20.996555+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:07:20.293607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'8","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:23.390782+0000","last_change":"2024-03-28T18:08:19.477559+0000","last_active":"2024-03-28T18:08:23.390782+0000","last_peered":"2024-03-28T18:08:23.390782+0000","last_clean":"2024-03-28T18:08:23.390782+0000","last_became_active":"2024-03-28T18:08:19.477217+0000","last_became_peered":"2024-03-28T18:08:19.477217+0000","last_unstale":"2024-03-28T18:08:23.390782+0000","last_undegraded":"2024-03-28T18:08:23.390782+0000","last_fullsized":"2024-03-28T18:08:23.390782+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:37.071671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.269994+0000","last_change":"2024-03-28T18:08:20.011303+0000","last_active":"2024-03-28T18:08:21.269994+0000","last_peered":"2024-03-28T18:08:21.269994+0000","last_clean":"2024-03-28T18:08:21.269994+0000","last_became_active":"2024-03-28T18:08:20.010997+0000","last_became_peered":"2024-03-28T18:08:20.010997+0000","last_unstale":"2024-03-28T18:08:21.269994+0000","last_undegraded":"2024-03-28T18:08:21.269994+0000","last_fullsized":"2024-03-28T18:08:21.269994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:10:56.036132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:24.419778+0000","last_change":"2024-03-28T18:08:19.477342+0000","last_active":"2024-03-28T18:08:24.419778+0000","last_peered":"2024-03-28T18:08:24.419778+0000","last_clean":"2024-03-28T18:08:24.419778+0000","last_became_active":"2024-03-28T18:08:19.476984+0000","last_became_peered":"2024-03-28T18:08:19.476984+0000","last_unstale":"2024-03-28T18:08:24.419778+0000","last_undegraded":"2024-03-28T18:08:24.419778+0000","last_fullsized":"2024-03-28T18:08:24.419778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:15:42.606213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996321+0000","last_change":"2024-03-28T18:08:20.030524+0000","last_active":"2024-03-28T18:08:20.996321+0000","last_peered":"2024-03-28T18:08:20.996321+0000","last_clean":"2024-03-28T18:08:20.996321+0000","last_became_active":"2024-03-28T18:08:20.030227+0000","last_became_peered":"2024-03-28T18:08:20.030227+0000","last_unstale":"2024-03-28T18:08:20.996321+0000","last_undegraded":"2024-03-28T18:08:20.996321+0000","last_fullsized":"2024-03-28T18:08:20.996321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:07:41.763739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:23.392978+0000","last_change":"2024-03-28T18:08:19.481169+0000","last_active":"2024-03-28T18:08:23.392978+0000","last_peered":"2024-03-28T18:08:23.392978+0000","last_clean":"2024-03-28T18:08:23.392978+0000","last_became_active":"2024-03-28T18:08:19.480634+0000","last_became_peered":"2024-03-28T18:08:19.480634+0000","last_unstale":"2024-03-28T18:08:23.392978+0000","last_undegraded":"2024-03-28T18:08:23.392978+0000","last_fullsized":"2024-03-28T18:08:23.392978+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:39.265215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539597+0000","last_change":"2024-03-28T18:08:20.015163+0000","last_active":"2024-03-28T18:08:21.539597+0000","last_peered":"2024-03-28T18:08:21.539597+0000","last_clean":"2024-03-28T18:08:21.539597+0000","last_became_active":"2024-03-28T18:08:20.014765+0000","last_became_peered":"2024-03-28T18:08:20.014765+0000","last_unstale":"2024-03-28T18:08:21.539597+0000","last_undegraded":"2024-03-28T18:08:21.539597+0000","last_fullsized":"2024-03-28T18:08:21.539597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:51.035532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:24.415785+0000","last_change":"2024-03-28T18:08:19.479086+0000","last_active":"2024-03-28T18:08:24.415785+0000","last_peered":"2024-03-28T18:08:24.415785+0000","last_clean":"2024-03-28T18:08:24.415785+0000","last_became_active":"2024-03-28T18:08:19.478748+0000","last_became_peered":"2024-03-28T18:08:19.478748+0000","last_unstale":"2024-03-28T18:08:24.415785+0000","last_undegraded":"2024-03-28T18:08:24.415785+0000","last_fullsized":"2024-03-28T18:08:24.415785+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:46:59.566998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539150+0000","last_change":"2024-03-28T18:08:20.016375+0000","last_active":"2024-03-28T18:08:21.539150+0000","last_peered":"2024-03-28T18:08:21.539150+0000","last_clean":"2024-03-28T18:08:21.539150+0000","last_became_active":"2024-03-28T18:08:20.015670+0000","last_became_peered":"2024-03-28T18:08:20.015670+0000","last_unstale":"2024-03-28T18:08:21.539150+0000","last_undegraded":"2024-03-28T18:08:21.539150+0000","last_fullsized":"2024-03-28T18:08:21.539150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:48.388617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:24.421364+0000","last_change":"2024-03-28T18:08:19.482934+0000","last_active":"2024-03-28T18:08:24.421364+0000","last_peered":"2024-03-28T18:08:24.421364+0000","last_clean":"2024-03-28T18:08:24.421364+0000","last_became_active":"2024-03-28T18:08:19.482623+0000","last_became_peered":"2024-03-28T18:08:19.482623+0000","last_unstale":"2024-03-28T18:08:24.421364+0000","last_undegraded":"2024-03-28T18:08:24.421364+0000","last_fullsized":"2024-03-28T18:08:24.421364+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:06:31.718675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.326366+0000","last_change":"2024-03-28T18:08:20.007457+0000","last_active":"2024-03-28T18:08:21.326366+0000","last_peered":"2024-03-28T18:08:21.326366+0000","last_clean":"2024-03-28T18:08:21.326366+0000","last_became_active":"2024-03-28T18:08:20.007090+0000","last_became_peered":"2024-03-28T18:08:20.007090+0000","last_unstale":"2024-03-28T18:08:21.326366+0000","last_undegraded":"2024-03-28T18:08:21.326366+0000","last_fullsized":"2024-03-28T18:08:21.326366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:22:45.702706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:25.412096+0000","last_change":"2024-03-28T18:08:19.475104+0000","last_active":"2024-03-28T18:08:25.412096+0000","last_peered":"2024-03-28T18:08:25.412096+0000","last_clean":"2024-03-28T18:08:25.412096+0000","last_became_active":"2024-03-28T18:08:19.474754+0000","last_became_peered":"2024-03-28T18:08:19.474754+0000","last_unstale":"2024-03-28T18:08:25.412096+0000","last_undegraded":"2024-03-28T18:08:25.412096+0000","last_fullsized":"2024-03-28T18:08:25.412096+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:19.085631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539338+0000","last_change":"2024-03-28T18:08:20.009347+0000","last_active":"2024-03-28T18:08:21.539338+0000","last_peered":"2024-03-28T18:08:21.539338+0000","last_clean":"2024-03-28T18:08:21.539338+0000","last_became_active":"2024-03-28T18:08:20.009012+0000","last_became_peered":"2024-03-28T18:08:20.009012+0000","last_unstale":"2024-03-28T18:08:21.539338+0000","last_undegraded":"2024-03-28T18:08:21.539338+0000","last_fullsized":"2024-03-28T18:08:21.539338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:04:26.625830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.522533+0000","last_change":"2024-03-28T18:08:19.482414+0000","last_active":"2024-03-28T18:08:21.522533+0000","last_peered":"2024-03-28T18:08:21.522533+0000","last_clean":"2024-03-28T18:08:21.522533+0000","last_became_active":"2024-03-28T18:08:19.481830+0000","last_became_peered":"2024-03-28T18:08:19.481830+0000","last_unstale":"2024-03-28T18:08:21.522533+0000","last_undegraded":"2024-03-28T18:08:21.522533+0000","last_fullsized":"2024-03-28T18:08:21.522533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:27:14.259355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539453+0000","last_change":"2024-03-28T18:08:20.030687+0000","last_active":"2024-03-28T18:08:21.539453+0000","last_peered":"2024-03-28T18:08:21.539453+0000","last_clean":"2024-03-28T18:08:21.539453+0000","last_became_active":"2024-03-28T18:08:20.030308+0000","last_became_peered":"2024-03-28T18:08:20.030308+0000","last_unstale":"2024-03-28T18:08:21.539453+0000","last_undegraded":"2024-03-28T18:08:21.539453+0000","last_fullsized":"2024-03-28T18:08:21.539453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:31.745952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539566+0000","last_change":"2024-03-28T18:08:19.494446+0000","last_active":"2024-03-28T18:08:21.539566+0000","last_peered":"2024-03-28T18:08:21.539566+0000","last_clean":"2024-03-28T18:08:21.539566+0000","last_became_active":"2024-03-28T18:08:19.494036+0000","last_became_peered":"2024-03-28T18:08:19.494036+0000","last_unstale":"2024-03-28T18:08:21.539566+0000","last_undegraded":"2024-03-28T18:08:21.539566+0000","last_fullsized":"2024-03-28T18:08:21.539566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:42:53.660741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539623+0000","last_change":"2024-03-28T18:08:20.025706+0000","last_active":"2024-03-28T18:08:21.539623+0000","last_peered":"2024-03-28T18:08:21.539623+0000","last_clean":"2024-03-28T18:08:21.539623+0000","last_became_active":"2024-03-28T18:08:20.025343+0000","last_became_peered":"2024-03-28T18:08:20.025343+0000","last_unstale":"2024-03-28T18:08:21.539623+0000","last_undegraded":"2024-03-28T18:08:21.539623+0000","last_fullsized":"2024-03-28T18:08:21.539623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:32:27.883940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.326590+0000","last_change":"2024-03-28T18:08:20.009636+0000","last_active":"2024-03-28T18:08:21.326590+0000","last_peered":"2024-03-28T18:08:21.326590+0000","last_clean":"2024-03-28T18:08:21.326590+0000","last_became_active":"2024-03-28T18:08:20.009178+0000","last_became_peered":"2024-03-28T18:08:20.009178+0000","last_unstale":"2024-03-28T18:08:21.326590+0000","last_undegraded":"2024-03-28T18:08:21.326590+0000","last_fullsized":"2024-03-28T18:08:21.326590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:28.461411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:24.422708+0000","last_change":"2024-03-28T18:08:19.501287+0000","last_active":"2024-03-28T18:08:24.422708+0000","last_peered":"2024-03-28T18:08:24.422708+0000","last_clean":"2024-03-28T18:08:24.422708+0000","last_became_active":"2024-03-28T18:08:19.500920+0000","last_became_peered":"2024-03-28T18:08:19.500920+0000","last_unstale":"2024-03-28T18:08:24.422708+0000","last_undegraded":"2024-03-28T18:08:24.422708+0000","last_fullsized":"2024-03-28T18:08:24.422708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:29.882164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.525536+0000","last_change":"2024-03-28T18:08:20.020811+0000","last_active":"2024-03-28T18:08:21.525536+0000","last_peered":"2024-03-28T18:08:21.525536+0000","last_clean":"2024-03-28T18:08:21.525536+0000","last_became_active":"2024-03-28T18:08:20.020381+0000","last_became_peered":"2024-03-28T18:08:20.020381+0000","last_unstale":"2024-03-28T18:08:21.525536+0000","last_undegraded":"2024-03-28T18:08:21.525536+0000","last_fullsized":"2024-03-28T18:08:21.525536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:10:35.999858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:23.395868+0000","last_change":"2024-03-28T18:08:19.478020+0000","last_active":"2024-03-28T18:08:23.395868+0000","last_peered":"2024-03-28T18:08:23.395868+0000","last_clean":"2024-03-28T18:08:23.395868+0000","last_became_active":"2024-03-28T18:08:19.477627+0000","last_became_peered":"2024-03-28T18:08:19.477627+0000","last_unstale":"2024-03-28T18:08:23.395868+0000","last_undegraded":"2024-03-28T18:08:23.395868+0000","last_fullsized":"2024-03-28T18:08:23.395868+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:00:01.649974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":349,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539190+0000","last_change":"2024-03-28T18:06:58.750932+0000","last_active":"2024-03-28T18:08:21.539190+0000","last_peered":"2024-03-28T18:08:21.539190+0000","last_clean":"2024-03-28T18:08:21.539190+0000","last_became_active":"2024-03-28T18:06:58.750309+0000","last_became_peered":"2024-03-28T18:06:58.750309+0000","last_unstale":"2024-03-28T18:08:21.539190+0000","last_undegraded":"2024-03-28T18:08:21.539190+0000","last_fullsized":"2024-03-28T18:08:21.539190+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:03:00.832683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:03:00.832683+0000","last_clean_scrub_stamp":"2024-03-28T18:03:00.832683+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:46.001886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.995824+0000","last_change":"2024-03-28T18:08:20.009573+0000","last_active":"2024-03-28T18:08:20.995824+0000","last_peered":"2024-03-28T18:08:20.995824+0000","last_clean":"2024-03-28T18:08:20.995824+0000","last_became_active":"2024-03-28T18:08:20.009143+0000","last_became_peered":"2024-03-28T18:08:20.009143+0000","last_unstale":"2024-03-28T18:08:20.995824+0000","last_undegraded":"2024-03-28T18:08:20.995824+0000","last_fullsized":"2024-03-28T18:08:20.995824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:37:50.643356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:24.416384+0000","last_change":"2024-03-28T18:08:19.483733+0000","last_active":"2024-03-28T18:08:24.416384+0000","last_peered":"2024-03-28T18:08:24.416384+0000","last_clean":"2024-03-28T18:08:24.416384+0000","last_became_active":"2024-03-28T18:08:19.483366+0000","last_became_peered":"2024-03-28T18:08:19.483366+0000","last_unstale":"2024-03-28T18:08:24.416384+0000","last_undegraded":"2024-03-28T18:08:24.416384+0000","last_fullsized":"2024-03-28T18:08:24.416384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:29.192442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:23.393731+0000","last_change":"2024-03-28T18:08:19.481844+0000","last_active":"2024-03-28T18:08:23.393731+0000","last_peered":"2024-03-28T18:08:23.393731+0000","last_clean":"2024-03-28T18:08:23.393731+0000","last_became_active":"2024-03-28T18:08:19.481430+0000","last_became_peered":"2024-03-28T18:08:19.481430+0000","last_unstale":"2024-03-28T18:08:23.393731+0000","last_undegraded":"2024-03-28T18:08:23.393731+0000","last_fullsized":"2024-03-28T18:08:23.393731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:21:28.789883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996326+0000","last_change":"2024-03-28T18:08:20.021037+0000","last_active":"2024-03-28T18:08:20.996326+0000","last_peered":"2024-03-28T18:08:20.996326+0000","last_clean":"2024-03-28T18:08:20.996326+0000","last_became_active":"2024-03-28T18:08:20.020680+0000","last_became_peered":"2024-03-28T18:08:20.020680+0000","last_unstale":"2024-03-28T18:08:20.996326+0000","last_undegraded":"2024-03-28T18:08:20.996326+0000","last_fullsized":"2024-03-28T18:08:20.996326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:08:28.825774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.522834+0000","last_change":"2024-03-28T18:08:19.495681+0000","last_active":"2024-03-28T18:08:21.522834+0000","last_peered":"2024-03-28T18:08:21.522834+0000","last_clean":"2024-03-28T18:08:21.522834+0000","last_became_active":"2024-03-28T18:08:19.495289+0000","last_became_peered":"2024-03-28T18:08:19.495289+0000","last_unstale":"2024-03-28T18:08:21.522834+0000","last_undegraded":"2024-03-28T18:08:21.522834+0000","last_fullsized":"2024-03-28T18:08:21.522834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:36:54.861716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.526414+0000","last_change":"2024-03-28T18:08:20.024938+0000","last_active":"2024-03-28T18:08:21.526414+0000","last_peered":"2024-03-28T18:08:21.526414+0000","last_clean":"2024-03-28T18:08:21.526414+0000","last_became_active":"2024-03-28T18:08:20.024536+0000","last_became_peered":"2024-03-28T18:08:20.024536+0000","last_unstale":"2024-03-28T18:08:21.526414+0000","last_undegraded":"2024-03-28T18:08:21.526414+0000","last_fullsized":"2024-03-28T18:08:21.526414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:46:27.080005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:23.391895+0000","last_change":"2024-03-28T18:08:19.480310+0000","last_active":"2024-03-28T18:08:23.391895+0000","last_peered":"2024-03-28T18:08:23.391895+0000","last_clean":"2024-03-28T18:08:23.391895+0000","last_became_active":"2024-03-28T18:08:19.479919+0000","last_became_peered":"2024-03-28T18:08:19.479919+0000","last_unstale":"2024-03-28T18:08:23.391895+0000","last_undegraded":"2024-03-28T18:08:23.391895+0000","last_fullsized":"2024-03-28T18:08:23.391895+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:36:26.239515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539455+0000","last_change":"2024-03-28T18:08:20.033974+0000","last_active":"2024-03-28T18:08:21.539455+0000","last_peered":"2024-03-28T18:08:21.539455+0000","last_clean":"2024-03-28T18:08:21.539455+0000","last_became_active":"2024-03-28T18:08:20.033544+0000","last_became_peered":"2024-03-28T18:08:20.033544+0000","last_unstale":"2024-03-28T18:08:21.539455+0000","last_undegraded":"2024-03-28T18:08:21.539455+0000","last_fullsized":"2024-03-28T18:08:21.539455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:45:09.617988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:23.389708+0000","last_change":"2024-03-28T18:08:19.482604+0000","last_active":"2024-03-28T18:08:23.389708+0000","last_peered":"2024-03-28T18:08:23.389708+0000","last_clean":"2024-03-28T18:08:23.389708+0000","last_became_active":"2024-03-28T18:08:19.482248+0000","last_became_peered":"2024-03-28T18:08:19.482248+0000","last_unstale":"2024-03-28T18:08:23.389708+0000","last_undegraded":"2024-03-28T18:08:23.389708+0000","last_fullsized":"2024-03-28T18:08:23.389708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:23.199196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.269570+0000","last_change":"2024-03-28T18:08:20.011198+0000","last_active":"2024-03-28T18:08:21.269570+0000","last_peered":"2024-03-28T18:08:21.269570+0000","last_clean":"2024-03-28T18:08:21.269570+0000","last_became_active":"2024-03-28T18:08:20.010866+0000","last_became_peered":"2024-03-28T18:08:20.010866+0000","last_unstale":"2024-03-28T18:08:21.269570+0000","last_undegraded":"2024-03-28T18:08:21.269570+0000","last_fullsized":"2024-03-28T18:08:21.269570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:26.342858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:23.397351+0000","last_change":"2024-03-28T18:08:19.493830+0000","last_active":"2024-03-28T18:08:23.397351+0000","last_peered":"2024-03-28T18:08:23.397351+0000","last_clean":"2024-03-28T18:08:23.397351+0000","last_became_active":"2024-03-28T18:08:19.493447+0000","last_became_peered":"2024-03-28T18:08:19.493447+0000","last_unstale":"2024-03-28T18:08:23.397351+0000","last_undegraded":"2024-03-28T18:08:23.397351+0000","last_fullsized":"2024-03-28T18:08:23.397351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:33:29.797186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539165+0000","last_change":"2024-03-28T18:08:20.025620+0000","last_active":"2024-03-28T18:08:21.539165+0000","last_peered":"2024-03-28T18:08:21.539165+0000","last_clean":"2024-03-28T18:08:21.539165+0000","last_became_active":"2024-03-28T18:08:20.025276+0000","last_became_peered":"2024-03-28T18:08:20.025276+0000","last_unstale":"2024-03-28T18:08:21.539165+0000","last_undegraded":"2024-03-28T18:08:21.539165+0000","last_fullsized":"2024-03-28T18:08:21.539165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:30:06.667707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.995575+0000","last_change":"2024-03-28T18:08:19.474204+0000","last_active":"2024-03-28T18:08:20.995575+0000","last_peered":"2024-03-28T18:08:20.995575+0000","last_clean":"2024-03-28T18:08:20.995575+0000","last_became_active":"2024-03-28T18:08:19.473688+0000","last_became_peered":"2024-03-28T18:08:19.473688+0000","last_unstale":"2024-03-28T18:08:20.995575+0000","last_undegraded":"2024-03-28T18:08:20.995575+0000","last_fullsized":"2024-03-28T18:08:20.995575+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:15:09.880974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.525731+0000","last_change":"2024-03-28T18:08:20.035920+0000","last_active":"2024-03-28T18:08:21.525731+0000","last_peered":"2024-03-28T18:08:21.525731+0000","last_clean":"2024-03-28T18:08:21.525731+0000","last_became_active":"2024-03-28T18:08:20.035600+0000","last_became_peered":"2024-03-28T18:08:20.035600+0000","last_unstale":"2024-03-28T18:08:21.525731+0000","last_undegraded":"2024-03-28T18:08:21.525731+0000","last_fullsized":"2024-03-28T18:08:21.525731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:41.280448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:23.389762+0000","last_change":"2024-03-28T18:08:19.481305+0000","last_active":"2024-03-28T18:08:23.389762+0000","last_peered":"2024-03-28T18:08:23.389762+0000","last_clean":"2024-03-28T18:08:23.389762+0000","last_became_active":"2024-03-28T18:08:19.480978+0000","last_became_peered":"2024-03-28T18:08:19.480978+0000","last_unstale":"2024-03-28T18:08:23.389762+0000","last_undegraded":"2024-03-28T18:08:23.389762+0000","last_fullsized":"2024-03-28T18:08:23.389762+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:14:06.464967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539915+0000","last_change":"2024-03-28T18:08:20.017660+0000","last_active":"2024-03-28T18:08:21.539915+0000","last_peered":"2024-03-28T18:08:21.539915+0000","last_clean":"2024-03-28T18:08:21.539915+0000","last_became_active":"2024-03-28T18:08:20.016083+0000","last_became_peered":"2024-03-28T18:08:20.016083+0000","last_unstale":"2024-03-28T18:08:21.539915+0000","last_undegraded":"2024-03-28T18:08:21.539915+0000","last_fullsized":"2024-03-28T18:08:21.539915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:33:12.104274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:24.421135+0000","last_change":"2024-03-28T18:08:19.479855+0000","last_active":"2024-03-28T18:08:24.421135+0000","last_peered":"2024-03-28T18:08:24.421135+0000","last_clean":"2024-03-28T18:08:24.421135+0000","last_became_active":"2024-03-28T18:08:19.479429+0000","last_became_peered":"2024-03-28T18:08:19.479429+0000","last_unstale":"2024-03-28T18:08:24.421135+0000","last_undegraded":"2024-03-28T18:08:24.421135+0000","last_fullsized":"2024-03-28T18:08:24.421135+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:06:23.286970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.540207+0000","last_change":"2024-03-28T18:08:20.015419+0000","last_active":"2024-03-28T18:08:21.540207+0000","last_peered":"2024-03-28T18:08:21.540207+0000","last_clean":"2024-03-28T18:08:21.540207+0000","last_became_active":"2024-03-28T18:08:20.015011+0000","last_became_peered":"2024-03-28T18:08:20.015011+0000","last_unstale":"2024-03-28T18:08:21.540207+0000","last_undegraded":"2024-03-28T18:08:21.540207+0000","last_fullsized":"2024-03-28T18:08:21.540207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:44:46.858352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.326528+0000","last_change":"2024-03-28T18:08:19.495541+0000","last_active":"2024-03-28T18:08:21.326528+0000","last_peered":"2024-03-28T18:08:21.326528+0000","last_clean":"2024-03-28T18:08:21.326528+0000","last_became_active":"2024-03-28T18:08:19.495173+0000","last_became_peered":"2024-03-28T18:08:19.495173+0000","last_unstale":"2024-03-28T18:08:21.326528+0000","last_undegraded":"2024-03-28T18:08:21.326528+0000","last_fullsized":"2024-03-28T18:08:21.326528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:42:06.423107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539845+0000","last_change":"2024-03-28T18:08:20.018582+0000","last_active":"2024-03-28T18:08:21.539845+0000","last_peered":"2024-03-28T18:08:21.539845+0000","last_clean":"2024-03-28T18:08:21.539845+0000","last_became_active":"2024-03-28T18:08:20.017936+0000","last_became_peered":"2024-03-28T18:08:20.017936+0000","last_unstale":"2024-03-28T18:08:21.539845+0000","last_undegraded":"2024-03-28T18:08:21.539845+0000","last_fullsized":"2024-03-28T18:08:21.539845+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:05:49.510246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996206+0000","last_change":"2024-03-28T18:08:19.497049+0000","last_active":"2024-03-28T18:08:20.996206+0000","last_peered":"2024-03-28T18:08:20.996206+0000","last_clean":"2024-03-28T18:08:20.996206+0000","last_became_active":"2024-03-28T18:08:19.496623+0000","last_became_peered":"2024-03-28T18:08:19.496623+0000","last_unstale":"2024-03-28T18:08:20.996206+0000","last_undegraded":"2024-03-28T18:08:20.996206+0000","last_fullsized":"2024-03-28T18:08:20.996206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:55:07.842439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.540348+0000","last_change":"2024-03-28T18:08:20.017419+0000","last_active":"2024-03-28T18:08:21.540348+0000","last_peered":"2024-03-28T18:08:21.540348+0000","last_clean":"2024-03-28T18:08:21.540348+0000","last_became_active":"2024-03-28T18:08:20.015916+0000","last_became_peered":"2024-03-28T18:08:20.015916+0000","last_unstale":"2024-03-28T18:08:21.540348+0000","last_undegraded":"2024-03-28T18:08:21.540348+0000","last_fullsized":"2024-03-28T18:08:21.540348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:10:55.927794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539204+0000","last_change":"2024-03-28T18:08:20.008093+0000","last_active":"2024-03-28T18:08:21.539204+0000","last_peered":"2024-03-28T18:08:21.539204+0000","last_clean":"2024-03-28T18:08:21.539204+0000","last_became_active":"2024-03-28T18:08:20.007801+0000","last_became_peered":"2024-03-28T18:08:20.007801+0000","last_unstale":"2024-03-28T18:08:21.539204+0000","last_undegraded":"2024-03-28T18:08:21.539204+0000","last_fullsized":"2024-03-28T18:08:21.539204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:58.976859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.995859+0000","last_change":"2024-03-28T18:08:20.012771+0000","last_active":"2024-03-28T18:08:20.995859+0000","last_peered":"2024-03-28T18:08:20.995859+0000","last_clean":"2024-03-28T18:08:20.995859+0000","last_became_active":"2024-03-28T18:08:20.012456+0000","last_became_peered":"2024-03-28T18:08:20.012456+0000","last_unstale":"2024-03-28T18:08:20.995859+0000","last_undegraded":"2024-03-28T18:08:20.995859+0000","last_fullsized":"2024-03-28T18:08:20.995859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:56.428332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.995728+0000","last_change":"2024-03-28T18:08:20.034872+0000","last_active":"2024-03-28T18:08:20.995728+0000","last_peered":"2024-03-28T18:08:20.995728+0000","last_clean":"2024-03-28T18:08:20.995728+0000","last_became_active":"2024-03-28T18:08:20.034614+0000","last_became_peered":"2024-03-28T18:08:20.034614+0000","last_unstale":"2024-03-28T18:08:20.995728+0000","last_undegraded":"2024-03-28T18:08:20.995728+0000","last_fullsized":"2024-03-28T18:08:20.995728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:35.658186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539267+0000","last_change":"2024-03-28T18:08:20.033119+0000","last_active":"2024-03-28T18:08:21.539267+0000","last_peered":"2024-03-28T18:08:21.539267+0000","last_clean":"2024-03-28T18:08:21.539267+0000","last_became_active":"2024-03-28T18:08:20.030694+0000","last_became_peered":"2024-03-28T18:08:20.030694+0000","last_unstale":"2024-03-28T18:08:21.539267+0000","last_undegraded":"2024-03-28T18:08:21.539267+0000","last_fullsized":"2024-03-28T18:08:21.539267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:51:50.082993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996315+0000","last_change":"2024-03-28T18:08:20.025725+0000","last_active":"2024-03-28T18:08:20.996315+0000","last_peered":"2024-03-28T18:08:20.996315+0000","last_clean":"2024-03-28T18:08:20.996315+0000","last_became_active":"2024-03-28T18:08:20.024929+0000","last_became_peered":"2024-03-28T18:08:20.024929+0000","last_unstale":"2024-03-28T18:08:20.996315+0000","last_undegraded":"2024-03-28T18:08:20.996315+0000","last_fullsized":"2024-03-28T18:08:20.996315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:04.894246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539333+0000","last_change":"2024-03-28T18:08:20.005293+0000","last_active":"2024-03-28T18:08:21.539333+0000","last_peered":"2024-03-28T18:08:21.539333+0000","last_clean":"2024-03-28T18:08:21.539333+0000","last_became_active":"2024-03-28T18:08:20.004938+0000","last_became_peered":"2024-03-28T18:08:20.004938+0000","last_unstale":"2024-03-28T18:08:21.539333+0000","last_undegraded":"2024-03-28T18:08:21.539333+0000","last_fullsized":"2024-03-28T18:08:21.539333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:36.100810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996060+0000","last_change":"2024-03-28T18:08:20.008741+0000","last_active":"2024-03-28T18:08:20.996060+0000","last_peered":"2024-03-28T18:08:20.996060+0000","last_clean":"2024-03-28T18:08:20.996060+0000","last_became_active":"2024-03-28T18:08:20.008411+0000","last_became_peered":"2024-03-28T18:08:20.008411+0000","last_unstale":"2024-03-28T18:08:20.996060+0000","last_undegraded":"2024-03-28T18:08:20.996060+0000","last_fullsized":"2024-03-28T18:08:20.996060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:28:52.276278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.526417+0000","last_change":"2024-03-28T18:08:20.028486+0000","last_active":"2024-03-28T18:08:21.526417+0000","last_peered":"2024-03-28T18:08:21.526417+0000","last_clean":"2024-03-28T18:08:21.526417+0000","last_became_active":"2024-03-28T18:08:20.028121+0000","last_became_peered":"2024-03-28T18:08:20.028121+0000","last_unstale":"2024-03-28T18:08:21.526417+0000","last_undegraded":"2024-03-28T18:08:21.526417+0000","last_fullsized":"2024-03-28T18:08:21.526417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:55:26.828305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.326334+0000","last_change":"2024-03-28T18:08:20.028085+0000","last_active":"2024-03-28T18:08:21.326334+0000","last_peered":"2024-03-28T18:08:21.326334+0000","last_clean":"2024-03-28T18:08:21.326334+0000","last_became_active":"2024-03-28T18:08:20.027721+0000","last_became_peered":"2024-03-28T18:08:20.027721+0000","last_unstale":"2024-03-28T18:08:21.326334+0000","last_undegraded":"2024-03-28T18:08:21.326334+0000","last_fullsized":"2024-03-28T18:08:21.326334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:31:36.674002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.525438+0000","last_change":"2024-03-28T18:08:20.023605+0000","last_active":"2024-03-28T18:08:21.525438+0000","last_peered":"2024-03-28T18:08:21.525438+0000","last_clean":"2024-03-28T18:08:21.525438+0000","last_became_active":"2024-03-28T18:08:20.018217+0000","last_became_peered":"2024-03-28T18:08:20.018217+0000","last_unstale":"2024-03-28T18:08:21.525438+0000","last_undegraded":"2024-03-28T18:08:21.525438+0000","last_fullsized":"2024-03-28T18:08:21.525438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:43.717751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996782+0000","last_change":"2024-03-28T18:08:20.023072+0000","last_active":"2024-03-28T18:08:20.996782+0000","last_peered":"2024-03-28T18:08:20.996782+0000","last_clean":"2024-03-28T18:08:20.996782+0000","last_became_active":"2024-03-28T18:08:20.022690+0000","last_became_peered":"2024-03-28T18:08:20.022690+0000","last_unstale":"2024-03-28T18:08:20.996782+0000","last_undegraded":"2024-03-28T18:08:20.996782+0000","last_fullsized":"2024-03-28T18:08:20.996782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:21.391415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996605+0000","last_change":"2024-03-28T18:08:20.024261+0000","last_active":"2024-03-28T18:08:20.996605+0000","last_peered":"2024-03-28T18:08:20.996605+0000","last_clean":"2024-03-28T18:08:20.996605+0000","last_became_active":"2024-03-28T18:08:20.023792+0000","last_became_peered":"2024-03-28T18:08:20.023792+0000","last_unstale":"2024-03-28T18:08:20.996605+0000","last_undegraded":"2024-03-28T18:08:20.996605+0000","last_fullsized":"2024-03-28T18:08:20.996605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:23:08.030982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539195+0000","last_change":"2024-03-28T18:08:20.008306+0000","last_active":"2024-03-28T18:08:21.539195+0000","last_peered":"2024-03-28T18:08:21.539195+0000","last_clean":"2024-03-28T18:08:21.539195+0000","last_became_active":"2024-03-28T18:08:20.007984+0000","last_became_peered":"2024-03-28T18:08:20.007984+0000","last_unstale":"2024-03-28T18:08:21.539195+0000","last_undegraded":"2024-03-28T18:08:21.539195+0000","last_fullsized":"2024-03-28T18:08:21.539195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:25:26.889600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":248983,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5222,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":115,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":22623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383695,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645332,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":1032079,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808857,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":439305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070496,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":439259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332134,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":1031072,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":438611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574920,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":440133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":439999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411394,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36264,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708888,"statfs":{"total":95995035648,"available":95957901312,"internally_reserved":0,"allocated":950272,"data_stored":441483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182461,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":438922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920824,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":438543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":991232,"data_stored":1030365,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34394,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34394,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":8212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:08:29.663 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:08:29.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:29 smithi037 ceph-mon[31460]: pgmap v322: 97 pgs: 97 active+clean; 582 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 4.7 KiB/s wr, 15 op/s 2024-03-28T18:08:29.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:29 smithi037 ceph-mon[31460]: from='client.24709 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:08:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:29 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/163413978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:29 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1826529871' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:08:29.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.934+0000 7fd779000640 1 -- 172.21.15.37:0/1833601415 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd774104ad0 msgr2=0x7fd774104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:29.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.934+0000 7fd779000640 1 --2- 172.21.15.37:0/1833601415 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd774104ad0 0x7fd774104f50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd760009900 tx=0x7fd76002bc50 comp rx=0 tx=0).stop 2024-03-28T18:08:29.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.935+0000 7fd779000640 1 -- 172.21.15.37:0/1833601415 shutdown_connections 2024-03-28T18:08:29.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.935+0000 7fd779000640 1 --2- 172.21.15.37:0/1833601415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd774105490 0x7fd77410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:29.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.935+0000 7fd779000640 1 --2- 172.21.15.37:0/1833601415 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd774104ad0 0x7fd774104f50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:29.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.935+0000 7fd779000640 1 --2- 172.21.15.37:0/1833601415 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7741038e0 0x7fd774103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:29.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.935+0000 7fd779000640 1 -- 172.21.15.37:0/1833601415 >> 172.21.15.37:0/1833601415 conn(0x7fd7740fee90 msgr2=0x7fd7741012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:29.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.935+0000 7fd779000640 1 -- 172.21.15.37:0/1833601415 shutdown_connections 2024-03-28T18:08:29.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.935+0000 7fd779000640 1 -- 172.21.15.37:0/1833601415 wait complete. 2024-03-28T18:08:29.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.936+0000 7fd779000640 1 Processor -- start 2024-03-28T18:08:29.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.936+0000 7fd779000640 1 -- start start 2024-03-28T18:08:29.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.937+0000 7fd779000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7741038e0 0x7fd774199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:29.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.937+0000 7fd779000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd774104ad0 0x7fd77419a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:29.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.937+0000 7fd779000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd774105490 0x7fd7741a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:29.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.937+0000 7fd779000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77410f5b0 con 0x7fd774105490 2024-03-28T18:08:29.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.937+0000 7fd779000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77410f430 con 0x7fd7741038e0 2024-03-28T18:08:29.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.937+0000 7fd779000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77410f730 con 0x7fd774104ad0 2024-03-28T18:08:29.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.937+0000 7fd771e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7741038e0 0x7fd774199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:29.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.937+0000 7fd771400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd774104ad0 0x7fd77419a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:29.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.937+0000 7fd771e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7741038e0 0x7fd774199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:57990/0 (socket says 172.21.15.37:57990) 2024-03-28T18:08:29.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.937+0000 7fd771e00640 1 -- 172.21.15.37:0/1388746086 learned_addr learned my addr 172.21.15.37:0/1388746086 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:29.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.937+0000 7fd772800640 1 --2- 172.21.15.37:0/1388746086 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd774105490 0x7fd7741a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:29.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.937+0000 7fd771e00640 1 -- 172.21.15.37:0/1388746086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd774104ad0 msgr2=0x7fd77419a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:29.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.937+0000 7fd771e00640 1 --2- 172.21.15.37:0/1388746086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd774104ad0 0x7fd77419a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:29.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.937+0000 7fd771e00640 1 -- 172.21.15.37:0/1388746086 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd774105490 msgr2=0x7fd7741a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:29.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.937+0000 7fd771e00640 1 --2- 172.21.15.37:0/1388746086 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd774105490 0x7fd7741a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:29.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.937+0000 7fd771e00640 1 -- 172.21.15.37:0/1388746086 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7741a17e0 con 0x7fd7741038e0 2024-03-28T18:08:29.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.938+0000 7fd772800640 1 --2- 172.21.15.37:0/1388746086 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd774105490 0x7fd7741a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:08:29.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.938+0000 7fd771400640 1 --2- 172.21.15.37:0/1388746086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd774104ad0 0x7fd77419a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:08:29.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.938+0000 7fd771e00640 1 --2- 172.21.15.37:0/1388746086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7741038e0 0x7fd774199bc0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd76400e9c0 tx=0x7fd76400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:29.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.938+0000 7fd76aa00640 1 -- 172.21.15.37:0/1388746086 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76400cd80 con 0x7fd7741038e0 2024-03-28T18:08:29.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.938+0000 7fd76aa00640 1 -- 172.21.15.37:0/1388746086 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7640035f0 con 0x7fd7741038e0 2024-03-28T18:08:29.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.938+0000 7fd76aa00640 1 -- 172.21.15.37:0/1388746086 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd764010690 con 0x7fd7741038e0 2024-03-28T18:08:29.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.938+0000 7fd779000640 1 -- 172.21.15.37:0/1388746086 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7741a1ad0 con 0x7fd7741038e0 2024-03-28T18:08:29.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.938+0000 7fd779000640 1 -- 172.21.15.37:0/1388746086 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7741a1ff0 con 0x7fd7741038e0 2024-03-28T18:08:29.940 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:29 smithi170 ceph-mon[36280]: pgmap v322: 97 pgs: 97 active+clean; 582 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 4.7 KiB/s wr, 15 op/s 2024-03-28T18:08:29.940 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:29 smithi170 ceph-mon[36280]: from='client.24709 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:08:29.940 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:29 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/163413978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:29.941 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:29 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1826529871' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:08:29.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.940+0000 7fd779000640 1 -- 172.21.15.37:0/1388746086 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7740688e0 con 0x7fd7741038e0 2024-03-28T18:08:29.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.943+0000 7fd76aa00640 1 -- 172.21.15.37:0/1388746086 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fd7640037a0 con 0x7fd7741038e0 2024-03-28T18:08:29.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.944+0000 7fd76aa00640 1 --2- 172.21.15.37:0/1388746086 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd7580a2890 0x7fd7580a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:29.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.944+0000 7fd76aa00640 1 -- 172.21.15.37:0/1388746086 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd7640cefa0 con 0x7fd7741038e0 2024-03-28T18:08:29.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.944+0000 7fd771400640 1 --2- 172.21.15.37:0/1388746086 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd7580a2890 0x7fd7580a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:29.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.945+0000 7fd771400640 1 --2- 172.21.15.37:0/1388746086 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd7580a2890 0x7fd7580a4d50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd76002c260 tx=0x7fd760000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:29.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:29.948+0000 7fd76aa00640 1 -- 172.21.15.37:0/1388746086 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd76409bf90 con 0x7fd7741038e0 2024-03-28T18:08:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:29 smithi059 ceph-mon[36190]: pgmap v322: 97 pgs: 97 active+clean; 582 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 4.7 KiB/s wr, 15 op/s 2024-03-28T18:08:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:29 smithi059 ceph-mon[36190]: from='client.24709 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:08:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:29 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/163413978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:08:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:29 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1826529871' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:08:30.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:30.148+0000 7fd779000640 1 -- 172.21.15.37:0/1388746086 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd774061ae0 con 0x7fd7580a2890 2024-03-28T18:08:30.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:30.165+0000 7fd76aa00640 1 -- 172.21.15.37:0/1388746086 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258906 (secure 0 0 0) 0x7fd774061ae0 con 0x7fd7580a2890 2024-03-28T18:08:30.169 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:08:30.170 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-03-28T18:08:30.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:30.172+0000 7fd779000640 1 -- 172.21.15.37:0/1388746086 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd7580a2890 msgr2=0x7fd7580a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:30.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:30.172+0000 7fd779000640 1 --2- 172.21.15.37:0/1388746086 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd7580a2890 0x7fd7580a4d50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd76002c260 tx=0x7fd760000bc0 comp rx=0 tx=0).stop 2024-03-28T18:08:30.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:30.172+0000 7fd779000640 1 -- 172.21.15.37:0/1388746086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7741038e0 msgr2=0x7fd774199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:30.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:30.172+0000 7fd779000640 1 --2- 172.21.15.37:0/1388746086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7741038e0 0x7fd774199bc0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd76400e9c0 tx=0x7fd76400ee80 comp rx=0 tx=0).stop 2024-03-28T18:08:30.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:30.172+0000 7fd779000640 1 -- 172.21.15.37:0/1388746086 shutdown_connections 2024-03-28T18:08:30.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:30.172+0000 7fd779000640 1 --2- 172.21.15.37:0/1388746086 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd7580a2890 0x7fd7580a4d50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:30.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:30.172+0000 7fd779000640 1 --2- 172.21.15.37:0/1388746086 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd774105490 0x7fd7741a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:30.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:30.172+0000 7fd779000640 1 --2- 172.21.15.37:0/1388746086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd774104ad0 0x7fd77419a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:30.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:30.172+0000 7fd779000640 1 --2- 172.21.15.37:0/1388746086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7741038e0 0x7fd774199bc0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:30.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:30.172+0000 7fd779000640 1 -- 172.21.15.37:0/1388746086 >> 172.21.15.37:0/1388746086 conn(0x7fd7740fee90 msgr2=0x7fd774107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:30.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:30.172+0000 7fd779000640 1 -- 172.21.15.37:0/1388746086 shutdown_connections 2024-03-28T18:08:30.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:30.173+0000 7fd779000640 1 -- 172.21.15.37:0/1388746086 wait complete. 2024-03-28T18:08:30.201 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-03-28T18:08:29.951926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":633723,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":177,"num_read_kb":185,"num_write":321,"num_write_kb":3019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":259,"ondisk_log_size":259,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376924,"kb_used_data":11128,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564900,"statfs":{"total":1151940427776,"available":1151554457600,"internally_reserved":0,"allocated":11395072,"data_stored":7087663,"data_compressed":109995,"data_compressed_allocated":139264,"data_compressed_original":1777664,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":43355,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":41,"num_read_kb":66,"num_write":140,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.959637"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539441+0000","last_change":"2024-03-28T18:08:20.025576+0000","last_active":"2024-03-28T18:08:21.539441+0000","last_peered":"2024-03-28T18:08:21.539441+0000","last_clean":"2024-03-28T18:08:21.539441+0000","last_became_active":"2024-03-28T18:08:20.025177+0000","last_became_peered":"2024-03-28T18:08:20.025177+0000","last_unstale":"2024-03-28T18:08:21.539441+0000","last_undegraded":"2024-03-28T18:08:21.539441+0000","last_fullsized":"2024-03-28T18:08:21.539441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:14.521170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539526+0000","last_change":"2024-03-28T18:08:20.027418+0000","last_active":"2024-03-28T18:08:21.539526+0000","last_peered":"2024-03-28T18:08:21.539526+0000","last_clean":"2024-03-28T18:08:21.539526+0000","last_became_active":"2024-03-28T18:08:20.026731+0000","last_became_peered":"2024-03-28T18:08:20.026731+0000","last_unstale":"2024-03-28T18:08:21.539526+0000","last_undegraded":"2024-03-28T18:08:21.539526+0000","last_fullsized":"2024-03-28T18:08:21.539526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:52:58.328738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539345+0000","last_change":"2024-03-28T18:08:20.029821+0000","last_active":"2024-03-28T18:08:21.539345+0000","last_peered":"2024-03-28T18:08:21.539345+0000","last_clean":"2024-03-28T18:08:21.539345+0000","last_became_active":"2024-03-28T18:08:20.028245+0000","last_became_peered":"2024-03-28T18:08:20.028245+0000","last_unstale":"2024-03-28T18:08:21.539345+0000","last_undegraded":"2024-03-28T18:08:21.539345+0000","last_fullsized":"2024-03-28T18:08:21.539345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:44.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.510241+0000","last_change":"2024-03-28T18:08:20.015682+0000","last_active":"2024-03-28T18:08:21.510241+0000","last_peered":"2024-03-28T18:08:21.510241+0000","last_clean":"2024-03-28T18:08:21.510241+0000","last_became_active":"2024-03-28T18:08:20.015303+0000","last_became_peered":"2024-03-28T18:08:20.015303+0000","last_unstale":"2024-03-28T18:08:21.510241+0000","last_undegraded":"2024-03-28T18:08:21.510241+0000","last_fullsized":"2024-03-28T18:08:21.510241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:57.456052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.269696+0000","last_change":"2024-03-28T18:08:20.020970+0000","last_active":"2024-03-28T18:08:21.269696+0000","last_peered":"2024-03-28T18:08:21.269696+0000","last_clean":"2024-03-28T18:08:21.269696+0000","last_became_active":"2024-03-28T18:08:20.020646+0000","last_became_peered":"2024-03-28T18:08:20.020646+0000","last_unstale":"2024-03-28T18:08:21.269696+0000","last_undegraded":"2024-03-28T18:08:21.269696+0000","last_fullsized":"2024-03-28T18:08:21.269696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:57:20.422546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539686+0000","last_change":"2024-03-28T18:08:20.022395+0000","last_active":"2024-03-28T18:08:21.539686+0000","last_peered":"2024-03-28T18:08:21.539686+0000","last_clean":"2024-03-28T18:08:21.539686+0000","last_became_active":"2024-03-28T18:08:20.022022+0000","last_became_peered":"2024-03-28T18:08:20.022022+0000","last_unstale":"2024-03-28T18:08:21.539686+0000","last_undegraded":"2024-03-28T18:08:21.539686+0000","last_fullsized":"2024-03-28T18:08:21.539686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:32.590751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539423+0000","last_change":"2024-03-28T18:08:20.019990+0000","last_active":"2024-03-28T18:08:21.539423+0000","last_peered":"2024-03-28T18:08:21.539423+0000","last_clean":"2024-03-28T18:08:21.539423+0000","last_became_active":"2024-03-28T18:08:20.019618+0000","last_became_peered":"2024-03-28T18:08:20.019618+0000","last_unstale":"2024-03-28T18:08:21.539423+0000","last_undegraded":"2024-03-28T18:08:21.539423+0000","last_fullsized":"2024-03-28T18:08:21.539423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:48.442579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.525075+0000","last_change":"2024-03-28T18:08:20.029701+0000","last_active":"2024-03-28T18:08:21.525075+0000","last_peered":"2024-03-28T18:08:21.525075+0000","last_clean":"2024-03-28T18:08:21.525075+0000","last_became_active":"2024-03-28T18:08:20.029181+0000","last_became_peered":"2024-03-28T18:08:20.029181+0000","last_unstale":"2024-03-28T18:08:21.525075+0000","last_undegraded":"2024-03-28T18:08:21.525075+0000","last_fullsized":"2024-03-28T18:08:21.525075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:00:41.213388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996886+0000","last_change":"2024-03-28T18:08:20.009732+0000","last_active":"2024-03-28T18:08:20.996886+0000","last_peered":"2024-03-28T18:08:20.996886+0000","last_clean":"2024-03-28T18:08:20.996886+0000","last_became_active":"2024-03-28T18:08:20.009288+0000","last_became_peered":"2024-03-28T18:08:20.009288+0000","last_unstale":"2024-03-28T18:08:20.996886+0000","last_undegraded":"2024-03-28T18:08:20.996886+0000","last_fullsized":"2024-03-28T18:08:20.996886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:20:54.917964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.540205+0000","last_change":"2024-03-28T18:08:20.009687+0000","last_active":"2024-03-28T18:08:21.540205+0000","last_peered":"2024-03-28T18:08:21.540205+0000","last_clean":"2024-03-28T18:08:21.540205+0000","last_became_active":"2024-03-28T18:08:20.008364+0000","last_became_peered":"2024-03-28T18:08:20.008364+0000","last_unstale":"2024-03-28T18:08:21.540205+0000","last_undegraded":"2024-03-28T18:08:21.540205+0000","last_fullsized":"2024-03-28T18:08:21.540205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:08.419202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539987+0000","last_change":"2024-03-28T18:08:20.011508+0000","last_active":"2024-03-28T18:08:21.539987+0000","last_peered":"2024-03-28T18:08:21.539987+0000","last_clean":"2024-03-28T18:08:21.539987+0000","last_became_active":"2024-03-28T18:08:20.011117+0000","last_became_peered":"2024-03-28T18:08:20.011117+0000","last_unstale":"2024-03-28T18:08:21.539987+0000","last_undegraded":"2024-03-28T18:08:21.539987+0000","last_fullsized":"2024-03-28T18:08:21.539987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:06:19.278931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.510180+0000","last_change":"2024-03-28T18:08:20.011403+0000","last_active":"2024-03-28T18:08:21.510180+0000","last_peered":"2024-03-28T18:08:21.510180+0000","last_clean":"2024-03-28T18:08:21.510180+0000","last_became_active":"2024-03-28T18:08:20.011025+0000","last_became_peered":"2024-03-28T18:08:20.011025+0000","last_unstale":"2024-03-28T18:08:21.510180+0000","last_undegraded":"2024-03-28T18:08:21.510180+0000","last_fullsized":"2024-03-28T18:08:21.510180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:13.599432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996683+0000","last_change":"2024-03-28T18:08:20.001664+0000","last_active":"2024-03-28T18:08:20.996683+0000","last_peered":"2024-03-28T18:08:20.996683+0000","last_clean":"2024-03-28T18:08:20.996683+0000","last_became_active":"2024-03-28T18:08:20.000910+0000","last_became_peered":"2024-03-28T18:08:20.000910+0000","last_unstale":"2024-03-28T18:08:20.996683+0000","last_undegraded":"2024-03-28T18:08:20.996683+0000","last_fullsized":"2024-03-28T18:08:20.996683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:00:23.796302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539047+0000","last_change":"2024-03-28T18:08:20.027174+0000","last_active":"2024-03-28T18:08:21.539047+0000","last_peered":"2024-03-28T18:08:21.539047+0000","last_clean":"2024-03-28T18:08:21.539047+0000","last_became_active":"2024-03-28T18:08:20.026113+0000","last_became_peered":"2024-03-28T18:08:20.026113+0000","last_unstale":"2024-03-28T18:08:21.539047+0000","last_undegraded":"2024-03-28T18:08:21.539047+0000","last_fullsized":"2024-03-28T18:08:21.539047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:55:00.061696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.538870+0000","last_change":"2024-03-28T18:08:20.020060+0000","last_active":"2024-03-28T18:08:21.538870+0000","last_peered":"2024-03-28T18:08:21.538870+0000","last_clean":"2024-03-28T18:08:21.538870+0000","last_became_active":"2024-03-28T18:08:20.019691+0000","last_became_peered":"2024-03-28T18:08:20.019691+0000","last_unstale":"2024-03-28T18:08:21.538870+0000","last_undegraded":"2024-03-28T18:08:21.538870+0000","last_fullsized":"2024-03-28T18:08:21.538870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:27.889737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996294+0000","last_change":"2024-03-28T18:08:20.024239+0000","last_active":"2024-03-28T18:08:20.996294+0000","last_peered":"2024-03-28T18:08:20.996294+0000","last_clean":"2024-03-28T18:08:20.996294+0000","last_became_active":"2024-03-28T18:08:20.023874+0000","last_became_peered":"2024-03-28T18:08:20.023874+0000","last_unstale":"2024-03-28T18:08:20.996294+0000","last_undegraded":"2024-03-28T18:08:20.996294+0000","last_fullsized":"2024-03-28T18:08:20.996294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:05:32.903195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539571+0000","last_change":"2024-03-28T18:08:20.033322+0000","last_active":"2024-03-28T18:08:21.539571+0000","last_peered":"2024-03-28T18:08:21.539571+0000","last_clean":"2024-03-28T18:08:21.539571+0000","last_became_active":"2024-03-28T18:08:20.033048+0000","last_became_peered":"2024-03-28T18:08:20.033048+0000","last_unstale":"2024-03-28T18:08:21.539571+0000","last_undegraded":"2024-03-28T18:08:21.539571+0000","last_fullsized":"2024-03-28T18:08:21.539571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:18:59.234141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.522842+0000","last_change":"2024-03-28T18:08:20.022093+0000","last_active":"2024-03-28T18:08:21.522842+0000","last_peered":"2024-03-28T18:08:21.522842+0000","last_clean":"2024-03-28T18:08:21.522842+0000","last_became_active":"2024-03-28T18:08:20.021411+0000","last_became_peered":"2024-03-28T18:08:20.021411+0000","last_unstale":"2024-03-28T18:08:21.522842+0000","last_undegraded":"2024-03-28T18:08:21.522842+0000","last_fullsized":"2024-03-28T18:08:21.522842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:09:40.366628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.995920+0000","last_change":"2024-03-28T18:08:20.028893+0000","last_active":"2024-03-28T18:08:20.995920+0000","last_peered":"2024-03-28T18:08:20.995920+0000","last_clean":"2024-03-28T18:08:20.995920+0000","last_became_active":"2024-03-28T18:08:20.028634+0000","last_became_peered":"2024-03-28T18:08:20.028634+0000","last_unstale":"2024-03-28T18:08:20.995920+0000","last_undegraded":"2024-03-28T18:08:20.995920+0000","last_fullsized":"2024-03-28T18:08:20.995920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:32:27.454406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:24.424132+0000","last_change":"2024-03-28T18:08:19.481368+0000","last_active":"2024-03-28T18:08:24.424132+0000","last_peered":"2024-03-28T18:08:24.424132+0000","last_clean":"2024-03-28T18:08:24.424132+0000","last_became_active":"2024-03-28T18:08:19.480980+0000","last_became_peered":"2024-03-28T18:08:19.480980+0000","last_unstale":"2024-03-28T18:08:24.424132+0000","last_undegraded":"2024-03-28T18:08:24.424132+0000","last_fullsized":"2024-03-28T18:08:24.424132+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:57.387755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.526039+0000","last_change":"2024-03-28T18:08:20.027855+0000","last_active":"2024-03-28T18:08:21.526039+0000","last_peered":"2024-03-28T18:08:21.526039+0000","last_clean":"2024-03-28T18:08:21.526039+0000","last_became_active":"2024-03-28T18:08:20.027565+0000","last_became_peered":"2024-03-28T18:08:20.027565+0000","last_unstale":"2024-03-28T18:08:21.526039+0000","last_undegraded":"2024-03-28T18:08:21.526039+0000","last_fullsized":"2024-03-28T18:08:21.526039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:36:37.596573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:22.540745+0000","last_change":"2024-03-28T18:08:19.499055+0000","last_active":"2024-03-28T18:08:22.540745+0000","last_peered":"2024-03-28T18:08:22.540745+0000","last_clean":"2024-03-28T18:08:22.540745+0000","last_became_active":"2024-03-28T18:08:19.497964+0000","last_became_peered":"2024-03-28T18:08:19.497964+0000","last_unstale":"2024-03-28T18:08:22.540745+0000","last_undegraded":"2024-03-28T18:08:22.540745+0000","last_fullsized":"2024-03-28T18:08:22.540745+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:38:45.620406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539645+0000","last_change":"2024-03-28T18:08:20.028967+0000","last_active":"2024-03-28T18:08:21.539645+0000","last_peered":"2024-03-28T18:08:21.539645+0000","last_clean":"2024-03-28T18:08:21.539645+0000","last_became_active":"2024-03-28T18:08:20.028607+0000","last_became_peered":"2024-03-28T18:08:20.028607+0000","last_unstale":"2024-03-28T18:08:21.539645+0000","last_undegraded":"2024-03-28T18:08:21.539645+0000","last_fullsized":"2024-03-28T18:08:21.539645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:21:23.994077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:24.415509+0000","last_change":"2024-03-28T18:08:19.481998+0000","last_active":"2024-03-28T18:08:24.415509+0000","last_peered":"2024-03-28T18:08:24.415509+0000","last_clean":"2024-03-28T18:08:24.415509+0000","last_became_active":"2024-03-28T18:08:19.481135+0000","last_became_peered":"2024-03-28T18:08:19.481135+0000","last_unstale":"2024-03-28T18:08:24.415509+0000","last_undegraded":"2024-03-28T18:08:24.415509+0000","last_fullsized":"2024-03-28T18:08:24.415509+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:37:26.171036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996593+0000","last_change":"2024-03-28T18:08:20.009878+0000","last_active":"2024-03-28T18:08:20.996593+0000","last_peered":"2024-03-28T18:08:20.996593+0000","last_clean":"2024-03-28T18:08:20.996593+0000","last_became_active":"2024-03-28T18:08:20.009534+0000","last_became_peered":"2024-03-28T18:08:20.009534+0000","last_unstale":"2024-03-28T18:08:20.996593+0000","last_undegraded":"2024-03-28T18:08:20.996593+0000","last_fullsized":"2024-03-28T18:08:20.996593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:58.975854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.269908+0000","last_change":"2024-03-28T18:08:19.472613+0000","last_active":"2024-03-28T18:08:21.269908+0000","last_peered":"2024-03-28T18:08:21.269908+0000","last_clean":"2024-03-28T18:08:21.269908+0000","last_became_active":"2024-03-28T18:08:19.472186+0000","last_became_peered":"2024-03-28T18:08:19.472186+0000","last_unstale":"2024-03-28T18:08:21.269908+0000","last_undegraded":"2024-03-28T18:08:21.269908+0000","last_fullsized":"2024-03-28T18:08:21.269908+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:50.807609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.326663+0000","last_change":"2024-03-28T18:08:20.008242+0000","last_active":"2024-03-28T18:08:21.326663+0000","last_peered":"2024-03-28T18:08:21.326663+0000","last_clean":"2024-03-28T18:08:21.326663+0000","last_became_active":"2024-03-28T18:08:20.007942+0000","last_became_peered":"2024-03-28T18:08:20.007942+0000","last_unstale":"2024-03-28T18:08:21.326663+0000","last_undegraded":"2024-03-28T18:08:21.326663+0000","last_fullsized":"2024-03-28T18:08:21.326663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:23.288978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:25.425258+0000","last_change":"2024-03-28T18:08:19.480869+0000","last_active":"2024-03-28T18:08:25.425258+0000","last_peered":"2024-03-28T18:08:25.425258+0000","last_clean":"2024-03-28T18:08:25.425258+0000","last_became_active":"2024-03-28T18:08:19.480441+0000","last_became_peered":"2024-03-28T18:08:19.480441+0000","last_unstale":"2024-03-28T18:08:25.425258+0000","last_undegraded":"2024-03-28T18:08:25.425258+0000","last_fullsized":"2024-03-28T18:08:25.425258+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:26.640605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.269760+0000","last_change":"2024-03-28T18:08:20.021530+0000","last_active":"2024-03-28T18:08:21.269760+0000","last_peered":"2024-03-28T18:08:21.269760+0000","last_clean":"2024-03-28T18:08:21.269760+0000","last_became_active":"2024-03-28T18:08:20.021179+0000","last_became_peered":"2024-03-28T18:08:20.021179+0000","last_unstale":"2024-03-28T18:08:21.269760+0000","last_undegraded":"2024-03-28T18:08:21.269760+0000","last_fullsized":"2024-03-28T18:08:21.269760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:43:07.300300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:24.417719+0000","last_change":"2024-03-28T18:08:19.494597+0000","last_active":"2024-03-28T18:08:24.417719+0000","last_peered":"2024-03-28T18:08:24.417719+0000","last_clean":"2024-03-28T18:08:24.417719+0000","last_became_active":"2024-03-28T18:08:19.494178+0000","last_became_peered":"2024-03-28T18:08:19.494178+0000","last_unstale":"2024-03-28T18:08:24.417719+0000","last_undegraded":"2024-03-28T18:08:24.417719+0000","last_fullsized":"2024-03-28T18:08:24.417719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:08:10.257768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.326878+0000","last_change":"2024-03-28T18:08:20.021062+0000","last_active":"2024-03-28T18:08:21.326878+0000","last_peered":"2024-03-28T18:08:21.326878+0000","last_clean":"2024-03-28T18:08:21.326878+0000","last_became_active":"2024-03-28T18:08:20.020758+0000","last_became_peered":"2024-03-28T18:08:20.020758+0000","last_unstale":"2024-03-28T18:08:21.326878+0000","last_undegraded":"2024-03-28T18:08:21.326878+0000","last_fullsized":"2024-03-28T18:08:21.326878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:47:39.656513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:24.422435+0000","last_change":"2024-03-28T18:08:19.480189+0000","last_active":"2024-03-28T18:08:24.422435+0000","last_peered":"2024-03-28T18:08:24.422435+0000","last_clean":"2024-03-28T18:08:24.422435+0000","last_became_active":"2024-03-28T18:08:19.479812+0000","last_became_peered":"2024-03-28T18:08:19.479812+0000","last_unstale":"2024-03-28T18:08:24.422435+0000","last_undegraded":"2024-03-28T18:08:24.422435+0000","last_fullsized":"2024-03-28T18:08:24.422435+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:46.408565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.525379+0000","last_change":"2024-03-28T18:08:20.023635+0000","last_active":"2024-03-28T18:08:21.525379+0000","last_peered":"2024-03-28T18:08:21.525379+0000","last_clean":"2024-03-28T18:08:21.525379+0000","last_became_active":"2024-03-28T18:08:20.023199+0000","last_became_peered":"2024-03-28T18:08:20.023199+0000","last_unstale":"2024-03-28T18:08:21.525379+0000","last_undegraded":"2024-03-28T18:08:21.525379+0000","last_fullsized":"2024-03-28T18:08:21.525379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:12.477899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.326295+0000","last_change":"2024-03-28T18:08:19.480858+0000","last_active":"2024-03-28T18:08:21.326295+0000","last_peered":"2024-03-28T18:08:21.326295+0000","last_clean":"2024-03-28T18:08:21.326295+0000","last_became_active":"2024-03-28T18:08:19.480418+0000","last_became_peered":"2024-03-28T18:08:19.480418+0000","last_unstale":"2024-03-28T18:08:21.326295+0000","last_undegraded":"2024-03-28T18:08:21.326295+0000","last_fullsized":"2024-03-28T18:08:21.326295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:59:57.788915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.522608+0000","last_change":"2024-03-28T18:08:20.031719+0000","last_active":"2024-03-28T18:08:21.522608+0000","last_peered":"2024-03-28T18:08:21.522608+0000","last_clean":"2024-03-28T18:08:21.522608+0000","last_became_active":"2024-03-28T18:08:20.029482+0000","last_became_peered":"2024-03-28T18:08:20.029482+0000","last_unstale":"2024-03-28T18:08:21.522608+0000","last_undegraded":"2024-03-28T18:08:21.522608+0000","last_fullsized":"2024-03-28T18:08:21.522608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:13.226596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:28.429879+0000","last_change":"2024-03-28T18:08:19.483346+0000","last_active":"2024-03-28T18:08:28.429879+0000","last_peered":"2024-03-28T18:08:28.429879+0000","last_clean":"2024-03-28T18:08:28.429879+0000","last_became_active":"2024-03-28T18:08:19.483016+0000","last_became_peered":"2024-03-28T18:08:19.483016+0000","last_unstale":"2024-03-28T18:08:28.429879+0000","last_undegraded":"2024-03-28T18:08:28.429879+0000","last_fullsized":"2024-03-28T18:08:28.429879+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:34:33.008477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996738+0000","last_change":"2024-03-28T18:08:20.020478+0000","last_active":"2024-03-28T18:08:20.996738+0000","last_peered":"2024-03-28T18:08:20.996738+0000","last_clean":"2024-03-28T18:08:20.996738+0000","last_became_active":"2024-03-28T18:08:20.020054+0000","last_became_peered":"2024-03-28T18:08:20.020054+0000","last_unstale":"2024-03-28T18:08:20.996738+0000","last_undegraded":"2024-03-28T18:08:20.996738+0000","last_fullsized":"2024-03-28T18:08:20.996738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:13:43.655549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:23.390504+0000","last_change":"2024-03-28T18:08:19.479386+0000","last_active":"2024-03-28T18:08:23.390504+0000","last_peered":"2024-03-28T18:08:23.390504+0000","last_clean":"2024-03-28T18:08:23.390504+0000","last_became_active":"2024-03-28T18:08:19.478985+0000","last_became_peered":"2024-03-28T18:08:19.478985+0000","last_unstale":"2024-03-28T18:08:23.390504+0000","last_undegraded":"2024-03-28T18:08:23.390504+0000","last_fullsized":"2024-03-28T18:08:23.390504+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:37:22.359475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539639+0000","last_change":"2024-03-28T18:08:20.021790+0000","last_active":"2024-03-28T18:08:21.539639+0000","last_peered":"2024-03-28T18:08:21.539639+0000","last_clean":"2024-03-28T18:08:21.539639+0000","last_became_active":"2024-03-28T18:08:20.021031+0000","last_became_peered":"2024-03-28T18:08:20.021031+0000","last_unstale":"2024-03-28T18:08:21.539639+0000","last_undegraded":"2024-03-28T18:08:21.539639+0000","last_fullsized":"2024-03-28T18:08:21.539639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:59:50.897692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:23.389970+0000","last_change":"2024-03-28T18:08:19.481576+0000","last_active":"2024-03-28T18:08:23.389970+0000","last_peered":"2024-03-28T18:08:23.389970+0000","last_clean":"2024-03-28T18:08:23.389970+0000","last_became_active":"2024-03-28T18:08:19.481226+0000","last_became_peered":"2024-03-28T18:08:19.481226+0000","last_unstale":"2024-03-28T18:08:23.389970+0000","last_undegraded":"2024-03-28T18:08:23.389970+0000","last_fullsized":"2024-03-28T18:08:23.389970+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:54:44.677343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996555+0000","last_change":"2024-03-28T18:08:20.013231+0000","last_active":"2024-03-28T18:08:20.996555+0000","last_peered":"2024-03-28T18:08:20.996555+0000","last_clean":"2024-03-28T18:08:20.996555+0000","last_became_active":"2024-03-28T18:08:20.012267+0000","last_became_peered":"2024-03-28T18:08:20.012267+0000","last_unstale":"2024-03-28T18:08:20.996555+0000","last_undegraded":"2024-03-28T18:08:20.996555+0000","last_fullsized":"2024-03-28T18:08:20.996555+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:07:20.293607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'19","reported_seq":48,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:29.434910+0000","last_change":"2024-03-28T18:08:19.477559+0000","last_active":"2024-03-28T18:08:29.434910+0000","last_peered":"2024-03-28T18:08:29.434910+0000","last_clean":"2024-03-28T18:08:29.434910+0000","last_became_active":"2024-03-28T18:08:19.477217+0000","last_became_peered":"2024-03-28T18:08:19.477217+0000","last_unstale":"2024-03-28T18:08:29.434910+0000","last_undegraded":"2024-03-28T18:08:29.434910+0000","last_fullsized":"2024-03-28T18:08:29.434910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:37.071671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39628,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":45,"num_write":20,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.269994+0000","last_change":"2024-03-28T18:08:20.011303+0000","last_active":"2024-03-28T18:08:21.269994+0000","last_peered":"2024-03-28T18:08:21.269994+0000","last_clean":"2024-03-28T18:08:21.269994+0000","last_became_active":"2024-03-28T18:08:20.010997+0000","last_became_peered":"2024-03-28T18:08:20.010997+0000","last_unstale":"2024-03-28T18:08:21.269994+0000","last_undegraded":"2024-03-28T18:08:21.269994+0000","last_fullsized":"2024-03-28T18:08:21.269994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:10:56.036132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:24.419778+0000","last_change":"2024-03-28T18:08:19.477342+0000","last_active":"2024-03-28T18:08:24.419778+0000","last_peered":"2024-03-28T18:08:24.419778+0000","last_clean":"2024-03-28T18:08:24.419778+0000","last_became_active":"2024-03-28T18:08:19.476984+0000","last_became_peered":"2024-03-28T18:08:19.476984+0000","last_unstale":"2024-03-28T18:08:24.419778+0000","last_undegraded":"2024-03-28T18:08:24.419778+0000","last_fullsized":"2024-03-28T18:08:24.419778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:15:42.606213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996321+0000","last_change":"2024-03-28T18:08:20.030524+0000","last_active":"2024-03-28T18:08:20.996321+0000","last_peered":"2024-03-28T18:08:20.996321+0000","last_clean":"2024-03-28T18:08:20.996321+0000","last_became_active":"2024-03-28T18:08:20.030227+0000","last_became_peered":"2024-03-28T18:08:20.030227+0000","last_unstale":"2024-03-28T18:08:20.996321+0000","last_undegraded":"2024-03-28T18:08:20.996321+0000","last_fullsized":"2024-03-28T18:08:20.996321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:07:41.763739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:23.392978+0000","last_change":"2024-03-28T18:08:19.481169+0000","last_active":"2024-03-28T18:08:23.392978+0000","last_peered":"2024-03-28T18:08:23.392978+0000","last_clean":"2024-03-28T18:08:23.392978+0000","last_became_active":"2024-03-28T18:08:19.480634+0000","last_became_peered":"2024-03-28T18:08:19.480634+0000","last_unstale":"2024-03-28T18:08:23.392978+0000","last_undegraded":"2024-03-28T18:08:23.392978+0000","last_fullsized":"2024-03-28T18:08:23.392978+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:39.265215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539597+0000","last_change":"2024-03-28T18:08:20.015163+0000","last_active":"2024-03-28T18:08:21.539597+0000","last_peered":"2024-03-28T18:08:21.539597+0000","last_clean":"2024-03-28T18:08:21.539597+0000","last_became_active":"2024-03-28T18:08:20.014765+0000","last_became_peered":"2024-03-28T18:08:20.014765+0000","last_unstale":"2024-03-28T18:08:21.539597+0000","last_undegraded":"2024-03-28T18:08:21.539597+0000","last_fullsized":"2024-03-28T18:08:21.539597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:51.035532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'3","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:28.430783+0000","last_change":"2024-03-28T18:08:19.479086+0000","last_active":"2024-03-28T18:08:28.430783+0000","last_peered":"2024-03-28T18:08:28.430783+0000","last_clean":"2024-03-28T18:08:28.430783+0000","last_became_active":"2024-03-28T18:08:19.478748+0000","last_became_peered":"2024-03-28T18:08:19.478748+0000","last_unstale":"2024-03-28T18:08:28.430783+0000","last_undegraded":"2024-03-28T18:08:28.430783+0000","last_fullsized":"2024-03-28T18:08:28.430783+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:46:59.566998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539150+0000","last_change":"2024-03-28T18:08:20.016375+0000","last_active":"2024-03-28T18:08:21.539150+0000","last_peered":"2024-03-28T18:08:21.539150+0000","last_clean":"2024-03-28T18:08:21.539150+0000","last_became_active":"2024-03-28T18:08:20.015670+0000","last_became_peered":"2024-03-28T18:08:20.015670+0000","last_unstale":"2024-03-28T18:08:21.539150+0000","last_undegraded":"2024-03-28T18:08:21.539150+0000","last_fullsized":"2024-03-28T18:08:21.539150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:48.388617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:24.421364+0000","last_change":"2024-03-28T18:08:19.482934+0000","last_active":"2024-03-28T18:08:24.421364+0000","last_peered":"2024-03-28T18:08:24.421364+0000","last_clean":"2024-03-28T18:08:24.421364+0000","last_became_active":"2024-03-28T18:08:19.482623+0000","last_became_peered":"2024-03-28T18:08:19.482623+0000","last_unstale":"2024-03-28T18:08:24.421364+0000","last_undegraded":"2024-03-28T18:08:24.421364+0000","last_fullsized":"2024-03-28T18:08:24.421364+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:06:31.718675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.326366+0000","last_change":"2024-03-28T18:08:20.007457+0000","last_active":"2024-03-28T18:08:21.326366+0000","last_peered":"2024-03-28T18:08:21.326366+0000","last_clean":"2024-03-28T18:08:21.326366+0000","last_became_active":"2024-03-28T18:08:20.007090+0000","last_became_peered":"2024-03-28T18:08:20.007090+0000","last_unstale":"2024-03-28T18:08:21.326366+0000","last_undegraded":"2024-03-28T18:08:21.326366+0000","last_fullsized":"2024-03-28T18:08:21.326366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:22:45.702706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:25.412096+0000","last_change":"2024-03-28T18:08:19.475104+0000","last_active":"2024-03-28T18:08:25.412096+0000","last_peered":"2024-03-28T18:08:25.412096+0000","last_clean":"2024-03-28T18:08:25.412096+0000","last_became_active":"2024-03-28T18:08:19.474754+0000","last_became_peered":"2024-03-28T18:08:19.474754+0000","last_unstale":"2024-03-28T18:08:25.412096+0000","last_undegraded":"2024-03-28T18:08:25.412096+0000","last_fullsized":"2024-03-28T18:08:25.412096+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:19.085631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539338+0000","last_change":"2024-03-28T18:08:20.009347+0000","last_active":"2024-03-28T18:08:21.539338+0000","last_peered":"2024-03-28T18:08:21.539338+0000","last_clean":"2024-03-28T18:08:21.539338+0000","last_became_active":"2024-03-28T18:08:20.009012+0000","last_became_peered":"2024-03-28T18:08:20.009012+0000","last_unstale":"2024-03-28T18:08:21.539338+0000","last_undegraded":"2024-03-28T18:08:21.539338+0000","last_fullsized":"2024-03-28T18:08:21.539338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:04:26.625830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.522533+0000","last_change":"2024-03-28T18:08:19.482414+0000","last_active":"2024-03-28T18:08:21.522533+0000","last_peered":"2024-03-28T18:08:21.522533+0000","last_clean":"2024-03-28T18:08:21.522533+0000","last_became_active":"2024-03-28T18:08:19.481830+0000","last_became_peered":"2024-03-28T18:08:19.481830+0000","last_unstale":"2024-03-28T18:08:21.522533+0000","last_undegraded":"2024-03-28T18:08:21.522533+0000","last_fullsized":"2024-03-28T18:08:21.522533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:27:14.259355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539514+0000","last_change":"2024-03-28T18:08:20.030687+0000","last_active":"2024-03-28T18:08:21.539514+0000","last_peered":"2024-03-28T18:08:21.539514+0000","last_clean":"2024-03-28T18:08:21.539514+0000","last_became_active":"2024-03-28T18:08:20.030308+0000","last_became_peered":"2024-03-28T18:08:20.030308+0000","last_unstale":"2024-03-28T18:08:21.539514+0000","last_undegraded":"2024-03-28T18:08:21.539514+0000","last_fullsized":"2024-03-28T18:08:21.539514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:31.745952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539566+0000","last_change":"2024-03-28T18:08:19.494446+0000","last_active":"2024-03-28T18:08:21.539566+0000","last_peered":"2024-03-28T18:08:21.539566+0000","last_clean":"2024-03-28T18:08:21.539566+0000","last_became_active":"2024-03-28T18:08:19.494036+0000","last_became_peered":"2024-03-28T18:08:19.494036+0000","last_unstale":"2024-03-28T18:08:21.539566+0000","last_undegraded":"2024-03-28T18:08:21.539566+0000","last_fullsized":"2024-03-28T18:08:21.539566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:42:53.660741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539623+0000","last_change":"2024-03-28T18:08:20.025706+0000","last_active":"2024-03-28T18:08:21.539623+0000","last_peered":"2024-03-28T18:08:21.539623+0000","last_clean":"2024-03-28T18:08:21.539623+0000","last_became_active":"2024-03-28T18:08:20.025343+0000","last_became_peered":"2024-03-28T18:08:20.025343+0000","last_unstale":"2024-03-28T18:08:21.539623+0000","last_undegraded":"2024-03-28T18:08:21.539623+0000","last_fullsized":"2024-03-28T18:08:21.539623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:32:27.883940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.326590+0000","last_change":"2024-03-28T18:08:20.009636+0000","last_active":"2024-03-28T18:08:21.326590+0000","last_peered":"2024-03-28T18:08:21.326590+0000","last_clean":"2024-03-28T18:08:21.326590+0000","last_became_active":"2024-03-28T18:08:20.009178+0000","last_became_peered":"2024-03-28T18:08:20.009178+0000","last_unstale":"2024-03-28T18:08:21.326590+0000","last_undegraded":"2024-03-28T18:08:21.326590+0000","last_fullsized":"2024-03-28T18:08:21.326590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:28.461411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:24.422708+0000","last_change":"2024-03-28T18:08:19.501287+0000","last_active":"2024-03-28T18:08:24.422708+0000","last_peered":"2024-03-28T18:08:24.422708+0000","last_clean":"2024-03-28T18:08:24.422708+0000","last_became_active":"2024-03-28T18:08:19.500920+0000","last_became_peered":"2024-03-28T18:08:19.500920+0000","last_unstale":"2024-03-28T18:08:24.422708+0000","last_undegraded":"2024-03-28T18:08:24.422708+0000","last_fullsized":"2024-03-28T18:08:24.422708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:29.882164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.525536+0000","last_change":"2024-03-28T18:08:20.020811+0000","last_active":"2024-03-28T18:08:21.525536+0000","last_peered":"2024-03-28T18:08:21.525536+0000","last_clean":"2024-03-28T18:08:21.525536+0000","last_became_active":"2024-03-28T18:08:20.020381+0000","last_became_peered":"2024-03-28T18:08:20.020381+0000","last_unstale":"2024-03-28T18:08:21.525536+0000","last_undegraded":"2024-03-28T18:08:21.525536+0000","last_fullsized":"2024-03-28T18:08:21.525536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:10:35.999858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:25.421704+0000","last_change":"2024-03-28T18:08:19.478020+0000","last_active":"2024-03-28T18:08:25.421704+0000","last_peered":"2024-03-28T18:08:25.421704+0000","last_clean":"2024-03-28T18:08:25.421704+0000","last_became_active":"2024-03-28T18:08:19.477627+0000","last_became_peered":"2024-03-28T18:08:19.477627+0000","last_unstale":"2024-03-28T18:08:25.421704+0000","last_undegraded":"2024-03-28T18:08:25.421704+0000","last_fullsized":"2024-03-28T18:08:25.421704+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:00:01.649974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":349,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539190+0000","last_change":"2024-03-28T18:06:58.750932+0000","last_active":"2024-03-28T18:08:21.539190+0000","last_peered":"2024-03-28T18:08:21.539190+0000","last_clean":"2024-03-28T18:08:21.539190+0000","last_became_active":"2024-03-28T18:06:58.750309+0000","last_became_peered":"2024-03-28T18:06:58.750309+0000","last_unstale":"2024-03-28T18:08:21.539190+0000","last_undegraded":"2024-03-28T18:08:21.539190+0000","last_fullsized":"2024-03-28T18:08:21.539190+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:03:00.832683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:03:00.832683+0000","last_clean_scrub_stamp":"2024-03-28T18:03:00.832683+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:46.001886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.995824+0000","last_change":"2024-03-28T18:08:20.009573+0000","last_active":"2024-03-28T18:08:20.995824+0000","last_peered":"2024-03-28T18:08:20.995824+0000","last_clean":"2024-03-28T18:08:20.995824+0000","last_became_active":"2024-03-28T18:08:20.009143+0000","last_became_peered":"2024-03-28T18:08:20.009143+0000","last_unstale":"2024-03-28T18:08:20.995824+0000","last_undegraded":"2024-03-28T18:08:20.995824+0000","last_fullsized":"2024-03-28T18:08:20.995824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:37:50.643356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:24.416384+0000","last_change":"2024-03-28T18:08:19.483733+0000","last_active":"2024-03-28T18:08:24.416384+0000","last_peered":"2024-03-28T18:08:24.416384+0000","last_clean":"2024-03-28T18:08:24.416384+0000","last_became_active":"2024-03-28T18:08:19.483366+0000","last_became_peered":"2024-03-28T18:08:19.483366+0000","last_unstale":"2024-03-28T18:08:24.416384+0000","last_undegraded":"2024-03-28T18:08:24.416384+0000","last_fullsized":"2024-03-28T18:08:24.416384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:29.192442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:23.393731+0000","last_change":"2024-03-28T18:08:19.481844+0000","last_active":"2024-03-28T18:08:23.393731+0000","last_peered":"2024-03-28T18:08:23.393731+0000","last_clean":"2024-03-28T18:08:23.393731+0000","last_became_active":"2024-03-28T18:08:19.481430+0000","last_became_peered":"2024-03-28T18:08:19.481430+0000","last_unstale":"2024-03-28T18:08:23.393731+0000","last_undegraded":"2024-03-28T18:08:23.393731+0000","last_fullsized":"2024-03-28T18:08:23.393731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:21:28.789883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996326+0000","last_change":"2024-03-28T18:08:20.021037+0000","last_active":"2024-03-28T18:08:20.996326+0000","last_peered":"2024-03-28T18:08:20.996326+0000","last_clean":"2024-03-28T18:08:20.996326+0000","last_became_active":"2024-03-28T18:08:20.020680+0000","last_became_peered":"2024-03-28T18:08:20.020680+0000","last_unstale":"2024-03-28T18:08:20.996326+0000","last_undegraded":"2024-03-28T18:08:20.996326+0000","last_fullsized":"2024-03-28T18:08:20.996326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:08:28.825774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:26.521323+0000","last_change":"2024-03-28T18:08:19.495681+0000","last_active":"2024-03-28T18:08:26.521323+0000","last_peered":"2024-03-28T18:08:26.521323+0000","last_clean":"2024-03-28T18:08:26.521323+0000","last_became_active":"2024-03-28T18:08:19.495289+0000","last_became_peered":"2024-03-28T18:08:19.495289+0000","last_unstale":"2024-03-28T18:08:26.521323+0000","last_undegraded":"2024-03-28T18:08:26.521323+0000","last_fullsized":"2024-03-28T18:08:26.521323+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:36:54.861716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.526414+0000","last_change":"2024-03-28T18:08:20.024938+0000","last_active":"2024-03-28T18:08:21.526414+0000","last_peered":"2024-03-28T18:08:21.526414+0000","last_clean":"2024-03-28T18:08:21.526414+0000","last_became_active":"2024-03-28T18:08:20.024536+0000","last_became_peered":"2024-03-28T18:08:20.024536+0000","last_unstale":"2024-03-28T18:08:21.526414+0000","last_undegraded":"2024-03-28T18:08:21.526414+0000","last_fullsized":"2024-03-28T18:08:21.526414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:46:27.080005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:23.391895+0000","last_change":"2024-03-28T18:08:19.480310+0000","last_active":"2024-03-28T18:08:23.391895+0000","last_peered":"2024-03-28T18:08:23.391895+0000","last_clean":"2024-03-28T18:08:23.391895+0000","last_became_active":"2024-03-28T18:08:19.479919+0000","last_became_peered":"2024-03-28T18:08:19.479919+0000","last_unstale":"2024-03-28T18:08:23.391895+0000","last_undegraded":"2024-03-28T18:08:23.391895+0000","last_fullsized":"2024-03-28T18:08:23.391895+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:36:26.239515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539455+0000","last_change":"2024-03-28T18:08:20.033974+0000","last_active":"2024-03-28T18:08:21.539455+0000","last_peered":"2024-03-28T18:08:21.539455+0000","last_clean":"2024-03-28T18:08:21.539455+0000","last_became_active":"2024-03-28T18:08:20.033544+0000","last_became_peered":"2024-03-28T18:08:20.033544+0000","last_unstale":"2024-03-28T18:08:21.539455+0000","last_undegraded":"2024-03-28T18:08:21.539455+0000","last_fullsized":"2024-03-28T18:08:21.539455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:45:09.617988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:23.389708+0000","last_change":"2024-03-28T18:08:19.482604+0000","last_active":"2024-03-28T18:08:23.389708+0000","last_peered":"2024-03-28T18:08:23.389708+0000","last_clean":"2024-03-28T18:08:23.389708+0000","last_became_active":"2024-03-28T18:08:19.482248+0000","last_became_peered":"2024-03-28T18:08:19.482248+0000","last_unstale":"2024-03-28T18:08:23.389708+0000","last_undegraded":"2024-03-28T18:08:23.389708+0000","last_fullsized":"2024-03-28T18:08:23.389708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:23.199196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.269570+0000","last_change":"2024-03-28T18:08:20.011198+0000","last_active":"2024-03-28T18:08:21.269570+0000","last_peered":"2024-03-28T18:08:21.269570+0000","last_clean":"2024-03-28T18:08:21.269570+0000","last_became_active":"2024-03-28T18:08:20.010866+0000","last_became_peered":"2024-03-28T18:08:20.010866+0000","last_unstale":"2024-03-28T18:08:21.269570+0000","last_undegraded":"2024-03-28T18:08:21.269570+0000","last_fullsized":"2024-03-28T18:08:21.269570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:26.342858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:23.397351+0000","last_change":"2024-03-28T18:08:19.493830+0000","last_active":"2024-03-28T18:08:23.397351+0000","last_peered":"2024-03-28T18:08:23.397351+0000","last_clean":"2024-03-28T18:08:23.397351+0000","last_became_active":"2024-03-28T18:08:19.493447+0000","last_became_peered":"2024-03-28T18:08:19.493447+0000","last_unstale":"2024-03-28T18:08:23.397351+0000","last_undegraded":"2024-03-28T18:08:23.397351+0000","last_fullsized":"2024-03-28T18:08:23.397351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:33:29.797186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539165+0000","last_change":"2024-03-28T18:08:20.025620+0000","last_active":"2024-03-28T18:08:21.539165+0000","last_peered":"2024-03-28T18:08:21.539165+0000","last_clean":"2024-03-28T18:08:21.539165+0000","last_became_active":"2024-03-28T18:08:20.025276+0000","last_became_peered":"2024-03-28T18:08:20.025276+0000","last_unstale":"2024-03-28T18:08:21.539165+0000","last_undegraded":"2024-03-28T18:08:21.539165+0000","last_fullsized":"2024-03-28T18:08:21.539165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:30:06.667707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.995575+0000","last_change":"2024-03-28T18:08:19.474204+0000","last_active":"2024-03-28T18:08:20.995575+0000","last_peered":"2024-03-28T18:08:20.995575+0000","last_clean":"2024-03-28T18:08:20.995575+0000","last_became_active":"2024-03-28T18:08:19.473688+0000","last_became_peered":"2024-03-28T18:08:19.473688+0000","last_unstale":"2024-03-28T18:08:20.995575+0000","last_undegraded":"2024-03-28T18:08:20.995575+0000","last_fullsized":"2024-03-28T18:08:20.995575+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:15:09.880974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.525731+0000","last_change":"2024-03-28T18:08:20.035920+0000","last_active":"2024-03-28T18:08:21.525731+0000","last_peered":"2024-03-28T18:08:21.525731+0000","last_clean":"2024-03-28T18:08:21.525731+0000","last_became_active":"2024-03-28T18:08:20.035600+0000","last_became_peered":"2024-03-28T18:08:20.035600+0000","last_unstale":"2024-03-28T18:08:21.525731+0000","last_undegraded":"2024-03-28T18:08:21.525731+0000","last_fullsized":"2024-03-28T18:08:21.525731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:41.280448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:23.389762+0000","last_change":"2024-03-28T18:08:19.481305+0000","last_active":"2024-03-28T18:08:23.389762+0000","last_peered":"2024-03-28T18:08:23.389762+0000","last_clean":"2024-03-28T18:08:23.389762+0000","last_became_active":"2024-03-28T18:08:19.480978+0000","last_became_peered":"2024-03-28T18:08:19.480978+0000","last_unstale":"2024-03-28T18:08:23.389762+0000","last_undegraded":"2024-03-28T18:08:23.389762+0000","last_fullsized":"2024-03-28T18:08:23.389762+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:14:06.464967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539915+0000","last_change":"2024-03-28T18:08:20.017660+0000","last_active":"2024-03-28T18:08:21.539915+0000","last_peered":"2024-03-28T18:08:21.539915+0000","last_clean":"2024-03-28T18:08:21.539915+0000","last_became_active":"2024-03-28T18:08:20.016083+0000","last_became_peered":"2024-03-28T18:08:20.016083+0000","last_unstale":"2024-03-28T18:08:21.539915+0000","last_undegraded":"2024-03-28T18:08:21.539915+0000","last_fullsized":"2024-03-28T18:08:21.539915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:33:12.104274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:24.421135+0000","last_change":"2024-03-28T18:08:19.479855+0000","last_active":"2024-03-28T18:08:24.421135+0000","last_peered":"2024-03-28T18:08:24.421135+0000","last_clean":"2024-03-28T18:08:24.421135+0000","last_became_active":"2024-03-28T18:08:19.479429+0000","last_became_peered":"2024-03-28T18:08:19.479429+0000","last_unstale":"2024-03-28T18:08:24.421135+0000","last_undegraded":"2024-03-28T18:08:24.421135+0000","last_fullsized":"2024-03-28T18:08:24.421135+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:06:23.286970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.540207+0000","last_change":"2024-03-28T18:08:20.015419+0000","last_active":"2024-03-28T18:08:21.540207+0000","last_peered":"2024-03-28T18:08:21.540207+0000","last_clean":"2024-03-28T18:08:21.540207+0000","last_became_active":"2024-03-28T18:08:20.015011+0000","last_became_peered":"2024-03-28T18:08:20.015011+0000","last_unstale":"2024-03-28T18:08:21.540207+0000","last_undegraded":"2024-03-28T18:08:21.540207+0000","last_fullsized":"2024-03-28T18:08:21.540207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:44:46.858352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.326528+0000","last_change":"2024-03-28T18:08:19.495541+0000","last_active":"2024-03-28T18:08:21.326528+0000","last_peered":"2024-03-28T18:08:21.326528+0000","last_clean":"2024-03-28T18:08:21.326528+0000","last_became_active":"2024-03-28T18:08:19.495173+0000","last_became_peered":"2024-03-28T18:08:19.495173+0000","last_unstale":"2024-03-28T18:08:21.326528+0000","last_undegraded":"2024-03-28T18:08:21.326528+0000","last_fullsized":"2024-03-28T18:08:21.326528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:42:06.423107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539845+0000","last_change":"2024-03-28T18:08:20.018582+0000","last_active":"2024-03-28T18:08:21.539845+0000","last_peered":"2024-03-28T18:08:21.539845+0000","last_clean":"2024-03-28T18:08:21.539845+0000","last_became_active":"2024-03-28T18:08:20.017936+0000","last_became_peered":"2024-03-28T18:08:20.017936+0000","last_unstale":"2024-03-28T18:08:21.539845+0000","last_undegraded":"2024-03-28T18:08:21.539845+0000","last_fullsized":"2024-03-28T18:08:21.539845+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:05:49.510246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996206+0000","last_change":"2024-03-28T18:08:19.497049+0000","last_active":"2024-03-28T18:08:20.996206+0000","last_peered":"2024-03-28T18:08:20.996206+0000","last_clean":"2024-03-28T18:08:20.996206+0000","last_became_active":"2024-03-28T18:08:19.496623+0000","last_became_peered":"2024-03-28T18:08:19.496623+0000","last_unstale":"2024-03-28T18:08:20.996206+0000","last_undegraded":"2024-03-28T18:08:20.996206+0000","last_fullsized":"2024-03-28T18:08:20.996206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:55:07.842439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.540348+0000","last_change":"2024-03-28T18:08:20.017419+0000","last_active":"2024-03-28T18:08:21.540348+0000","last_peered":"2024-03-28T18:08:21.540348+0000","last_clean":"2024-03-28T18:08:21.540348+0000","last_became_active":"2024-03-28T18:08:20.015916+0000","last_became_peered":"2024-03-28T18:08:20.015916+0000","last_unstale":"2024-03-28T18:08:21.540348+0000","last_undegraded":"2024-03-28T18:08:21.540348+0000","last_fullsized":"2024-03-28T18:08:21.540348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:10:55.927794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539204+0000","last_change":"2024-03-28T18:08:20.008093+0000","last_active":"2024-03-28T18:08:21.539204+0000","last_peered":"2024-03-28T18:08:21.539204+0000","last_clean":"2024-03-28T18:08:21.539204+0000","last_became_active":"2024-03-28T18:08:20.007801+0000","last_became_peered":"2024-03-28T18:08:20.007801+0000","last_unstale":"2024-03-28T18:08:21.539204+0000","last_undegraded":"2024-03-28T18:08:21.539204+0000","last_fullsized":"2024-03-28T18:08:21.539204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:58.976859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.995859+0000","last_change":"2024-03-28T18:08:20.012771+0000","last_active":"2024-03-28T18:08:20.995859+0000","last_peered":"2024-03-28T18:08:20.995859+0000","last_clean":"2024-03-28T18:08:20.995859+0000","last_became_active":"2024-03-28T18:08:20.012456+0000","last_became_peered":"2024-03-28T18:08:20.012456+0000","last_unstale":"2024-03-28T18:08:20.995859+0000","last_undegraded":"2024-03-28T18:08:20.995859+0000","last_fullsized":"2024-03-28T18:08:20.995859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:56.428332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.995728+0000","last_change":"2024-03-28T18:08:20.034872+0000","last_active":"2024-03-28T18:08:20.995728+0000","last_peered":"2024-03-28T18:08:20.995728+0000","last_clean":"2024-03-28T18:08:20.995728+0000","last_became_active":"2024-03-28T18:08:20.034614+0000","last_became_peered":"2024-03-28T18:08:20.034614+0000","last_unstale":"2024-03-28T18:08:20.995728+0000","last_undegraded":"2024-03-28T18:08:20.995728+0000","last_fullsized":"2024-03-28T18:08:20.995728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:35.658186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539267+0000","last_change":"2024-03-28T18:08:20.033119+0000","last_active":"2024-03-28T18:08:21.539267+0000","last_peered":"2024-03-28T18:08:21.539267+0000","last_clean":"2024-03-28T18:08:21.539267+0000","last_became_active":"2024-03-28T18:08:20.030694+0000","last_became_peered":"2024-03-28T18:08:20.030694+0000","last_unstale":"2024-03-28T18:08:21.539267+0000","last_undegraded":"2024-03-28T18:08:21.539267+0000","last_fullsized":"2024-03-28T18:08:21.539267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:51:50.082993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996315+0000","last_change":"2024-03-28T18:08:20.025725+0000","last_active":"2024-03-28T18:08:20.996315+0000","last_peered":"2024-03-28T18:08:20.996315+0000","last_clean":"2024-03-28T18:08:20.996315+0000","last_became_active":"2024-03-28T18:08:20.024929+0000","last_became_peered":"2024-03-28T18:08:20.024929+0000","last_unstale":"2024-03-28T18:08:20.996315+0000","last_undegraded":"2024-03-28T18:08:20.996315+0000","last_fullsized":"2024-03-28T18:08:20.996315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:04.894246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539333+0000","last_change":"2024-03-28T18:08:20.005293+0000","last_active":"2024-03-28T18:08:21.539333+0000","last_peered":"2024-03-28T18:08:21.539333+0000","last_clean":"2024-03-28T18:08:21.539333+0000","last_became_active":"2024-03-28T18:08:20.004938+0000","last_became_peered":"2024-03-28T18:08:20.004938+0000","last_unstale":"2024-03-28T18:08:21.539333+0000","last_undegraded":"2024-03-28T18:08:21.539333+0000","last_fullsized":"2024-03-28T18:08:21.539333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:36.100810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996060+0000","last_change":"2024-03-28T18:08:20.008741+0000","last_active":"2024-03-28T18:08:20.996060+0000","last_peered":"2024-03-28T18:08:20.996060+0000","last_clean":"2024-03-28T18:08:20.996060+0000","last_became_active":"2024-03-28T18:08:20.008411+0000","last_became_peered":"2024-03-28T18:08:20.008411+0000","last_unstale":"2024-03-28T18:08:20.996060+0000","last_undegraded":"2024-03-28T18:08:20.996060+0000","last_fullsized":"2024-03-28T18:08:20.996060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:28:52.276278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.526417+0000","last_change":"2024-03-28T18:08:20.028486+0000","last_active":"2024-03-28T18:08:21.526417+0000","last_peered":"2024-03-28T18:08:21.526417+0000","last_clean":"2024-03-28T18:08:21.526417+0000","last_became_active":"2024-03-28T18:08:20.028121+0000","last_became_peered":"2024-03-28T18:08:20.028121+0000","last_unstale":"2024-03-28T18:08:21.526417+0000","last_undegraded":"2024-03-28T18:08:21.526417+0000","last_fullsized":"2024-03-28T18:08:21.526417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:55:26.828305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.326334+0000","last_change":"2024-03-28T18:08:20.028085+0000","last_active":"2024-03-28T18:08:21.326334+0000","last_peered":"2024-03-28T18:08:21.326334+0000","last_clean":"2024-03-28T18:08:21.326334+0000","last_became_active":"2024-03-28T18:08:20.027721+0000","last_became_peered":"2024-03-28T18:08:20.027721+0000","last_unstale":"2024-03-28T18:08:21.326334+0000","last_undegraded":"2024-03-28T18:08:21.326334+0000","last_fullsized":"2024-03-28T18:08:21.326334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:31:36.674002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.525438+0000","last_change":"2024-03-28T18:08:20.023605+0000","last_active":"2024-03-28T18:08:21.525438+0000","last_peered":"2024-03-28T18:08:21.525438+0000","last_clean":"2024-03-28T18:08:21.525438+0000","last_became_active":"2024-03-28T18:08:20.018217+0000","last_became_peered":"2024-03-28T18:08:20.018217+0000","last_unstale":"2024-03-28T18:08:21.525438+0000","last_undegraded":"2024-03-28T18:08:21.525438+0000","last_fullsized":"2024-03-28T18:08:21.525438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:43.717751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996782+0000","last_change":"2024-03-28T18:08:20.023072+0000","last_active":"2024-03-28T18:08:20.996782+0000","last_peered":"2024-03-28T18:08:20.996782+0000","last_clean":"2024-03-28T18:08:20.996782+0000","last_became_active":"2024-03-28T18:08:20.022690+0000","last_became_peered":"2024-03-28T18:08:20.022690+0000","last_unstale":"2024-03-28T18:08:20.996782+0000","last_undegraded":"2024-03-28T18:08:20.996782+0000","last_fullsized":"2024-03-28T18:08:20.996782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:21.391415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:20.996605+0000","last_change":"2024-03-28T18:08:20.024261+0000","last_active":"2024-03-28T18:08:20.996605+0000","last_peered":"2024-03-28T18:08:20.996605+0000","last_clean":"2024-03-28T18:08:20.996605+0000","last_became_active":"2024-03-28T18:08:20.023792+0000","last_became_peered":"2024-03-28T18:08:20.023792+0000","last_unstale":"2024-03-28T18:08:20.996605+0000","last_undegraded":"2024-03-28T18:08:20.996605+0000","last_fullsized":"2024-03-28T18:08:20.996605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:23:08.030982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T18:08:21.539195+0000","last_change":"2024-03-28T18:08:20.008306+0000","last_active":"2024-03-28T18:08:21.539195+0000","last_peered":"2024-03-28T18:08:21.539195+0000","last_clean":"2024-03-28T18:08:21.539195+0000","last_became_active":"2024-03-28T18:08:20.007984+0000","last_became_peered":"2024-03-28T18:08:20.007984+0000","last_unstale":"2024-03-28T18:08:21.539195+0000","last_undegraded":"2024-03-28T18:08:21.539195+0000","last_fullsized":"2024-03-28T18:08:21.539195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:25:26.889600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":248983,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":43238,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":41,"num_read_kb":66,"num_write":139,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":60673,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":121,"ondisk_log_size":121,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383695,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645333,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":1070212,"data_compressed":37917,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808858,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":439339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070496,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":439259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332134,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":1031072,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":438611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574920,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":440133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":439999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411394,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36264,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708888,"statfs":{"total":95995035648,"available":95957901312,"internally_reserved":0,"allocated":950272,"data_stored":441483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182461,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":438922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920825,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":439165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":991232,"data_stored":1030365,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34394,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34394,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":8212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":41206,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:08:30.206 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T18:08:30.207 INFO:tasks.cephadm:Setting up client nodes... 2024-03-28T18:08:30.207 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-28T18:08:30.728 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:08:30.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:30 smithi037 ceph-mon[31460]: from='client.24701 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:08:31.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:30 smithi059 ceph-mon[36190]: from='client.24701 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:08:31.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:30 smithi170 ceph-mon[36280]: from='client.24701 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:08:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:31 smithi037 ceph-mon[31460]: pgmap v323: 97 pgs: 97 active+clean; 619 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 8.1 KiB/s wr, 18 op/s 2024-03-28T18:08:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:31 smithi037 ceph-mon[31460]: from='client.24707 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:08:31.935 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:31 smithi059 ceph-mon[36190]: pgmap v323: 97 pgs: 97 active+clean; 619 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 8.1 KiB/s wr, 18 op/s 2024-03-28T18:08:31.935 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:31 smithi059 ceph-mon[36190]: from='client.24707 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:08:32.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:31 smithi170 ceph-mon[36280]: pgmap v323: 97 pgs: 97 active+clean; 619 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 8.1 KiB/s wr, 18 op/s 2024-03-28T18:08:32.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:31 smithi170 ceph-mon[36280]: from='client.24707 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:08:33.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.262+0000 7fa462c00700 1 -- 172.21.15.37:0/4224610524 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa45c10a0c0 msgr2=0x7fa45c10c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:33.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.262+0000 7fa462c00700 1 --2- 172.21.15.37:0/4224610524 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa45c10a0c0 0x7fa45c10c550 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa454009f90 tx=0x7fa45400b330 comp rx=0 tx=0).stop 2024-03-28T18:08:33.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.263+0000 7fa462c00700 1 -- 172.21.15.37:0/4224610524 shutdown_connections 2024-03-28T18:08:33.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.263+0000 7fa462c00700 1 --2- 172.21.15.37:0/4224610524 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa45c10a0c0 0x7fa45c10c550 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:33.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.263+0000 7fa462c00700 1 --2- 172.21.15.37:0/4224610524 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa45c0761c0 0x7fa45c074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:33.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.263+0000 7fa462c00700 1 --2- 172.21.15.37:0/4224610524 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa45c074bf0 0x7fa45c075070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:33.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.263+0000 7fa462c00700 1 -- 172.21.15.37:0/4224610524 >> 172.21.15.37:0/4224610524 conn(0x7fa45c0fd770 msgr2=0x7fa45c0ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:33.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.263+0000 7fa462c00700 1 -- 172.21.15.37:0/4224610524 shutdown_connections 2024-03-28T18:08:33.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.263+0000 7fa462c00700 1 -- 172.21.15.37:0/4224610524 wait complete. 2024-03-28T18:08:33.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.264+0000 7fa462c00700 1 Processor -- start 2024-03-28T18:08:33.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.264+0000 7fa462c00700 1 -- start start 2024-03-28T18:08:33.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.264+0000 7fa462c00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa45c074bf0 0x7fa45c19a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:33.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.265+0000 7fa462c00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa45c0761c0 0x7fa45c19aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:33.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.265+0000 7fa462c00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa45c10a0c0 0x7fa45c1a1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:33.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.265+0000 7fa462c00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c110020 con 0x7fa45c10a0c0 2024-03-28T18:08:33.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.265+0000 7fa462c00700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c10fea0 con 0x7fa45c074bf0 2024-03-28T18:08:33.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.265+0000 7fa462c00700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c1101a0 con 0x7fa45c0761c0 2024-03-28T18:08:33.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.265+0000 7fa45be00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa45c074bf0 0x7fa45c19a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:33.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.265+0000 7fa45be00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa45c074bf0 0x7fa45c19a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:58006/0 (socket says 172.21.15.37:58006) 2024-03-28T18:08:33.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.265+0000 7fa45be00700 1 -- 172.21.15.37:0/1817438820 learned_addr learned my addr 172.21.15.37:0/1817438820 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:33.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.265+0000 7fa453400700 1 --2- 172.21.15.37:0/1817438820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa45c0761c0 0x7fa45c19aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:33.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.265+0000 7fa460c00700 1 --2- 172.21.15.37:0/1817438820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa45c10a0c0 0x7fa45c1a1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:33.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.265+0000 7fa45be00700 1 -- 172.21.15.37:0/1817438820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa45c0761c0 msgr2=0x7fa45c19aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:33.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.265+0000 7fa45be00700 1 --2- 172.21.15.37:0/1817438820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa45c0761c0 0x7fa45c19aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:33.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.265+0000 7fa45be00700 1 -- 172.21.15.37:0/1817438820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa45c10a0c0 msgr2=0x7fa45c1a1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:33.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.265+0000 7fa45be00700 1 --2- 172.21.15.37:0/1817438820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa45c10a0c0 0x7fa45c1a1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:33.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.265+0000 7fa45be00700 1 -- 172.21.15.37:0/1817438820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45c1a2140 con 0x7fa45c074bf0 2024-03-28T18:08:33.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.266+0000 7fa45be00700 1 --2- 172.21.15.37:0/1817438820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa45c074bf0 0x7fa45c19a4e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa44c00ec70 tx=0x7fa44c00ef80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:33.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.266+0000 7fa459600700 1 -- 172.21.15.37:0/1817438820 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44c00cd40 con 0x7fa45c074bf0 2024-03-28T18:08:33.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.266+0000 7fa459600700 1 -- 172.21.15.37:0/1817438820 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa44c004500 con 0x7fa45c074bf0 2024-03-28T18:08:33.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.266+0000 7fa459600700 1 -- 172.21.15.37:0/1817438820 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44c010690 con 0x7fa45c074bf0 2024-03-28T18:08:33.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.266+0000 7fa462c00700 1 -- 172.21.15.37:0/1817438820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa45c1a2430 con 0x7fa45c074bf0 2024-03-28T18:08:33.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.266+0000 7fa462c00700 1 -- 172.21.15.37:0/1817438820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45c1a29c0 con 0x7fa45c074bf0 2024-03-28T18:08:33.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.267+0000 7fa462c00700 1 -- 172.21.15.37:0/1817438820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa45c0506a0 con 0x7fa45c074bf0 2024-03-28T18:08:33.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.269+0000 7fa459600700 1 -- 172.21.15.37:0/1817438820 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fa44c003f10 con 0x7fa45c074bf0 2024-03-28T18:08:33.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.271+0000 7fa459600700 1 --2- 172.21.15.37:0/1817438820 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa4400a2a50 0x7fa4400a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:33.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.271+0000 7fa459600700 1 -- 172.21.15.37:0/1817438820 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa44c0ceeb0 con 0x7fa45c074bf0 2024-03-28T18:08:33.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.271+0000 7fa453400700 1 --2- 172.21.15.37:0/1817438820 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa4400a2a50 0x7fa4400a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:33.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.272+0000 7fa453400700 1 --2- 172.21.15.37:0/1817438820 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa4400a2a50 0x7fa4400a4f10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa45c075ef0 tx=0x7fa444009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:33.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.272+0000 7fa459600700 1 -- 172.21.15.37:0/1817438820 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa44c09bf30 con 0x7fa45c074bf0 2024-03-28T18:08:33.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.602+0000 7fa462c00700 1 -- 172.21.15.37:0/1817438820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fa45c02dad0 con 0x7fa45c074bf0 2024-03-28T18:08:33.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.609+0000 7fa459600700 1 -- 172.21.15.37:0/1817438820 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7fa44c09b830 con 0x7fa45c074bf0 2024-03-28T18:08:33.609 INFO:teuthology.orchestra.run.smithi037.stdout:[client.0] 2024-03-28T18:08:33.609 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAhsgVmyzkHJBAA5+P6v8xjh2s2WIhEw0jtSQ== 2024-03-28T18:08:33.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.612+0000 7fa462c00700 1 -- 172.21.15.37:0/1817438820 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa4400a2a50 msgr2=0x7fa4400a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:33.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.612+0000 7fa462c00700 1 --2- 172.21.15.37:0/1817438820 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa4400a2a50 0x7fa4400a4f10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa45c075ef0 tx=0x7fa444009260 comp rx=0 tx=0).stop 2024-03-28T18:08:33.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.612+0000 7fa462c00700 1 -- 172.21.15.37:0/1817438820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa45c074bf0 msgr2=0x7fa45c19a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:33.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.612+0000 7fa462c00700 1 --2- 172.21.15.37:0/1817438820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa45c074bf0 0x7fa45c19a4e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa44c00ec70 tx=0x7fa44c00ef80 comp rx=0 tx=0).stop 2024-03-28T18:08:33.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.612+0000 7fa462c00700 1 -- 172.21.15.37:0/1817438820 shutdown_connections 2024-03-28T18:08:33.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.612+0000 7fa462c00700 1 --2- 172.21.15.37:0/1817438820 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa4400a2a50 0x7fa4400a4f10 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:33.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.612+0000 7fa462c00700 1 --2- 172.21.15.37:0/1817438820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa45c10a0c0 0x7fa45c1a1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:33.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.612+0000 7fa462c00700 1 --2- 172.21.15.37:0/1817438820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa45c074bf0 0x7fa45c19a4e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:33.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.612+0000 7fa462c00700 1 --2- 172.21.15.37:0/1817438820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa45c0761c0 0x7fa45c19aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:33.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.612+0000 7fa462c00700 1 -- 172.21.15.37:0/1817438820 >> 172.21.15.37:0/1817438820 conn(0x7fa45c0fd770 msgr2=0x7fa45c105fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:33.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.612+0000 7fa462c00700 1 -- 172.21.15.37:0/1817438820 shutdown_connections 2024-03-28T18:08:33.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:33.612+0000 7fa462c00700 1 -- 172.21.15.37:0/1817438820 wait complete. 2024-03-28T18:08:33.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:33 smithi037 ceph-mon[31460]: pgmap v324: 97 pgs: 97 active+clean; 619 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 7.4 KiB/s wr, 17 op/s 2024-03-28T18:08:34.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:33 smithi059 ceph-mon[36190]: pgmap v324: 97 pgs: 97 active+clean; 619 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 7.4 KiB/s wr, 17 op/s 2024-03-28T18:08:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:33 smithi170 ceph-mon[36280]: pgmap v324: 97 pgs: 97 active+clean; 619 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 7.4 KiB/s wr, 17 op/s 2024-03-28T18:08:34.570 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-28T18:08:34.571 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-28T18:08:34.571 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-28T18:08:34.653 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:34 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1817438820' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T18:08:34.653 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:34 smithi037 ceph-mon[31460]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T18:08:34.653 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:34 smithi037 ceph-mon[31460]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T18:08:34.719 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-28T18:08:34.720 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-28T18:08:34.720 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-28T18:08:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:34 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1817438820' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T18:08:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:34 smithi059 ceph-mon[36190]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T18:08:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:34 smithi059 ceph-mon[36190]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T18:08:35.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:34 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1817438820' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T18:08:35.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:34 smithi170 ceph-mon[36280]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T18:08:35.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:34 smithi170 ceph-mon[36280]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T18:08:35.253 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:08:35.736 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:35 smithi037 ceph-mon[31460]: pgmap v325: 97 pgs: 97 active+clean; 619 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.8 KiB/s wr, 16 op/s 2024-03-28T18:08:36.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:35 smithi059 ceph-mon[36190]: pgmap v325: 97 pgs: 97 active+clean; 619 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.8 KiB/s wr, 16 op/s 2024-03-28T18:08:36.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:35 smithi170 ceph-mon[36280]: pgmap v325: 97 pgs: 97 active+clean; 619 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.8 KiB/s wr, 16 op/s 2024-03-28T18:08:37.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.554+0000 7f6468a00700 1 -- 172.21.15.37:0/3943133553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6464102410 msgr2=0x7f64641027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.554+0000 7f6468a00700 1 --2- 172.21.15.37:0/3943133553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6464102410 0x7f64641027f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f645400b0a0 tx=0x7f645400b3b0 comp rx=0 tx=0).stop 2024-03-28T18:08:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.554+0000 7f6468a00700 1 -- 172.21.15.37:0/3943133553 shutdown_connections 2024-03-28T18:08:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.554+0000 7f6468a00700 1 --2- 172.21.15.37:0/3943133553 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64641074f0 0x7f646410f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.554+0000 7f6468a00700 1 --2- 172.21.15.37:0/3943133553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6464102410 0x7f64641027f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.554+0000 7f6468a00700 1 --2- 172.21.15.37:0/3943133553 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6464102dc0 0x7f6464106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.554+0000 7f6468a00700 1 -- 172.21.15.37:0/3943133553 >> 172.21.15.37:0/3943133553 conn(0x7f64640fdc60 msgr2=0x7f6464100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.554+0000 7f6468a00700 1 -- 172.21.15.37:0/3943133553 shutdown_connections 2024-03-28T18:08:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.554+0000 7f6468a00700 1 -- 172.21.15.37:0/3943133553 wait complete. 2024-03-28T18:08:37.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.555+0000 7f6468a00700 1 Processor -- start 2024-03-28T18:08:37.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.555+0000 7f6468a00700 1 -- start start 2024-03-28T18:08:37.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.556+0000 7f6468a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6464102410 0x7f6464074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.556+0000 7f6468a00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6464102dc0 0x7f64640749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.556+0000 7f6468a00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64641074f0 0x7f6464074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.556+0000 7f6468a00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6464112520 con 0x7f6464102410 2024-03-28T18:08:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.556+0000 7f6468a00700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64641123a0 con 0x7f64641074f0 2024-03-28T18:08:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.556+0000 7f6468a00700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64641126a0 con 0x7f6464102dc0 2024-03-28T18:08:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.556+0000 7f6461e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6464102410 0x7f6464074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.556+0000 7f6462800700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64641074f0 0x7f6464074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.556+0000 7f6461400700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6464102dc0 0x7f64640749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.556+0000 7f6461e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6464102410 0x7f6464074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41680/0 (socket says 172.21.15.37:41680) 2024-03-28T18:08:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.556+0000 7f6462800700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64641074f0 0x7f6464074f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:34234/0 (socket says 172.21.15.37:34234) 2024-03-28T18:08:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.556+0000 7f6461e00700 1 -- 172.21.15.37:0/1466572360 learned_addr learned my addr 172.21.15.37:0/1466572360 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.556+0000 7f6461400700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6464102dc0 0x7f64640749d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:37602/0 (socket says 172.21.15.37:37602) 2024-03-28T18:08:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.557+0000 7f6461400700 1 -- 172.21.15.37:0/1466572360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64641074f0 msgr2=0x7f6464074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.557+0000 7f6461400700 1 --2- 172.21.15.37:0/1466572360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64641074f0 0x7f6464074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.557+0000 7f6461400700 1 -- 172.21.15.37:0/1466572360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6464102410 msgr2=0x7f6464074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.557+0000 7f6461400700 1 --2- 172.21.15.37:0/1466572360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6464102410 0x7f6464074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.557+0000 7f6461400700 1 -- 172.21.15.37:0/1466572360 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64641a6f00 con 0x7f6464102dc0 2024-03-28T18:08:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.557+0000 7f6461400700 1 --2- 172.21.15.37:0/1466572360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6464102dc0 0x7f64640749d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f645000e9c0 tx=0x7f645000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.557+0000 7f645aa00700 1 -- 172.21.15.37:0/1466572360 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645000c9c0 con 0x7f6464102dc0 2024-03-28T18:08:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.557+0000 7f6468a00700 1 -- 172.21.15.37:0/1466572360 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64641a71f0 con 0x7f6464102dc0 2024-03-28T18:08:37.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.557+0000 7f645aa00700 1 -- 172.21.15.37:0/1466572360 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f645000cb60 con 0x7f6464102dc0 2024-03-28T18:08:37.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.557+0000 7f645aa00700 1 -- 172.21.15.37:0/1466572360 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6450018680 con 0x7f6464102dc0 2024-03-28T18:08:37.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.557+0000 7f6468a00700 1 -- 172.21.15.37:0/1466572360 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64641a7730 con 0x7f6464102dc0 2024-03-28T18:08:37.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.559+0000 7f6468a00700 1 -- 172.21.15.37:0/1466572360 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6464068e50 con 0x7f6464102dc0 2024-03-28T18:08:37.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.560+0000 7f645aa00700 1 -- 172.21.15.37:0/1466572360 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f6450010430 con 0x7f6464102dc0 2024-03-28T18:08:37.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.562+0000 7f645aa00700 1 --2- 172.21.15.37:0/1466572360 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f64480a2920 0x7f64480a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:37.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.562+0000 7f645aa00700 1 -- 172.21.15.37:0/1466572360 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f64500cec80 con 0x7f6464102dc0 2024-03-28T18:08:37.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.562+0000 7f6461e00700 1 --2- 172.21.15.37:0/1466572360 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f64480a2920 0x7f64480a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:37.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.563+0000 7f6461e00700 1 --2- 172.21.15.37:0/1466572360 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f64480a2920 0x7f64480a4de0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6454009e60 tx=0x7f6454006210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:37.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.566+0000 7f645aa00700 1 -- 172.21.15.37:0/1466572360 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f645009bc70 con 0x7f6464102dc0 2024-03-28T18:08:37.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.848+0000 7f6468a00700 1 -- 172.21.15.37:0/1466572360 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f646402da30 con 0x7f6464102dc0 2024-03-28T18:08:37.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.854+0000 7f645aa00700 1 -- 172.21.15.37:0/1466572360 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+262655 (secure 0 0 0) 0x7f645009b570 con 0x7f6464102dc0 2024-03-28T18:08:37.860 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:08:37.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.860+0000 7f6468a00700 1 -- 172.21.15.37:0/1466572360 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f64480a2920 msgr2=0x7f64480a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:37.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.860+0000 7f6468a00700 1 --2- 172.21.15.37:0/1466572360 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f64480a2920 0x7f64480a4de0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6454009e60 tx=0x7f6454006210 comp rx=0 tx=0).stop 2024-03-28T18:08:37.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.860+0000 7f6468a00700 1 -- 172.21.15.37:0/1466572360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6464102dc0 msgr2=0x7f64640749d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:37.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.860+0000 7f6468a00700 1 --2- 172.21.15.37:0/1466572360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6464102dc0 0x7f64640749d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f645000e9c0 tx=0x7f645000ecd0 comp rx=0 tx=0).stop 2024-03-28T18:08:37.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.860+0000 7f6468a00700 1 -- 172.21.15.37:0/1466572360 shutdown_connections 2024-03-28T18:08:37.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.860+0000 7f6468a00700 1 --2- 172.21.15.37:0/1466572360 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f64480a2920 0x7f64480a4de0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:37.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.860+0000 7f6468a00700 1 --2- 172.21.15.37:0/1466572360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64641074f0 0x7f6464074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:37.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.860+0000 7f6468a00700 1 --2- 172.21.15.37:0/1466572360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6464102410 0x7f6464074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:37.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.860+0000 7f6468a00700 1 --2- 172.21.15.37:0/1466572360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6464102dc0 0x7f64640749d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:37.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.860+0000 7f6468a00700 1 -- 172.21.15.37:0/1466572360 >> 172.21.15.37:0/1466572360 conn(0x7f64640fdc60 msgr2=0x7f64640fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:37.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.860+0000 7f6468a00700 1 -- 172.21.15.37:0/1466572360 shutdown_connections 2024-03-28T18:08:37.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:37.861+0000 7f6468a00700 1 -- 172.21.15.37:0/1466572360 wait complete. 2024-03-28T18:08:37.864 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:37 smithi037 ceph-mon[31460]: pgmap v326: 97 pgs: 97 active+clean; 624 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.1 KiB/s wr, 17 op/s 2024-03-28T18:08:38.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:37 smithi059 ceph-mon[36190]: pgmap v326: 97 pgs: 97 active+clean; 624 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.1 KiB/s wr, 17 op/s 2024-03-28T18:08:38.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:37 smithi170 ceph-mon[36280]: pgmap v326: 97 pgs: 97 active+clean; 624 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.1 KiB/s wr, 17 op/s 2024-03-28T18:08:38.653 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":521857560},{"type":"v1","addr":"172.21.15.37:6801","nonce":521857560}]},"active_addr":"172.21.15.37:6801/521857560","active_change":"2024-03-28T18:00:13.809539+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.37:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.37:0","nonce":2571469571}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.37:0","nonce":767215028}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.37:0","nonce":375309539}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.37:0","nonce":3957151624}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.37:0","nonce":466001570}]}]} 2024-03-28T18:08:38.658 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-28T18:08:38.658 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-28T18:08:38.659 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T18:08:38.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:38 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1466572360' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T18:08:39.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:38 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1466572360' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T18:08:39.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:38 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1466572360' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T18:08:39.190 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:08:40.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:39 smithi059 ceph-mon[36190]: pgmap v327: 97 pgs: 97 active+clean; 624 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.1 KiB/s wr, 11 op/s 2024-03-28T18:08:40.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:39 smithi170 ceph-mon[36280]: pgmap v327: 97 pgs: 97 active+clean; 624 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.1 KiB/s wr, 11 op/s 2024-03-28T18:08:40.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:39 smithi037 ceph-mon[31460]: pgmap v327: 97 pgs: 97 active+clean; 624 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.1 KiB/s wr, 11 op/s 2024-03-28T18:08:41.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.420+0000 7fdb29800700 1 -- 172.21.15.37:0/3097634117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb241081e0 msgr2=0x7fdb241085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:41.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.420+0000 7fdb29800700 1 --2- 172.21.15.37:0/3097634117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb241081e0 0x7fdb241085c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdb14009920 tx=0x7fdb14009c30 comp rx=0 tx=0).stop 2024-03-28T18:08:41.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.420+0000 7fdb29800700 1 -- 172.21.15.37:0/3097634117 shutdown_connections 2024-03-28T18:08:41.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.420+0000 7fdb29800700 1 --2- 172.21.15.37:0/3097634117 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb2406a7e0 0x7fdb2410f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:41.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.421+0000 7fdb29800700 1 --2- 172.21.15.37:0/3097634117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb241081e0 0x7fdb241085c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:41.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.421+0000 7fdb29800700 1 --2- 172.21.15.37:0/3097634117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb24069e40 0x7fdb2406a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:41.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.421+0000 7fdb29800700 1 -- 172.21.15.37:0/3097634117 >> 172.21.15.37:0/3097634117 conn(0x7fdb24077fc0 msgr2=0x7fdb240783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:41.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.421+0000 7fdb29800700 1 -- 172.21.15.37:0/3097634117 shutdown_connections 2024-03-28T18:08:41.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.421+0000 7fdb29800700 1 -- 172.21.15.37:0/3097634117 wait complete. 2024-03-28T18:08:41.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.422+0000 7fdb29800700 1 Processor -- start 2024-03-28T18:08:41.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.422+0000 7fdb29800700 1 -- start start 2024-03-28T18:08:41.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.423+0000 7fdb29800700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb24069e40 0x7fdb24101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:41.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.423+0000 7fdb29800700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb2406a7e0 0x7fdb241016a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:41.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.423+0000 7fdb29800700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb241081e0 0x7fdb24101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:41.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.423+0000 7fdb29800700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb24112470 con 0x7fdb2406a7e0 2024-03-28T18:08:41.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.423+0000 7fdb29800700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb241122f0 con 0x7fdb24069e40 2024-03-28T18:08:41.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.423+0000 7fdb29800700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb241125f0 con 0x7fdb241081e0 2024-03-28T18:08:41.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.423+0000 7fdb22a00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb24069e40 0x7fdb24101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:41.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.423+0000 7fdb22a00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb24069e40 0x7fdb24101160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:34244/0 (socket says 172.21.15.37:34244) 2024-03-28T18:08:41.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.423+0000 7fdb22000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb2406a7e0 0x7fdb241016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:41.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.423+0000 7fdb22a00700 1 -- 172.21.15.37:0/2138230077 learned_addr learned my addr 172.21.15.37:0/2138230077 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:41.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.424+0000 7fdb23400700 1 --2- 172.21.15.37:0/2138230077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb241081e0 0x7fdb24101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:41.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.424+0000 7fdb22a00700 1 -- 172.21.15.37:0/2138230077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb241081e0 msgr2=0x7fdb24101be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:41.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.424+0000 7fdb22a00700 1 --2- 172.21.15.37:0/2138230077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb241081e0 0x7fdb24101be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:41.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.424+0000 7fdb22a00700 1 -- 172.21.15.37:0/2138230077 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb2406a7e0 msgr2=0x7fdb241016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:41.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.424+0000 7fdb22a00700 1 --2- 172.21.15.37:0/2138230077 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb2406a7e0 0x7fdb241016a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:41.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.424+0000 7fdb22a00700 1 -- 172.21.15.37:0/2138230077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb241a6ef0 con 0x7fdb24069e40 2024-03-28T18:08:41.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.424+0000 7fdb22a00700 1 --2- 172.21.15.37:0/2138230077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb24069e40 0x7fdb24101160 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdb14000c00 tx=0x7fdb1400fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:41.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.424+0000 7fdb1b400700 1 -- 172.21.15.37:0/2138230077 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb14027070 con 0x7fdb24069e40 2024-03-28T18:08:41.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.425+0000 7fdb1b400700 1 -- 172.21.15.37:0/2138230077 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb14005660 con 0x7fdb24069e40 2024-03-28T18:08:41.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.425+0000 7fdb1b400700 1 -- 172.21.15.37:0/2138230077 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb14004930 con 0x7fdb24069e40 2024-03-28T18:08:41.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.425+0000 7fdb29800700 1 -- 172.21.15.37:0/2138230077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb241a7180 con 0x7fdb24069e40 2024-03-28T18:08:41.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.425+0000 7fdb29800700 1 -- 172.21.15.37:0/2138230077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb241a7580 con 0x7fdb24069e40 2024-03-28T18:08:41.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.426+0000 7fdb29800700 1 -- 172.21.15.37:0/2138230077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb24068e50 con 0x7fdb24069e40 2024-03-28T18:08:41.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.428+0000 7fdb1b400700 1 -- 172.21.15.37:0/2138230077 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fdb1402a070 con 0x7fdb24069e40 2024-03-28T18:08:41.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.429+0000 7fdb1b400700 1 --2- 172.21.15.37:0/2138230077 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fdb080a2920 0x7fdb080a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:41.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.429+0000 7fdb1b400700 1 -- 172.21.15.37:0/2138230077 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fdb140cf4f0 con 0x7fdb24069e40 2024-03-28T18:08:41.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.429+0000 7fdb22000700 1 --2- 172.21.15.37:0/2138230077 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fdb080a2920 0x7fdb080a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:41.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.430+0000 7fdb22000700 1 --2- 172.21.15.37:0/2138230077 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fdb080a2920 0x7fdb080a4de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdb10009750 tx=0x7fdb10009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:41.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.433+0000 7fdb1b400700 1 -- 172.21.15.37:0/2138230077 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdb1405e740 con 0x7fdb24069e40 2024-03-28T18:08:41.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.663+0000 7fdb29800700 1 -- 172.21.15.37:0/2138230077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb2402da30 con 0x7fdb24069e40 2024-03-28T18:08:41.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.665+0000 7fdb1b400700 1 -- 172.21.15.37:0/2138230077 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22196 (secure 0 0 0) 0x7fdb14013030 con 0x7fdb24069e40 2024-03-28T18:08:41.666 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:08:41.667 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":85,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","created":"2024-03-28T17:59:05.949847+0000","modified":"2024-03-28T18:08:20.991162+0000","last_up_change":"2024-03-28T18:07:21.021030+0000","last_in_change":"2024-03-28T18:07:01.416192+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:03:00.517967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:08:18.365827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:08:19.446282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db60618d-56f7-4130-8a90-84cbc480965b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6803","nonce":3562602035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6805","nonce":3562602035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6809","nonce":3562602035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6807","nonce":3562602035}]},"public_addr":"172.21.15.37:6803/3562602035","cluster_addr":"172.21.15.37:6805/3562602035","heartbeat_back_addr":"172.21.15.37:6809/3562602035","heartbeat_front_addr":"172.21.15.37:6807/3562602035","state":["exists","up"]},{"osd":1,"uuid":"7a4b7ad6-905e-4258-9574-693d4277dfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6801","nonce":3326537211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6803","nonce":3326537211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6807","nonce":3326537211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6805","nonce":3326537211}]},"public_addr":"172.21.15.59:6801/3326537211","cluster_addr":"172.21.15.59:6803/3326537211","heartbeat_back_addr":"172.21.15.59:6807/3326537211","heartbeat_front_addr":"172.21.15.59:6805/3326537211","state":["exists","up"]},{"osd":2,"uuid":"8b76ec1c-46bf-4336-89b5-12d82906e8b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6801","nonce":1269398427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6803","nonce":1269398427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6807","nonce":1269398427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6805","nonce":1269398427}]},"public_addr":"172.21.15.170:6801/1269398427","cluster_addr":"172.21.15.170:6803/1269398427","heartbeat_back_addr":"172.21.15.170:6807/1269398427","heartbeat_front_addr":"172.21.15.170:6805/1269398427","state":["exists","up"]},{"osd":3,"uuid":"66eca1da-2fbd-44e1-8bbf-5635a2c70ad5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6811","nonce":317805240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6813","nonce":317805240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6817","nonce":317805240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6815","nonce":317805240}]},"public_addr":"172.21.15.37:6811/317805240","cluster_addr":"172.21.15.37:6813/317805240","heartbeat_back_addr":"172.21.15.37:6817/317805240","heartbeat_front_addr":"172.21.15.37:6815/317805240","state":["exists","up"]},{"osd":4,"uuid":"4c877335-e42d-4d8e-9f48-a1d0a89ae438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6809","nonce":2410632623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6811","nonce":2410632623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6815","nonce":2410632623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6813","nonce":2410632623}]},"public_addr":"172.21.15.59:6809/2410632623","cluster_addr":"172.21.15.59:6811/2410632623","heartbeat_back_addr":"172.21.15.59:6815/2410632623","heartbeat_front_addr":"172.21.15.59:6813/2410632623","state":["exists","up"]},{"osd":5,"uuid":"b884842a-86e2-455b-9570-ff163b3bc0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6809","nonce":1218943899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6811","nonce":1218943899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6815","nonce":1218943899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6813","nonce":1218943899}]},"public_addr":"172.21.15.170:6809/1218943899","cluster_addr":"172.21.15.170:6811/1218943899","heartbeat_back_addr":"172.21.15.170:6815/1218943899","heartbeat_front_addr":"172.21.15.170:6813/1218943899","state":["exists","up"]},{"osd":6,"uuid":"249a1200-77f4-44ad-981f-1d352f732b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6819","nonce":1899652046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6821","nonce":1899652046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6825","nonce":1899652046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6823","nonce":1899652046}]},"public_addr":"172.21.15.37:6819/1899652046","cluster_addr":"172.21.15.37:6821/1899652046","heartbeat_back_addr":"172.21.15.37:6825/1899652046","heartbeat_front_addr":"172.21.15.37:6823/1899652046","state":["exists","up"]},{"osd":7,"uuid":"c73338e6-f505-4517-a84b-f35e04750a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6817","nonce":1343675690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6819","nonce":1343675690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6823","nonce":1343675690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6821","nonce":1343675690}]},"public_addr":"172.21.15.59:6817/1343675690","cluster_addr":"172.21.15.59:6819/1343675690","heartbeat_back_addr":"172.21.15.59:6823/1343675690","heartbeat_front_addr":"172.21.15.59:6821/1343675690","state":["exists","up"]},{"osd":8,"uuid":"149b8297-ff2d-4a88-9b1e-0def021cc516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6817","nonce":3235794556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6819","nonce":3235794556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6823","nonce":3235794556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6821","nonce":3235794556}]},"public_addr":"172.21.15.170:6817/3235794556","cluster_addr":"172.21.15.170:6819/3235794556","heartbeat_back_addr":"172.21.15.170:6823/3235794556","heartbeat_front_addr":"172.21.15.170:6821/3235794556","state":["exists","up"]},{"osd":9,"uuid":"55c27cc2-c771-44b7-90fe-8f0feca44162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6827","nonce":2307153008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6829","nonce":2307153008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6833","nonce":2307153008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6831","nonce":2307153008}]},"public_addr":"172.21.15.37:6827/2307153008","cluster_addr":"172.21.15.37:6829/2307153008","heartbeat_back_addr":"172.21.15.37:6833/2307153008","heartbeat_front_addr":"172.21.15.37:6831/2307153008","state":["exists","up"]},{"osd":10,"uuid":"c0595135-ab81-49d3-8c23-19f960c03e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6825","nonce":2113830275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6827","nonce":2113830275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6831","nonce":2113830275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6829","nonce":2113830275}]},"public_addr":"172.21.15.59:6825/2113830275","cluster_addr":"172.21.15.59:6827/2113830275","heartbeat_back_addr":"172.21.15.59:6831/2113830275","heartbeat_front_addr":"172.21.15.59:6829/2113830275","state":["exists","up"]},{"osd":11,"uuid":"5871e177-c17b-4637-a797-4fa5125920dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6825","nonce":3082439360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6827","nonce":3082439360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6831","nonce":3082439360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6829","nonce":3082439360}]},"public_addr":"172.21.15.170:6825/3082439360","cluster_addr":"172.21.15.170:6827/3082439360","heartbeat_back_addr":"172.21.15.170:6831/3082439360","heartbeat_front_addr":"172.21.15.170:6829/3082439360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:07.690545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:33.183396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:57.400263+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:30.597211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:57.259044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:22.627823+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:57.585863+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:25.000476+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:50.876688+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:25.859761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:53.563146+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:07:19.304115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3023379428":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1443710231":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6800/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6801/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6801/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6800/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1007625490":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/2895698584":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/497313034":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/3293943475":"2024-03-29T17:59:39.162038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:08:41.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.668+0000 7fdb29800700 1 -- 172.21.15.37:0/2138230077 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fdb080a2920 msgr2=0x7fdb080a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:41.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.668+0000 7fdb29800700 1 --2- 172.21.15.37:0/2138230077 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fdb080a2920 0x7fdb080a4de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdb10009750 tx=0x7fdb10009310 comp rx=0 tx=0).stop 2024-03-28T18:08:41.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.668+0000 7fdb29800700 1 -- 172.21.15.37:0/2138230077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb24069e40 msgr2=0x7fdb24101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:41.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.668+0000 7fdb29800700 1 --2- 172.21.15.37:0/2138230077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb24069e40 0x7fdb24101160 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdb14000c00 tx=0x7fdb1400fa70 comp rx=0 tx=0).stop 2024-03-28T18:08:41.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.669+0000 7fdb29800700 1 -- 172.21.15.37:0/2138230077 shutdown_connections 2024-03-28T18:08:41.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.669+0000 7fdb29800700 1 --2- 172.21.15.37:0/2138230077 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fdb080a2920 0x7fdb080a4de0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:41.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.669+0000 7fdb29800700 1 --2- 172.21.15.37:0/2138230077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb241081e0 0x7fdb24101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:41.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.669+0000 7fdb29800700 1 --2- 172.21.15.37:0/2138230077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb24069e40 0x7fdb24101160 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:41.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.669+0000 7fdb29800700 1 --2- 172.21.15.37:0/2138230077 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb2406a7e0 0x7fdb241016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:41.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.669+0000 7fdb29800700 1 -- 172.21.15.37:0/2138230077 >> 172.21.15.37:0/2138230077 conn(0x7fdb24077fc0 msgr2=0x7fdb240ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:41.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.669+0000 7fdb29800700 1 -- 172.21.15.37:0/2138230077 shutdown_connections 2024-03-28T18:08:41.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:41.669+0000 7fdb29800700 1 -- 172.21.15.37:0/2138230077 wait complete. 2024-03-28T18:08:42.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:41 smithi059 ceph-mon[36190]: pgmap v328: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.6 KiB/s wr, 11 op/s 2024-03-28T18:08:42.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:41 smithi170 ceph-mon[36280]: pgmap v328: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.6 KiB/s wr, 11 op/s 2024-03-28T18:08:42.111 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:41 smithi037 ceph-mon[31460]: pgmap v328: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.6 KiB/s wr, 11 op/s 2024-03-28T18:08:42.512 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-28T18:08:42.513 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T18:08:42.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:42 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2138230077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:08:42.993 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:08:43.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:42 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2138230077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:08:43.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:42 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2138230077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:08:43.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:43 smithi037 ceph-mon[31460]: pgmap v329: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-28T18:08:44.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:43 smithi059 ceph-mon[36190]: pgmap v329: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-28T18:08:44.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:43 smithi170 ceph-mon[36280]: pgmap v329: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-28T18:08:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.160+0000 7fba6e200700 1 -- 172.21.15.37:0/3911997639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba68104f90 msgr2=0x7fba68108fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.160+0000 7fba6e200700 1 --2- 172.21.15.37:0/3911997639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba68104f90 0x7fba68108fe0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fba50009990 tx=0x7fba50009ca0 comp rx=0 tx=0).stop 2024-03-28T18:08:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.160+0000 7fba6e200700 1 -- 172.21.15.37:0/3911997639 shutdown_connections 2024-03-28T18:08:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.160+0000 7fba6e200700 1 --2- 172.21.15.37:0/3911997639 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba681096c0 0x7fba68111290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.160+0000 7fba6e200700 1 --2- 172.21.15.37:0/3911997639 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba681045e0 0x7fba681049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.160+0000 7fba6e200700 1 --2- 172.21.15.37:0/3911997639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba68104f90 0x7fba68108fe0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:45.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.160+0000 7fba6e200700 1 -- 172.21.15.37:0/3911997639 >> 172.21.15.37:0/3911997639 conn(0x7fba680ffe90 msgr2=0x7fba681022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:45.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.160+0000 7fba6e200700 1 -- 172.21.15.37:0/3911997639 shutdown_connections 2024-03-28T18:08:45.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.160+0000 7fba6e200700 1 -- 172.21.15.37:0/3911997639 wait complete. 2024-03-28T18:08:45.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.161+0000 7fba6e200700 1 Processor -- start 2024-03-28T18:08:45.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.161+0000 7fba6e200700 1 -- start start 2024-03-28T18:08:45.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.162+0000 7fba6e200700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba681045e0 0x7fba681a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:45.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.162+0000 7fba6e200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba68104f90 0x7fba681a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:45.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.162+0000 7fba6e200700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba681096c0 0x7fba6819a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:45.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.162+0000 7fba6e200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba681146f0 con 0x7fba68104f90 2024-03-28T18:08:45.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.162+0000 7fba6e200700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba68114570 con 0x7fba681096c0 2024-03-28T18:08:45.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.162+0000 7fba6e200700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba68114870 con 0x7fba681045e0 2024-03-28T18:08:45.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.162+0000 7fba67400700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba681045e0 0x7fba681a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:45.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.162+0000 7fba67400700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba681045e0 0x7fba681a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:45690/0 (socket says 172.21.15.37:45690) 2024-03-28T18:08:45.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.162+0000 7fba67400700 1 -- 172.21.15.37:0/902765605 learned_addr learned my addr 172.21.15.37:0/902765605 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:45.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.162+0000 7fba67e00700 1 --2- 172.21.15.37:0/902765605 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba681096c0 0x7fba6819a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:45.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.162+0000 7fba5fe00700 1 --2- 172.21.15.37:0/902765605 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba68104f90 0x7fba681a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:45.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.163+0000 7fba67400700 1 -- 172.21.15.37:0/902765605 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba681096c0 msgr2=0x7fba6819a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:45.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.163+0000 7fba67400700 1 --2- 172.21.15.37:0/902765605 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba681096c0 0x7fba6819a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:45.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.163+0000 7fba67400700 1 -- 172.21.15.37:0/902765605 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba68104f90 msgr2=0x7fba681a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:45.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.163+0000 7fba67400700 1 --2- 172.21.15.37:0/902765605 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba68104f90 0x7fba681a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:45.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.163+0000 7fba67400700 1 -- 172.21.15.37:0/902765605 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6819b150 con 0x7fba681045e0 2024-03-28T18:08:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.163+0000 7fba67400700 1 --2- 172.21.15.37:0/902765605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba681045e0 0x7fba681a0830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fba5800e9c0 tx=0x7fba5800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.163+0000 7fba64c00700 1 -- 172.21.15.37:0/902765605 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5800cbd0 con 0x7fba681045e0 2024-03-28T18:08:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.163+0000 7fba64c00700 1 -- 172.21.15.37:0/902765605 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba5800cd70 con 0x7fba681045e0 2024-03-28T18:08:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.163+0000 7fba6e200700 1 -- 172.21.15.37:0/902765605 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba6819b440 con 0x7fba681045e0 2024-03-28T18:08:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.163+0000 7fba64c00700 1 -- 172.21.15.37:0/902765605 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58010430 con 0x7fba681045e0 2024-03-28T18:08:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.163+0000 7fba6e200700 1 -- 172.21.15.37:0/902765605 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba681a7700 con 0x7fba681045e0 2024-03-28T18:08:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.164+0000 7fba6e200700 1 -- 172.21.15.37:0/902765605 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba68068e50 con 0x7fba681045e0 2024-03-28T18:08:45.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.168+0000 7fba64c00700 1 -- 172.21.15.37:0/902765605 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fba58004750 con 0x7fba681045e0 2024-03-28T18:08:45.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.169+0000 7fba64c00700 1 --2- 172.21.15.37:0/902765605 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fba4c0a29a0 0x7fba4c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:45.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.169+0000 7fba64c00700 1 -- 172.21.15.37:0/902765605 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fba580ced80 con 0x7fba681045e0 2024-03-28T18:08:45.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.169+0000 7fba5fe00700 1 --2- 172.21.15.37:0/902765605 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fba4c0a29a0 0x7fba4c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:45.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.170+0000 7fba5fe00700 1 --2- 172.21.15.37:0/902765605 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fba4c0a29a0 0x7fba4c0a4e60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fba5000b2e0 tx=0x7fba5001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:45.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.173+0000 7fba64c00700 1 -- 172.21.15.37:0/902765605 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fba5809bd70 con 0x7fba681045e0 2024-03-28T18:08:45.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.461+0000 7fba6e200700 1 -- 172.21.15.37:0/902765605 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba6819bf50 con 0x7fba681045e0 2024-03-28T18:08:45.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.462+0000 7fba64c00700 1 -- 172.21.15.37:0/902765605 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22196 (secure 0 0 0) 0x7fba6819bf50 con 0x7fba681045e0 2024-03-28T18:08:45.464 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:08:45.464 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":85,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","created":"2024-03-28T17:59:05.949847+0000","modified":"2024-03-28T18:08:20.991162+0000","last_up_change":"2024-03-28T18:07:21.021030+0000","last_in_change":"2024-03-28T18:07:01.416192+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:03:00.517967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:08:18.365827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:08:19.446282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db60618d-56f7-4130-8a90-84cbc480965b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6803","nonce":3562602035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6805","nonce":3562602035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6809","nonce":3562602035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6807","nonce":3562602035}]},"public_addr":"172.21.15.37:6803/3562602035","cluster_addr":"172.21.15.37:6805/3562602035","heartbeat_back_addr":"172.21.15.37:6809/3562602035","heartbeat_front_addr":"172.21.15.37:6807/3562602035","state":["exists","up"]},{"osd":1,"uuid":"7a4b7ad6-905e-4258-9574-693d4277dfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6801","nonce":3326537211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6803","nonce":3326537211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6807","nonce":3326537211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6805","nonce":3326537211}]},"public_addr":"172.21.15.59:6801/3326537211","cluster_addr":"172.21.15.59:6803/3326537211","heartbeat_back_addr":"172.21.15.59:6807/3326537211","heartbeat_front_addr":"172.21.15.59:6805/3326537211","state":["exists","up"]},{"osd":2,"uuid":"8b76ec1c-46bf-4336-89b5-12d82906e8b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6801","nonce":1269398427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6803","nonce":1269398427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6807","nonce":1269398427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6805","nonce":1269398427}]},"public_addr":"172.21.15.170:6801/1269398427","cluster_addr":"172.21.15.170:6803/1269398427","heartbeat_back_addr":"172.21.15.170:6807/1269398427","heartbeat_front_addr":"172.21.15.170:6805/1269398427","state":["exists","up"]},{"osd":3,"uuid":"66eca1da-2fbd-44e1-8bbf-5635a2c70ad5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6811","nonce":317805240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6813","nonce":317805240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6817","nonce":317805240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6815","nonce":317805240}]},"public_addr":"172.21.15.37:6811/317805240","cluster_addr":"172.21.15.37:6813/317805240","heartbeat_back_addr":"172.21.15.37:6817/317805240","heartbeat_front_addr":"172.21.15.37:6815/317805240","state":["exists","up"]},{"osd":4,"uuid":"4c877335-e42d-4d8e-9f48-a1d0a89ae438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6809","nonce":2410632623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6811","nonce":2410632623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6815","nonce":2410632623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6813","nonce":2410632623}]},"public_addr":"172.21.15.59:6809/2410632623","cluster_addr":"172.21.15.59:6811/2410632623","heartbeat_back_addr":"172.21.15.59:6815/2410632623","heartbeat_front_addr":"172.21.15.59:6813/2410632623","state":["exists","up"]},{"osd":5,"uuid":"b884842a-86e2-455b-9570-ff163b3bc0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6809","nonce":1218943899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6811","nonce":1218943899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6815","nonce":1218943899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6813","nonce":1218943899}]},"public_addr":"172.21.15.170:6809/1218943899","cluster_addr":"172.21.15.170:6811/1218943899","heartbeat_back_addr":"172.21.15.170:6815/1218943899","heartbeat_front_addr":"172.21.15.170:6813/1218943899","state":["exists","up"]},{"osd":6,"uuid":"249a1200-77f4-44ad-981f-1d352f732b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6819","nonce":1899652046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6821","nonce":1899652046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6825","nonce":1899652046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6823","nonce":1899652046}]},"public_addr":"172.21.15.37:6819/1899652046","cluster_addr":"172.21.15.37:6821/1899652046","heartbeat_back_addr":"172.21.15.37:6825/1899652046","heartbeat_front_addr":"172.21.15.37:6823/1899652046","state":["exists","up"]},{"osd":7,"uuid":"c73338e6-f505-4517-a84b-f35e04750a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6817","nonce":1343675690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6819","nonce":1343675690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6823","nonce":1343675690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6821","nonce":1343675690}]},"public_addr":"172.21.15.59:6817/1343675690","cluster_addr":"172.21.15.59:6819/1343675690","heartbeat_back_addr":"172.21.15.59:6823/1343675690","heartbeat_front_addr":"172.21.15.59:6821/1343675690","state":["exists","up"]},{"osd":8,"uuid":"149b8297-ff2d-4a88-9b1e-0def021cc516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6817","nonce":3235794556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6819","nonce":3235794556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6823","nonce":3235794556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6821","nonce":3235794556}]},"public_addr":"172.21.15.170:6817/3235794556","cluster_addr":"172.21.15.170:6819/3235794556","heartbeat_back_addr":"172.21.15.170:6823/3235794556","heartbeat_front_addr":"172.21.15.170:6821/3235794556","state":["exists","up"]},{"osd":9,"uuid":"55c27cc2-c771-44b7-90fe-8f0feca44162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6827","nonce":2307153008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6829","nonce":2307153008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6833","nonce":2307153008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6831","nonce":2307153008}]},"public_addr":"172.21.15.37:6827/2307153008","cluster_addr":"172.21.15.37:6829/2307153008","heartbeat_back_addr":"172.21.15.37:6833/2307153008","heartbeat_front_addr":"172.21.15.37:6831/2307153008","state":["exists","up"]},{"osd":10,"uuid":"c0595135-ab81-49d3-8c23-19f960c03e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6825","nonce":2113830275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6827","nonce":2113830275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6831","nonce":2113830275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6829","nonce":2113830275}]},"public_addr":"172.21.15.59:6825/2113830275","cluster_addr":"172.21.15.59:6827/2113830275","heartbeat_back_addr":"172.21.15.59:6831/2113830275","heartbeat_front_addr":"172.21.15.59:6829/2113830275","state":["exists","up"]},{"osd":11,"uuid":"5871e177-c17b-4637-a797-4fa5125920dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6825","nonce":3082439360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6827","nonce":3082439360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6831","nonce":3082439360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6829","nonce":3082439360}]},"public_addr":"172.21.15.170:6825/3082439360","cluster_addr":"172.21.15.170:6827/3082439360","heartbeat_back_addr":"172.21.15.170:6831/3082439360","heartbeat_front_addr":"172.21.15.170:6829/3082439360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:07.690545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:33.183396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:57.400263+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:30.597211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:57.259044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:22.627823+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:57.585863+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:25.000476+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:50.876688+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:25.859761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:53.563146+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:07:19.304115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3023379428":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1443710231":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6800/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6801/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6801/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6800/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1007625490":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/2895698584":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/497313034":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/3293943475":"2024-03-29T17:59:39.162038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:08:45.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.465+0000 7fba6e200700 1 -- 172.21.15.37:0/902765605 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fba4c0a29a0 msgr2=0x7fba4c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:45.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.465+0000 7fba6e200700 1 --2- 172.21.15.37:0/902765605 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fba4c0a29a0 0x7fba4c0a4e60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fba5000b2e0 tx=0x7fba5001a040 comp rx=0 tx=0).stop 2024-03-28T18:08:45.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.465+0000 7fba6e200700 1 -- 172.21.15.37:0/902765605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba681045e0 msgr2=0x7fba681a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:45.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.465+0000 7fba6e200700 1 --2- 172.21.15.37:0/902765605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba681045e0 0x7fba681a0830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fba5800e9c0 tx=0x7fba5800ecd0 comp rx=0 tx=0).stop 2024-03-28T18:08:45.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.465+0000 7fba6e200700 1 -- 172.21.15.37:0/902765605 shutdown_connections 2024-03-28T18:08:45.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.465+0000 7fba6e200700 1 --2- 172.21.15.37:0/902765605 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fba4c0a29a0 0x7fba4c0a4e60 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:45.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.465+0000 7fba6e200700 1 --2- 172.21.15.37:0/902765605 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba681096c0 0x7fba6819a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:45.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.465+0000 7fba6e200700 1 --2- 172.21.15.37:0/902765605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba681045e0 0x7fba681a0830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:45.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.465+0000 7fba6e200700 1 --2- 172.21.15.37:0/902765605 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba68104f90 0x7fba681a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:45.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.465+0000 7fba6e200700 1 -- 172.21.15.37:0/902765605 >> 172.21.15.37:0/902765605 conn(0x7fba680ffe90 msgr2=0x7fba68102180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:45.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.466+0000 7fba6e200700 1 -- 172.21.15.37:0/902765605 shutdown_connections 2024-03-28T18:08:45.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:45.466+0000 7fba6e200700 1 -- 172.21.15.37:0/902765605 wait complete. 2024-03-28T18:08:45.847 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:45 smithi037 ceph-mon[31460]: pgmap v330: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-28T18:08:45.847 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:45 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/902765605' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:08:46.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:45 smithi059 ceph-mon[36190]: pgmap v330: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-28T18:08:46.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:45 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/902765605' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:08:46.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:45 smithi170 ceph-mon[36280]: pgmap v330: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-28T18:08:46.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:45 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/902765605' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:08:46.349 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-28T18:08:46.349 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-28T18:08:46.350 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-28T18:08:46.350 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-28T18:08:46.350 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-28T18:08:46.351 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-28T18:08:46.351 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-28T18:08:46.351 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-28T18:08:46.352 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-28T18:08:46.352 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-28T18:08:46.353 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-28T18:08:46.353 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-28T18:08:47.615 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:08:47.646 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:08:47.647 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:08:47.664 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:08:47.664 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:08:47.671 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:08:47.802 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:47 smithi037 ceph-mon[31460]: pgmap v331: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-28T18:08:48.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:47 smithi059 ceph-mon[36190]: pgmap v331: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-28T18:08:48.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:47 smithi170 ceph-mon[36280]: pgmap v331: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-28T18:08:49.527 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:08:49.530 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:08:49.531 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:08:49.534 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:08:50.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:49 smithi059 ceph-mon[36190]: pgmap v332: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-28T18:08:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:49 smithi170 ceph-mon[36280]: pgmap v332: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-28T18:08:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:49 smithi037 ceph-mon[31460]: pgmap v332: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-28T18:08:51.933 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:51 smithi037 ceph-mon[31460]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-28T18:08:52.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:51 smithi059 ceph-mon[36190]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-28T18:08:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:51 smithi170 ceph-mon[36280]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-28T18:08:52.898 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:08:53.836 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:53 smithi037 ceph-mon[31460]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T18:08:54.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:53 smithi059 ceph-mon[36190]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T18:08:54.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:53 smithi170 ceph-mon[36280]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T18:08:55.999 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:55 smithi037 ceph-mon[31460]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:08:56.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:55 smithi059 ceph-mon[36190]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:08:56.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:55 smithi170 ceph-mon[36280]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:08:56.704 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:08:57.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.883+0000 7f9ef4400700 1 -- 172.21.15.37:0/448903910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9eec104fb0 msgr2=0x7f9eec109000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:57.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.883+0000 7f9ef4400700 1 --2- 172.21.15.37:0/448903910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9eec104fb0 0x7f9eec109000 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8009990 tx=0x7f9ed8009ca0 comp rx=0 tx=0).stop 2024-03-28T18:08:57.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.884+0000 7f9ef4400700 1 -- 172.21.15.37:0/448903910 shutdown_connections 2024-03-28T18:08:57.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.884+0000 7f9ef4400700 1 --2- 172.21.15.37:0/448903910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eec1096e0 0x7f9eec1112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:57.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.884+0000 7f9ef4400700 1 --2- 172.21.15.37:0/448903910 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9eec104600 0x7f9eec1049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:57.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.884+0000 7f9ef4400700 1 --2- 172.21.15.37:0/448903910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9eec104fb0 0x7f9eec109000 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:57.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.884+0000 7f9ef4400700 1 -- 172.21.15.37:0/448903910 >> 172.21.15.37:0/448903910 conn(0x7f9eec0ffe90 msgr2=0x7f9eec1022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:57.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.884+0000 7f9ef4400700 1 -- 172.21.15.37:0/448903910 shutdown_connections 2024-03-28T18:08:57.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.884+0000 7f9ef4400700 1 -- 172.21.15.37:0/448903910 wait complete. 2024-03-28T18:08:57.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.885+0000 7f9ef4400700 1 Processor -- start 2024-03-28T18:08:57.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.885+0000 7f9ef4400700 1 -- start start 2024-03-28T18:08:57.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.886+0000 7f9ef4400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9eec104600 0x7f9eec1a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:57.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.886+0000 7f9ef4400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9eec104fb0 0x7f9eec1a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:57.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.886+0000 7f9ef4400700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eec1096e0 0x7f9eec19aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:57.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.886+0000 7f9ef4400700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec114710 con 0x7f9eec104600 2024-03-28T18:08:57.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.886+0000 7f9ef4400700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec114590 con 0x7f9eec104fb0 2024-03-28T18:08:57.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.886+0000 7f9ef4400700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec114890 con 0x7f9eec1096e0 2024-03-28T18:08:57.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.886+0000 7f9ef2400700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eec1096e0 0x7f9eec19aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:57.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.886+0000 7f9ef1000700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9eec104fb0 0x7f9eec1a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:57.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.886+0000 7f9ef1a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9eec104600 0x7f9eec1a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:57.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.886+0000 7f9ef1000700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9eec104fb0 0x7f9eec1a0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:34288/0 (socket says 172.21.15.37:34288) 2024-03-28T18:08:57.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.886+0000 7f9ef1000700 1 -- 172.21.15.37:0/2086679127 learned_addr learned my addr 172.21.15.37:0/2086679127 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:57.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.886+0000 7f9ef1a00700 1 --2- 172.21.15.37:0/2086679127 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9eec104600 0x7f9eec1a0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40402/0 (socket says 172.21.15.37:40402) 2024-03-28T18:08:57.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.886+0000 7f9ef2400700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eec1096e0 0x7f9eec19aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:38770/0 (socket says 172.21.15.37:38770) 2024-03-28T18:08:57.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.887+0000 7f9ef1000700 1 -- 172.21.15.37:0/2086679127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eec1096e0 msgr2=0x7f9eec19aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:57.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.887+0000 7f9ef1000700 1 --2- 172.21.15.37:0/2086679127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eec1096e0 0x7f9eec19aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:57.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.887+0000 7f9ef1000700 1 -- 172.21.15.37:0/2086679127 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9eec104600 msgr2=0x7f9eec1a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:57.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.887+0000 7f9ef1000700 1 --2- 172.21.15.37:0/2086679127 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9eec104600 0x7f9eec1a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:57.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.887+0000 7f9ef1000700 1 -- 172.21.15.37:0/2086679127 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eec19b190 con 0x7f9eec104fb0 2024-03-28T18:08:57.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.887+0000 7f9ef1000700 1 --2- 172.21.15.37:0/2086679127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9eec104fb0 0x7f9eec1a0db0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8000c00 tx=0x7f9ed8005590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:57.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.887+0000 7f9eea000700 1 -- 172.21.15.37:0/2086679127 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8027070 con 0x7f9eec104fb0 2024-03-28T18:08:57.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.887+0000 7f9eea000700 1 -- 172.21.15.37:0/2086679127 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ed800fda0 con 0x7f9eec104fb0 2024-03-28T18:08:57.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.887+0000 7f9eea000700 1 -- 172.21.15.37:0/2086679127 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed80173f0 con 0x7f9eec104fb0 2024-03-28T18:08:57.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.888+0000 7f9ef4400700 1 -- 172.21.15.37:0/2086679127 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eec19b420 con 0x7f9eec104fb0 2024-03-28T18:08:57.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.888+0000 7f9ef4400700 1 -- 172.21.15.37:0/2086679127 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eec1a7740 con 0x7f9eec104fb0 2024-03-28T18:08:57.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.889+0000 7f9ef4400700 1 -- 172.21.15.37:0/2086679127 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9eec0628a0 con 0x7f9eec104fb0 2024-03-28T18:08:57.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.892+0000 7f9eea000700 1 -- 172.21.15.37:0/2086679127 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f9ed802a070 con 0x7f9eec104fb0 2024-03-28T18:08:57.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.894+0000 7f9eea000700 1 --2- 172.21.15.37:0/2086679127 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9ed40a29a0 0x7f9ed40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:57.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.894+0000 7f9eea000700 1 -- 172.21.15.37:0/2086679127 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f9ed80cf7a0 con 0x7f9eec104fb0 2024-03-28T18:08:57.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.894+0000 7f9ef1a00700 1 --2- 172.21.15.37:0/2086679127 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9ed40a29a0 0x7f9ed40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:57.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.894+0000 7f9eea000700 1 --2- 172.21.15.37:0/2086679127 >> [v2:172.21.15.59:6808/2410632623,v1:172.21.15.59:6809/2410632623] conn(0x7f9ed40ac180 0x7f9ed40ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:57.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.894+0000 7f9eea000700 1 -- 172.21.15.37:0/2086679127 --> [v2:172.21.15.59:6808/2410632623,v1:172.21.15.59:6809/2410632623] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ed8016e70 con 0x7f9ed40ac180 2024-03-28T18:08:57.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.894+0000 7f9eea000700 1 -- 172.21.15.37:0/2086679127 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ed80cfc80 con 0x7f9eec104fb0 2024-03-28T18:08:57.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.895+0000 7f9ef2400700 1 --2- 172.21.15.37:0/2086679127 >> [v2:172.21.15.59:6808/2410632623,v1:172.21.15.59:6809/2410632623] conn(0x7f9ed40ac180 0x7f9ed40ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:57.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.895+0000 7f9ef1a00700 1 --2- 172.21.15.37:0/2086679127 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9ed40a29a0 0x7f9ed40a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9eec19bb70 tx=0x7f9ee0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:57.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.895+0000 7f9ef2400700 1 --2- 172.21.15.37:0/2086679127 >> [v2:172.21.15.59:6808/2410632623,v1:172.21.15.59:6809/2410632623] conn(0x7f9ed40ac180 0x7f9ed40ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:57.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.896+0000 7f9eea000700 1 -- 172.21.15.37:0/2086679127 <== osd.4 v2:172.21.15.59:6808/2410632623 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9ed8016e70 con 0x7f9ed40ac180 2024-03-28T18:08:57.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.921+0000 7f9ef4400700 1 -- 172.21.15.37:0/2086679127 --> [v2:172.21.15.59:6808/2410632623,v1:172.21.15.59:6809/2410632623] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9eec062d70 con 0x7f9ed40ac180 2024-03-28T18:08:57.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.921+0000 7f9eea000700 1 -- 172.21.15.37:0/2086679127 <== osd.4 v2:172.21.15.59:6808/2410632623 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f9eec062d70 con 0x7f9ed40ac180 2024-03-28T18:08:57.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.922+0000 7f9ef4400700 1 -- 172.21.15.37:0/2086679127 >> [v2:172.21.15.59:6808/2410632623,v1:172.21.15.59:6809/2410632623] conn(0x7f9ed40ac180 msgr2=0x7f9ed40ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:57.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.922+0000 7f9ef4400700 1 --2- 172.21.15.37:0/2086679127 >> [v2:172.21.15.59:6808/2410632623,v1:172.21.15.59:6809/2410632623] conn(0x7f9ed40ac180 0x7f9ed40ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:57.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.922+0000 7f9ef4400700 1 -- 172.21.15.37:0/2086679127 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9ed40a29a0 msgr2=0x7f9ed40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:57.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.922+0000 7f9ef4400700 1 --2- 172.21.15.37:0/2086679127 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9ed40a29a0 0x7f9ed40a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9eec19bb70 tx=0x7f9ee0009260 comp rx=0 tx=0).stop 2024-03-28T18:08:57.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.922+0000 7f9ef4400700 1 -- 172.21.15.37:0/2086679127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9eec104fb0 msgr2=0x7f9eec1a0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:57.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.922+0000 7f9ef4400700 1 --2- 172.21.15.37:0/2086679127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9eec104fb0 0x7f9eec1a0db0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8000c00 tx=0x7f9ed8005590 comp rx=0 tx=0).stop 2024-03-28T18:08:57.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.922+0000 7f9ef2400700 1 -- 172.21.15.37:0/2086679127 reap_dead start 2024-03-28T18:08:57.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.922+0000 7f9ef4400700 1 -- 172.21.15.37:0/2086679127 shutdown_connections 2024-03-28T18:08:57.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.922+0000 7f9ef4400700 1 -- 172.21.15.37:0/2086679127 >> 172.21.15.37:0/2086679127 conn(0x7f9eec0ffe90 msgr2=0x7f9eec100a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:57.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.922+0000 7f9ef4400700 1 -- 172.21.15.37:0/2086679127 shutdown_connections 2024-03-28T18:08:57.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:56.922+0000 7f9ef4400700 1 -- 172.21.15.37:0/2086679127 wait complete. 2024-03-28T18:08:57.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:57 smithi037 ceph-mon[31460]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-28T18:08:58.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:57 smithi059 ceph-mon[36190]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-28T18:08:58.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:57 smithi170 ceph-mon[36280]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-28T18:08:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.384+0000 7f338e800700 1 -- 172.21.15.37:0/862704946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3388102ba0 msgr2=0x7f338810efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.384+0000 7f338e800700 1 --2- 172.21.15.37:0/862704946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3388102ba0 0x7f338810efe0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f338000b0d0 tx=0x7f338000b3e0 comp rx=0 tx=0).stop 2024-03-28T18:08:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.384+0000 7f338e800700 1 -- 172.21.15.37:0/862704946 shutdown_connections 2024-03-28T18:08:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.384+0000 7f338e800700 1 --2- 172.21.15.37:0/862704946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3388102ba0 0x7f338810efe0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.384+0000 7f338e800700 1 --2- 172.21.15.37:0/862704946 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33881081c0 0x7f33881085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.384+0000 7f338e800700 1 --2- 172.21.15.37:0/862704946 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33881021e0 0x7f3388102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.384+0000 7f338e800700 1 -- 172.21.15.37:0/862704946 >> 172.21.15.37:0/862704946 conn(0x7f33880792c0 msgr2=0x7f33880796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.384+0000 7f338e800700 1 -- 172.21.15.37:0/862704946 shutdown_connections 2024-03-28T18:08:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.385+0000 7f338e800700 1 -- 172.21.15.37:0/862704946 wait complete. 2024-03-28T18:08:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.386+0000 7f338e800700 1 Processor -- start 2024-03-28T18:08:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.386+0000 7f338e800700 1 -- start start 2024-03-28T18:08:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.386+0000 7f338e800700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33881021e0 0x7f3388074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.386+0000 7f338e800700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3388102ba0 0x7f33880749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.386+0000 7f338e800700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33881081c0 0x7f3388074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.386+0000 7f338e800700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3388112490 con 0x7f33881021e0 2024-03-28T18:08:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.386+0000 7f338e800700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3388112310 con 0x7f3388102ba0 2024-03-28T18:08:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.386+0000 7f338e800700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3388112610 con 0x7f33881081c0 2024-03-28T18:08:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.387+0000 7f3386a00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3388102ba0 0x7f33880749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.387+0000 7f3387400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33881021e0 0x7f3388074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.387+0000 7f3387e00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33881081c0 0x7f3388074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.387+0000 7f3387e00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33881081c0 0x7f3388074f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:38814/0 (socket says 172.21.15.37:38814) 2024-03-28T18:08:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.387+0000 7f3387e00700 1 -- 172.21.15.37:0/2368898308 learned_addr learned my addr 172.21.15.37:0/2368898308 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:08:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.387+0000 7f3387400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33881021e0 0x7f3388074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40434/0 (socket says 172.21.15.37:40434) 2024-03-28T18:08:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.387+0000 7f3386a00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3388102ba0 0x7f33880749d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:34330/0 (socket says 172.21.15.37:34330) 2024-03-28T18:08:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.387+0000 7f3387e00700 1 -- 172.21.15.37:0/2368898308 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3388102ba0 msgr2=0x7f33880749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.387+0000 7f3387e00700 1 --2- 172.21.15.37:0/2368898308 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3388102ba0 0x7f33880749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.387+0000 7f3387e00700 1 -- 172.21.15.37:0/2368898308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33881021e0 msgr2=0x7f3388074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.387+0000 7f3387e00700 1 --2- 172.21.15.37:0/2368898308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33881021e0 0x7f3388074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.387+0000 7f3387e00700 1 -- 172.21.15.37:0/2368898308 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33881a6d70 con 0x7f33881081c0 2024-03-28T18:08:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.387+0000 7f3387e00700 1 --2- 172.21.15.37:0/2368898308 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33881081c0 0x7f3388074f10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f338000d010 tx=0x7f33800079a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.388+0000 7f337fe00700 1 -- 172.21.15.37:0/2368898308 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3380003c00 con 0x7f33881081c0 2024-03-28T18:08:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.388+0000 7f337fe00700 1 -- 172.21.15.37:0/2368898308 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3380022070 con 0x7f33881081c0 2024-03-28T18:08:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.388+0000 7f337fe00700 1 -- 172.21.15.37:0/2368898308 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338001b6c0 con 0x7f33881081c0 2024-03-28T18:08:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.388+0000 7f338e800700 1 -- 172.21.15.37:0/2368898308 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33881a7000 con 0x7f33881081c0 2024-03-28T18:08:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.388+0000 7f338e800700 1 -- 172.21.15.37:0/2368898308 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33881a7400 con 0x7f33881081c0 2024-03-28T18:08:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.389+0000 7f338e800700 1 -- 172.21.15.37:0/2368898308 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f33881016c0 con 0x7f33881081c0 2024-03-28T18:08:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.391+0000 7f337fe00700 1 -- 172.21.15.37:0/2368898308 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f3380009ba0 con 0x7f33881081c0 2024-03-28T18:08:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.392+0000 7f337fe00700 1 --2- 172.21.15.37:0/2368898308 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f336c0a29a0 0x7f336c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.392+0000 7f337fe00700 1 -- 172.21.15.37:0/2368898308 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f338000f020 con 0x7f33881081c0 2024-03-28T18:08:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.393+0000 7f3387400700 1 --2- 172.21.15.37:0/2368898308 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f336c0a29a0 0x7f336c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.393+0000 7f337fe00700 1 --2- 172.21.15.37:0/2368898308 >> [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035] conn(0x7f336c0ac180 0x7f336c0ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:08:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.393+0000 7f337fe00700 1 -- 172.21.15.37:0/2368898308 --> [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3380028e70 con 0x7f336c0ac180 2024-03-28T18:08:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.393+0000 7f337fe00700 1 -- 172.21.15.37:0/2368898308 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33800d0110 con 0x7f33881081c0 2024-03-28T18:08:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.393+0000 7f3386a00700 1 --2- 172.21.15.37:0/2368898308 >> [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035] conn(0x7f336c0ac180 0x7f336c0ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:08:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.393+0000 7f3387400700 1 --2- 172.21.15.37:0/2368898308 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f336c0a29a0 0x7f336c0a4e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3378009be0 tx=0x7f3378009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.394+0000 7f3386a00700 1 --2- 172.21.15.37:0/2368898308 >> [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035] conn(0x7f336c0ac180 0x7f336c0ae5a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:08:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.395+0000 7f337fe00700 1 -- 172.21.15.37:0/2368898308 <== osd.0 v2:172.21.15.37:6802/3562602035 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3380028e70 con 0x7f336c0ac180 2024-03-28T18:08:59.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.415+0000 7f338e800700 1 -- 172.21.15.37:0/2368898308 --> [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3388103870 con 0x7f336c0ac180 2024-03-28T18:08:59.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.415+0000 7f337fe00700 1 -- 172.21.15.37:0/2368898308 <== osd.0 v2:172.21.15.37:6802/3562602035 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3388103870 con 0x7f336c0ac180 2024-03-28T18:08:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.416+0000 7f338e800700 1 -- 172.21.15.37:0/2368898308 >> [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035] conn(0x7f336c0ac180 msgr2=0x7f336c0ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.416+0000 7f338e800700 1 --2- 172.21.15.37:0/2368898308 >> [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035] conn(0x7f336c0ac180 0x7f336c0ae5a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:08:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.416+0000 7f338e800700 1 -- 172.21.15.37:0/2368898308 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f336c0a29a0 msgr2=0x7f336c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.416+0000 7f338e800700 1 --2- 172.21.15.37:0/2368898308 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f336c0a29a0 0x7f336c0a4e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3378009be0 tx=0x7f3378009260 comp rx=0 tx=0).stop 2024-03-28T18:08:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.416+0000 7f338e800700 1 -- 172.21.15.37:0/2368898308 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33881081c0 msgr2=0x7f3388074f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:08:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.416+0000 7f338e800700 1 --2- 172.21.15.37:0/2368898308 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33881081c0 0x7f3388074f10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f338000d010 tx=0x7f33800079a0 comp rx=0 tx=0).stop 2024-03-28T18:08:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.416+0000 7f3387e00700 1 -- 172.21.15.37:0/2368898308 reap_dead start 2024-03-28T18:08:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.416+0000 7f338e800700 1 -- 172.21.15.37:0/2368898308 shutdown_connections 2024-03-28T18:08:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.416+0000 7f338e800700 1 -- 172.21.15.37:0/2368898308 >> 172.21.15.37:0/2368898308 conn(0x7f33880792c0 msgr2=0x7f3388100c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:08:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.416+0000 7f338e800700 1 -- 172.21.15.37:0/2368898308 shutdown_connections 2024-03-28T18:08:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:59.416+0000 7f338e800700 1 -- 172.21.15.37:0/2368898308 wait complete. 2024-03-28T18:08:59.847 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:08:59 smithi037 ceph-mon[31460]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:09:00.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:08:59 smithi059 ceph-mon[36190]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:09:00.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:08:59 smithi170 ceph-mon[36280]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:09:00.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.884+0000 7f5d9ac00700 1 -- 172.21.15.37:0/486168643 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d94104f90 msgr2=0x7f5d94108fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:00.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.884+0000 7f5d9ac00700 1 --2- 172.21.15.37:0/486168643 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d94104f90 0x7f5d94108fe0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5d80009990 tx=0x7f5d80009ca0 comp rx=0 tx=0).stop 2024-03-28T18:09:00.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.884+0000 7f5d9ac00700 1 -- 172.21.15.37:0/486168643 shutdown_connections 2024-03-28T18:09:00.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.884+0000 7f5d9ac00700 1 --2- 172.21.15.37:0/486168643 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d941096c0 0x7f5d94111290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:00.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.884+0000 7f5d9ac00700 1 --2- 172.21.15.37:0/486168643 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d941045e0 0x7f5d941049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:00.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.884+0000 7f5d9ac00700 1 --2- 172.21.15.37:0/486168643 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d94104f90 0x7f5d94108fe0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:00.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.884+0000 7f5d9ac00700 1 -- 172.21.15.37:0/486168643 >> 172.21.15.37:0/486168643 conn(0x7f5d940ffe70 msgr2=0x7f5d94102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:00.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.884+0000 7f5d9ac00700 1 -- 172.21.15.37:0/486168643 shutdown_connections 2024-03-28T18:09:00.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.884+0000 7f5d9ac00700 1 -- 172.21.15.37:0/486168643 wait complete. 2024-03-28T18:09:00.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.885+0000 7f5d9ac00700 1 Processor -- start 2024-03-28T18:09:00.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.885+0000 7f5d9ac00700 1 -- start start 2024-03-28T18:09:00.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.886+0000 7f5d9ac00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d941045e0 0x7f5d941a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:00.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.886+0000 7f5d9ac00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d94104f90 0x7f5d941a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:00.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.886+0000 7f5d9ac00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d941096c0 0x7f5d9419a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:00.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.886+0000 7f5d9ac00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d941146f0 con 0x7f5d94104f90 2024-03-28T18:09:00.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.886+0000 7f5d9ac00700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d94114570 con 0x7f5d941096c0 2024-03-28T18:09:00.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.886+0000 7f5d9ac00700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d94114870 con 0x7f5d941045e0 2024-03-28T18:09:00.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.886+0000 7f5d93e00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d941045e0 0x7f5d941a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:00.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.886+0000 7f5d93400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d94104f90 0x7f5d941a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:00.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.886+0000 7f5d93e00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d941045e0 0x7f5d941a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:38786/0 (socket says 172.21.15.37:38786) 2024-03-28T18:09:00.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.886+0000 7f5d98c00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d941096c0 0x7f5d9419a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:00.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.886+0000 7f5d93e00700 1 -- 172.21.15.37:0/2914372561 learned_addr learned my addr 172.21.15.37:0/2914372561 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:00.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.887+0000 7f5d98c00700 1 -- 172.21.15.37:0/2914372561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d941045e0 msgr2=0x7f5d941a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:00.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.887+0000 7f5d98c00700 1 --2- 172.21.15.37:0/2914372561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d941045e0 0x7f5d941a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:00.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.887+0000 7f5d98c00700 1 -- 172.21.15.37:0/2914372561 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d94104f90 msgr2=0x7f5d941a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:00.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.887+0000 7f5d98c00700 1 --2- 172.21.15.37:0/2914372561 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d94104f90 0x7f5d941a0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:00.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.887+0000 7f5d98c00700 1 -- 172.21.15.37:0/2914372561 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d9419b070 con 0x7f5d941096c0 2024-03-28T18:09:00.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.887+0000 7f5d98c00700 1 --2- 172.21.15.37:0/2914372561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d941096c0 0x7f5d9419a910 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c00efc0 tx=0x7f5d8c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:00.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.887+0000 7f5d90c00700 1 -- 172.21.15.37:0/2914372561 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c0092e0 con 0x7f5d941096c0 2024-03-28T18:09:00.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.887+0000 7f5d90c00700 1 -- 172.21.15.37:0/2914372561 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5d8c004bb0 con 0x7f5d941096c0 2024-03-28T18:09:00.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.887+0000 7f5d9ac00700 1 -- 172.21.15.37:0/2914372561 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d9419b300 con 0x7f5d941096c0 2024-03-28T18:09:00.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.887+0000 7f5d90c00700 1 -- 172.21.15.37:0/2914372561 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c004280 con 0x7f5d941096c0 2024-03-28T18:09:00.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.887+0000 7f5d9ac00700 1 -- 172.21.15.37:0/2914372561 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d941a7620 con 0x7f5d941096c0 2024-03-28T18:09:00.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.888+0000 7f5d9ac00700 1 -- 172.21.15.37:0/2914372561 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5d94062920 con 0x7f5d941096c0 2024-03-28T18:09:00.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.890+0000 7f5d90c00700 1 -- 172.21.15.37:0/2914372561 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f5d8c01b030 con 0x7f5d941096c0 2024-03-28T18:09:00.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.891+0000 7f5d90c00700 1 --2- 172.21.15.37:0/2914372561 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5d780a29a0 0x7f5d780a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:00.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.892+0000 7f5d90c00700 1 -- 172.21.15.37:0/2914372561 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f5d8c0ce5b0 con 0x7f5d941096c0 2024-03-28T18:09:00.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.892+0000 7f5d93e00700 1 --2- 172.21.15.37:0/2914372561 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5d780a29a0 0x7f5d780a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:00.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.892+0000 7f5d90c00700 1 --2- 172.21.15.37:0/2914372561 >> [v2:172.21.15.170:6816/3235794556,v1:172.21.15.170:6817/3235794556] conn(0x7f5d780ac180 0x7f5d780ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:00.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.892+0000 7f5d90c00700 1 -- 172.21.15.37:0/2914372561 --> [v2:172.21.15.170:6816/3235794556,v1:172.21.15.170:6817/3235794556] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d8c0048a0 con 0x7f5d780ac180 2024-03-28T18:09:00.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.892+0000 7f5d90c00700 1 -- 172.21.15.37:0/2914372561 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d8c0d5050 con 0x7f5d941096c0 2024-03-28T18:09:00.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.892+0000 7f5d93400700 1 --2- 172.21.15.37:0/2914372561 >> [v2:172.21.15.170:6816/3235794556,v1:172.21.15.170:6817/3235794556] conn(0x7f5d780ac180 0x7f5d780ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:00.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.892+0000 7f5d93e00700 1 --2- 172.21.15.37:0/2914372561 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5d780a29a0 0x7f5d780a4e60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5d9419ba50 tx=0x7f5d84009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:00.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.893+0000 7f5d93400700 1 --2- 172.21.15.37:0/2914372561 >> [v2:172.21.15.170:6816/3235794556,v1:172.21.15.170:6817/3235794556] conn(0x7f5d780ac180 0x7f5d780ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:00.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.894+0000 7f5d90c00700 1 -- 172.21.15.37:0/2914372561 <== osd.8 v2:172.21.15.170:6816/3235794556 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5d8c0048a0 con 0x7f5d780ac180 2024-03-28T18:09:00.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.912+0000 7f5d9ac00700 1 -- 172.21.15.37:0/2914372561 --> [v2:172.21.15.170:6816/3235794556,v1:172.21.15.170:6817/3235794556] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5d94062d20 con 0x7f5d780ac180 2024-03-28T18:09:00.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.912+0000 7f5d90c00700 1 -- 172.21.15.37:0/2914372561 <== osd.8 v2:172.21.15.170:6816/3235794556 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5d94062d20 con 0x7f5d780ac180 2024-03-28T18:09:00.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.913+0000 7f5d9ac00700 1 -- 172.21.15.37:0/2914372561 >> [v2:172.21.15.170:6816/3235794556,v1:172.21.15.170:6817/3235794556] conn(0x7f5d780ac180 msgr2=0x7f5d780ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:00.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.913+0000 7f5d9ac00700 1 --2- 172.21.15.37:0/2914372561 >> [v2:172.21.15.170:6816/3235794556,v1:172.21.15.170:6817/3235794556] conn(0x7f5d780ac180 0x7f5d780ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:00.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.913+0000 7f5d9ac00700 1 -- 172.21.15.37:0/2914372561 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5d780a29a0 msgr2=0x7f5d780a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:00.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.913+0000 7f5d9ac00700 1 --2- 172.21.15.37:0/2914372561 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5d780a29a0 0x7f5d780a4e60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5d9419ba50 tx=0x7f5d84009260 comp rx=0 tx=0).stop 2024-03-28T18:09:00.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.913+0000 7f5d9ac00700 1 -- 172.21.15.37:0/2914372561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d941096c0 msgr2=0x7f5d9419a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:00.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.913+0000 7f5d9ac00700 1 --2- 172.21.15.37:0/2914372561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d941096c0 0x7f5d9419a910 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c00efc0 tx=0x7f5d8c00c3a0 comp rx=0 tx=0).stop 2024-03-28T18:09:00.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.913+0000 7f5d98c00700 1 -- 172.21.15.37:0/2914372561 reap_dead start 2024-03-28T18:09:00.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.913+0000 7f5d9ac00700 1 -- 172.21.15.37:0/2914372561 shutdown_connections 2024-03-28T18:09:00.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.913+0000 7f5d9ac00700 1 -- 172.21.15.37:0/2914372561 >> 172.21.15.37:0/2914372561 conn(0x7f5d940ffe70 msgr2=0x7f5d94100980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:00.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.913+0000 7f5d9ac00700 1 -- 172.21.15.37:0/2914372561 shutdown_connections 2024-03-28T18:09:00.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:08:58.913+0000 7f5d9ac00700 1 -- 172.21.15.37:0/2914372561 wait complete. 2024-03-28T18:09:01.793 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:01 smithi037 ceph-mon[31460]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T18:09:02.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.969+0000 7fb903e00700 1 -- 172.21.15.37:0/2905600832 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb90410c660 msgr2=0x7fb90410eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:02.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.969+0000 7fb903e00700 1 --2- 172.21.15.37:0/2905600832 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb90410c660 0x7fb90410eb20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc00b0a0 tx=0x7fb8fc00b3b0 comp rx=0 tx=0).stop 2024-03-28T18:09:02.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.969+0000 7fb903e00700 1 -- 172.21.15.37:0/2905600832 shutdown_connections 2024-03-28T18:09:02.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.969+0000 7fb903e00700 1 --2- 172.21.15.37:0/2905600832 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb90410c660 0x7fb90410eb20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:02.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.969+0000 7fb903e00700 1 --2- 172.21.15.37:0/2905600832 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb904103230 0x7fb904103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:02.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.969+0000 7fb903e00700 1 --2- 172.21.15.37:0/2905600832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb904103b50 0x7fb90410c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:02.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.969+0000 7fb903e00700 1 -- 172.21.15.37:0/2905600832 >> 172.21.15.37:0/2905600832 conn(0x7fb9040fca60 msgr2=0x7fb9040fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:02.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.970+0000 7fb903e00700 1 -- 172.21.15.37:0/2905600832 shutdown_connections 2024-03-28T18:09:02.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.970+0000 7fb903e00700 1 -- 172.21.15.37:0/2905600832 wait complete. 2024-03-28T18:09:02.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.971+0000 7fb903e00700 1 Processor -- start 2024-03-28T18:09:02.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.971+0000 7fb903e00700 1 -- start start 2024-03-28T18:09:02.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.971+0000 7fb903e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb904103230 0x7fb9041a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:02.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.971+0000 7fb903e00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb904103b50 0x7fb9041a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:02.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.972+0000 7fb903e00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb90410c660 0x7fb90419a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:02.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.972+0000 7fb903e00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb904112400 con 0x7fb904103230 2024-03-28T18:09:02.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.972+0000 7fb903e00700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb904112280 con 0x7fb90410c660 2024-03-28T18:09:02.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.972+0000 7fb903e00700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb904112580 con 0x7fb904103b50 2024-03-28T18:09:02.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.972+0000 7fb8fbe00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb904103b50 0x7fb9041a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:02.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.972+0000 7fb8fbe00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb904103b50 0x7fb9041a0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:38830/0 (socket says 172.21.15.37:38830) 2024-03-28T18:09:02.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.972+0000 7fb8fbe00700 1 -- 172.21.15.37:0/2791481095 learned_addr learned my addr 172.21.15.37:0/2791481095 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:02.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.972+0000 7fb902a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb904103230 0x7fb9041a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:02.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.972+0000 7fb903400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb90410c660 0x7fb90419a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:02.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.972+0000 7fb8fbe00700 1 -- 172.21.15.37:0/2791481095 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb90410c660 msgr2=0x7fb90419a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:02.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.972+0000 7fb8fbe00700 1 --2- 172.21.15.37:0/2791481095 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb90410c660 0x7fb90419a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:02.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.972+0000 7fb8fbe00700 1 -- 172.21.15.37:0/2791481095 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb904103230 msgr2=0x7fb9041a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:02.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.972+0000 7fb8fbe00700 1 --2- 172.21.15.37:0/2791481095 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb904103230 0x7fb9041a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:02.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.972+0000 7fb8fbe00700 1 -- 172.21.15.37:0/2791481095 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90419aff0 con 0x7fb904103b50 2024-03-28T18:09:02.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.972+0000 7fb903400700 1 --2- 172.21.15.37:0/2791481095 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb90410c660 0x7fb90419a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:09:02.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.973+0000 7fb8fbe00700 1 --2- 172.21.15.37:0/2791481095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb904103b50 0x7fb9041a0c10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00e9c0 tx=0x7fb8ec00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:02.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.973+0000 7fb908a00700 1 -- 172.21.15.37:0/2791481095 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec004a30 con 0x7fb904103b50 2024-03-28T18:09:02.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.973+0000 7fb908a00700 1 -- 172.21.15.37:0/2791481095 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb8ec00cca0 con 0x7fb904103b50 2024-03-28T18:09:02.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.973+0000 7fb908a00700 1 -- 172.21.15.37:0/2791481095 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec005610 con 0x7fb904103b50 2024-03-28T18:09:02.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.973+0000 7fb903e00700 1 -- 172.21.15.37:0/2791481095 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90419b2e0 con 0x7fb904103b50 2024-03-28T18:09:02.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.973+0000 7fb903e00700 1 -- 172.21.15.37:0/2791481095 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9041a75a0 con 0x7fb904103b50 2024-03-28T18:09:02.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.974+0000 7fb903e00700 1 -- 172.21.15.37:0/2791481095 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb9041028c0 con 0x7fb904103b50 2024-03-28T18:09:02.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.976+0000 7fb908a00700 1 -- 172.21.15.37:0/2791481095 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fb8ec00c7c0 con 0x7fb904103b50 2024-03-28T18:09:02.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.978+0000 7fb908a00700 1 --2- 172.21.15.37:0/2791481095 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb8e80a29a0 0x7fb8e80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:02.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.979+0000 7fb908a00700 1 -- 172.21.15.37:0/2791481095 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb8ec0ceb60 con 0x7fb904103b50 2024-03-28T18:09:02.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.979+0000 7fb902a00700 1 --2- 172.21.15.37:0/2791481095 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb8e80a29a0 0x7fb8e80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:02.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.979+0000 7fb908a00700 1 --2- 172.21.15.37:0/2791481095 >> [v2:172.21.15.59:6816/1343675690,v1:172.21.15.59:6817/1343675690] conn(0x7fb8e80ac180 0x7fb8e80ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:02.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.979+0000 7fb908a00700 1 -- 172.21.15.37:0/2791481095 --> [v2:172.21.15.59:6816/1343675690,v1:172.21.15.59:6817/1343675690] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8ec004d50 con 0x7fb8e80ac180 2024-03-28T18:09:02.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.979+0000 7fb908a00700 1 -- 172.21.15.37:0/2791481095 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb8ec0d32a0 con 0x7fb904103b50 2024-03-28T18:09:02.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.979+0000 7fb903400700 1 --2- 172.21.15.37:0/2791481095 >> [v2:172.21.15.59:6816/1343675690,v1:172.21.15.59:6817/1343675690] conn(0x7fb8e80ac180 0x7fb8e80ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:02.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.979+0000 7fb902a00700 1 --2- 172.21.15.37:0/2791481095 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb8e80a29a0 0x7fb8e80a4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb90419b9d0 tx=0x7fb8f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:02.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.980+0000 7fb903400700 1 --2- 172.21.15.37:0/2791481095 >> [v2:172.21.15.59:6816/1343675690,v1:172.21.15.59:6817/1343675690] conn(0x7fb8e80ac180 0x7fb8e80ae5a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:02.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:01.981+0000 7fb908a00700 1 -- 172.21.15.37:0/2791481095 <== osd.7 v2:172.21.15.59:6816/1343675690 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb8ec004d50 con 0x7fb8e80ac180 2024-03-28T18:09:02.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:02.010+0000 7fb903e00700 1 -- 172.21.15.37:0/2791481095 --> [v2:172.21.15.59:6816/1343675690,v1:172.21.15.59:6817/1343675690] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb90406a720 con 0x7fb8e80ac180 2024-03-28T18:09:02.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:02.011+0000 7fb908a00700 1 -- 172.21.15.37:0/2791481095 <== osd.7 v2:172.21.15.59:6816/1343675690 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb90406a720 con 0x7fb8e80ac180 2024-03-28T18:09:02.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:02.011+0000 7fb903e00700 1 -- 172.21.15.37:0/2791481095 >> [v2:172.21.15.59:6816/1343675690,v1:172.21.15.59:6817/1343675690] conn(0x7fb8e80ac180 msgr2=0x7fb8e80ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:02.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:02.011+0000 7fb903e00700 1 --2- 172.21.15.37:0/2791481095 >> [v2:172.21.15.59:6816/1343675690,v1:172.21.15.59:6817/1343675690] conn(0x7fb8e80ac180 0x7fb8e80ae5a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:02.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:02.011+0000 7fb903e00700 1 -- 172.21.15.37:0/2791481095 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb8e80a29a0 msgr2=0x7fb8e80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:02.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:02.011+0000 7fb903e00700 1 --2- 172.21.15.37:0/2791481095 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb8e80a29a0 0x7fb8e80a4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb90419b9d0 tx=0x7fb8f4009260 comp rx=0 tx=0).stop 2024-03-28T18:09:02.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:02.011+0000 7fb903e00700 1 -- 172.21.15.37:0/2791481095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb904103b50 msgr2=0x7fb9041a0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:02.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:02.011+0000 7fb903e00700 1 --2- 172.21.15.37:0/2791481095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb904103b50 0x7fb9041a0c10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00e9c0 tx=0x7fb8ec00ecd0 comp rx=0 tx=0).stop 2024-03-28T18:09:02.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:02.012+0000 7fb903400700 1 -- 172.21.15.37:0/2791481095 reap_dead start 2024-03-28T18:09:02.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:02.012+0000 7fb903e00700 1 -- 172.21.15.37:0/2791481095 shutdown_connections 2024-03-28T18:09:02.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:02.012+0000 7fb903e00700 1 -- 172.21.15.37:0/2791481095 >> 172.21.15.37:0/2791481095 conn(0x7fb9040fca60 msgr2=0x7fb90410e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:02.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:02.012+0000 7fb903e00700 1 -- 172.21.15.37:0/2791481095 shutdown_connections 2024-03-28T18:09:02.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:02.012+0000 7fb903e00700 1 -- 172.21.15.37:0/2791481095 wait complete. 2024-03-28T18:09:02.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:01 smithi059 ceph-mon[36190]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T18:09:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:01 smithi170 ceph-mon[36280]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T18:09:03.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:03 smithi037 ceph-mon[31460]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T18:09:04.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:03 smithi059 ceph-mon[36190]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T18:09:04.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:03 smithi170 ceph-mon[36280]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T18:09:04.416 INFO:teuthology.orchestra.run.smithi037.stdout:141733920831 2024-03-28T18:09:04.417 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-28T18:09:05.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:05 smithi037 ceph-mon[31460]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T18:09:06.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:05 smithi059 ceph-mon[36190]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T18:09:06.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:05 smithi170 ceph-mon[36280]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T18:09:06.038 INFO:teuthology.orchestra.run.smithi037.stdout:34359738453 2024-03-28T18:09:06.038 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-28T18:09:06.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.255+0000 7f5117400700 1 -- 172.21.15.37:0/604795775 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5110104fb0 msgr2=0x7f5110109000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:06.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.255+0000 7f5117400700 1 --2- 172.21.15.37:0/604795775 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5110104fb0 0x7f5110109000 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f50fc009990 tx=0x7f50fc009ca0 comp rx=0 tx=0).stop 2024-03-28T18:09:06.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.255+0000 7f5117400700 1 -- 172.21.15.37:0/604795775 shutdown_connections 2024-03-28T18:09:06.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.255+0000 7f5117400700 1 --2- 172.21.15.37:0/604795775 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51101096e0 0x7f51101112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:06.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.255+0000 7f5117400700 1 --2- 172.21.15.37:0/604795775 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5110104600 0x7f51101049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:06.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.255+0000 7f5117400700 1 --2- 172.21.15.37:0/604795775 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5110104fb0 0x7f5110109000 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:06.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.255+0000 7f5117400700 1 -- 172.21.15.37:0/604795775 >> 172.21.15.37:0/604795775 conn(0x7f51100ffe90 msgr2=0x7f51101022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:06.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.255+0000 7f5117400700 1 -- 172.21.15.37:0/604795775 shutdown_connections 2024-03-28T18:09:06.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.255+0000 7f5117400700 1 -- 172.21.15.37:0/604795775 wait complete. 2024-03-28T18:09:06.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.257+0000 7f5117400700 1 Processor -- start 2024-03-28T18:09:06.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.257+0000 7f5117400700 1 -- start start 2024-03-28T18:09:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.257+0000 7f5117400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5110104600 0x7f51101a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.257+0000 7f5117400700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5110104fb0 0x7f51101a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.257+0000 7f5117400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51101096e0 0x7f511019a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.257+0000 7f5117400700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51101146c0 con 0x7f51101096e0 2024-03-28T18:09:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.257+0000 7f5117400700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5110114540 con 0x7f5110104600 2024-03-28T18:09:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.257+0000 7f5117400700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5110114840 con 0x7f5110104fb0 2024-03-28T18:09:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.258+0000 7f5114a00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5110104600 0x7f51101a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.258+0000 7f510fe00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5110104fb0 0x7f51101a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.258+0000 7f5114a00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5110104600 0x7f51101a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:56786/0 (socket says 172.21.15.37:56786) 2024-03-28T18:09:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.258+0000 7f5114a00700 1 -- 172.21.15.37:0/545017012 learned_addr learned my addr 172.21.15.37:0/545017012 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.258+0000 7f5115400700 1 --2- 172.21.15.37:0/545017012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51101096e0 0x7f511019a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:06.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.258+0000 7f510fe00700 1 -- 172.21.15.37:0/545017012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5110104600 msgr2=0x7f51101a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.258+0000 7f510fe00700 1 --2- 172.21.15.37:0/545017012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5110104600 0x7f51101a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.258+0000 7f510fe00700 1 -- 172.21.15.37:0/545017012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51101096e0 msgr2=0x7f511019a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.258+0000 7f510fe00700 1 --2- 172.21.15.37:0/545017012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51101096e0 0x7f511019a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.258+0000 7f510fe00700 1 -- 172.21.15.37:0/545017012 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511019b140 con 0x7f5110104fb0 2024-03-28T18:09:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.258+0000 7f5114a00700 1 --2- 172.21.15.37:0/545017012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5110104600 0x7f51101a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:09:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.258+0000 7f510fe00700 1 --2- 172.21.15.37:0/545017012 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5110104fb0 0x7f51101a0d60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f50fc0059d0 tx=0x7f50fc005590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.259+0000 7f510d600700 1 -- 172.21.15.37:0/545017012 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50fc027070 con 0x7f5110104fb0 2024-03-28T18:09:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.259+0000 7f510d600700 1 -- 172.21.15.37:0/545017012 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50fc0056c0 con 0x7f5110104fb0 2024-03-28T18:09:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.259+0000 7f5117400700 1 -- 172.21.15.37:0/545017012 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f511019b3d0 con 0x7f5110104fb0 2024-03-28T18:09:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.259+0000 7f510d600700 1 -- 172.21.15.37:0/545017012 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50fc0173f0 con 0x7f5110104fb0 2024-03-28T18:09:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.259+0000 7f5117400700 1 -- 172.21.15.37:0/545017012 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51101a76f0 con 0x7f5110104fb0 2024-03-28T18:09:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.260+0000 7f5117400700 1 -- 172.21.15.37:0/545017012 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f51100628a0 con 0x7f5110104fb0 2024-03-28T18:09:06.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.262+0000 7f510d600700 1 -- 172.21.15.37:0/545017012 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f50fc02a070 con 0x7f5110104fb0 2024-03-28T18:09:06.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.263+0000 7f510d600700 1 --2- 172.21.15.37:0/545017012 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f50f80a29a0 0x7f50f80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:06.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.263+0000 7f510d600700 1 -- 172.21.15.37:0/545017012 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f50fc0cf9e0 con 0x7f5110104fb0 2024-03-28T18:09:06.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.264+0000 7f5114a00700 1 --2- 172.21.15.37:0/545017012 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f50f80a29a0 0x7f50f80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:06.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.264+0000 7f510d600700 1 --2- 172.21.15.37:0/545017012 >> [v2:172.21.15.59:6800/3326537211,v1:172.21.15.59:6801/3326537211] conn(0x7f50f80ac050 0x7f50f80ae470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:06.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.264+0000 7f510d600700 1 -- 172.21.15.37:0/545017012 --> [v2:172.21.15.59:6800/3326537211,v1:172.21.15.59:6801/3326537211] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50fc0053e0 con 0x7f50f80ac050 2024-03-28T18:09:06.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.264+0000 7f510d600700 1 -- 172.21.15.37:0/545017012 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50fc0cfd90 con 0x7f5110104fb0 2024-03-28T18:09:06.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.264+0000 7f5115400700 1 --2- 172.21.15.37:0/545017012 >> [v2:172.21.15.59:6800/3326537211,v1:172.21.15.59:6801/3326537211] conn(0x7f50f80ac050 0x7f50f80ae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:06.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.264+0000 7f5115400700 1 --2- 172.21.15.37:0/545017012 >> [v2:172.21.15.59:6800/3326537211,v1:172.21.15.59:6801/3326537211] conn(0x7f50f80ac050 0x7f50f80ae470 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:06.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.264+0000 7f5114a00700 1 --2- 172.21.15.37:0/545017012 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f50f80a29a0 0x7f50f80a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f511019bb20 tx=0x7f5104009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:06.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.265+0000 7f510d600700 1 -- 172.21.15.37:0/545017012 <== osd.1 v2:172.21.15.59:6800/3326537211 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f50fc0053e0 con 0x7f50f80ac050 2024-03-28T18:09:06.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.286+0000 7f5117400700 1 -- 172.21.15.37:0/545017012 --> [v2:172.21.15.59:6800/3326537211,v1:172.21.15.59:6801/3326537211] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5110062d20 con 0x7f50f80ac050 2024-03-28T18:09:06.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.286+0000 7f510d600700 1 -- 172.21.15.37:0/545017012 <== osd.1 v2:172.21.15.59:6800/3326537211 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5110062d20 con 0x7f50f80ac050 2024-03-28T18:09:06.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.286+0000 7f5117400700 1 -- 172.21.15.37:0/545017012 >> [v2:172.21.15.59:6800/3326537211,v1:172.21.15.59:6801/3326537211] conn(0x7f50f80ac050 msgr2=0x7f50f80ae470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:06.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.286+0000 7f5117400700 1 --2- 172.21.15.37:0/545017012 >> [v2:172.21.15.59:6800/3326537211,v1:172.21.15.59:6801/3326537211] conn(0x7f50f80ac050 0x7f50f80ae470 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:06.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.287+0000 7f5117400700 1 -- 172.21.15.37:0/545017012 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f50f80a29a0 msgr2=0x7f50f80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:06.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.287+0000 7f5117400700 1 --2- 172.21.15.37:0/545017012 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f50f80a29a0 0x7f50f80a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f511019bb20 tx=0x7f5104009260 comp rx=0 tx=0).stop 2024-03-28T18:09:06.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.287+0000 7f5117400700 1 -- 172.21.15.37:0/545017012 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5110104fb0 msgr2=0x7f51101a0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:06.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.287+0000 7f5117400700 1 --2- 172.21.15.37:0/545017012 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5110104fb0 0x7f51101a0d60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f50fc0059d0 tx=0x7f50fc005590 comp rx=0 tx=0).stop 2024-03-28T18:09:06.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.287+0000 7f5115400700 1 -- 172.21.15.37:0/545017012 reap_dead start 2024-03-28T18:09:06.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.287+0000 7f5117400700 1 -- 172.21.15.37:0/545017012 shutdown_connections 2024-03-28T18:09:06.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.287+0000 7f5117400700 1 -- 172.21.15.37:0/545017012 >> 172.21.15.37:0/545017012 conn(0x7f51100ffe90 msgr2=0x7f5110100a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:06.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.287+0000 7f5117400700 1 -- 172.21.15.37:0/545017012 shutdown_connections 2024-03-28T18:09:06.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:06.287+0000 7f5117400700 1 -- 172.21.15.37:0/545017012 wait complete. 2024-03-28T18:09:07.252 INFO:teuthology.orchestra.run.smithi037.stdout:219043332142 2024-03-28T18:09:07.252 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-28T18:09:07.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:07 smithi037 ceph-mon[31460]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-28T18:09:08.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:07 smithi059 ceph-mon[36190]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-28T18:09:08.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:07 smithi170 ceph-mon[36280]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-28T18:09:08.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.063+0000 7f39c4a00700 1 -- 172.21.15.37:0/1739023756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39c0069e40 msgr2=0x7f39c006a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:08.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.063+0000 7f39c4a00700 1 --2- 172.21.15.37:0/1739023756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39c0069e40 0x7f39c006a2a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f39ac0099c0 tx=0x7f39ac009cd0 comp rx=0 tx=0).stop 2024-03-28T18:09:08.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.063+0000 7f39c4a00700 1 -- 172.21.15.37:0/1739023756 shutdown_connections 2024-03-28T18:09:08.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.063+0000 7f39c4a00700 1 --2- 172.21.15.37:0/1739023756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39c006a7e0 0x7f39c010f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:08.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.063+0000 7f39c4a00700 1 --2- 172.21.15.37:0/1739023756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39c01081e0 0x7f39c01085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:08.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.063+0000 7f39c4a00700 1 --2- 172.21.15.37:0/1739023756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39c0069e40 0x7f39c006a2a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:08.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.063+0000 7f39c4a00700 1 -- 172.21.15.37:0/1739023756 >> 172.21.15.37:0/1739023756 conn(0x7f39c0077fc0 msgr2=0x7f39c00783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:08.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.063+0000 7f39c4a00700 1 -- 172.21.15.37:0/1739023756 shutdown_connections 2024-03-28T18:09:08.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.063+0000 7f39c4a00700 1 -- 172.21.15.37:0/1739023756 wait complete. 2024-03-28T18:09:08.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.065+0000 7f39c4a00700 1 Processor -- start 2024-03-28T18:09:08.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.065+0000 7f39c4a00700 1 -- start start 2024-03-28T18:09:08.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.066+0000 7f39c4a00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39c0069e40 0x7f39c01a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:08.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.066+0000 7f39c4a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39c006a7e0 0x7f39c01a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:08.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.066+0000 7f39c4a00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39c01081e0 0x7f39c019a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:08.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.066+0000 7f39c4a00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c0112440 con 0x7f39c006a7e0 2024-03-28T18:09:08.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.066+0000 7f39c4a00700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c01122c0 con 0x7f39c01081e0 2024-03-28T18:09:08.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.066+0000 7f39c4a00700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c01125c0 con 0x7f39c0069e40 2024-03-28T18:09:08.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.066+0000 7f39bde00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39c0069e40 0x7f39c01a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:08.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.066+0000 7f39bde00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39c0069e40 0x7f39c01a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:60650/0 (socket says 172.21.15.37:60650) 2024-03-28T18:09:08.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.066+0000 7f39bde00700 1 -- 172.21.15.37:0/4034525503 learned_addr learned my addr 172.21.15.37:0/4034525503 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.066+0000 7f39be800700 1 --2- 172.21.15.37:0/4034525503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39c01081e0 0x7f39c019a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.066+0000 7f39bd400700 1 --2- 172.21.15.37:0/4034525503 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39c006a7e0 0x7f39c01a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.066+0000 7f39bde00700 1 -- 172.21.15.37:0/4034525503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39c01081e0 msgr2=0x7f39c019a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.066+0000 7f39bde00700 1 --2- 172.21.15.37:0/4034525503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39c01081e0 0x7f39c019a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.066+0000 7f39bde00700 1 -- 172.21.15.37:0/4034525503 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39c006a7e0 msgr2=0x7f39c01a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.066+0000 7f39bde00700 1 --2- 172.21.15.37:0/4034525503 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39c006a7e0 0x7f39c01a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.066+0000 7f39bde00700 1 -- 172.21.15.37:0/4034525503 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c019b0b0 con 0x7f39c0069e40 2024-03-28T18:09:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.066+0000 7f39bd400700 1 --2- 172.21.15.37:0/4034525503 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39c006a7e0 0x7f39c01a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:09:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.066+0000 7f39be800700 1 --2- 172.21.15.37:0/4034525503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39c01081e0 0x7f39c019a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:09:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.067+0000 7f39bde00700 1 --2- 172.21.15.37:0/4034525503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39c0069e40 0x7f39c01a0790 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f39b000e9a0 tx=0x7f39b000ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.067+0000 7f39b6a00700 1 -- 172.21.15.37:0/4034525503 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b000cbd0 con 0x7f39c0069e40 2024-03-28T18:09:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.067+0000 7f39b6a00700 1 -- 172.21.15.37:0/4034525503 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39b000cd70 con 0x7f39c0069e40 2024-03-28T18:09:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.067+0000 7f39c4a00700 1 -- 172.21.15.37:0/4034525503 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c019b3a0 con 0x7f39c0069e40 2024-03-28T18:09:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.067+0000 7f39b6a00700 1 -- 172.21.15.37:0/4034525503 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b00187c0 con 0x7f39c0069e40 2024-03-28T18:09:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.067+0000 7f39c4a00700 1 -- 172.21.15.37:0/4034525503 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c01a7660 con 0x7f39c0069e40 2024-03-28T18:09:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.068+0000 7f39c4a00700 1 -- 172.21.15.37:0/4034525503 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f39c010dd50 con 0x7f39c0069e40 2024-03-28T18:09:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.070+0000 7f39b6a00700 1 -- 172.21.15.37:0/4034525503 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f39b0010430 con 0x7f39c0069e40 2024-03-28T18:09:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.072+0000 7f39b6a00700 1 --2- 172.21.15.37:0/4034525503 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f39a40a29a0 0x7f39a40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.073+0000 7f39b6a00700 1 -- 172.21.15.37:0/4034525503 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f39b00ced20 con 0x7f39c0069e40 2024-03-28T18:09:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.073+0000 7f39bd400700 1 --2- 172.21.15.37:0/4034525503 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f39a40a29a0 0x7f39a40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.073+0000 7f39b6a00700 1 --2- 172.21.15.37:0/4034525503 >> [v2:172.21.15.170:6824/3082439360,v1:172.21.15.170:6825/3082439360] conn(0x7f39a40ac180 0x7f39a40ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.073+0000 7f39b6a00700 1 -- 172.21.15.37:0/4034525503 --> [v2:172.21.15.170:6824/3082439360,v1:172.21.15.170:6825/3082439360] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f39a40aec50 con 0x7f39a40ac180 2024-03-28T18:09:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.073+0000 7f39b6a00700 1 -- 172.21.15.37:0/4034525503 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39b00d4070 con 0x7f39c0069e40 2024-03-28T18:09:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.073+0000 7f39be800700 1 --2- 172.21.15.37:0/4034525503 >> [v2:172.21.15.170:6824/3082439360,v1:172.21.15.170:6825/3082439360] conn(0x7f39a40ac180 0x7f39a40ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.074+0000 7f39bd400700 1 --2- 172.21.15.37:0/4034525503 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f39a40a29a0 0x7f39a40a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f39c0103070 tx=0x7f39ac005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.074+0000 7f39be800700 1 --2- 172.21.15.37:0/4034525503 >> [v2:172.21.15.170:6824/3082439360,v1:172.21.15.170:6825/3082439360] conn(0x7f39a40ac180 0x7f39a40ae5a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.075+0000 7f39b6a00700 1 -- 172.21.15.37:0/4034525503 <== osd.11 v2:172.21.15.170:6824/3082439360 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f39a40aec50 con 0x7f39a40ac180 2024-03-28T18:09:08.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.109+0000 7f39c4a00700 1 -- 172.21.15.37:0/4034525503 --> [v2:172.21.15.170:6824/3082439360,v1:172.21.15.170:6825/3082439360] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f39c0103890 con 0x7f39a40ac180 2024-03-28T18:09:08.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.109+0000 7f39b6a00700 1 -- 172.21.15.37:0/4034525503 <== osd.11 v2:172.21.15.170:6824/3082439360 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f39c0103890 con 0x7f39a40ac180 2024-03-28T18:09:08.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.110+0000 7f39c4a00700 1 -- 172.21.15.37:0/4034525503 >> [v2:172.21.15.170:6824/3082439360,v1:172.21.15.170:6825/3082439360] conn(0x7f39a40ac180 msgr2=0x7f39a40ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:08.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.110+0000 7f39c4a00700 1 --2- 172.21.15.37:0/4034525503 >> [v2:172.21.15.170:6824/3082439360,v1:172.21.15.170:6825/3082439360] conn(0x7f39a40ac180 0x7f39a40ae5a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:08.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.110+0000 7f39c4a00700 1 -- 172.21.15.37:0/4034525503 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f39a40a29a0 msgr2=0x7f39a40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:08.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.110+0000 7f39c4a00700 1 --2- 172.21.15.37:0/4034525503 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f39a40a29a0 0x7f39a40a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f39c0103070 tx=0x7f39ac005fd0 comp rx=0 tx=0).stop 2024-03-28T18:09:08.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.110+0000 7f39c4a00700 1 -- 172.21.15.37:0/4034525503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39c0069e40 msgr2=0x7f39c01a0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:08.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.110+0000 7f39c4a00700 1 --2- 172.21.15.37:0/4034525503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39c0069e40 0x7f39c01a0790 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f39b000e9a0 tx=0x7f39b000ed60 comp rx=0 tx=0).stop 2024-03-28T18:09:08.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.110+0000 7f39be800700 1 -- 172.21.15.37:0/4034525503 reap_dead start 2024-03-28T18:09:08.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.110+0000 7f39c4a00700 1 -- 172.21.15.37:0/4034525503 shutdown_connections 2024-03-28T18:09:08.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.110+0000 7f39c4a00700 1 -- 172.21.15.37:0/4034525503 >> 172.21.15.37:0/4034525503 conn(0x7f39c0077fc0 msgr2=0x7f39c010d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:08.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.111+0000 7f39c4a00700 1 -- 172.21.15.37:0/4034525503 shutdown_connections 2024-03-28T18:09:08.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.111+0000 7f39c4a00700 1 -- 172.21.15.37:0/4034525503 wait complete. 2024-03-28T18:09:08.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.918+0000 7f75cb200700 1 -- 172.21.15.37:0/257136494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75c41081e0 msgr2=0x7f75c41085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:08.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.918+0000 7f75cb200700 1 --2- 172.21.15.37:0/257136494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75c41081e0 0x7f75c41085c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f75b400b0a0 tx=0x7f75b400b3b0 comp rx=0 tx=0).stop 2024-03-28T18:09:08.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.918+0000 7f75cb200700 1 -- 172.21.15.37:0/257136494 shutdown_connections 2024-03-28T18:09:08.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.918+0000 7f75cb200700 1 --2- 172.21.15.37:0/257136494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75c406a7e0 0x7f75c410f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:08.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.918+0000 7f75cb200700 1 --2- 172.21.15.37:0/257136494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75c41081e0 0x7f75c41085c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:08.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.918+0000 7f75cb200700 1 --2- 172.21.15.37:0/257136494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75c4069e40 0x7f75c406a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:08.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.918+0000 7f75cb200700 1 -- 172.21.15.37:0/257136494 >> 172.21.15.37:0/257136494 conn(0x7f75c4077fc0 msgr2=0x7f75c40783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:08.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.919+0000 7f75cb200700 1 -- 172.21.15.37:0/257136494 shutdown_connections 2024-03-28T18:09:08.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.919+0000 7f75cb200700 1 -- 172.21.15.37:0/257136494 wait complete. 2024-03-28T18:09:08.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.920+0000 7f75cb200700 1 Processor -- start 2024-03-28T18:09:08.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.920+0000 7f75cb200700 1 -- start start 2024-03-28T18:09:08.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.920+0000 7f75cb200700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75c4069e40 0x7f75c4101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:08.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.920+0000 7f75cb200700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75c406a7e0 0x7f75c41016a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:08.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.920+0000 7f75cb200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75c41081e0 0x7f75c4101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:08.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.920+0000 7f75cb200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c4112470 con 0x7f75c41081e0 2024-03-28T18:09:08.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.920+0000 7f75cb200700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c41122f0 con 0x7f75c406a7e0 2024-03-28T18:09:08.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.920+0000 7f75cb200700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c41125f0 con 0x7f75c4069e40 2024-03-28T18:09:08.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.921+0000 7f75c3e00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75c4069e40 0x7f75c4101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:08.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.921+0000 7f75c3400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75c406a7e0 0x7f75c41016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:08.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.921+0000 7f75c3e00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75c4069e40 0x7f75c4101160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:60634/0 (socket says 172.21.15.37:60634) 2024-03-28T18:09:08.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.921+0000 7f75c3e00700 1 -- 172.21.15.37:0/1329173446 learned_addr learned my addr 172.21.15.37:0/1329173446 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:08.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.921+0000 7f75c9200700 1 --2- 172.21.15.37:0/1329173446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75c41081e0 0x7f75c4101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:08.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.921+0000 7f75c3e00700 1 -- 172.21.15.37:0/1329173446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75c406a7e0 msgr2=0x7f75c41016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:08.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.921+0000 7f75c3e00700 1 --2- 172.21.15.37:0/1329173446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75c406a7e0 0x7f75c41016a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:08.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.921+0000 7f75c3e00700 1 -- 172.21.15.37:0/1329173446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75c41081e0 msgr2=0x7f75c4101be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:08.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.921+0000 7f75c3e00700 1 --2- 172.21.15.37:0/1329173446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75c41081e0 0x7f75c4101be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:08.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.921+0000 7f75c3e00700 1 -- 172.21.15.37:0/1329173446 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c41a6e10 con 0x7f75c4069e40 2024-03-28T18:09:08.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.921+0000 7f75c9200700 1 --2- 172.21.15.37:0/1329173446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75c41081e0 0x7f75c4101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:09:08.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.921+0000 7f75c3400700 1 --2- 172.21.15.37:0/1329173446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75c406a7e0 0x7f75c41016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:09:08.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.921+0000 7f75c3e00700 1 --2- 172.21.15.37:0/1329173446 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75c4069e40 0x7f75c4101160 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f75b40062a0 tx=0x7f75b4004ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:08.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.922+0000 7f75c0c00700 1 -- 172.21.15.37:0/1329173446 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b4021070 con 0x7f75c4069e40 2024-03-28T18:09:08.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.922+0000 7f75c0c00700 1 -- 172.21.15.37:0/1329173446 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75b4007b80 con 0x7f75c4069e40 2024-03-28T18:09:08.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.922+0000 7f75c0c00700 1 -- 172.21.15.37:0/1329173446 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b40052a0 con 0x7f75c4069e40 2024-03-28T18:09:08.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.922+0000 7f75cb200700 1 -- 172.21.15.37:0/1329173446 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c41a70a0 con 0x7f75c4069e40 2024-03-28T18:09:08.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.922+0000 7f75cb200700 1 -- 172.21.15.37:0/1329173446 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c41a75a0 con 0x7f75c4069e40 2024-03-28T18:09:08.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.923+0000 7f75cb200700 1 -- 172.21.15.37:0/1329173446 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f75c40628a0 con 0x7f75c4069e40 2024-03-28T18:09:08.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.925+0000 7f75c0c00700 1 -- 172.21.15.37:0/1329173446 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f75b4024070 con 0x7f75c4069e40 2024-03-28T18:09:08.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.926+0000 7f75c0c00700 1 --2- 172.21.15.37:0/1329173446 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f75a80a29a0 0x7f75a80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:08.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.926+0000 7f75c0c00700 1 -- 172.21.15.37:0/1329173446 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f75b400f020 con 0x7f75c4069e40 2024-03-28T18:09:08.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.926+0000 7f75c3400700 1 --2- 172.21.15.37:0/1329173446 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f75a80a29a0 0x7f75a80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:08.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.927+0000 7f75c0c00700 1 --2- 172.21.15.37:0/1329173446 >> [v2:172.21.15.170:6808/1218943899,v1:172.21.15.170:6809/1218943899] conn(0x7f75a80ac180 0x7f75a80ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:08.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.927+0000 7f75c0c00700 1 -- 172.21.15.37:0/1329173446 --> [v2:172.21.15.170:6808/1218943899,v1:172.21.15.170:6809/1218943899] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f75b4020ec0 con 0x7f75a80ac180 2024-03-28T18:09:08.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.927+0000 7f75c0c00700 1 -- 172.21.15.37:0/1329173446 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75b40d2490 con 0x7f75c4069e40 2024-03-28T18:09:08.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.927+0000 7f75c9200700 1 --2- 172.21.15.37:0/1329173446 >> [v2:172.21.15.170:6808/1218943899,v1:172.21.15.170:6809/1218943899] conn(0x7f75a80ac180 0x7f75a80ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:08.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.927+0000 7f75c3400700 1 --2- 172.21.15.37:0/1329173446 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f75a80a29a0 0x7f75a80a4e60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f75c4078550 tx=0x7f75b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:08.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.928+0000 7f75c9200700 1 --2- 172.21.15.37:0/1329173446 >> [v2:172.21.15.170:6808/1218943899,v1:172.21.15.170:6809/1218943899] conn(0x7f75a80ac180 0x7f75a80ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:08.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.928+0000 7f75c0c00700 1 -- 172.21.15.37:0/1329173446 <== osd.5 v2:172.21.15.170:6808/1218943899 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f75b4020ec0 con 0x7f75a80ac180 2024-03-28T18:09:08.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.954+0000 7f75cb200700 1 -- 172.21.15.37:0/1329173446 --> [v2:172.21.15.170:6808/1218943899,v1:172.21.15.170:6809/1218943899] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f75c4062d20 con 0x7f75a80ac180 2024-03-28T18:09:08.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.955+0000 7f75c0c00700 1 -- 172.21.15.37:0/1329173446 <== osd.5 v2:172.21.15.170:6808/1218943899 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f75c4062d20 con 0x7f75a80ac180 2024-03-28T18:09:08.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.955+0000 7f75cb200700 1 -- 172.21.15.37:0/1329173446 >> [v2:172.21.15.170:6808/1218943899,v1:172.21.15.170:6809/1218943899] conn(0x7f75a80ac180 msgr2=0x7f75a80ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:08.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.955+0000 7f75cb200700 1 --2- 172.21.15.37:0/1329173446 >> [v2:172.21.15.170:6808/1218943899,v1:172.21.15.170:6809/1218943899] conn(0x7f75a80ac180 0x7f75a80ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:08.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.956+0000 7f75cb200700 1 -- 172.21.15.37:0/1329173446 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f75a80a29a0 msgr2=0x7f75a80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:08.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.956+0000 7f75cb200700 1 --2- 172.21.15.37:0/1329173446 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f75a80a29a0 0x7f75a80a4e60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f75c4078550 tx=0x7f75b0009260 comp rx=0 tx=0).stop 2024-03-28T18:09:08.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.956+0000 7f75cb200700 1 -- 172.21.15.37:0/1329173446 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75c4069e40 msgr2=0x7f75c4101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:08.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.956+0000 7f75cb200700 1 --2- 172.21.15.37:0/1329173446 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75c4069e40 0x7f75c4101160 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f75b40062a0 tx=0x7f75b4004ce0 comp rx=0 tx=0).stop 2024-03-28T18:09:08.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.956+0000 7f75c9200700 1 -- 172.21.15.37:0/1329173446 reap_dead start 2024-03-28T18:09:08.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.956+0000 7f75cb200700 1 -- 172.21.15.37:0/1329173446 shutdown_connections 2024-03-28T18:09:08.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.956+0000 7f75cb200700 1 -- 172.21.15.37:0/1329173446 >> 172.21.15.37:0/1329173446 conn(0x7f75c4077fc0 msgr2=0x7f75c40ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:08.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.956+0000 7f75cb200700 1 -- 172.21.15.37:0/1329173446 shutdown_connections 2024-03-28T18:09:08.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:07.956+0000 7f75cb200700 1 -- 172.21.15.37:0/1329173446 wait complete. 2024-03-28T18:09:09.035 INFO:teuthology.orchestra.run.smithi037.stdout:253403070503 2024-03-28T18:09:09.035 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-28T18:09:09.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.441+0000 7fb6d8800700 1 -- 172.21.15.37:0/1156712840 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6d01087a0 msgr2=0x7fb6d0108c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:09.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.441+0000 7fb6d8800700 1 --2- 172.21.15.37:0/1156712840 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6d01087a0 0x7fb6d0108c20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0009990 tx=0x7fb6c0009ca0 comp rx=0 tx=0).stop 2024-03-28T18:09:09.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.442+0000 7fb6d8800700 1 -- 172.21.15.37:0/1156712840 shutdown_connections 2024-03-28T18:09:09.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.442+0000 7fb6d8800700 1 --2- 172.21.15.37:0/1156712840 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6d0109160 0x7fb6d0115670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:09.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.442+0000 7fb6d8800700 1 --2- 172.21.15.37:0/1156712840 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6d010e7a0 0x7fb6d010eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:09.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.442+0000 7fb6d8800700 1 --2- 172.21.15.37:0/1156712840 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6d01087a0 0x7fb6d0108c20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:09.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.442+0000 7fb6d8800700 1 -- 172.21.15.37:0/1156712840 >> 172.21.15.37:0/1156712840 conn(0x7fb6d0075960 msgr2=0x7fb6d0077d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:09.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.442+0000 7fb6d8800700 1 -- 172.21.15.37:0/1156712840 shutdown_connections 2024-03-28T18:09:09.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.442+0000 7fb6d8800700 1 -- 172.21.15.37:0/1156712840 wait complete. 2024-03-28T18:09:09.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.443+0000 7fb6d8800700 1 Processor -- start 2024-03-28T18:09:09.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.443+0000 7fb6d8800700 1 -- start start 2024-03-28T18:09:09.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.443+0000 7fb6d8800700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6d01087a0 0x7fb6d019ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:09.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.443+0000 7fb6d8800700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6d0109160 0x7fb6d019f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:09.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.443+0000 7fb6d8800700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6d010e7a0 0x7fb6d01a3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:09.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.443+0000 7fb6d8800700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d0118b80 con 0x7fb6d01087a0 2024-03-28T18:09:09.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.443+0000 7fb6d8800700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d0118a00 con 0x7fb6d010e7a0 2024-03-28T18:09:09.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.443+0000 7fb6d8800700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d0118d00 con 0x7fb6d0109160 2024-03-28T18:09:09.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.444+0000 7fb6d5e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6d01087a0 0x7fb6d019ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:09.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.444+0000 7fb6d5400700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6d0109160 0x7fb6d019f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:09.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.444+0000 7fb6d5e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6d01087a0 0x7fb6d019ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41566/0 (socket says 172.21.15.37:41566) 2024-03-28T18:09:09.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.444+0000 7fb6d5e00700 1 -- 172.21.15.37:0/1763625426 learned_addr learned my addr 172.21.15.37:0/1763625426 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:09.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.444+0000 7fb6d6800700 1 --2- 172.21.15.37:0/1763625426 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6d010e7a0 0x7fb6d01a3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:09.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.444+0000 7fb6d5400700 1 -- 172.21.15.37:0/1763625426 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6d010e7a0 msgr2=0x7fb6d01a3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:09.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.444+0000 7fb6d5400700 1 --2- 172.21.15.37:0/1763625426 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6d010e7a0 0x7fb6d01a3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:09.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.444+0000 7fb6d5400700 1 -- 172.21.15.37:0/1763625426 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6d01087a0 msgr2=0x7fb6d019ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:09.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.444+0000 7fb6d5400700 1 --2- 172.21.15.37:0/1763625426 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6d01087a0 0x7fb6d019ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:09.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.444+0000 7fb6d5400700 1 -- 172.21.15.37:0/1763625426 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d01a3f60 con 0x7fb6d0109160 2024-03-28T18:09:09.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.444+0000 7fb6d5400700 1 --2- 172.21.15.37:0/1763625426 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6d0109160 0x7fb6d019f490 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0006010 tx=0x7fb6c000bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:09.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.445+0000 7fb6caa00700 1 -- 172.21.15.37:0/1763625426 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c0027070 con 0x7fb6d0109160 2024-03-28T18:09:09.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.445+0000 7fb6caa00700 1 -- 172.21.15.37:0/1763625426 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6c0003d70 con 0x7fb6d0109160 2024-03-28T18:09:09.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.445+0000 7fb6d8800700 1 -- 172.21.15.37:0/1763625426 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6d01a41f0 con 0x7fb6d0109160 2024-03-28T18:09:09.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.445+0000 7fb6caa00700 1 -- 172.21.15.37:0/1763625426 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c000f460 con 0x7fb6d0109160 2024-03-28T18:09:09.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.445+0000 7fb6d8800700 1 -- 172.21.15.37:0/1763625426 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d01abad0 con 0x7fb6d0109160 2024-03-28T18:09:09.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.446+0000 7fb6d8800700 1 -- 172.21.15.37:0/1763625426 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb6d0062920 con 0x7fb6d0109160 2024-03-28T18:09:09.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.448+0000 7fb6caa00700 1 -- 172.21.15.37:0/1763625426 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fb6c002a070 con 0x7fb6d0109160 2024-03-28T18:09:09.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.449+0000 7fb6caa00700 1 --2- 172.21.15.37:0/1763625426 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb6b80a29a0 0x7fb6b80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:09.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.449+0000 7fb6caa00700 1 -- 172.21.15.37:0/1763625426 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb6c00cfa20 con 0x7fb6d0109160 2024-03-28T18:09:09.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.449+0000 7fb6d5e00700 1 --2- 172.21.15.37:0/1763625426 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb6b80a29a0 0x7fb6b80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:09.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.449+0000 7fb6caa00700 1 --2- 172.21.15.37:0/1763625426 >> [v2:172.21.15.170:6800/1269398427,v1:172.21.15.170:6801/1269398427] conn(0x7fb6b80ac180 0x7fb6b80ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:09.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.450+0000 7fb6caa00700 1 -- 172.21.15.37:0/1763625426 --> [v2:172.21.15.170:6800/1269398427,v1:172.21.15.170:6801/1269398427] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb6b80aec50 con 0x7fb6b80ac180 2024-03-28T18:09:09.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.450+0000 7fb6caa00700 1 -- 172.21.15.37:0/1763625426 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6c00d42a0 con 0x7fb6d0109160 2024-03-28T18:09:09.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.450+0000 7fb6d6800700 1 --2- 172.21.15.37:0/1763625426 >> [v2:172.21.15.170:6800/1269398427,v1:172.21.15.170:6801/1269398427] conn(0x7fb6b80ac180 0x7fb6b80ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:09.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.450+0000 7fb6d5e00700 1 --2- 172.21.15.37:0/1763625426 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb6b80a29a0 0x7fb6b80a4e60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4009ca0 tx=0x7fb6c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:09.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.450+0000 7fb6d6800700 1 --2- 172.21.15.37:0/1763625426 >> [v2:172.21.15.170:6800/1269398427,v1:172.21.15.170:6801/1269398427] conn(0x7fb6b80ac180 0x7fb6b80ae5a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:09.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.451+0000 7fb6caa00700 1 -- 172.21.15.37:0/1763625426 <== osd.2 v2:172.21.15.170:6800/1269398427 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb6b80aec50 con 0x7fb6b80ac180 2024-03-28T18:09:09.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.475+0000 7fb6d8800700 1 -- 172.21.15.37:0/1763625426 --> [v2:172.21.15.170:6800/1269398427,v1:172.21.15.170:6801/1269398427] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb6d0062d20 con 0x7fb6b80ac180 2024-03-28T18:09:09.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.475+0000 7fb6caa00700 1 -- 172.21.15.37:0/1763625426 <== osd.2 v2:172.21.15.170:6800/1269398427 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb6d0062d20 con 0x7fb6b80ac180 2024-03-28T18:09:09.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.476+0000 7fb6d8800700 1 -- 172.21.15.37:0/1763625426 >> [v2:172.21.15.170:6800/1269398427,v1:172.21.15.170:6801/1269398427] conn(0x7fb6b80ac180 msgr2=0x7fb6b80ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:09.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.476+0000 7fb6d8800700 1 --2- 172.21.15.37:0/1763625426 >> [v2:172.21.15.170:6800/1269398427,v1:172.21.15.170:6801/1269398427] conn(0x7fb6b80ac180 0x7fb6b80ae5a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:09.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.476+0000 7fb6d8800700 1 -- 172.21.15.37:0/1763625426 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb6b80a29a0 msgr2=0x7fb6b80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:09.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.476+0000 7fb6d8800700 1 --2- 172.21.15.37:0/1763625426 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb6b80a29a0 0x7fb6b80a4e60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4009ca0 tx=0x7fb6c4009260 comp rx=0 tx=0).stop 2024-03-28T18:09:09.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.476+0000 7fb6d8800700 1 -- 172.21.15.37:0/1763625426 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6d0109160 msgr2=0x7fb6d019f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:09.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.476+0000 7fb6d8800700 1 --2- 172.21.15.37:0/1763625426 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6d0109160 0x7fb6d019f490 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0006010 tx=0x7fb6c000bab0 comp rx=0 tx=0).stop 2024-03-28T18:09:09.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.476+0000 7fb6d6800700 1 -- 172.21.15.37:0/1763625426 reap_dead start 2024-03-28T18:09:09.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.476+0000 7fb6d8800700 1 -- 172.21.15.37:0/1763625426 shutdown_connections 2024-03-28T18:09:09.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.476+0000 7fb6d8800700 1 -- 172.21.15.37:0/1763625426 >> 172.21.15.37:0/1763625426 conn(0x7fb6d0075960 msgr2=0x7fb6d0075ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:09.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.476+0000 7fb6d8800700 1 -- 172.21.15.37:0/1763625426 shutdown_connections 2024-03-28T18:09:09.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:08.476+0000 7fb6d8800700 1 -- 172.21.15.37:0/1763625426 wait complete. 2024-03-28T18:09:10.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:09 smithi059 ceph-mon[36190]: pgmap v342: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T18:09:10.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:09 smithi170 ceph-mon[36280]: pgmap v342: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T18:09:10.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:09 smithi037 ceph-mon[31460]: pgmap v342: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T18:09:11.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:11 smithi037 ceph-mon[31460]: pgmap v343: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T18:09:11.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.830+0000 7fa1a2400700 1 -- 172.21.15.37:0/3064373908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa19c102440 msgr2=0x7fa19c102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:11.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.830+0000 7fa1a2400700 1 --2- 172.21.15.37:0/3064373908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa19c102440 0x7fa19c102820 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa18c009990 tx=0x7fa18c009ca0 comp rx=0 tx=0).stop 2024-03-28T18:09:11.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.830+0000 7fa1a2400700 1 -- 172.21.15.37:0/3064373908 shutdown_connections 2024-03-28T18:09:11.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.830+0000 7fa1a2400700 1 --2- 172.21.15.37:0/3064373908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa19c107510 0x7fa19c10f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:11.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.830+0000 7fa1a2400700 1 --2- 172.21.15.37:0/3064373908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa19c102440 0x7fa19c102820 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:11.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.830+0000 7fa1a2400700 1 --2- 172.21.15.37:0/3064373908 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa19c102df0 0x7fa19c106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:11.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.830+0000 7fa1a2400700 1 -- 172.21.15.37:0/3064373908 >> 172.21.15.37:0/3064373908 conn(0x7fa19c0792f0 msgr2=0x7fa19c079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:11.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.830+0000 7fa1a2400700 1 -- 172.21.15.37:0/3064373908 shutdown_connections 2024-03-28T18:09:11.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.830+0000 7fa1a2400700 1 -- 172.21.15.37:0/3064373908 wait complete. 2024-03-28T18:09:11.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.831+0000 7fa1a2400700 1 Processor -- start 2024-03-28T18:09:11.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.831+0000 7fa1a2400700 1 -- start start 2024-03-28T18:09:11.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.832+0000 7fa1a2400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa19c102440 0x7fa19c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:11.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.832+0000 7fa1a2400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa19c102df0 0x7fa19c19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:11.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.832+0000 7fa1a2400700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa19c107510 0x7fa19c1988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:11.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.832+0000 7fa1a2400700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa19c112540 con 0x7fa19c102df0 2024-03-28T18:09:11.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.832+0000 7fa1a2400700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa19c1123c0 con 0x7fa19c102440 2024-03-28T18:09:11.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.832+0000 7fa1a2400700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa19c1126c0 con 0x7fa19c107510 2024-03-28T18:09:11.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.832+0000 7fa19b400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa19c102440 0x7fa19c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:11.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.832+0000 7fa19b400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa19c102440 0x7fa19c19e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:56858/0 (socket says 172.21.15.37:56858) 2024-03-28T18:09:11.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.832+0000 7fa19b400700 1 -- 172.21.15.37:0/216799011 learned_addr learned my addr 172.21.15.37:0/216799011 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:11.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.832+0000 7fa19be00700 1 --2- 172.21.15.37:0/216799011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa19c107510 0x7fa19c1988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:11.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.832+0000 7fa19aa00700 1 --2- 172.21.15.37:0/216799011 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa19c102df0 0x7fa19c19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:11.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.833+0000 7fa19be00700 1 -- 172.21.15.37:0/216799011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa19c102440 msgr2=0x7fa19c19e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:11.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.833+0000 7fa19be00700 1 --2- 172.21.15.37:0/216799011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa19c102440 0x7fa19c19e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:11.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.833+0000 7fa19be00700 1 -- 172.21.15.37:0/216799011 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa19c102df0 msgr2=0x7fa19c19ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:11.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.833+0000 7fa19be00700 1 --2- 172.21.15.37:0/216799011 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa19c102df0 0x7fa19c19ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:11.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.833+0000 7fa19be00700 1 -- 172.21.15.37:0/216799011 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19c199080 con 0x7fa19c107510 2024-03-28T18:09:11.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.833+0000 7fa19b400700 1 --2- 172.21.15.37:0/216799011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa19c102440 0x7fa19c19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:09:11.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.833+0000 7fa19be00700 1 --2- 172.21.15.37:0/216799011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa19c107510 0x7fa19c1988a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa19400efc0 tx=0x7fa19400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:11.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.833+0000 7fa193e00700 1 -- 172.21.15.37:0/216799011 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1940092e0 con 0x7fa19c107510 2024-03-28T18:09:11.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.833+0000 7fa193e00700 1 -- 172.21.15.37:0/216799011 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa194004bb0 con 0x7fa19c107510 2024-03-28T18:09:11.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.833+0000 7fa1a2400700 1 -- 172.21.15.37:0/216799011 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa19c199310 con 0x7fa19c107510 2024-03-28T18:09:11.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.833+0000 7fa193e00700 1 -- 172.21.15.37:0/216799011 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa194004280 con 0x7fa19c107510 2024-03-28T18:09:11.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.833+0000 7fa1a2400700 1 -- 172.21.15.37:0/216799011 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa19c1a5530 con 0x7fa19c107510 2024-03-28T18:09:11.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.835+0000 7fa191600700 1 -- 172.21.15.37:0/216799011 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa19c0628a0 con 0x7fa19c107510 2024-03-28T18:09:11.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.837+0000 7fa193e00700 1 -- 172.21.15.37:0/216799011 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fa19401b030 con 0x7fa19c107510 2024-03-28T18:09:11.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.838+0000 7fa193e00700 1 --2- 172.21.15.37:0/216799011 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa1800a29a0 0x7fa1800a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:11.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.838+0000 7fa193e00700 1 -- 172.21.15.37:0/216799011 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa1940ce240 con 0x7fa19c107510 2024-03-28T18:09:11.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.838+0000 7fa19b400700 1 --2- 172.21.15.37:0/216799011 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa1800a29a0 0x7fa1800a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:11.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.839+0000 7fa193e00700 1 --2- 172.21.15.37:0/216799011 >> [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046] conn(0x7fa1800ac180 0x7fa1800ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:11.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.839+0000 7fa193e00700 1 -- 172.21.15.37:0/216799011 --> [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa19400ce50 con 0x7fa1800ac180 2024-03-28T18:09:11.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.839+0000 7fa193e00700 1 -- 172.21.15.37:0/216799011 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1940d5050 con 0x7fa19c107510 2024-03-28T18:09:11.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.839+0000 7fa19b400700 1 --2- 172.21.15.37:0/216799011 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa1800a29a0 0x7fa1800a4e60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa18c009750 tx=0x7fa18c00b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:11.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.839+0000 7fa19aa00700 1 --2- 172.21.15.37:0/216799011 >> [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046] conn(0x7fa1800ac180 0x7fa1800ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:11.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.840+0000 7fa19aa00700 1 --2- 172.21.15.37:0/216799011 >> [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046] conn(0x7fa1800ac180 0x7fa1800ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:11.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.841+0000 7fa193e00700 1 -- 172.21.15.37:0/216799011 <== osd.6 v2:172.21.15.37:6818/1899652046 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa19400ce50 con 0x7fa1800ac180 2024-03-28T18:09:11.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.859+0000 7fa191600700 1 -- 172.21.15.37:0/216799011 --> [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa19c062ca0 con 0x7fa1800ac180 2024-03-28T18:09:11.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.860+0000 7fa193e00700 1 -- 172.21.15.37:0/216799011 <== osd.6 v2:172.21.15.37:6818/1899652046 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa19c062ca0 con 0x7fa1800ac180 2024-03-28T18:09:11.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.860+0000 7fa191600700 1 -- 172.21.15.37:0/216799011 >> [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046] conn(0x7fa1800ac180 msgr2=0x7fa1800ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:11.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.860+0000 7fa191600700 1 --2- 172.21.15.37:0/216799011 >> [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046] conn(0x7fa1800ac180 0x7fa1800ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:11.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.861+0000 7fa191600700 1 -- 172.21.15.37:0/216799011 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa1800a29a0 msgr2=0x7fa1800a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:11.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.861+0000 7fa191600700 1 --2- 172.21.15.37:0/216799011 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa1800a29a0 0x7fa1800a4e60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa18c009750 tx=0x7fa18c00b280 comp rx=0 tx=0).stop 2024-03-28T18:09:11.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.861+0000 7fa191600700 1 -- 172.21.15.37:0/216799011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa19c107510 msgr2=0x7fa19c1988a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:11.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.861+0000 7fa191600700 1 --2- 172.21.15.37:0/216799011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa19c107510 0x7fa19c1988a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa19400efc0 tx=0x7fa19400c3a0 comp rx=0 tx=0).stop 2024-03-28T18:09:11.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.861+0000 7fa19be00700 1 -- 172.21.15.37:0/216799011 reap_dead start 2024-03-28T18:09:11.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.861+0000 7fa191600700 1 -- 172.21.15.37:0/216799011 shutdown_connections 2024-03-28T18:09:11.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.861+0000 7fa191600700 1 -- 172.21.15.37:0/216799011 >> 172.21.15.37:0/216799011 conn(0x7fa19c0792f0 msgr2=0x7fa19c1061f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:11.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.861+0000 7fa191600700 1 -- 172.21.15.37:0/216799011 shutdown_connections 2024-03-28T18:09:11.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.861+0000 7fa191600700 1 -- 172.21.15.37:0/216799011 wait complete. 2024-03-28T18:09:12.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:11 smithi059 ceph-mon[36190]: pgmap v343: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T18:09:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:11 smithi170 ceph-mon[36280]: pgmap v343: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T18:09:12.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.950+0000 7f754de00700 1 -- 172.21.15.37:0/912329809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75481022e0 msgr2=0x7f75481026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:12.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.950+0000 7f754de00700 1 --2- 172.21.15.37:0/912329809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75481022e0 0x7f75481026c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f753c009990 tx=0x7f753c009ca0 comp rx=0 tx=0).stop 2024-03-28T18:09:12.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.950+0000 7f754de00700 1 -- 172.21.15.37:0/912329809 shutdown_connections 2024-03-28T18:09:12.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.950+0000 7f754de00700 1 --2- 172.21.15.37:0/912329809 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75481073c0 0x7f754810ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:12.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.950+0000 7f754de00700 1 --2- 172.21.15.37:0/912329809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75481022e0 0x7f75481026c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:12.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.950+0000 7f754de00700 1 --2- 172.21.15.37:0/912329809 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7548102c90 0x7f7548106ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:12.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.950+0000 7f754de00700 1 -- 172.21.15.37:0/912329809 >> 172.21.15.37:0/912329809 conn(0x7f75480fdb90 msgr2=0x7f75480fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:12.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.950+0000 7f754de00700 1 -- 172.21.15.37:0/912329809 shutdown_connections 2024-03-28T18:09:12.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.951+0000 7f754de00700 1 -- 172.21.15.37:0/912329809 wait complete. 2024-03-28T18:09:12.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.952+0000 7f754de00700 1 Processor -- start 2024-03-28T18:09:12.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.952+0000 7f754de00700 1 -- start start 2024-03-28T18:09:12.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.952+0000 7f754de00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75481022e0 0x7f75481a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:12.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.952+0000 7f754de00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7548102c90 0x7f75481a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:12.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.952+0000 7f754de00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75481073c0 0x7f754819a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:12.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.952+0000 7f754de00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75481123f0 con 0x7f75481022e0 2024-03-28T18:09:12.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.952+0000 7f754de00700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7548112270 con 0x7f75481073c0 2024-03-28T18:09:12.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.952+0000 7f754de00700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7548112570 con 0x7f7548102c90 2024-03-28T18:09:12.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.952+0000 7f754d400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75481073c0 0x7f754819a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:12.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.953+0000 7f7547e00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7548102c90 0x7f75481a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:12.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.953+0000 7f754ca00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75481022e0 0x7f75481a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:12.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.953+0000 7f754d400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75481073c0 0x7f754819a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:56866/0 (socket says 172.21.15.37:56866) 2024-03-28T18:09:12.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.953+0000 7f7547e00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7548102c90 0x7f75481a0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:60684/0 (socket says 172.21.15.37:60684) 2024-03-28T18:09:12.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.953+0000 7f754d400700 1 -- 172.21.15.37:0/1645761634 learned_addr learned my addr 172.21.15.37:0/1645761634 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:12.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.953+0000 7f7547e00700 1 -- 172.21.15.37:0/1645761634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75481073c0 msgr2=0x7f754819a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:12.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.953+0000 7f7547e00700 1 --2- 172.21.15.37:0/1645761634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75481073c0 0x7f754819a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:12.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.953+0000 7f7547e00700 1 -- 172.21.15.37:0/1645761634 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75481022e0 msgr2=0x7f75481a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:12.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.953+0000 7f7547e00700 1 --2- 172.21.15.37:0/1645761634 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75481022e0 0x7f75481a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:12.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.953+0000 7f7547e00700 1 -- 172.21.15.37:0/1645761634 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754819afc0 con 0x7f7548102c90 2024-03-28T18:09:12.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.953+0000 7f754d400700 1 --2- 172.21.15.37:0/1645761634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75481073c0 0x7f754819a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:09:12.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.953+0000 7f7547e00700 1 --2- 172.21.15.37:0/1645761634 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7548102c90 0x7f75481a0be0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f753400e9c0 tx=0x7f753400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:12.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.953+0000 7f7545600700 1 -- 172.21.15.37:0/1645761634 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753400cb70 con 0x7f7548102c90 2024-03-28T18:09:12.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.954+0000 7f7545600700 1 -- 172.21.15.37:0/1645761634 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7534004500 con 0x7f7548102c90 2024-03-28T18:09:12.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.954+0000 7f754de00700 1 -- 172.21.15.37:0/1645761634 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f754819b2b0 con 0x7f7548102c90 2024-03-28T18:09:12.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.954+0000 7f7545600700 1 -- 172.21.15.37:0/1645761634 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7534003de0 con 0x7f7548102c90 2024-03-28T18:09:12.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.954+0000 7f754de00700 1 -- 172.21.15.37:0/1645761634 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75481a7570 con 0x7f7548102c90 2024-03-28T18:09:12.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.954+0000 7f754de00700 1 -- 172.21.15.37:0/1645761634 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f754810ecc0 con 0x7f7548102c90 2024-03-28T18:09:12.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.959+0000 7f7545600700 1 -- 172.21.15.37:0/1645761634 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f7534003fc0 con 0x7f7548102c90 2024-03-28T18:09:12.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.960+0000 7f7545600700 1 --2- 172.21.15.37:0/1645761634 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f75300a29a0 0x7f75300a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:12.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.960+0000 7f7545600700 1 -- 172.21.15.37:0/1645761634 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f75340ced40 con 0x7f7548102c90 2024-03-28T18:09:12.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.960+0000 7f754ca00700 1 --2- 172.21.15.37:0/1645761634 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f75300a29a0 0x7f75300a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:12.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.960+0000 7f7545600700 1 --2- 172.21.15.37:0/1645761634 >> [v2:172.21.15.59:6824/2113830275,v1:172.21.15.59:6825/2113830275] conn(0x7f75300ac180 0x7f75300ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:12.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.960+0000 7f7545600700 1 -- 172.21.15.37:0/1645761634 --> [v2:172.21.15.59:6824/2113830275,v1:172.21.15.59:6825/2113830275] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f75300aec50 con 0x7f75300ac180 2024-03-28T18:09:12.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.960+0000 7f7545600700 1 -- 172.21.15.37:0/1645761634 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75340cf0f0 con 0x7f7548102c90 2024-03-28T18:09:12.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.961+0000 7f754d400700 1 --2- 172.21.15.37:0/1645761634 >> [v2:172.21.15.59:6824/2113830275,v1:172.21.15.59:6825/2113830275] conn(0x7f75300ac180 0x7f75300ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:12.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.961+0000 7f754ca00700 1 --2- 172.21.15.37:0/1645761634 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f75300a29a0 0x7f75300a4e60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f754819b9a0 tx=0x7f753c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:12.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.961+0000 7f754d400700 1 --2- 172.21.15.37:0/1645761634 >> [v2:172.21.15.59:6824/2113830275,v1:172.21.15.59:6825/2113830275] conn(0x7f75300ac180 0x7f75300ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:12.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.962+0000 7f7545600700 1 -- 172.21.15.37:0/1645761634 <== osd.10 v2:172.21.15.59:6824/2113830275 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f75300aec50 con 0x7f75300ac180 2024-03-28T18:09:12.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.981+0000 7f754de00700 1 -- 172.21.15.37:0/1645761634 --> [v2:172.21.15.59:6824/2113830275,v1:172.21.15.59:6825/2113830275] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f75481037c0 con 0x7f75300ac180 2024-03-28T18:09:12.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.981+0000 7f7545600700 1 -- 172.21.15.37:0/1645761634 <== osd.10 v2:172.21.15.59:6824/2113830275 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f75481037c0 con 0x7f75300ac180 2024-03-28T18:09:12.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.981+0000 7f754de00700 1 -- 172.21.15.37:0/1645761634 >> [v2:172.21.15.59:6824/2113830275,v1:172.21.15.59:6825/2113830275] conn(0x7f75300ac180 msgr2=0x7f75300ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:12.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.981+0000 7f754de00700 1 --2- 172.21.15.37:0/1645761634 >> [v2:172.21.15.59:6824/2113830275,v1:172.21.15.59:6825/2113830275] conn(0x7f75300ac180 0x7f75300ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:12.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.982+0000 7f754de00700 1 -- 172.21.15.37:0/1645761634 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f75300a29a0 msgr2=0x7f75300a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:12.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.982+0000 7f754de00700 1 --2- 172.21.15.37:0/1645761634 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f75300a29a0 0x7f75300a4e60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f754819b9a0 tx=0x7f753c01a040 comp rx=0 tx=0).stop 2024-03-28T18:09:12.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.982+0000 7f754de00700 1 -- 172.21.15.37:0/1645761634 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7548102c90 msgr2=0x7f75481a0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:12.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.982+0000 7f754de00700 1 --2- 172.21.15.37:0/1645761634 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7548102c90 0x7f75481a0be0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f753400e9c0 tx=0x7f753400ecd0 comp rx=0 tx=0).stop 2024-03-28T18:09:12.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.982+0000 7f754d400700 1 -- 172.21.15.37:0/1645761634 reap_dead start 2024-03-28T18:09:12.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.982+0000 7f754de00700 1 -- 172.21.15.37:0/1645761634 shutdown_connections 2024-03-28T18:09:12.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.982+0000 7f754de00700 1 -- 172.21.15.37:0/1645761634 >> 172.21.15.37:0/1645761634 conn(0x7f75480fdb90 msgr2=0x7f754810e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:12.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.982+0000 7f754de00700 1 -- 172.21.15.37:0/1645761634 shutdown_connections 2024-03-28T18:09:12.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:11.982+0000 7f754de00700 1 -- 172.21.15.37:0/1645761634 wait complete. 2024-03-28T18:09:13.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.993+0000 7fd1e8000700 1 -- 172.21.15.37:0/2141986121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1e0102410 msgr2=0x7fd1e01027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:13.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.993+0000 7fd1e8000700 1 --2- 172.21.15.37:0/2141986121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1e0102410 0x7fd1e01027f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd1d400b0d0 tx=0x7fd1d400b3e0 comp rx=0 tx=0).stop 2024-03-28T18:09:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.993+0000 7fd1e8000700 1 -- 172.21.15.37:0/2141986121 shutdown_connections 2024-03-28T18:09:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.993+0000 7fd1e8000700 1 --2- 172.21.15.37:0/2141986121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1e01074f0 0x7fd1e010f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.993+0000 7fd1e8000700 1 --2- 172.21.15.37:0/2141986121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1e0102410 0x7fd1e01027f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.993+0000 7fd1e8000700 1 --2- 172.21.15.37:0/2141986121 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1e0102dc0 0x7fd1e0106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.993+0000 7fd1e8000700 1 -- 172.21.15.37:0/2141986121 >> 172.21.15.37:0/2141986121 conn(0x7fd1e00fdc60 msgr2=0x7fd1e0100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.993+0000 7fd1e8000700 1 -- 172.21.15.37:0/2141986121 shutdown_connections 2024-03-28T18:09:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.994+0000 7fd1e8000700 1 -- 172.21.15.37:0/2141986121 wait complete. 2024-03-28T18:09:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.995+0000 7fd1e8000700 1 Processor -- start 2024-03-28T18:09:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.995+0000 7fd1e8000700 1 -- start start 2024-03-28T18:09:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.995+0000 7fd1e8000700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1e0102410 0x7fd1e01a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.995+0000 7fd1e8000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1e0102dc0 0x7fd1e01a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.995+0000 7fd1e8000700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1e01074f0 0x7fd1e019a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.995+0000 7fd1e8000700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e0112520 con 0x7fd1e0102dc0 2024-03-28T18:09:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.995+0000 7fd1e8000700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e01123a0 con 0x7fd1e0102410 2024-03-28T18:09:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.995+0000 7fd1e8000700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e01126a0 con 0x7fd1e01074f0 2024-03-28T18:09:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.996+0000 7fd1e5600700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1e0102410 0x7fd1e01a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.996+0000 7fd1e5600700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1e0102410 0x7fd1e01a07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:56890/0 (socket says 172.21.15.37:56890) 2024-03-28T18:09:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.996+0000 7fd1e4c00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1e0102dc0 0x7fd1e01a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.996+0000 7fd1e5600700 1 -- 172.21.15.37:0/1683274756 learned_addr learned my addr 172.21.15.37:0/1683274756 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.996+0000 7fd1e6000700 1 --2- 172.21.15.37:0/1683274756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1e01074f0 0x7fd1e019a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.996+0000 7fd1e5600700 1 -- 172.21.15.37:0/1683274756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1e01074f0 msgr2=0x7fd1e019a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.996+0000 7fd1e5600700 1 --2- 172.21.15.37:0/1683274756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1e01074f0 0x7fd1e019a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.996+0000 7fd1e5600700 1 -- 172.21.15.37:0/1683274756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1e0102dc0 msgr2=0x7fd1e01a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.996+0000 7fd1e5600700 1 --2- 172.21.15.37:0/1683274756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1e0102dc0 0x7fd1e01a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.996+0000 7fd1e5600700 1 -- 172.21.15.37:0/1683274756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e019b0e0 con 0x7fd1e0102410 2024-03-28T18:09:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.996+0000 7fd1e5600700 1 --2- 172.21.15.37:0/1683274756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1e0102410 0x7fd1e01a07c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4007a20 tx=0x7fd1d4007a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.997+0000 7fd1de000700 1 -- 172.21.15.37:0/1683274756 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d4021020 con 0x7fd1e0102410 2024-03-28T18:09:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.997+0000 7fd1de000700 1 -- 172.21.15.37:0/1683274756 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1d4004c50 con 0x7fd1e0102410 2024-03-28T18:09:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.997+0000 7fd1de000700 1 -- 172.21.15.37:0/1683274756 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d40052b0 con 0x7fd1e0102410 2024-03-28T18:09:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.997+0000 7fd1e8000700 1 -- 172.21.15.37:0/1683274756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e019b370 con 0x7fd1e0102410 2024-03-28T18:09:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.997+0000 7fd1e8000700 1 -- 172.21.15.37:0/1683274756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e01a7690 con 0x7fd1e0102410 2024-03-28T18:09:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:12.998+0000 7fd1e8000700 1 -- 172.21.15.37:0/1683274756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd1e010edf0 con 0x7fd1e0102410 2024-03-28T18:09:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.000+0000 7fd1de000700 1 -- 172.21.15.37:0/1683274756 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fd1d4024020 con 0x7fd1e0102410 2024-03-28T18:09:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.001+0000 7fd1de000700 1 --2- 172.21.15.37:0/1683274756 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd1c80a29a0 0x7fd1c80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.002+0000 7fd1de000700 1 -- 172.21.15.37:0/1683274756 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd1d400f030 con 0x7fd1e0102410 2024-03-28T18:09:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.002+0000 7fd1e4c00700 1 --2- 172.21.15.37:0/1683274756 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd1c80a29a0 0x7fd1c80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.002+0000 7fd1de000700 1 --2- 172.21.15.37:0/1683274756 >> [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240] conn(0x7fd1c80ac050 0x7fd1c80ae470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.002+0000 7fd1de000700 1 -- 172.21.15.37:0/1683274756 --> [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd1d4027e50 con 0x7fd1c80ac050 2024-03-28T18:09:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.002+0000 7fd1de000700 1 -- 172.21.15.37:0/1683274756 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1d40cfe30 con 0x7fd1e0102410 2024-03-28T18:09:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.002+0000 7fd1e6000700 1 --2- 172.21.15.37:0/1683274756 >> [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240] conn(0x7fd1c80ac050 0x7fd1c80ae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.003+0000 7fd1e4c00700 1 --2- 172.21.15.37:0/1683274756 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd1c80a29a0 0x7fd1c80a4e60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00ff310 tx=0x7fd1cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:13.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.003+0000 7fd1e6000700 1 --2- 172.21.15.37:0/1683274756 >> [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240] conn(0x7fd1c80ac050 0x7fd1c80ae470 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:13.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.004+0000 7fd1de000700 1 -- 172.21.15.37:0/1683274756 <== osd.3 v2:172.21.15.37:6810/317805240 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd1d4027e50 con 0x7fd1c80ac050 2024-03-28T18:09:13.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.039+0000 7fd1e8000700 1 -- 172.21.15.37:0/1683274756 --> [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd1e01038f0 con 0x7fd1c80ac050 2024-03-28T18:09:13.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.040+0000 7fd1de000700 1 -- 172.21.15.37:0/1683274756 <== osd.3 v2:172.21.15.37:6810/317805240 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fd1e01038f0 con 0x7fd1c80ac050 2024-03-28T18:09:13.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.041+0000 7fd1e8000700 1 -- 172.21.15.37:0/1683274756 >> [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240] conn(0x7fd1c80ac050 msgr2=0x7fd1c80ae470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:13.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.041+0000 7fd1e8000700 1 --2- 172.21.15.37:0/1683274756 >> [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240] conn(0x7fd1c80ac050 0x7fd1c80ae470 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:13.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.041+0000 7fd1e8000700 1 -- 172.21.15.37:0/1683274756 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd1c80a29a0 msgr2=0x7fd1c80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:13.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.041+0000 7fd1e8000700 1 --2- 172.21.15.37:0/1683274756 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd1c80a29a0 0x7fd1c80a4e60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00ff310 tx=0x7fd1cc009260 comp rx=0 tx=0).stop 2024-03-28T18:09:13.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.041+0000 7fd1e8000700 1 -- 172.21.15.37:0/1683274756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1e0102410 msgr2=0x7fd1e01a07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:13.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.041+0000 7fd1e8000700 1 --2- 172.21.15.37:0/1683274756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1e0102410 0x7fd1e01a07c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4007a20 tx=0x7fd1d4007a50 comp rx=0 tx=0).stop 2024-03-28T18:09:13.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.041+0000 7fd1e6000700 1 -- 172.21.15.37:0/1683274756 reap_dead start 2024-03-28T18:09:13.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.041+0000 7fd1e8000700 1 -- 172.21.15.37:0/1683274756 shutdown_connections 2024-03-28T18:09:13.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.041+0000 7fd1e8000700 1 -- 172.21.15.37:0/1683274756 >> 172.21.15.37:0/1683274756 conn(0x7fd1e00fdc60 msgr2=0x7fd1e010e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:13.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.041+0000 7fd1e8000700 1 -- 172.21.15.37:0/1683274756 shutdown_connections 2024-03-28T18:09:13.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:13.041+0000 7fd1e8000700 1 -- 172.21.15.37:0/1683274756 wait complete. 2024-03-28T18:09:13.713 INFO:teuthology.orchestra.run.smithi037.stdout:77309411403 2024-03-28T18:09:13.713 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-28T18:09:13.801 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:13 smithi037 ceph-mon[31460]: pgmap v344: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-28T18:09:14.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:13 smithi059 ceph-mon[36190]: pgmap v344: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-28T18:09:14.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:13 smithi170 ceph-mon[36280]: pgmap v344: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-28T18:09:14.045 INFO:teuthology.orchestra.run.smithi037.stdout:309237645342 2024-03-28T18:09:14.045 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-28T18:09:14.532 INFO:teuthology.orchestra.run.smithi037.stdout:343597383704 2024-03-28T18:09:14.532 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-28T18:09:14.863 INFO:teuthology.orchestra.run.smithi037.stdout:163208757306 2024-03-28T18:09:14.864 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-28T18:09:14.956 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-28T18:09:14.956 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]: dispatch 2024-03-28T18:09:14.956 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-03-28T18:09:14.956 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-03-28T18:09:14.957 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-28T18:09:14.957 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-03-28T18:09:14.957 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-28T18:09:14.957 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:14 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-03-28T18:09:15.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-28T18:09:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]: dispatch 2024-03-28T18:09:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-03-28T18:09:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-03-28T18:09:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-28T18:09:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-03-28T18:09:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-28T18:09:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:14 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-03-28T18:09:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-28T18:09:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]: dispatch 2024-03-28T18:09:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-03-28T18:09:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-03-28T18:09:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-28T18:09:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-03-28T18:09:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-28T18:09:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:14 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-03-28T18:09:15.335 INFO:teuthology.orchestra.run.smithi037.stdout:197568495669 2024-03-28T18:09:15.335 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-28T18:09:15.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:15 smithi037 ceph-mon[31460]: pgmap v345: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:09:15.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:15 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-28T18:09:15.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:15 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-03-28T18:09:15.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:15 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-03-28T18:09:15.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:15 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-03-28T18:09:15.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:15 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-28T18:09:15.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:15 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-03-28T18:09:15.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:15 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-28T18:09:15.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:15 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-03-28T18:09:15.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:15 smithi037 ceph-mon[31460]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T18:09:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:15 smithi059 ceph-mon[36190]: pgmap v345: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:09:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:15 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-28T18:09:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:15 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-03-28T18:09:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:15 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-03-28T18:09:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:15 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-03-28T18:09:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:15 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-28T18:09:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:15 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-03-28T18:09:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:15 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-28T18:09:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:15 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-03-28T18:09:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:15 smithi059 ceph-mon[36190]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T18:09:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:15 smithi170 ceph-mon[36280]: pgmap v345: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:09:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:15 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-28T18:09:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:15 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-03-28T18:09:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:15 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-03-28T18:09:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:15 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-03-28T18:09:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:15 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-28T18:09:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:15 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-03-28T18:09:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:15 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-28T18:09:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:15 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-03-28T18:09:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:15 smithi170 ceph-mon[36280]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T18:09:16.188 INFO:teuthology.orchestra.run.smithi037.stdout:107374182471 2024-03-28T18:09:16.188 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-28T18:09:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:15.998+0000 7f215e000700 1 -- 172.21.15.37:0/3181509743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2158102ba0 msgr2=0x7f215810efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:15.998+0000 7f215e000700 1 --2- 172.21.15.37:0/3181509743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2158102ba0 0x7f215810efe0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f215000b0d0 tx=0x7f215000b3e0 comp rx=0 tx=0).stop 2024-03-28T18:09:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:15.998+0000 7f215e000700 1 -- 172.21.15.37:0/3181509743 shutdown_connections 2024-03-28T18:09:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:15.998+0000 7f215e000700 1 --2- 172.21.15.37:0/3181509743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2158102ba0 0x7f215810efe0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:15.998+0000 7f215e000700 1 --2- 172.21.15.37:0/3181509743 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21581081c0 0x7f21581085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:15.998+0000 7f215e000700 1 --2- 172.21.15.37:0/3181509743 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21581021e0 0x7f2158102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:15.998+0000 7f215e000700 1 -- 172.21.15.37:0/3181509743 >> 172.21.15.37:0/3181509743 conn(0x7f21580792c0 msgr2=0x7f21580796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:15.998+0000 7f215e000700 1 -- 172.21.15.37:0/3181509743 shutdown_connections 2024-03-28T18:09:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:15.998+0000 7f215e000700 1 -- 172.21.15.37:0/3181509743 wait complete. 2024-03-28T18:09:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:15.999+0000 7f215e000700 1 Processor -- start 2024-03-28T18:09:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.000+0000 7f215e000700 1 -- start start 2024-03-28T18:09:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.000+0000 7f215e000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21581021e0 0x7f21581a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.000+0000 7f215e000700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2158102ba0 0x7f21581a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.000+0000 7f215e000700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21581081c0 0x7f215819a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.000+0000 7f215e000700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2158112490 con 0x7f21581021e0 2024-03-28T18:09:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.000+0000 7f215e000700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2158112310 con 0x7f21581081c0 2024-03-28T18:09:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.000+0000 7f215e000700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2158112610 con 0x7f2158102ba0 2024-03-28T18:09:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.000+0000 7f2156a00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2158102ba0 0x7f21581a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.000+0000 7f2156a00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2158102ba0 0x7f21581a0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:37480/0 (socket says 172.21.15.37:37480) 2024-03-28T18:09:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.000+0000 7f2157e00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21581081c0 0x7f215819a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.000+0000 7f2156a00700 1 -- 172.21.15.37:0/2685080644 learned_addr learned my addr 172.21.15.37:0/2685080644 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.000+0000 7f2157400700 1 --2- 172.21.15.37:0/2685080644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21581021e0 0x7f21581a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.001+0000 7f2156a00700 1 -- 172.21.15.37:0/2685080644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21581081c0 msgr2=0x7f215819a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.001+0000 7f2156a00700 1 --2- 172.21.15.37:0/2685080644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21581081c0 0x7f215819a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.001+0000 7f2156a00700 1 -- 172.21.15.37:0/2685080644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21581021e0 msgr2=0x7f21581a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.001+0000 7f2156a00700 1 --2- 172.21.15.37:0/2685080644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21581021e0 0x7f21581a0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.001+0000 7f2156a00700 1 -- 172.21.15.37:0/2685080644 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215819b080 con 0x7f2158102ba0 2024-03-28T18:09:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.001+0000 7f2157e00700 1 --2- 172.21.15.37:0/2685080644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21581081c0 0x7f215819a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:09:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.001+0000 7f2157400700 1 --2- 172.21.15.37:0/2685080644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21581021e0 0x7f21581a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:09:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.001+0000 7f2156a00700 1 --2- 172.21.15.37:0/2685080644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2158102ba0 0x7f21581a0ca0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f214400e9a0 tx=0x7f214400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.001+0000 7f214fe00700 1 -- 172.21.15.37:0/2685080644 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214400cbd0 con 0x7f2158102ba0 2024-03-28T18:09:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.001+0000 7f214fe00700 1 -- 172.21.15.37:0/2685080644 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f214400cd70 con 0x7f2158102ba0 2024-03-28T18:09:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.001+0000 7f215e000700 1 -- 172.21.15.37:0/2685080644 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f215819b370 con 0x7f2158102ba0 2024-03-28T18:09:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.002+0000 7f214fe00700 1 -- 172.21.15.37:0/2685080644 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2144018750 con 0x7f2158102ba0 2024-03-28T18:09:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.002+0000 7f215e000700 1 -- 172.21.15.37:0/2685080644 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21581a7630 con 0x7f2158102ba0 2024-03-28T18:09:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.003+0000 7f215e000700 1 -- 172.21.15.37:0/2685080644 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f215810ed80 con 0x7f2158102ba0 2024-03-28T18:09:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.004+0000 7f214fe00700 1 -- 172.21.15.37:0/2685080644 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f2144010430 con 0x7f2158102ba0 2024-03-28T18:09:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.006+0000 7f214fe00700 1 --2- 172.21.15.37:0/2685080644 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f213c0a29a0 0x7f213c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.006+0000 7f214fe00700 1 -- 172.21.15.37:0/2685080644 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f21440cee80 con 0x7f2158102ba0 2024-03-28T18:09:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.006+0000 7f2157400700 1 --2- 172.21.15.37:0/2685080644 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f213c0a29a0 0x7f213c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.006+0000 7f214fe00700 1 --2- 172.21.15.37:0/2685080644 >> [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008] conn(0x7f213c0ac4e0 0x7f213c0ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.006+0000 7f214fe00700 1 -- 172.21.15.37:0/2685080644 --> [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f214401ee50 con 0x7f213c0ac4e0 2024-03-28T18:09:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.006+0000 7f214fe00700 1 -- 172.21.15.37:0/2685080644 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21440d14a0 con 0x7f2158102ba0 2024-03-28T18:09:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.007+0000 7f2157e00700 1 --2- 172.21.15.37:0/2685080644 >> [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008] conn(0x7f213c0ac4e0 0x7f213c0ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.007+0000 7f2157400700 1 --2- 172.21.15.37:0/2685080644 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f213c0a29a0 0x7f213c0a4e60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f215819ba60 tx=0x7f2148009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.007+0000 7f2157e00700 1 --2- 172.21.15.37:0/2685080644 >> [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008] conn(0x7f213c0ac4e0 0x7f213c0ae980 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.008+0000 7f214fe00700 1 -- 172.21.15.37:0/2685080644 <== osd.9 v2:172.21.15.37:6826/2307153008 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f214401ee50 con 0x7f213c0ac4e0 2024-03-28T18:09:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.030+0000 7f215e000700 1 -- 172.21.15.37:0/2685080644 --> [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2158103870 con 0x7f213c0ac4e0 2024-03-28T18:09:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.031+0000 7f214fe00700 1 -- 172.21.15.37:0/2685080644 <== osd.9 v2:172.21.15.37:6826/2307153008 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2158103870 con 0x7f213c0ac4e0 2024-03-28T18:09:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.031+0000 7f215e000700 1 -- 172.21.15.37:0/2685080644 >> [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008] conn(0x7f213c0ac4e0 msgr2=0x7f213c0ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.031+0000 7f215e000700 1 --2- 172.21.15.37:0/2685080644 >> [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008] conn(0x7f213c0ac4e0 0x7f213c0ae980 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.032+0000 7f215e000700 1 -- 172.21.15.37:0/2685080644 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f213c0a29a0 msgr2=0x7f213c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:16.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.032+0000 7f215e000700 1 --2- 172.21.15.37:0/2685080644 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f213c0a29a0 0x7f213c0a4e60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f215819ba60 tx=0x7f2148009260 comp rx=0 tx=0).stop 2024-03-28T18:09:16.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.032+0000 7f215e000700 1 -- 172.21.15.37:0/2685080644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2158102ba0 msgr2=0x7f21581a0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:16.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.032+0000 7f215e000700 1 --2- 172.21.15.37:0/2685080644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2158102ba0 0x7f21581a0ca0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f214400e9a0 tx=0x7f214400ecb0 comp rx=0 tx=0).stop 2024-03-28T18:09:16.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.032+0000 7f2157e00700 1 -- 172.21.15.37:0/2685080644 reap_dead start 2024-03-28T18:09:16.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.032+0000 7f215e000700 1 -- 172.21.15.37:0/2685080644 shutdown_connections 2024-03-28T18:09:16.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.032+0000 7f215e000700 1 -- 172.21.15.37:0/2685080644 >> 172.21.15.37:0/2685080644 conn(0x7f21580792c0 msgr2=0x7f215810d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:16.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.032+0000 7f215e000700 1 -- 172.21.15.37:0/2685080644 shutdown_connections 2024-03-28T18:09:16.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:16.032+0000 7f215e000700 1 -- 172.21.15.37:0/2685080644 wait complete. 2024-03-28T18:09:16.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:16 smithi037 ceph-mon[31460]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T18:09:17.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:16 smithi059 ceph-mon[36190]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T18:09:17.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:16 smithi170 ceph-mon[36280]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T18:09:17.101 INFO:teuthology.orchestra.run.smithi037.stdout:287762808868 2024-03-28T18:09:17.102 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-28T18:09:17.431 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:09:17.448 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:09:17.501 INFO:teuthology.orchestra.run.smithi037.stdout:55834574929 2024-03-28T18:09:17.502 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-28T18:09:17.546 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:09:18.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:17 smithi059 ceph-mon[36190]: pgmap v348: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:09:18.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:17 smithi170 ceph-mon[36280]: pgmap v348: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:09:18.037 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:17 smithi037 ceph-mon[31460]: pgmap v348: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:09:19.068 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:09:19.073 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:09:19.076 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:09:19.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:19 smithi037 ceph-mon[31460]: pgmap v349: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 86 B/s, 2 keys/s, 1 objects/s recovering 2024-03-28T18:09:20.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:19 smithi059 ceph-mon[36190]: pgmap v349: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 86 B/s, 2 keys/s, 1 objects/s recovering 2024-03-28T18:09:20.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:19 smithi170 ceph-mon[36280]: pgmap v349: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 86 B/s, 2 keys/s, 1 objects/s recovering 2024-03-28T18:09:21.034 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:20 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:09:21.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:20 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:09:21.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:20 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:09:21.955 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:21 smithi037 ceph-mon[31460]: pgmap v350: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 86 B/s, 2 keys/s, 1 objects/s recovering 2024-03-28T18:09:22.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:21 smithi059 ceph-mon[36190]: pgmap v350: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 86 B/s, 2 keys/s, 1 objects/s recovering 2024-03-28T18:09:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:21 smithi170 ceph-mon[36280]: pgmap v350: 97 pgs: 97 active+clean; 641 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 86 B/s, 2 keys/s, 1 objects/s recovering 2024-03-28T18:09:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:23 smithi059 ceph-mon[36190]: pgmap v351: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 86 B/s, 2 keys/s, 1 objects/s recovering 2024-03-28T18:09:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:23 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:23 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:23 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:23 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:23.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:23 smithi170 ceph-mon[36280]: pgmap v351: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 86 B/s, 2 keys/s, 1 objects/s recovering 2024-03-28T18:09:23.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:23.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:23 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:23.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:23 smithi037 ceph-mon[31460]: pgmap v351: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 86 B/s, 2 keys/s, 1 objects/s recovering 2024-03-28T18:09:23.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:23.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:23 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:23.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.880+0000 7fc440800700 1 -- 172.21.15.37:0/2920681244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc438102d80 msgr2=0x7fc438106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:23.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.880+0000 7fc440800700 1 --2- 172.21.15.37:0/2920681244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc438102d80 0x7fc438106dd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc4280098f0 tx=0x7fc428009c00 comp rx=0 tx=0).stop 2024-03-28T18:09:23.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.880+0000 7fc440800700 1 -- 172.21.15.37:0/2920681244 shutdown_connections 2024-03-28T18:09:23.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.880+0000 7fc440800700 1 --2- 172.21.15.37:0/2920681244 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4381074b0 0x7fc43810f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:23.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.880+0000 7fc440800700 1 --2- 172.21.15.37:0/2920681244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4381023d0 0x7fc4381027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:23.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.880+0000 7fc440800700 1 --2- 172.21.15.37:0/2920681244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc438102d80 0x7fc438106dd0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:23.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.880+0000 7fc440800700 1 -- 172.21.15.37:0/2920681244 >> 172.21.15.37:0/2920681244 conn(0x7fc4380fdc40 msgr2=0x7fc438100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:23.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.880+0000 7fc440800700 1 -- 172.21.15.37:0/2920681244 shutdown_connections 2024-03-28T18:09:23.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.880+0000 7fc440800700 1 -- 172.21.15.37:0/2920681244 wait complete. 2024-03-28T18:09:23.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.881+0000 7fc440800700 1 Processor -- start 2024-03-28T18:09:23.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.881+0000 7fc440800700 1 -- start start 2024-03-28T18:09:23.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.881+0000 7fc440800700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4381023d0 0x7fc4381a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:23.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.881+0000 7fc440800700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc438102d80 0x7fc4381a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:23.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.881+0000 7fc440800700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4381074b0 0x7fc43819aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:23.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.881+0000 7fc440800700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4381124c0 con 0x7fc4381074b0 2024-03-28T18:09:23.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.882+0000 7fc440800700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc438112340 con 0x7fc4381023d0 2024-03-28T18:09:23.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.882+0000 7fc440800700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc438112640 con 0x7fc438102d80 2024-03-28T18:09:23.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.882+0000 7fc43de00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4381023d0 0x7fc4381a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:23.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.882+0000 7fc43de00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4381023d0 0x7fc4381a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:36166/0 (socket says 172.21.15.37:36166) 2024-03-28T18:09:23.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.882+0000 7fc43de00700 1 -- 172.21.15.37:0/3727836581 learned_addr learned my addr 172.21.15.37:0/3727836581 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:23.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.882+0000 7fc43d400700 1 --2- 172.21.15.37:0/3727836581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc438102d80 0x7fc4381a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:23.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.882+0000 7fc43e800700 1 --2- 172.21.15.37:0/3727836581 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4381074b0 0x7fc43819aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:23.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.882+0000 7fc43de00700 1 -- 172.21.15.37:0/3727836581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc438102d80 msgr2=0x7fc4381a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:23.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.882+0000 7fc43de00700 1 --2- 172.21.15.37:0/3727836581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc438102d80 0x7fc4381a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:23.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.882+0000 7fc43de00700 1 -- 172.21.15.37:0/3727836581 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4381074b0 msgr2=0x7fc43819aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:23.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.882+0000 7fc43de00700 1 --2- 172.21.15.37:0/3727836581 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4381074b0 0x7fc43819aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:23.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.882+0000 7fc43de00700 1 -- 172.21.15.37:0/3727836581 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43819b160 con 0x7fc4381023d0 2024-03-28T18:09:23.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.882+0000 7fc43e800700 1 --2- 172.21.15.37:0/3727836581 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4381074b0 0x7fc43819aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:09:23.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.883+0000 7fc43de00700 1 --2- 172.21.15.37:0/3727836581 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4381023d0 0x7fc4381a0840 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc42c00e9c0 tx=0x7fc42c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:23.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.883+0000 7fc432a00700 1 -- 172.21.15.37:0/3727836581 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c00cbd0 con 0x7fc4381023d0 2024-03-28T18:09:23.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.883+0000 7fc440800700 1 -- 172.21.15.37:0/3727836581 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc43819b450 con 0x7fc4381023d0 2024-03-28T18:09:23.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.883+0000 7fc432a00700 1 -- 172.21.15.37:0/3727836581 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc42c00cd70 con 0x7fc4381023d0 2024-03-28T18:09:23.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.883+0000 7fc432a00700 1 -- 172.21.15.37:0/3727836581 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c005660 con 0x7fc4381023d0 2024-03-28T18:09:23.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.883+0000 7fc440800700 1 -- 172.21.15.37:0/3727836581 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4381a7710 con 0x7fc4381023d0 2024-03-28T18:09:23.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.884+0000 7fc440800700 1 -- 172.21.15.37:0/3727836581 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc438068e50 con 0x7fc4381023d0 2024-03-28T18:09:23.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.886+0000 7fc432a00700 1 -- 172.21.15.37:0/3727836581 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fc42c004750 con 0x7fc4381023d0 2024-03-28T18:09:23.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.888+0000 7fc432a00700 1 --2- 172.21.15.37:0/3727836581 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc4200a2920 0x7fc4200a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:23.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.888+0000 7fc432a00700 1 -- 172.21.15.37:0/3727836581 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc42c0cedd0 con 0x7fc4381023d0 2024-03-28T18:09:23.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.888+0000 7fc43d400700 1 --2- 172.21.15.37:0/3727836581 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc4200a2920 0x7fc4200a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:23.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.889+0000 7fc43d400700 1 --2- 172.21.15.37:0/3727836581 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc4200a2920 0x7fc4200a4de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc428009fd0 tx=0x7fc42801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:23.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:23.896+0000 7fc432a00700 1 -- 172.21.15.37:0/3727836581 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc42c09bcc0 con 0x7fc4381023d0 2024-03-28T18:09:24.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.153+0000 7fc440800700 1 -- 172.21.15.37:0/3727836581 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fc43819bf30 con 0x7fc4381023d0 2024-03-28T18:09:24.153 INFO:teuthology.orchestra.run.smithi037.stdout:141733920836 2024-03-28T18:09:24.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.153+0000 7fc432a00700 1 -- 172.21.15.37:0/3727836581 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc42c09b5c0 con 0x7fc4381023d0 2024-03-28T18:09:24.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.156+0000 7fc440800700 1 -- 172.21.15.37:0/3727836581 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc4200a2920 msgr2=0x7fc4200a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:24.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.156+0000 7fc440800700 1 --2- 172.21.15.37:0/3727836581 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc4200a2920 0x7fc4200a4de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc428009fd0 tx=0x7fc42801a040 comp rx=0 tx=0).stop 2024-03-28T18:09:24.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.156+0000 7fc440800700 1 -- 172.21.15.37:0/3727836581 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4381023d0 msgr2=0x7fc4381a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:24.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.156+0000 7fc440800700 1 --2- 172.21.15.37:0/3727836581 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4381023d0 0x7fc4381a0840 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc42c00e9c0 tx=0x7fc42c00ecd0 comp rx=0 tx=0).stop 2024-03-28T18:09:24.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.156+0000 7fc440800700 1 -- 172.21.15.37:0/3727836581 shutdown_connections 2024-03-28T18:09:24.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.156+0000 7fc440800700 1 --2- 172.21.15.37:0/3727836581 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fc4200a2920 0x7fc4200a4de0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:24.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.156+0000 7fc440800700 1 --2- 172.21.15.37:0/3727836581 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4381074b0 0x7fc43819aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:24.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.156+0000 7fc440800700 1 --2- 172.21.15.37:0/3727836581 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4381023d0 0x7fc4381a0840 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:24.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.156+0000 7fc440800700 1 --2- 172.21.15.37:0/3727836581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc438102d80 0x7fc4381a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:24.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.156+0000 7fc440800700 1 -- 172.21.15.37:0/3727836581 >> 172.21.15.37:0/3727836581 conn(0x7fc4380fdc40 msgr2=0x7fc4380fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:24.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.156+0000 7fc440800700 1 -- 172.21.15.37:0/3727836581 shutdown_connections 2024-03-28T18:09:24.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.157+0000 7fc440800700 1 -- 172.21.15.37:0/3727836581 wait complete. 2024-03-28T18:09:25.085 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:25 smithi037 ceph-mon[31460]: pgmap v352: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s; 74 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T18:09:25.085 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:25 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3727836581' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T18:09:25.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:25 smithi170 ceph-mon[36280]: pgmap v352: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s; 74 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T18:09:25.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:25 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3727836581' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T18:09:25.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.410+0000 7f69fc200700 1 -- 172.21.15.37:0/2567669564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69f41096d0 msgr2=0x7f69f41112a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:25.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.410+0000 7f69fc200700 1 --2- 172.21.15.37:0/2567669564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69f41096d0 0x7f69f41112a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f69ec00b0a0 tx=0x7f69ec00b3b0 comp rx=0 tx=0).stop 2024-03-28T18:09:25.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.410+0000 7f69fc200700 1 -- 172.21.15.37:0/2567669564 shutdown_connections 2024-03-28T18:09:25.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.410+0000 7f69fc200700 1 --2- 172.21.15.37:0/2567669564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69f41096d0 0x7f69f41112a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:25.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.410+0000 7f69fc200700 1 --2- 172.21.15.37:0/2567669564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69f41045f0 0x7f69f41049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:25.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.410+0000 7f69fc200700 1 --2- 172.21.15.37:0/2567669564 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69f4104fa0 0x7f69f4108ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:25.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.410+0000 7f69fc200700 1 -- 172.21.15.37:0/2567669564 >> 172.21.15.37:0/2567669564 conn(0x7f69f40ffe60 msgr2=0x7f69f4102280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:25.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.411+0000 7f69fc200700 1 -- 172.21.15.37:0/2567669564 shutdown_connections 2024-03-28T18:09:25.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.411+0000 7f69fc200700 1 -- 172.21.15.37:0/2567669564 wait complete. 2024-03-28T18:09:25.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.412+0000 7f69fc200700 1 Processor -- start 2024-03-28T18:09:25.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.412+0000 7f69fc200700 1 -- start start 2024-03-28T18:09:25.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.413+0000 7f69fc200700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69f41045f0 0x7f69f41a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:25.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.413+0000 7f69fc200700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69f4104fa0 0x7f69f41a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:25.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.413+0000 7f69fc200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69f41096d0 0x7f69f419a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:25.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.413+0000 7f69fc200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f4114730 con 0x7f69f41096d0 2024-03-28T18:09:25.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.413+0000 7f69fc200700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f41145b0 con 0x7f69f41045f0 2024-03-28T18:09:25.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.413+0000 7f69fc200700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f41148b0 con 0x7f69f4104fa0 2024-03-28T18:09:25.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.413+0000 7f69f9800700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69f41045f0 0x7f69f41a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:25.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.413+0000 7f69f9800700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69f41045f0 0x7f69f41a07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:36204/0 (socket says 172.21.15.37:36204) 2024-03-28T18:09:25.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.413+0000 7f69f9800700 1 -- 172.21.15.37:0/1327415441 learned_addr learned my addr 172.21.15.37:0/1327415441 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:25.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.413+0000 7f69f8e00700 1 --2- 172.21.15.37:0/1327415441 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69f4104fa0 0x7f69f41a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:25.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.413+0000 7f69fa200700 1 --2- 172.21.15.37:0/1327415441 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69f41096d0 0x7f69f419a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:25.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.413+0000 7f69f9800700 1 -- 172.21.15.37:0/1327415441 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69f4104fa0 msgr2=0x7f69f41a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:25.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.413+0000 7f69f9800700 1 --2- 172.21.15.37:0/1327415441 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69f4104fa0 0x7f69f41a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:25.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.413+0000 7f69f9800700 1 -- 172.21.15.37:0/1327415441 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69f41096d0 msgr2=0x7f69f419a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:25.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.413+0000 7f69f9800700 1 --2- 172.21.15.37:0/1327415441 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69f41096d0 0x7f69f419a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:25.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.413+0000 7f69f9800700 1 -- 172.21.15.37:0/1327415441 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f419b0d0 con 0x7f69f41045f0 2024-03-28T18:09:25.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.414+0000 7f69f9800700 1 --2- 172.21.15.37:0/1327415441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69f41045f0 0x7f69f41a07b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f69e800e9c0 tx=0x7f69e800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:25.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.414+0000 7f69f2000700 1 -- 172.21.15.37:0/1327415441 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e800cbd0 con 0x7f69f41045f0 2024-03-28T18:09:25.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.414+0000 7f69f2000700 1 -- 172.21.15.37:0/1327415441 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69e800ee60 con 0x7f69f41045f0 2024-03-28T18:09:25.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.414+0000 7f69fc200700 1 -- 172.21.15.37:0/1327415441 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f419b3c0 con 0x7f69f41045f0 2024-03-28T18:09:25.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.414+0000 7f69f2000700 1 -- 172.21.15.37:0/1327415441 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8005630 con 0x7f69f41045f0 2024-03-28T18:09:25.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.414+0000 7f69fc200700 1 -- 172.21.15.37:0/1327415441 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f41a7680 con 0x7f69f41045f0 2024-03-28T18:09:25.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.415+0000 7f69fc200700 1 -- 172.21.15.37:0/1327415441 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f4068e50 con 0x7f69f41045f0 2024-03-28T18:09:25.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.417+0000 7f69f2000700 1 -- 172.21.15.37:0/1327415441 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f69e8004750 con 0x7f69f41045f0 2024-03-28T18:09:25.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.419+0000 7f69f2000700 1 --2- 172.21.15.37:0/1327415441 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f69dc0a29a0 0x7f69dc0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:25.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.419+0000 7f69f2000700 1 -- 172.21.15.37:0/1327415441 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f69e80cee20 con 0x7f69f41045f0 2024-03-28T18:09:25.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.419+0000 7f69f8e00700 1 --2- 172.21.15.37:0/1327415441 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f69dc0a29a0 0x7f69dc0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:25.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.420+0000 7f69f8e00700 1 --2- 172.21.15.37:0/1327415441 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f69dc0a29a0 0x7f69dc0a4e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f69f4103570 tx=0x7f69e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:25.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.423+0000 7f69f2000700 1 -- 172.21.15.37:0/1327415441 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f69e809bd10 con 0x7f69f41045f0 2024-03-28T18:09:25.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:25 smithi059 ceph-mon[36190]: pgmap v352: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s; 74 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T18:09:25.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:25 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3727836581' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T18:09:25.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.659+0000 7f69fc200700 1 -- 172.21.15.37:0/1327415441 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f69f419bed0 con 0x7f69f41045f0 2024-03-28T18:09:25.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.660+0000 7f69f2000700 1 -- 172.21.15.37:0/1327415441 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f69f419bed0 con 0x7f69f41045f0 2024-03-28T18:09:25.660 INFO:teuthology.orchestra.run.smithi037.stdout:253403070508 2024-03-28T18:09:25.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.663+0000 7f69fc200700 1 -- 172.21.15.37:0/1327415441 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f69dc0a29a0 msgr2=0x7f69dc0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:25.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.663+0000 7f69fc200700 1 --2- 172.21.15.37:0/1327415441 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f69dc0a29a0 0x7f69dc0a4e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f69f4103570 tx=0x7f69e0009260 comp rx=0 tx=0).stop 2024-03-28T18:09:25.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.663+0000 7f69fc200700 1 -- 172.21.15.37:0/1327415441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69f41045f0 msgr2=0x7f69f41a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:25.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.663+0000 7f69fc200700 1 --2- 172.21.15.37:0/1327415441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69f41045f0 0x7f69f41a07b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f69e800e9c0 tx=0x7f69e800ecd0 comp rx=0 tx=0).stop 2024-03-28T18:09:25.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.663+0000 7f69fc200700 1 -- 172.21.15.37:0/1327415441 shutdown_connections 2024-03-28T18:09:25.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.663+0000 7f69fc200700 1 --2- 172.21.15.37:0/1327415441 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f69dc0a29a0 0x7f69dc0a4e60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:25.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.663+0000 7f69fc200700 1 --2- 172.21.15.37:0/1327415441 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69f41096d0 0x7f69f419a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:25.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.663+0000 7f69fc200700 1 --2- 172.21.15.37:0/1327415441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69f41045f0 0x7f69f41a07b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:25.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.663+0000 7f69fc200700 1 --2- 172.21.15.37:0/1327415441 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69f4104fa0 0x7f69f41a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:25.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.663+0000 7f69fc200700 1 -- 172.21.15.37:0/1327415441 >> 172.21.15.37:0/1327415441 conn(0x7f69f40ffe60 msgr2=0x7f69f4100940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:25.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.663+0000 7f69fc200700 1 -- 172.21.15.37:0/1327415441 shutdown_connections 2024-03-28T18:09:25.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.664+0000 7f69fc200700 1 -- 172.21.15.37:0/1327415441 wait complete. 2024-03-28T18:09:26.117 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920831 got 141733920836 for osd.4 2024-03-28T18:09:26.118 DEBUG:teuthology.parallel:result is None 2024-03-28T18:09:26.306 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:26 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/4141622782' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T18:09:26.306 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:26 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1327415441' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T18:09:26.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.825+0000 7f9edd000700 1 -- 172.21.15.37:0/2284164597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ed8102dc0 msgr2=0x7f9ed8106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:26.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.825+0000 7f9edd000700 1 --2- 172.21.15.37:0/2284164597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ed8102dc0 0x7f9ed8106e10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9ec40099c0 tx=0x7f9ec4009cd0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.825+0000 7f9edd000700 1 -- 172.21.15.37:0/2284164597 shutdown_connections 2024-03-28T18:09:26.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.825+0000 7f9edd000700 1 --2- 172.21.15.37:0/2284164597 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ed81074f0 0x7f9ed810f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.825+0000 7f9edd000700 1 --2- 172.21.15.37:0/2284164597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ed8102410 0x7f9ed81027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.825+0000 7f9edd000700 1 --2- 172.21.15.37:0/2284164597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ed8102dc0 0x7f9ed8106e10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.825+0000 7f9edd000700 1 -- 172.21.15.37:0/2284164597 >> 172.21.15.37:0/2284164597 conn(0x7f9ed80fdc60 msgr2=0x7f9ed8100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:26.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.825+0000 7f9edd000700 1 -- 172.21.15.37:0/2284164597 shutdown_connections 2024-03-28T18:09:26.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.826+0000 7f9edd000700 1 -- 172.21.15.37:0/2284164597 wait complete. 2024-03-28T18:09:26.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.827+0000 7f9edd000700 1 Processor -- start 2024-03-28T18:09:26.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.827+0000 7f9edd000700 1 -- start start 2024-03-28T18:09:26.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.828+0000 7f9edd000700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ed8102410 0x7f9ed819e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:26.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.828+0000 7f9edd000700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ed8102dc0 0x7f9ed819ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:26.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.828+0000 7f9edd000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ed81074f0 0x7f9ed8198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:26.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.828+0000 7f9edd000700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed8112520 con 0x7f9ed81074f0 2024-03-28T18:09:26.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.828+0000 7f9edd000700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed81123a0 con 0x7f9ed8102410 2024-03-28T18:09:26.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.828+0000 7f9edd000700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed81126a0 con 0x7f9ed8102dc0 2024-03-28T18:09:26.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.828+0000 7f9ed5e00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ed8102410 0x7f9ed819e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:26.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.828+0000 7f9ed5e00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ed8102410 0x7f9ed819e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:36186/0 (socket says 172.21.15.37:36186) 2024-03-28T18:09:26.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.828+0000 7f9ed5e00700 1 -- 172.21.15.37:0/4141622782 learned_addr learned my addr 172.21.15.37:0/4141622782 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:26.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.828+0000 7f9ed5400700 1 --2- 172.21.15.37:0/4141622782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ed8102dc0 0x7f9ed819ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:26.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.828+0000 7f9ed6800700 1 --2- 172.21.15.37:0/4141622782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ed81074f0 0x7f9ed8198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:26.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.828+0000 7f9ed5e00700 1 -- 172.21.15.37:0/4141622782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ed8102dc0 msgr2=0x7f9ed819ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:26.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.828+0000 7f9ed5e00700 1 --2- 172.21.15.37:0/4141622782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ed8102dc0 0x7f9ed819ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.828+0000 7f9ed5e00700 1 -- 172.21.15.37:0/4141622782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ed81074f0 msgr2=0x7f9ed8198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:26.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.828+0000 7f9ed5e00700 1 --2- 172.21.15.37:0/4141622782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ed81074f0 0x7f9ed8198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.828+0000 7f9ed5e00700 1 -- 172.21.15.37:0/4141622782 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed8198fe0 con 0x7f9ed8102410 2024-03-28T18:09:26.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.828+0000 7f9ed5400700 1 --2- 172.21.15.37:0/4141622782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ed8102dc0 0x7f9ed819ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:09:26.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.829+0000 7f9ed5e00700 1 --2- 172.21.15.37:0/4141622782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ed8102410 0x7f9ed819e680 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9ec800e9c0 tx=0x7f9ec800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:26.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.829+0000 7f9ecea00700 1 -- 172.21.15.37:0/4141622782 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec800cbd0 con 0x7f9ed8102410 2024-03-28T18:09:26.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.829+0000 7f9ecea00700 1 -- 172.21.15.37:0/4141622782 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ec800ee60 con 0x7f9ed8102410 2024-03-28T18:09:26.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.829+0000 7f9edd000700 1 -- 172.21.15.37:0/4141622782 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed81992d0 con 0x7f9ed8102410 2024-03-28T18:09:26.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.829+0000 7f9ecea00700 1 -- 172.21.15.37:0/4141622782 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec8005630 con 0x7f9ed8102410 2024-03-28T18:09:26.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.829+0000 7f9edd000700 1 -- 172.21.15.37:0/4141622782 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed81a54e0 con 0x7f9ed8102410 2024-03-28T18:09:26.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.830+0000 7f9edd000700 1 -- 172.21.15.37:0/4141622782 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed81038f0 con 0x7f9ed8102410 2024-03-28T18:09:26.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.834+0000 7f9ecea00700 1 -- 172.21.15.37:0/4141622782 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f9ec8004750 con 0x7f9ed8102410 2024-03-28T18:09:26.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.835+0000 7f9ecea00700 1 --2- 172.21.15.37:0/4141622782 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9ebc0a2920 0x7f9ebc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:26.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.836+0000 7f9ecea00700 1 -- 172.21.15.37:0/4141622782 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9ec80ceec0 con 0x7f9ed8102410 2024-03-28T18:09:26.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.836+0000 7f9ed5400700 1 --2- 172.21.15.37:0/4141622782 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9ebc0a2920 0x7f9ebc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:26.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.836+0000 7f9ed5400700 1 --2- 172.21.15.37:0/4141622782 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9ebc0a2920 0x7f9ebc0a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4000c00 tx=0x7f9ec4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:26.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:24.839+0000 7f9ecea00700 1 -- 172.21.15.37:0/4141622782 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9ec809bdb0 con 0x7f9ed8102410 2024-03-28T18:09:26.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.084+0000 7f9edd000700 1 -- 172.21.15.37:0/4141622782 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f9ed802da30 con 0x7f9ed8102410 2024-03-28T18:09:26.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.085+0000 7f9ecea00700 1 -- 172.21.15.37:0/4141622782 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9ec80bb030 con 0x7f9ed8102410 2024-03-28T18:09:26.420 INFO:teuthology.orchestra.run.smithi037.stdout:219043332146 2024-03-28T18:09:26.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.088+0000 7f9edd000700 1 -- 172.21.15.37:0/4141622782 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9ebc0a2920 msgr2=0x7f9ebc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:26.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.088+0000 7f9edd000700 1 --2- 172.21.15.37:0/4141622782 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9ebc0a2920 0x7f9ebc0a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4000c00 tx=0x7f9ec4005fb0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.088+0000 7f9edd000700 1 -- 172.21.15.37:0/4141622782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ed8102410 msgr2=0x7f9ed819e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:26.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.088+0000 7f9edd000700 1 --2- 172.21.15.37:0/4141622782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ed8102410 0x7f9ed819e680 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9ec800e9c0 tx=0x7f9ec800ecd0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.088+0000 7f9edd000700 1 -- 172.21.15.37:0/4141622782 shutdown_connections 2024-03-28T18:09:26.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.088+0000 7f9edd000700 1 --2- 172.21.15.37:0/4141622782 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9ebc0a2920 0x7f9ebc0a4de0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.088+0000 7f9edd000700 1 --2- 172.21.15.37:0/4141622782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ed81074f0 0x7f9ed8198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.088+0000 7f9edd000700 1 --2- 172.21.15.37:0/4141622782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ed8102410 0x7f9ed819e680 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.088+0000 7f9edd000700 1 --2- 172.21.15.37:0/4141622782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ed8102dc0 0x7f9ed819ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.088+0000 7f9edd000700 1 -- 172.21.15.37:0/4141622782 >> 172.21.15.37:0/4141622782 conn(0x7f9ed80fdc60 msgr2=0x7f9ed80fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:26.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.088+0000 7f9edd000700 1 -- 172.21.15.37:0/4141622782 shutdown_connections 2024-03-28T18:09:26.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:25.089+0000 7f9edd000700 1 -- 172.21.15.37:0/4141622782 wait complete. 2024-03-28T18:09:26.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:26 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/4141622782' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T18:09:26.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:26 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1327415441' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T18:09:26.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:26 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/4141622782' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T18:09:26.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:26 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1327415441' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T18:09:26.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.709+0000 7f9816a00700 1 -- 172.21.15.37:0/3194975862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9810069e40 msgr2=0x7f981006a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:26.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.709+0000 7f9816a00700 1 --2- 172.21.15.37:0/3194975862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9810069e40 0x7f981006a2a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f97fc009990 tx=0x7f97fc009ca0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.709+0000 7f9816a00700 1 -- 172.21.15.37:0/3194975862 shutdown_connections 2024-03-28T18:09:26.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.709+0000 7f9816a00700 1 --2- 172.21.15.37:0/3194975862 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f981006a7e0 0x7f981010f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.709+0000 7f9816a00700 1 --2- 172.21.15.37:0/3194975862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98101081e0 0x7f98101085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.709+0000 7f9816a00700 1 --2- 172.21.15.37:0/3194975862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9810069e40 0x7f981006a2a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.709+0000 7f9816a00700 1 -- 172.21.15.37:0/3194975862 >> 172.21.15.37:0/3194975862 conn(0x7f9810077fc0 msgr2=0x7f98100783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:26.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.710+0000 7f9816a00700 1 -- 172.21.15.37:0/3194975862 shutdown_connections 2024-03-28T18:09:26.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.710+0000 7f9816a00700 1 -- 172.21.15.37:0/3194975862 wait complete. 2024-03-28T18:09:26.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.711+0000 7f9816a00700 1 Processor -- start 2024-03-28T18:09:26.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.711+0000 7f9816a00700 1 -- start start 2024-03-28T18:09:26.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.711+0000 7f9816a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9810069e40 0x7f9810101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:26.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.711+0000 7f9816a00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f981006a7e0 0x7f98101016a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:26.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.711+0000 7f9816a00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98101081e0 0x7f9810101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:26.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.711+0000 7f9816a00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9810112490 con 0x7f9810069e40 2024-03-28T18:09:26.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.711+0000 7f9816a00700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9810112310 con 0x7f981006a7e0 2024-03-28T18:09:26.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.711+0000 7f9816a00700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9810112610 con 0x7f98101081e0 2024-03-28T18:09:26.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.711+0000 7f9814a00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98101081e0 0x7f9810101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:26.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.711+0000 7f980f400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f981006a7e0 0x7f98101016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:26.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.711+0000 7f980fe00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9810069e40 0x7f9810101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:26.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.712+0000 7f980f400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f981006a7e0 0x7f98101016a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:36218/0 (socket says 172.21.15.37:36218) 2024-03-28T18:09:26.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.712+0000 7f980f400700 1 -- 172.21.15.37:0/1169519933 learned_addr learned my addr 172.21.15.37:0/1169519933 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:26.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.712+0000 7f9814a00700 1 -- 172.21.15.37:0/1169519933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f981006a7e0 msgr2=0x7f98101016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:26.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.712+0000 7f9814a00700 1 --2- 172.21.15.37:0/1169519933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f981006a7e0 0x7f98101016a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.712+0000 7f9814a00700 1 -- 172.21.15.37:0/1169519933 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9810069e40 msgr2=0x7f9810101160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:26.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.712+0000 7f9814a00700 1 --2- 172.21.15.37:0/1169519933 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9810069e40 0x7f9810101160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.712+0000 7f9814a00700 1 -- 172.21.15.37:0/1169519933 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98101a6ef0 con 0x7f98101081e0 2024-03-28T18:09:26.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.712+0000 7f980f400700 1 --2- 172.21.15.37:0/1169519933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f981006a7e0 0x7f98101016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:09:26.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.712+0000 7f9814a00700 1 --2- 172.21.15.37:0/1169519933 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98101081e0 0x7f9810101be0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f980800efc0 tx=0x7f980800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:26.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.712+0000 7f980cc00700 1 -- 172.21.15.37:0/1169519933 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98080092e0 con 0x7f98101081e0 2024-03-28T18:09:26.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.713+0000 7f980cc00700 1 -- 172.21.15.37:0/1169519933 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9808004bb0 con 0x7f98101081e0 2024-03-28T18:09:26.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.713+0000 7f9816a00700 1 -- 172.21.15.37:0/1169519933 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98101a7180 con 0x7f98101081e0 2024-03-28T18:09:26.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.713+0000 7f980cc00700 1 -- 172.21.15.37:0/1169519933 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9808004280 con 0x7f98101081e0 2024-03-28T18:09:26.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.713+0000 7f9816a00700 1 -- 172.21.15.37:0/1169519933 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98101a76c0 con 0x7f98101081e0 2024-03-28T18:09:26.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.714+0000 7f9816a00700 1 -- 172.21.15.37:0/1169519933 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9810068e50 con 0x7f98101081e0 2024-03-28T18:09:26.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.717+0000 7f980cc00700 1 -- 172.21.15.37:0/1169519933 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f980801b030 con 0x7f98101081e0 2024-03-28T18:09:26.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.718+0000 7f980cc00700 1 --2- 172.21.15.37:0/1169519933 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f97f40a29a0 0x7f97f40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:26.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.719+0000 7f980cc00700 1 -- 172.21.15.37:0/1169519933 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f98080ce720 con 0x7f98101081e0 2024-03-28T18:09:26.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.719+0000 7f980fe00700 1 --2- 172.21.15.37:0/1169519933 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f97f40a29a0 0x7f97f40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:26.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.719+0000 7f980fe00700 1 --2- 172.21.15.37:0/1169519933 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f97f40a29a0 0x7f97f40a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9800009680 tx=0x7f9800009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:26.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.722+0000 7f980cc00700 1 -- 172.21.15.37:0/1169519933 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9808022020 con 0x7f98101081e0 2024-03-28T18:09:26.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.958+0000 7f9816a00700 1 -- 172.21.15.37:0/1169519933 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f981002da30 con 0x7f98101081e0 2024-03-28T18:09:26.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.959+0000 7f980cc00700 1 -- 172.21.15.37:0/1169519933 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9808010040 con 0x7f98101081e0 2024-03-28T18:09:26.959 INFO:teuthology.orchestra.run.smithi037.stdout:34359738459 2024-03-28T18:09:26.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.964+0000 7f9816a00700 1 -- 172.21.15.37:0/1169519933 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f97f40a29a0 msgr2=0x7f97f40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:26.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.964+0000 7f9816a00700 1 --2- 172.21.15.37:0/1169519933 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f97f40a29a0 0x7f97f40a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9800009680 tx=0x7f9800009260 comp rx=0 tx=0).stop 2024-03-28T18:09:26.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.964+0000 7f9816a00700 1 -- 172.21.15.37:0/1169519933 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98101081e0 msgr2=0x7f9810101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:26.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.964+0000 7f9816a00700 1 --2- 172.21.15.37:0/1169519933 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98101081e0 0x7f9810101be0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f980800efc0 tx=0x7f980800c3a0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.964+0000 7f9816a00700 1 -- 172.21.15.37:0/1169519933 shutdown_connections 2024-03-28T18:09:26.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.964+0000 7f9816a00700 1 --2- 172.21.15.37:0/1169519933 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f97f40a29a0 0x7f97f40a4e60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.964+0000 7f9816a00700 1 --2- 172.21.15.37:0/1169519933 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98101081e0 0x7f9810101be0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.964+0000 7f9816a00700 1 --2- 172.21.15.37:0/1169519933 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9810069e40 0x7f9810101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.964+0000 7f9816a00700 1 --2- 172.21.15.37:0/1169519933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f981006a7e0 0x7f98101016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:26.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.964+0000 7f9816a00700 1 -- 172.21.15.37:0/1169519933 >> 172.21.15.37:0/1169519933 conn(0x7f9810077fc0 msgr2=0x7f98100ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:26.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.964+0000 7f9816a00700 1 -- 172.21.15.37:0/1169519933 shutdown_connections 2024-03-28T18:09:26.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:26.964+0000 7f9816a00700 1 -- 172.21.15.37:0/1169519933 wait complete. 2024-03-28T18:09:27.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:27 smithi037 ceph-mon[31460]: pgmap v353: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 67 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T18:09:27.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:27 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1169519933' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T18:09:27.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:27 smithi059 ceph-mon[36190]: pgmap v353: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 67 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T18:09:27.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:27 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1169519933' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T18:09:27.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:27 smithi170 ceph-mon[36280]: pgmap v353: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 67 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T18:09:27.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:27 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1169519933' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T18:09:27.758 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332142 got 219043332146 for osd.7 2024-03-28T18:09:27.758 DEBUG:teuthology.parallel:result is None 2024-03-28T18:09:29.038 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070503 got 253403070508 for osd.8 2024-03-28T18:09:29.038 DEBUG:teuthology.parallel:result is None 2024-03-28T18:09:29.302 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:29 smithi037 ceph-mon[31460]: pgmap v354: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 57 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T18:09:29.337 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:29 smithi170 ceph-mon[36280]: pgmap v354: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 57 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T18:09:29.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:29 smithi059 ceph-mon[36190]: pgmap v354: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 57 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T18:09:30.207 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738453 got 34359738459 for osd.0 2024-03-28T18:09:30.207 DEBUG:teuthology.parallel:result is None 2024-03-28T18:09:30.432 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:09:30.433 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:09:30.434 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:09:31.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:31 smithi037 ceph-mon[31460]: pgmap v355: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:09:31.336 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:31 smithi059 ceph-mon[36190]: pgmap v355: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:09:31.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:31 smithi170 ceph-mon[36280]: pgmap v355: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:09:31.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.625+0000 7ffa96000700 1 -- 172.21.15.37:0/352688123 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa90100ad0 msgr2=0x7ffa90100f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:31.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.625+0000 7ffa96000700 1 --2- 172.21.15.37:0/352688123 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa90100ad0 0x7ffa90100f50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009990 tx=0x7ffa7c009ca0 comp rx=0 tx=0).stop 2024-03-28T18:09:31.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.625+0000 7ffa96000700 1 -- 172.21.15.37:0/352688123 shutdown_connections 2024-03-28T18:09:31.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.625+0000 7ffa96000700 1 --2- 172.21.15.37:0/352688123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa90101490 0x7ffa9010d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:31.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.625+0000 7ffa96000700 1 --2- 172.21.15.37:0/352688123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa901068b0 0x7ffa90106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.625+0000 7ffa96000700 1 --2- 172.21.15.37:0/352688123 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa90100ad0 0x7ffa90100f50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.625+0000 7ffa96000700 1 -- 172.21.15.37:0/352688123 >> 172.21.15.37:0/352688123 conn(0x7ffa900763d0 msgr2=0x7ffa900767e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.625+0000 7ffa96000700 1 -- 172.21.15.37:0/352688123 shutdown_connections 2024-03-28T18:09:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.625+0000 7ffa96000700 1 -- 172.21.15.37:0/352688123 wait complete. 2024-03-28T18:09:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.626+0000 7ffa96000700 1 Processor -- start 2024-03-28T18:09:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.626+0000 7ffa96000700 1 -- start start 2024-03-28T18:09:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.627+0000 7ffa96000700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa90100ad0 0x7ffa9019e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.627+0000 7ffa96000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa90101490 0x7ffa9019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.627+0000 7ffa96000700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa901068b0 0x7ffa901986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.627+0000 7ffa96000700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa90110350 con 0x7ffa90101490 2024-03-28T18:09:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.627+0000 7ffa96000700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa901101d0 con 0x7ffa90100ad0 2024-03-28T18:09:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.627+0000 7ffa96000700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa901104d0 con 0x7ffa901068b0 2024-03-28T18:09:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.627+0000 7ffa8f400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa90100ad0 0x7ffa9019e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.627+0000 7ffa8fe00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa901068b0 0x7ffa901986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.627+0000 7ffa8f400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa90100ad0 0x7ffa9019e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:36240/0 (socket says 172.21.15.37:36240) 2024-03-28T18:09:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.627+0000 7ffa8f400700 1 -- 172.21.15.37:0/601605901 learned_addr learned my addr 172.21.15.37:0/601605901 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.627+0000 7ffa8ea00700 1 --2- 172.21.15.37:0/601605901 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa90101490 0x7ffa9019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.628+0000 7ffa8f400700 1 -- 172.21.15.37:0/601605901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa901068b0 msgr2=0x7ffa901986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.628+0000 7ffa8f400700 1 --2- 172.21.15.37:0/601605901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa901068b0 0x7ffa901986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.628+0000 7ffa8f400700 1 -- 172.21.15.37:0/601605901 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa90101490 msgr2=0x7ffa9019eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.628+0000 7ffa8f400700 1 --2- 172.21.15.37:0/601605901 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa90101490 0x7ffa9019eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.628+0000 7ffa8f400700 1 -- 172.21.15.37:0/601605901 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa90198fc0 con 0x7ffa90100ad0 2024-03-28T18:09:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.628+0000 7ffa8fe00700 1 --2- 172.21.15.37:0/601605901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa901068b0 0x7ffa901986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:09:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.628+0000 7ffa8ea00700 1 --2- 172.21.15.37:0/601605901 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa90101490 0x7ffa9019eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:09:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.628+0000 7ffa8f400700 1 --2- 172.21.15.37:0/601605901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa90100ad0 0x7ffa9019e610 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffa8000eab0 tx=0x7ffa8000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.628+0000 7ffa87e00700 1 -- 172.21.15.37:0/601605901 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8001b070 con 0x7ffa90100ad0 2024-03-28T18:09:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.629+0000 7ffa87e00700 1 -- 172.21.15.37:0/601605901 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa80016410 con 0x7ffa90100ad0 2024-03-28T18:09:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.629+0000 7ffa87e00700 1 -- 172.21.15.37:0/601605901 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8000b5a0 con 0x7ffa90100ad0 2024-03-28T18:09:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.629+0000 7ffa96000700 1 -- 172.21.15.37:0/601605901 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa901992b0 con 0x7ffa90100ad0 2024-03-28T18:09:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.629+0000 7ffa96000700 1 -- 172.21.15.37:0/601605901 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa901a5590 con 0x7ffa90100ad0 2024-03-28T18:09:31.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.630+0000 7ffa96000700 1 -- 172.21.15.37:0/601605901 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa90101f40 con 0x7ffa90100ad0 2024-03-28T18:09:31.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.632+0000 7ffa87e00700 1 -- 172.21.15.37:0/601605901 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7ffa8001e070 con 0x7ffa90100ad0 2024-03-28T18:09:31.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.633+0000 7ffa87e00700 1 --2- 172.21.15.37:0/601605901 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ffa740a29a0 0x7ffa740a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:31.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.633+0000 7ffa87e00700 1 -- 172.21.15.37:0/601605901 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ffa800ce610 con 0x7ffa90100ad0 2024-03-28T18:09:31.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.633+0000 7ffa8ea00700 1 --2- 172.21.15.37:0/601605901 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ffa740a29a0 0x7ffa740a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:31.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.634+0000 7ffa8ea00700 1 --2- 172.21.15.37:0/601605901 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ffa740a29a0 0x7ffa740a4e60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00b2e0 tx=0x7ffa7c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:31.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.637+0000 7ffa87e00700 1 -- 172.21.15.37:0/601605901 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffa8009b500 con 0x7ffa90100ad0 2024-03-28T18:09:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.875+0000 7ffa96000700 1 -- 172.21.15.37:0/601605901 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7ffa90199c10 con 0x7ffa90100ad0 2024-03-28T18:09:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.876+0000 7ffa87e00700 1 -- 172.21.15.37:0/601605901 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ffa8009ae00 con 0x7ffa90100ad0 2024-03-28T18:09:31.876 INFO:teuthology.orchestra.run.smithi037.stdout:163208757310 2024-03-28T18:09:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.879+0000 7ffa96000700 1 -- 172.21.15.37:0/601605901 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ffa740a29a0 msgr2=0x7ffa740a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.879+0000 7ffa96000700 1 --2- 172.21.15.37:0/601605901 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ffa740a29a0 0x7ffa740a4e60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00b2e0 tx=0x7ffa7c01a040 comp rx=0 tx=0).stop 2024-03-28T18:09:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.879+0000 7ffa96000700 1 -- 172.21.15.37:0/601605901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa90100ad0 msgr2=0x7ffa9019e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.879+0000 7ffa96000700 1 --2- 172.21.15.37:0/601605901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa90100ad0 0x7ffa9019e610 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffa8000eab0 tx=0x7ffa8000edc0 comp rx=0 tx=0).stop 2024-03-28T18:09:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.879+0000 7ffa96000700 1 -- 172.21.15.37:0/601605901 shutdown_connections 2024-03-28T18:09:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.879+0000 7ffa96000700 1 --2- 172.21.15.37:0/601605901 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ffa740a29a0 0x7ffa740a4e60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.879+0000 7ffa96000700 1 --2- 172.21.15.37:0/601605901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa901068b0 0x7ffa901986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.879+0000 7ffa96000700 1 --2- 172.21.15.37:0/601605901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa90100ad0 0x7ffa9019e610 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.879+0000 7ffa96000700 1 --2- 172.21.15.37:0/601605901 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa90101490 0x7ffa9019eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.879+0000 7ffa96000700 1 -- 172.21.15.37:0/601605901 >> 172.21.15.37:0/601605901 conn(0x7ffa900763d0 msgr2=0x7ffa900fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.879+0000 7ffa96000700 1 -- 172.21.15.37:0/601605901 shutdown_connections 2024-03-28T18:09:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:31.879+0000 7ffa96000700 1 -- 172.21.15.37:0/601605901 wait complete. 2024-03-28T18:09:32.349 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:32 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/601605901' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T18:09:32.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:32 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/601605901' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T18:09:32.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:32 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/601605901' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T18:09:33.214 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:09:33.215 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:09:33.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.275+0000 7f47d9c00700 1 -- 172.21.15.37:0/243136661 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47d406a760 msgr2=0x7f47d410e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:33.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.275+0000 7f47d9c00700 1 --2- 172.21.15.37:0/243136661 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47d406a760 0x7f47d410e700 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f47c0009990 tx=0x7f47c0009ca0 comp rx=0 tx=0).stop 2024-03-28T18:09:33.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.275+0000 7f47d9c00700 1 -- 172.21.15.37:0/243136661 shutdown_connections 2024-03-28T18:09:33.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.275+0000 7f47d9c00700 1 --2- 172.21.15.37:0/243136661 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f47d410ec40 0x7f47d4111030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:33.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.275+0000 7f47d9c00700 1 --2- 172.21.15.37:0/243136661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47d4069e40 0x7f47d406a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:33.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.275+0000 7f47d9c00700 1 --2- 172.21.15.37:0/243136661 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47d406a760 0x7f47d410e700 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:33.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.275+0000 7f47d9c00700 1 -- 172.21.15.37:0/243136661 >> 172.21.15.37:0/243136661 conn(0x7f47d4077fc0 msgr2=0x7f47d40783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:33.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.275+0000 7f47d9c00700 1 -- 172.21.15.37:0/243136661 shutdown_connections 2024-03-28T18:09:33.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.275+0000 7f47d9c00700 1 -- 172.21.15.37:0/243136661 wait complete. 2024-03-28T18:09:33.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.276+0000 7f47d9c00700 1 Processor -- start 2024-03-28T18:09:33.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.276+0000 7f47d9c00700 1 -- start start 2024-03-28T18:09:33.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.277+0000 7f47d9c00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f47d4069e40 0x7f47d41a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:33.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.277+0000 7f47d9c00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47d406a760 0x7f47d41a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:33.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.277+0000 7f47d9c00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47d410ec40 0x7f47d419aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:33.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.277+0000 7f47d9c00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d41146b0 con 0x7f47d4069e40 2024-03-28T18:09:33.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.277+0000 7f47d9c00700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d4114530 con 0x7f47d406a760 2024-03-28T18:09:33.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.277+0000 7f47d9c00700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d4114830 con 0x7f47d410ec40 2024-03-28T18:09:33.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.277+0000 7f47d2000700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47d406a760 0x7f47d41a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:33.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.277+0000 7f47d2a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f47d4069e40 0x7f47d41a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:33.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.277+0000 7f47d2000700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47d406a760 0x7f47d41a0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:36252/0 (socket says 172.21.15.37:36252) 2024-03-28T18:09:33.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.277+0000 7f47d3400700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47d410ec40 0x7f47d419aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:33.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.277+0000 7f47d2000700 1 -- 172.21.15.37:0/680160147 learned_addr learned my addr 172.21.15.37:0/680160147 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:33.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.278+0000 7f47d2000700 1 -- 172.21.15.37:0/680160147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47d410ec40 msgr2=0x7f47d419aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:33.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.278+0000 7f47d2000700 1 --2- 172.21.15.37:0/680160147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47d410ec40 0x7f47d419aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:33.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.278+0000 7f47d2000700 1 -- 172.21.15.37:0/680160147 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f47d4069e40 msgr2=0x7f47d41a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:33.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.278+0000 7f47d2000700 1 --2- 172.21.15.37:0/680160147 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f47d4069e40 0x7f47d41a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:33.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.278+0000 7f47d2000700 1 -- 172.21.15.37:0/680160147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d419b190 con 0x7f47d406a760 2024-03-28T18:09:33.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.278+0000 7f47d2000700 1 --2- 172.21.15.37:0/680160147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47d406a760 0x7f47d41a0db0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f47c0005910 tx=0x7f47c000bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:33.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.278+0000 7f47cb400700 1 -- 172.21.15.37:0/680160147 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c0027070 con 0x7f47d406a760 2024-03-28T18:09:33.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.278+0000 7f47cb400700 1 -- 172.21.15.37:0/680160147 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47c0004650 con 0x7f47d406a760 2024-03-28T18:09:33.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.278+0000 7f47cb400700 1 -- 172.21.15.37:0/680160147 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c000f460 con 0x7f47d406a760 2024-03-28T18:09:33.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.278+0000 7f47d9c00700 1 -- 172.21.15.37:0/680160147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d419b420 con 0x7f47d406a760 2024-03-28T18:09:33.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.279+0000 7f47d9c00700 1 -- 172.21.15.37:0/680160147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d41a7740 con 0x7f47d406a760 2024-03-28T18:09:33.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.280+0000 7f47d9c00700 1 -- 172.21.15.37:0/680160147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d4068e50 con 0x7f47d406a760 2024-03-28T18:09:33.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.281+0000 7f47cb400700 1 -- 172.21.15.37:0/680160147 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f47c002a070 con 0x7f47d406a760 2024-03-28T18:09:33.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.284+0000 7f47cb400700 1 --2- 172.21.15.37:0/680160147 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f47b80a29a0 0x7f47b80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:33.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.284+0000 7f47cb400700 1 -- 172.21.15.37:0/680160147 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f47c00cfa60 con 0x7f47d406a760 2024-03-28T18:09:33.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.284+0000 7f47d2a00700 1 --2- 172.21.15.37:0/680160147 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f47b80a29a0 0x7f47b80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:33.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.285+0000 7f47d2a00700 1 --2- 172.21.15.37:0/680160147 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f47b80a29a0 0x7f47b80a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f47d419bb70 tx=0x7f47c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:33.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.292+0000 7f47cb400700 1 -- 172.21.15.37:0/680160147 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f47c005e740 con 0x7f47d406a760 2024-03-28T18:09:33.325 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:33 smithi037 ceph-mon[31460]: pgmap v356: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:09:33.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:33 smithi059 ceph-mon[36190]: pgmap v356: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:09:33.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:33 smithi170 ceph-mon[36280]: pgmap v356: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:09:33.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.550+0000 7f47d9c00700 1 -- 172.21.15.37:0/680160147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f47d402da30 con 0x7f47d406a760 2024-03-28T18:09:33.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.551+0000 7f47cb400700 1 -- 172.21.15.37:0/680160147 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f47c0013070 con 0x7f47d406a760 2024-03-28T18:09:33.551 INFO:teuthology.orchestra.run.smithi037.stdout:107374182474 2024-03-28T18:09:33.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.554+0000 7f47d9c00700 1 -- 172.21.15.37:0/680160147 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f47b80a29a0 msgr2=0x7f47b80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:33.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.554+0000 7f47d9c00700 1 --2- 172.21.15.37:0/680160147 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f47b80a29a0 0x7f47b80a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f47d419bb70 tx=0x7f47c4009260 comp rx=0 tx=0).stop 2024-03-28T18:09:33.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.554+0000 7f47d9c00700 1 -- 172.21.15.37:0/680160147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47d406a760 msgr2=0x7f47d41a0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:33.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.554+0000 7f47d9c00700 1 --2- 172.21.15.37:0/680160147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47d406a760 0x7f47d41a0db0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f47c0005910 tx=0x7f47c000bab0 comp rx=0 tx=0).stop 2024-03-28T18:09:33.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.554+0000 7f47d9c00700 1 -- 172.21.15.37:0/680160147 shutdown_connections 2024-03-28T18:09:33.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.554+0000 7f47d9c00700 1 --2- 172.21.15.37:0/680160147 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f47b80a29a0 0x7f47b80a4e60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:33.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.554+0000 7f47d9c00700 1 --2- 172.21.15.37:0/680160147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47d410ec40 0x7f47d419aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:33.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.554+0000 7f47d9c00700 1 --2- 172.21.15.37:0/680160147 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f47d4069e40 0x7f47d41a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:33.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.554+0000 7f47d9c00700 1 --2- 172.21.15.37:0/680160147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47d406a760 0x7f47d41a0db0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:33.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.554+0000 7f47d9c00700 1 -- 172.21.15.37:0/680160147 >> 172.21.15.37:0/680160147 conn(0x7f47d4077fc0 msgr2=0x7f47d41100f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:33.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.554+0000 7f47d9c00700 1 -- 172.21.15.37:0/680160147 shutdown_connections 2024-03-28T18:09:33.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:33.555+0000 7f47d9c00700 1 -- 172.21.15.37:0/680160147 wait complete. 2024-03-28T18:09:34.277 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:34 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/680160147' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T18:09:34.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:34 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/680160147' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T18:09:34.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:34 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/680160147' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T18:09:34.845 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:09:35.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:35 smithi037 ceph-mon[31460]: pgmap v357: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:35.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:35 smithi059 ceph-mon[36190]: pgmap v357: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:35.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:35 smithi170 ceph-mon[36280]: pgmap v357: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:35.647 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182471 got 107374182474 for osd.3 2024-03-28T18:09:35.647 DEBUG:teuthology.parallel:result is None 2024-03-28T18:09:37.250 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:37 smithi037 ceph-mon[31460]: pgmap v358: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:09:37.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:37 smithi059 ceph-mon[36190]: pgmap v358: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:09:37.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:37 smithi170 ceph-mon[36280]: pgmap v358: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:09:37.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.701+0000 7fb3cfa00700 1 -- 172.21.15.37:0/1025813859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c81023d0 msgr2=0x7fb3c81027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:37.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.701+0000 7fb3cfa00700 1 --2- 172.21.15.37:0/1025813859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c81023d0 0x7fb3c81027b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc009990 tx=0x7fb3bc009ca0 comp rx=0 tx=0).stop 2024-03-28T18:09:37.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.701+0000 7fb3cfa00700 1 -- 172.21.15.37:0/1025813859 shutdown_connections 2024-03-28T18:09:37.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.701+0000 7fb3cfa00700 1 --2- 172.21.15.37:0/1025813859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3c81074b0 0x7fb3c810f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:37.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.701+0000 7fb3cfa00700 1 --2- 172.21.15.37:0/1025813859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c81023d0 0x7fb3c81027b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:37.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.701+0000 7fb3cfa00700 1 --2- 172.21.15.37:0/1025813859 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c8102d80 0x7fb3c8106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:37.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.701+0000 7fb3cfa00700 1 -- 172.21.15.37:0/1025813859 >> 172.21.15.37:0/1025813859 conn(0x7fb3c80fdc40 msgr2=0x7fb3c8100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:37.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.701+0000 7fb3cfa00700 1 -- 172.21.15.37:0/1025813859 shutdown_connections 2024-03-28T18:09:37.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.702+0000 7fb3cfa00700 1 -- 172.21.15.37:0/1025813859 wait complete. 2024-03-28T18:09:37.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.703+0000 7fb3cfa00700 1 Processor -- start 2024-03-28T18:09:37.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.703+0000 7fb3cfa00700 1 -- start start 2024-03-28T18:09:37.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.704+0000 7fb3cfa00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3c81023d0 0x7fb3c81a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:37.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.704+0000 7fb3cfa00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c8102d80 0x7fb3c81a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:37.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.704+0000 7fb3cfa00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c81074b0 0x7fb3c819a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:37.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.704+0000 7fb3cfa00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c81124e0 con 0x7fb3c8102d80 2024-03-28T18:09:37.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.704+0000 7fb3cfa00700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c8112360 con 0x7fb3c81074b0 2024-03-28T18:09:37.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.704+0000 7fb3cfa00700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c8112660 con 0x7fb3c81023d0 2024-03-28T18:09:37.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.704+0000 7fb3cd000700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3c81023d0 0x7fb3c81a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:37.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.704+0000 7fb3cd000700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3c81023d0 0x7fb3c81a07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:41368/0 (socket says 172.21.15.37:41368) 2024-03-28T18:09:37.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.704+0000 7fb3cd000700 1 -- 172.21.15.37:0/231381818 learned_addr learned my addr 172.21.15.37:0/231381818 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:37.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.704+0000 7fb3cda00700 1 --2- 172.21.15.37:0/231381818 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c81074b0 0x7fb3c819a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:37.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.704+0000 7fb3c7e00700 1 --2- 172.21.15.37:0/231381818 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c8102d80 0x7fb3c81a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:37.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.704+0000 7fb3cd000700 1 -- 172.21.15.37:0/231381818 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c81074b0 msgr2=0x7fb3c819a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:37.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.704+0000 7fb3cd000700 1 --2- 172.21.15.37:0/231381818 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c81074b0 0x7fb3c819a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:37.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.704+0000 7fb3cd000700 1 -- 172.21.15.37:0/231381818 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c8102d80 msgr2=0x7fb3c81a0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:37.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.704+0000 7fb3cd000700 1 --2- 172.21.15.37:0/231381818 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c8102d80 0x7fb3c81a0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:37.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.704+0000 7fb3cd000700 1 -- 172.21.15.37:0/231381818 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c819b110 con 0x7fb3c81023d0 2024-03-28T18:09:37.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.704+0000 7fb3c7e00700 1 --2- 172.21.15.37:0/231381818 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c8102d80 0x7fb3c81a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:09:37.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.704+0000 7fb3cda00700 1 --2- 172.21.15.37:0/231381818 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c81074b0 0x7fb3c819a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:09:37.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.705+0000 7fb3cd000700 1 --2- 172.21.15.37:0/231381818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3c81023d0 0x7fb3c81a07f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc005910 tx=0x7fb3bc00b6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:37.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.705+0000 7fb3c5600700 1 -- 172.21.15.37:0/231381818 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc027070 con 0x7fb3c81023d0 2024-03-28T18:09:37.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.705+0000 7fb3c5600700 1 -- 172.21.15.37:0/231381818 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3bc022470 con 0x7fb3c81023d0 2024-03-28T18:09:37.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.705+0000 7fb3cfa00700 1 -- 172.21.15.37:0/231381818 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c819b3a0 con 0x7fb3c81023d0 2024-03-28T18:09:37.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.705+0000 7fb3cfa00700 1 -- 172.21.15.37:0/231381818 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c81a76c0 con 0x7fb3c81023d0 2024-03-28T18:09:37.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.705+0000 7fb3c5600700 1 -- 172.21.15.37:0/231381818 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc017650 con 0x7fb3c81023d0 2024-03-28T18:09:37.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.707+0000 7fb3cfa00700 1 -- 172.21.15.37:0/231381818 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c8068e50 con 0x7fb3c81023d0 2024-03-28T18:09:37.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.708+0000 7fb3c5600700 1 -- 172.21.15.37:0/231381818 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fb3bc02a070 con 0x7fb3c81023d0 2024-03-28T18:09:37.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.710+0000 7fb3c5600700 1 --2- 172.21.15.37:0/231381818 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb3b00a2920 0x7fb3b00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:37.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.710+0000 7fb3c5600700 1 -- 172.21.15.37:0/231381818 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb3bc0cf8e0 con 0x7fb3c81023d0 2024-03-28T18:09:37.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.710+0000 7fb3c7e00700 1 --2- 172.21.15.37:0/231381818 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb3b00a2920 0x7fb3b00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:37.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.711+0000 7fb3c7e00700 1 --2- 172.21.15.37:0/231381818 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb3b00a2920 0x7fb3b00a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb3c80ff440 tx=0x7fb3b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:37.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.712+0000 7fb3c5600700 1 -- 172.21.15.37:0/231381818 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb3bc013070 con 0x7fb3c81023d0 2024-03-28T18:09:37.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.988+0000 7fb3cfa00700 1 -- 172.21.15.37:0/231381818 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7fb3c81a7890 con 0x7fb3c81023d0 2024-03-28T18:09:37.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.988+0000 7fb3c5600700 1 -- 172.21.15.37:0/231381818 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb3bc09c860 con 0x7fb3c81023d0 2024-03-28T18:09:37.988 INFO:teuthology.orchestra.run.smithi037.stdout:287762808872 2024-03-28T18:09:37.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.991+0000 7fb3cfa00700 1 -- 172.21.15.37:0/231381818 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb3b00a2920 msgr2=0x7fb3b00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:37.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.991+0000 7fb3cfa00700 1 --2- 172.21.15.37:0/231381818 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb3b00a2920 0x7fb3b00a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb3c80ff440 tx=0x7fb3b4009260 comp rx=0 tx=0).stop 2024-03-28T18:09:37.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.991+0000 7fb3cfa00700 1 -- 172.21.15.37:0/231381818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3c81023d0 msgr2=0x7fb3c81a07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:37.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.991+0000 7fb3cfa00700 1 --2- 172.21.15.37:0/231381818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3c81023d0 0x7fb3c81a07f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc005910 tx=0x7fb3bc00b6c0 comp rx=0 tx=0).stop 2024-03-28T18:09:37.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.991+0000 7fb3cfa00700 1 -- 172.21.15.37:0/231381818 shutdown_connections 2024-03-28T18:09:37.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.991+0000 7fb3cfa00700 1 --2- 172.21.15.37:0/231381818 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb3b00a2920 0x7fb3b00a4de0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:37.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.991+0000 7fb3cfa00700 1 --2- 172.21.15.37:0/231381818 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c81074b0 0x7fb3c819a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:37.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.991+0000 7fb3cfa00700 1 --2- 172.21.15.37:0/231381818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3c81023d0 0x7fb3c81a07f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:37.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.991+0000 7fb3cfa00700 1 --2- 172.21.15.37:0/231381818 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c8102d80 0x7fb3c81a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:37.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.991+0000 7fb3cfa00700 1 -- 172.21.15.37:0/231381818 >> 172.21.15.37:0/231381818 conn(0x7fb3c80fdc40 msgr2=0x7fb3c810e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:37.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.991+0000 7fb3cfa00700 1 -- 172.21.15.37:0/231381818 shutdown_connections 2024-03-28T18:09:37.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:37.992+0000 7fb3cfa00700 1 -- 172.21.15.37:0/231381818 wait complete. 2024-03-28T18:09:38.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:38 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/231381818' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T18:09:38.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:38 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/231381818' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T18:09:38.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:38 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/231381818' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T18:09:39.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:39 smithi037 ceph-mon[31460]: pgmap v359: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:39.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:39 smithi059 ceph-mon[36190]: pgmap v359: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:39.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:39 smithi170 ceph-mon[36280]: pgmap v359: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:40.715 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757306 got 163208757310 for osd.5 2024-03-28T18:09:40.715 DEBUG:teuthology.parallel:result is None 2024-03-28T18:09:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:41 smithi037 ceph-mon[31460]: pgmap v360: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:09:41.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:41 smithi059 ceph-mon[36190]: pgmap v360: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:09:41.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:41 smithi170 ceph-mon[36280]: pgmap v360: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:09:42.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.397+0000 7f202d200700 1 -- 172.21.15.37:0/3425268323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f202806a7e0 msgr2=0x7f202810f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:42.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.397+0000 7f202d200700 1 --2- 172.21.15.37:0/3425268323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f202806a7e0 0x7f202810f020 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f202000b0a0 tx=0x7f202000b3b0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.397+0000 7f202d200700 1 -- 172.21.15.37:0/3425268323 shutdown_connections 2024-03-28T18:09:42.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.397+0000 7f202d200700 1 --2- 172.21.15.37:0/3425268323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f202806a7e0 0x7f202810f020 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.397+0000 7f202d200700 1 --2- 172.21.15.37:0/3425268323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20281081e0 0x7f20281085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.397+0000 7f202d200700 1 --2- 172.21.15.37:0/3425268323 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2028069e40 0x7f202806a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.397+0000 7f202d200700 1 -- 172.21.15.37:0/3425268323 >> 172.21.15.37:0/3425268323 conn(0x7f2028077fc0 msgr2=0x7f20280783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:42.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.397+0000 7f202d200700 1 -- 172.21.15.37:0/3425268323 shutdown_connections 2024-03-28T18:09:42.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.397+0000 7f202d200700 1 -- 172.21.15.37:0/3425268323 wait complete. 2024-03-28T18:09:42.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.398+0000 7f202d200700 1 Processor -- start 2024-03-28T18:09:42.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.399+0000 7f202d200700 1 -- start start 2024-03-28T18:09:42.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.399+0000 7f202d200700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2028069e40 0x7f20281a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:42.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.399+0000 7f202d200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f202806a7e0 0x7f20281a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:42.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.399+0000 7f202d200700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20281081e0 0x7f202819a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:42.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.399+0000 7f202d200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2028112460 con 0x7f202806a7e0 2024-03-28T18:09:42.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.399+0000 7f202d200700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20281122e0 con 0x7f2028069e40 2024-03-28T18:09:42.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.399+0000 7f202d200700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20281125e0 con 0x7f20281081e0 2024-03-28T18:09:42.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.399+0000 7f2025e00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2028069e40 0x7f20281a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:42.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.399+0000 7f2025e00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2028069e40 0x7f20281a0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:45658/0 (socket says 172.21.15.37:45658) 2024-03-28T18:09:42.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.399+0000 7f2025e00700 1 -- 172.21.15.37:0/3731451006 learned_addr learned my addr 172.21.15.37:0/3731451006 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:42.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.399+0000 7f2026800700 1 --2- 172.21.15.37:0/3731451006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20281081e0 0x7f202819a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.400+0000 7f2025400700 1 --2- 172.21.15.37:0/3731451006 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f202806a7e0 0x7f20281a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.400+0000 7f2025e00700 1 -- 172.21.15.37:0/3731451006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20281081e0 msgr2=0x7f202819a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.400+0000 7f2025e00700 1 --2- 172.21.15.37:0/3731451006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20281081e0 0x7f202819a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.400+0000 7f2025e00700 1 -- 172.21.15.37:0/3731451006 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f202806a7e0 msgr2=0x7f20281a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.400+0000 7f2025e00700 1 --2- 172.21.15.37:0/3731451006 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f202806a7e0 0x7f20281a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.400+0000 7f2025e00700 1 -- 172.21.15.37:0/3731451006 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f202819b030 con 0x7f2028069e40 2024-03-28T18:09:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.400+0000 7f2026800700 1 --2- 172.21.15.37:0/3731451006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20281081e0 0x7f202819a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:09:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.400+0000 7f2025400700 1 --2- 172.21.15.37:0/3731451006 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f202806a7e0 0x7f20281a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:09:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.400+0000 7f2025e00700 1 --2- 172.21.15.37:0/3731451006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2028069e40 0x7f20281a0710 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f201800e9c0 tx=0x7f201800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.400+0000 7f201ea00700 1 -- 172.21.15.37:0/3731451006 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201800cbd0 con 0x7f2028069e40 2024-03-28T18:09:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.400+0000 7f201ea00700 1 -- 172.21.15.37:0/3731451006 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f201800cd70 con 0x7f2028069e40 2024-03-28T18:09:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.400+0000 7f201ea00700 1 -- 172.21.15.37:0/3731451006 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2018010430 con 0x7f2028069e40 2024-03-28T18:09:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.400+0000 7f202d200700 1 -- 172.21.15.37:0/3731451006 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f202819b320 con 0x7f2028069e40 2024-03-28T18:09:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.401+0000 7f202d200700 1 -- 172.21.15.37:0/3731451006 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20281a75e0 con 0x7f2028069e40 2024-03-28T18:09:42.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.402+0000 7f202d200700 1 -- 172.21.15.37:0/3731451006 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2028068e50 con 0x7f2028069e40 2024-03-28T18:09:42.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.405+0000 7f201ea00700 1 -- 172.21.15.37:0/3731451006 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f2018004750 con 0x7f2028069e40 2024-03-28T18:09:42.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.406+0000 7f201ea00700 1 --2- 172.21.15.37:0/3731451006 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f200c0a29a0 0x7f200c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:42.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.407+0000 7f201ea00700 1 -- 172.21.15.37:0/3731451006 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f20180cee10 con 0x7f2028069e40 2024-03-28T18:09:42.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.407+0000 7f2025400700 1 --2- 172.21.15.37:0/3731451006 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f200c0a29a0 0x7f200c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:42.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.408+0000 7f2025400700 1 --2- 172.21.15.37:0/3731451006 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f200c0a29a0 0x7f200c0a4e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f20281030e0 tx=0x7f2014009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:42.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.411+0000 7f201ea00700 1 -- 172.21.15.37:0/3731451006 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f20180d2030 con 0x7f2028069e40 2024-03-28T18:09:42.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.629+0000 7f932a800700 1 -- 172.21.15.37:0/854670369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9324074ce0 msgr2=0x7f9324112e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:42.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.629+0000 7f932a800700 1 --2- 172.21.15.37:0/854670369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9324074ce0 0x7f9324112e20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9310009990 tx=0x7f9310009ca0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.629+0000 7f932a800700 1 -- 172.21.15.37:0/854670369 shutdown_connections 2024-03-28T18:09:42.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.629+0000 7f932a800700 1 --2- 172.21.15.37:0/854670369 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9324113360 0x7f9324115750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.629+0000 7f932a800700 1 --2- 172.21.15.37:0/854670369 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93240743c0 0x7f93240747a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.629+0000 7f932a800700 1 --2- 172.21.15.37:0/854670369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9324074ce0 0x7f9324112e20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.629+0000 7f932a800700 1 -- 172.21.15.37:0/854670369 >> 172.21.15.37:0/854670369 conn(0x7f93240792f0 msgr2=0x7f9324079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:42.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.629+0000 7f932a800700 1 -- 172.21.15.37:0/854670369 shutdown_connections 2024-03-28T18:09:42.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.629+0000 7f932a800700 1 -- 172.21.15.37:0/854670369 wait complete. 2024-03-28T18:09:42.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.631+0000 7f932a800700 1 Processor -- start 2024-03-28T18:09:42.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.631+0000 7f932a800700 1 -- start start 2024-03-28T18:09:42.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.632+0000 7f932a800700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93240743c0 0x7f932419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:42.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.632+0000 7f932a800700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9324074ce0 0x7f932419f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:42.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.632+0000 7f932a800700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9324113360 0x7f93241a38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:42.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.632+0000 7f932a800700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9324118ba0 con 0x7f9324113360 2024-03-28T18:09:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.632+0000 7f932a800700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9324118a20 con 0x7f93240743c0 2024-03-28T18:09:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.632+0000 7f932a800700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9324118d20 con 0x7f9324074ce0 2024-03-28T18:09:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.632+0000 7f9323400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93240743c0 0x7f932419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.632+0000 7f9323400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93240743c0 0x7f932419efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:45676/0 (socket says 172.21.15.37:45676) 2024-03-28T18:09:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.632+0000 7f9323400700 1 -- 172.21.15.37:0/2158566836 learned_addr learned my addr 172.21.15.37:0/2158566836 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.632+0000 7f9322a00700 1 --2- 172.21.15.37:0/2158566836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9324074ce0 0x7f932419f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.633+0000 7f9323400700 1 -- 172.21.15.37:0/2158566836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9324074ce0 msgr2=0x7f932419f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.633+0000 7f9323400700 1 --2- 172.21.15.37:0/2158566836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9324074ce0 0x7f932419f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.633+0000 7f9323400700 1 -- 172.21.15.37:0/2158566836 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9324113360 msgr2=0x7f93241a38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.633+0000 7f9323400700 1 --2- 172.21.15.37:0/2158566836 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9324113360 0x7f93241a38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.633+0000 7f9323400700 1 -- 172.21.15.37:0/2158566836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93241a3fe0 con 0x7f93240743c0 2024-03-28T18:09:42.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.633+0000 7f9323400700 1 --2- 172.21.15.37:0/2158566836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93240743c0 0x7f932419efd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f931400e9a0 tx=0x7f931400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:42.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.633+0000 7f931be00700 1 -- 172.21.15.37:0/2158566836 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931400cb50 con 0x7f93240743c0 2024-03-28T18:09:42.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.633+0000 7f931be00700 1 -- 172.21.15.37:0/2158566836 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f931400ccf0 con 0x7f93240743c0 2024-03-28T18:09:42.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.633+0000 7f931be00700 1 -- 172.21.15.37:0/2158566836 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9314018750 con 0x7f93240743c0 2024-03-28T18:09:42.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.633+0000 7f932a800700 1 -- 172.21.15.37:0/2158566836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93241a42d0 con 0x7f93240743c0 2024-03-28T18:09:42.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.634+0000 7f932a800700 1 -- 172.21.15.37:0/2158566836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93241abb50 con 0x7f93240743c0 2024-03-28T18:09:42.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.635+0000 7f932a800700 1 -- 172.21.15.37:0/2158566836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9324068e50 con 0x7f93240743c0 2024-03-28T18:09:42.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.638+0000 7f931be00700 1 -- 172.21.15.37:0/2158566836 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f93140109a0 con 0x7f93240743c0 2024-03-28T18:09:42.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.639+0000 7f931be00700 1 --2- 172.21.15.37:0/2158566836 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f93080a2920 0x7f93080a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:42.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.639+0000 7f931be00700 1 -- 172.21.15.37:0/2158566836 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f93140cef80 con 0x7f93240743c0 2024-03-28T18:09:42.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.640+0000 7f9322a00700 1 --2- 172.21.15.37:0/2158566836 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f93080a2920 0x7f93080a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:42.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.640+0000 7f9322a00700 1 --2- 172.21.15.37:0/2158566836 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f93080a2920 0x7f93080a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f931000b2e0 tx=0x7f931001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:42.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.643+0000 7f931be00700 1 -- 172.21.15.37:0/2158566836 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f931409be70 con 0x7f93240743c0 2024-03-28T18:09:42.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.675+0000 7f202d200700 1 -- 172.21.15.37:0/3731451006 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f20281a78d0 con 0x7f2028069e40 2024-03-28T18:09:42.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.676+0000 7f201ea00700 1 -- 172.21.15.37:0/3731451006 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f201809bd00 con 0x7f2028069e40 2024-03-28T18:09:42.676 INFO:teuthology.orchestra.run.smithi037.stdout:77309411410 2024-03-28T18:09:42.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.678+0000 7f202d200700 1 -- 172.21.15.37:0/3731451006 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f200c0a29a0 msgr2=0x7f200c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:42.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.679+0000 7f202d200700 1 --2- 172.21.15.37:0/3731451006 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f200c0a29a0 0x7f200c0a4e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f20281030e0 tx=0x7f2014009260 comp rx=0 tx=0).stop 2024-03-28T18:09:42.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.679+0000 7f202d200700 1 -- 172.21.15.37:0/3731451006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2028069e40 msgr2=0x7f20281a0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:42.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.679+0000 7f202d200700 1 --2- 172.21.15.37:0/3731451006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2028069e40 0x7f20281a0710 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f201800e9c0 tx=0x7f201800ecd0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.679+0000 7f202d200700 1 -- 172.21.15.37:0/3731451006 shutdown_connections 2024-03-28T18:09:42.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.679+0000 7f202d200700 1 --2- 172.21.15.37:0/3731451006 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f200c0a29a0 0x7f200c0a4e60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.679+0000 7f202d200700 1 --2- 172.21.15.37:0/3731451006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20281081e0 0x7f202819a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.679+0000 7f202d200700 1 --2- 172.21.15.37:0/3731451006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2028069e40 0x7f20281a0710 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.679+0000 7f202d200700 1 --2- 172.21.15.37:0/3731451006 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f202806a7e0 0x7f20281a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.679+0000 7f202d200700 1 -- 172.21.15.37:0/3731451006 >> 172.21.15.37:0/3731451006 conn(0x7f2028077fc0 msgr2=0x7f202810d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:42.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.679+0000 7f202d200700 1 -- 172.21.15.37:0/3731451006 shutdown_connections 2024-03-28T18:09:42.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.679+0000 7f202d200700 1 -- 172.21.15.37:0/3731451006 wait complete. 2024-03-28T18:09:42.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.897+0000 7f932a800700 1 -- 172.21.15.37:0/2158566836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f93241abd20 con 0x7f93240743c0 2024-03-28T18:09:42.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.897+0000 7f931be00700 1 -- 172.21.15.37:0/2158566836 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f931409b770 con 0x7f93240743c0 2024-03-28T18:09:42.898 INFO:teuthology.orchestra.run.smithi037.stdout:197568495675 2024-03-28T18:09:42.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.901+0000 7f932a800700 1 -- 172.21.15.37:0/2158566836 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f93080a2920 msgr2=0x7f93080a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:42.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.901+0000 7f932a800700 1 --2- 172.21.15.37:0/2158566836 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f93080a2920 0x7f93080a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f931000b2e0 tx=0x7f931001a040 comp rx=0 tx=0).stop 2024-03-28T18:09:42.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.901+0000 7f932a800700 1 -- 172.21.15.37:0/2158566836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93240743c0 msgr2=0x7f932419efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:42.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.901+0000 7f932a800700 1 --2- 172.21.15.37:0/2158566836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93240743c0 0x7f932419efd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f931400e9a0 tx=0x7f931400ed60 comp rx=0 tx=0).stop 2024-03-28T18:09:42.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.901+0000 7f932a800700 1 -- 172.21.15.37:0/2158566836 shutdown_connections 2024-03-28T18:09:42.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.901+0000 7f932a800700 1 --2- 172.21.15.37:0/2158566836 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f93080a2920 0x7f93080a4de0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.901+0000 7f932a800700 1 --2- 172.21.15.37:0/2158566836 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9324113360 0x7f93241a38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.901+0000 7f932a800700 1 --2- 172.21.15.37:0/2158566836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93240743c0 0x7f932419efd0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.901+0000 7f932a800700 1 --2- 172.21.15.37:0/2158566836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9324074ce0 0x7f932419f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:42.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.901+0000 7f932a800700 1 -- 172.21.15.37:0/2158566836 >> 172.21.15.37:0/2158566836 conn(0x7f93240792f0 msgr2=0x7f9324112120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:42.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.901+0000 7f932a800700 1 -- 172.21.15.37:0/2158566836 shutdown_connections 2024-03-28T18:09:42.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:42.901+0000 7f932a800700 1 -- 172.21.15.37:0/2158566836 wait complete. 2024-03-28T18:09:43.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:43 smithi037 ceph-mon[31460]: pgmap v361: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:09:43.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:43 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3731451006' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T18:09:43.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:43 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2158566836' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T18:09:43.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:43 smithi059 ceph-mon[36190]: pgmap v361: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:09:43.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:43 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3731451006' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T18:09:43.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:43 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2158566836' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T18:09:43.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:43 smithi170 ceph-mon[36280]: pgmap v361: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:09:43.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:43 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3731451006' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T18:09:43.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:43 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2158566836' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T18:09:44.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.745+0000 7fce7ae00700 1 -- 172.21.15.37:0/1062161567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce74074530 msgr2=0x7fce74074910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:44.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.745+0000 7fce7ae00700 1 --2- 172.21.15.37:0/1062161567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce74074530 0x7fce74074910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fce6800b0a0 tx=0x7fce6800b3b0 comp rx=0 tx=0).stop 2024-03-28T18:09:44.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.745+0000 7fce7ae00700 1 -- 172.21.15.37:0/1062161567 shutdown_connections 2024-03-28T18:09:44.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.745+0000 7fce7ae00700 1 --2- 172.21.15.37:0/1062161567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce74113080 0x7fce74115470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:44.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.745+0000 7fce7ae00700 1 --2- 172.21.15.37:0/1062161567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce74074530 0x7fce74074910 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:44.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.745+0000 7fce7ae00700 1 --2- 172.21.15.37:0/1062161567 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce74074e50 0x7fce74112b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:44.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.745+0000 7fce7ae00700 1 -- 172.21.15.37:0/1062161567 >> 172.21.15.37:0/1062161567 conn(0x7fce740fdb60 msgr2=0x7fce740fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:44.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.745+0000 7fce7ae00700 1 -- 172.21.15.37:0/1062161567 shutdown_connections 2024-03-28T18:09:44.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.745+0000 7fce7ae00700 1 -- 172.21.15.37:0/1062161567 wait complete. 2024-03-28T18:09:44.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.747+0000 7fce7ae00700 1 Processor -- start 2024-03-28T18:09:44.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.747+0000 7fce7ae00700 1 -- start start 2024-03-28T18:09:44.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.747+0000 7fce7ae00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce74074530 0x7fce7419eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:44.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.747+0000 7fce7ae00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce74074e50 0x7fce7419f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:44.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.747+0000 7fce7ae00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce74113080 0x7fce741a3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:44.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.747+0000 7fce7ae00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce741189f0 con 0x7fce74074e50 2024-03-28T18:09:44.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.747+0000 7fce7ae00700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce74118870 con 0x7fce74113080 2024-03-28T18:09:44.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.747+0000 7fce7ae00700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce74118b70 con 0x7fce74074530 2024-03-28T18:09:44.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.748+0000 7fce79a00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce74074530 0x7fce7419eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:44.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.748+0000 7fce7a400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce74113080 0x7fce741a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:44.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.748+0000 7fce79000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce74074e50 0x7fce7419f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:44.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.748+0000 7fce79a00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce74074530 0x7fce7419eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:43276/0 (socket says 172.21.15.37:43276) 2024-03-28T18:09:44.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.748+0000 7fce79a00700 1 -- 172.21.15.37:0/2744891625 learned_addr learned my addr 172.21.15.37:0/2744891625 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:44.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.748+0000 7fce79a00700 1 -- 172.21.15.37:0/2744891625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce74113080 msgr2=0x7fce741a3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:44.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.748+0000 7fce79a00700 1 --2- 172.21.15.37:0/2744891625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce74113080 0x7fce741a3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:44.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.748+0000 7fce79a00700 1 -- 172.21.15.37:0/2744891625 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce74074e50 msgr2=0x7fce7419f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:44.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.748+0000 7fce79a00700 1 --2- 172.21.15.37:0/2744891625 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce74074e50 0x7fce7419f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:44.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.748+0000 7fce79a00700 1 -- 172.21.15.37:0/2744891625 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce741a3ea0 con 0x7fce74074530 2024-03-28T18:09:44.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.748+0000 7fce79a00700 1 --2- 172.21.15.37:0/2744891625 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce74074530 0x7fce7419eee0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fce6800b070 tx=0x7fce68004c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:44.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.749+0000 7fce72000700 1 -- 172.21.15.37:0/2744891625 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce68021070 con 0x7fce74074530 2024-03-28T18:09:44.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.749+0000 7fce72000700 1 -- 172.21.15.37:0/2744891625 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce68003c00 con 0x7fce74074530 2024-03-28T18:09:44.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.749+0000 7fce72000700 1 -- 172.21.15.37:0/2744891625 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce68004780 con 0x7fce74074530 2024-03-28T18:09:44.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.749+0000 7fce7ae00700 1 -- 172.21.15.37:0/2744891625 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce741a4130 con 0x7fce74074530 2024-03-28T18:09:44.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.749+0000 7fce7ae00700 1 -- 172.21.15.37:0/2744891625 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce741aba10 con 0x7fce74074530 2024-03-28T18:09:44.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.750+0000 7fce7ae00700 1 -- 172.21.15.37:0/2744891625 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce74075c70 con 0x7fce74074530 2024-03-28T18:09:44.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.752+0000 7fce72000700 1 -- 172.21.15.37:0/2744891625 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fce68009ba0 con 0x7fce74074530 2024-03-28T18:09:44.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.753+0000 7fce72000700 1 --2- 172.21.15.37:0/2744891625 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fce5c0a29a0 0x7fce5c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:44.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.753+0000 7fce72000700 1 -- 172.21.15.37:0/2744891625 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fce6800f020 con 0x7fce74074530 2024-03-28T18:09:44.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.753+0000 7fce79000700 1 --2- 172.21.15.37:0/2744891625 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fce5c0a29a0 0x7fce5c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:44.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.754+0000 7fce79000700 1 --2- 172.21.15.37:0/2744891625 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fce5c0a29a0 0x7fce5c0a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fce60009680 tx=0x7fce60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:44.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:43.757+0000 7fce72000700 1 -- 172.21.15.37:0/2744891625 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fce6802a020 con 0x7fce74074530 2024-03-28T18:09:44.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:44.045+0000 7fce7ae00700 1 -- 172.21.15.37:0/2744891625 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fce741a01f0 con 0x7fce74074530 2024-03-28T18:09:44.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:44.046+0000 7fce72000700 1 -- 172.21.15.37:0/2744891625 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fce6809cf00 con 0x7fce74074530 2024-03-28T18:09:44.119 INFO:teuthology.orchestra.run.smithi037.stdout:55834574936 2024-03-28T18:09:44.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:44.048+0000 7fce7ae00700 1 -- 172.21.15.37:0/2744891625 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fce5c0a29a0 msgr2=0x7fce5c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:44.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:44.048+0000 7fce7ae00700 1 --2- 172.21.15.37:0/2744891625 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fce5c0a29a0 0x7fce5c0a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fce60009680 tx=0x7fce60009260 comp rx=0 tx=0).stop 2024-03-28T18:09:44.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:44.048+0000 7fce7ae00700 1 -- 172.21.15.37:0/2744891625 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce74074530 msgr2=0x7fce7419eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:44.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:44.048+0000 7fce7ae00700 1 --2- 172.21.15.37:0/2744891625 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce74074530 0x7fce7419eee0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fce6800b070 tx=0x7fce68004c80 comp rx=0 tx=0).stop 2024-03-28T18:09:44.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:44.049+0000 7fce7ae00700 1 -- 172.21.15.37:0/2744891625 shutdown_connections 2024-03-28T18:09:44.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:44.049+0000 7fce7ae00700 1 --2- 172.21.15.37:0/2744891625 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fce5c0a29a0 0x7fce5c0a4e60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:44.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:44.049+0000 7fce7ae00700 1 --2- 172.21.15.37:0/2744891625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce74113080 0x7fce741a3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:44.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:44.049+0000 7fce7ae00700 1 --2- 172.21.15.37:0/2744891625 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce74074530 0x7fce7419eee0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:44.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:44.049+0000 7fce7ae00700 1 --2- 172.21.15.37:0/2744891625 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce74074e50 0x7fce7419f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:44.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:44.049+0000 7fce7ae00700 1 -- 172.21.15.37:0/2744891625 >> 172.21.15.37:0/2744891625 conn(0x7fce740fdb60 msgr2=0x7fce740ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:44.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:44.049+0000 7fce7ae00700 1 -- 172.21.15.37:0/2744891625 shutdown_connections 2024-03-28T18:09:44.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:44.049+0000 7fce7ae00700 1 -- 172.21.15.37:0/2744891625 wait complete. 2024-03-28T18:09:44.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:44 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2744891625' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T18:09:44.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:44 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2744891625' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T18:09:44.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:44 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2744891625' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T18:09:45.009 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808868 got 287762808872 for osd.9 2024-03-28T18:09:45.009 DEBUG:teuthology.parallel:result is None 2024-03-28T18:09:45.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:45 smithi037 ceph-mon[31460]: pgmap v362: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:45.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:45 smithi059 ceph-mon[36190]: pgmap v362: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:45.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:45 smithi170 ceph-mon[36280]: pgmap v362: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:45.603 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411403 got 77309411410 for osd.2 2024-03-28T18:09:45.603 DEBUG:teuthology.parallel:result is None 2024-03-28T18:09:46.335 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495669 got 197568495675 for osd.6 2024-03-28T18:09:46.335 DEBUG:teuthology.parallel:result is None 2024-03-28T18:09:46.344 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:46 smithi037 ceph-mon[31460]: mgrmap e18: x(active, since 9m), standbys: y, z 2024-03-28T18:09:46.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:46 smithi059 ceph-mon[36190]: mgrmap e18: x(active, since 9m), standbys: y, z 2024-03-28T18:09:46.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:46 smithi170 ceph-mon[36280]: mgrmap e18: x(active, since 9m), standbys: y, z 2024-03-28T18:09:47.103 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574929 got 55834574936 for osd.1 2024-03-28T18:09:47.103 DEBUG:teuthology.parallel:result is None 2024-03-28T18:09:47.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.094+0000 7fb3c6600700 1 -- 172.21.15.37:0/4259638406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3c00766c0 msgr2=0x7fb3c0076aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:47.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.094+0000 7fb3c6600700 1 --2- 172.21.15.37:0/4259638406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3c00766c0 0x7fb3c0076aa0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0009990 tx=0x7fb3b0009ca0 comp rx=0 tx=0).stop 2024-03-28T18:09:47.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.094+0000 7fb3c6600700 1 -- 172.21.15.37:0/4259638406 shutdown_connections 2024-03-28T18:09:47.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.094+0000 7fb3c6600700 1 --2- 172.21.15.37:0/4259638406 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c0113020 0x7fb3c0115410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:47.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.094+0000 7fb3c6600700 1 --2- 172.21.15.37:0/4259638406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3c00766c0 0x7fb3c0076aa0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:47.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.094+0000 7fb3c6600700 1 --2- 172.21.15.37:0/4259638406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c0076fe0 0x7fb3c0112ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:47.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.094+0000 7fb3c6600700 1 -- 172.21.15.37:0/4259638406 >> 172.21.15.37:0/4259638406 conn(0x7fb3c00ffe90 msgr2=0x7fb3c01022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:47.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.094+0000 7fb3c6600700 1 -- 172.21.15.37:0/4259638406 shutdown_connections 2024-03-28T18:09:47.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.094+0000 7fb3c6600700 1 -- 172.21.15.37:0/4259638406 wait complete. 2024-03-28T18:09:47.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.096+0000 7fb3c6600700 1 Processor -- start 2024-03-28T18:09:47.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.096+0000 7fb3c6600700 1 -- start start 2024-03-28T18:09:47.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.097+0000 7fb3c6600700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c00766c0 0x7fb3c019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:47.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.097+0000 7fb3c6600700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c0076fe0 0x7fb3c019f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:47.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.097+0000 7fb3c6600700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3c0113020 0x7fb3c01a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:47.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.097+0000 7fb3c6600700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c0118b30 con 0x7fb3c00766c0 2024-03-28T18:09:47.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.097+0000 7fb3c6600700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c01189b0 con 0x7fb3c0076fe0 2024-03-28T18:09:47.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.097+0000 7fb3c6600700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c0118cb0 con 0x7fb3c0113020 2024-03-28T18:09:47.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.097+0000 7fb3bea00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c0076fe0 0x7fb3c019f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:47.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.097+0000 7fb3bea00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c0076fe0 0x7fb3c019f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:55958/0 (socket says 172.21.15.37:55958) 2024-03-28T18:09:47.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.097+0000 7fb3bea00700 1 -- 172.21.15.37:0/4006963820 learned_addr learned my addr 172.21.15.37:0/4006963820 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:47.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.097+0000 7fb3bf400700 1 --2- 172.21.15.37:0/4006963820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c00766c0 0x7fb3c019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:47.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.097+0000 7fb3bfe00700 1 --2- 172.21.15.37:0/4006963820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3c0113020 0x7fb3c01a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:47.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.097+0000 7fb3bea00700 1 -- 172.21.15.37:0/4006963820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3c0113020 msgr2=0x7fb3c01a3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:47.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.097+0000 7fb3bea00700 1 --2- 172.21.15.37:0/4006963820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3c0113020 0x7fb3c01a3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:47.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.097+0000 7fb3bea00700 1 -- 172.21.15.37:0/4006963820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c00766c0 msgr2=0x7fb3c019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:47.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.097+0000 7fb3bea00700 1 --2- 172.21.15.37:0/4006963820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c00766c0 0x7fb3c019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:47.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.097+0000 7fb3bea00700 1 -- 172.21.15.37:0/4006963820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c01a3fd0 con 0x7fb3c0076fe0 2024-03-28T18:09:47.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.098+0000 7fb3bea00700 1 --2- 172.21.15.37:0/4006963820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c0076fe0 0x7fb3c019f500 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac00e9a0 tx=0x7fb3ac00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:47.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.098+0000 7fb3b7e00700 1 -- 172.21.15.37:0/4006963820 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac00ca10 con 0x7fb3c0076fe0 2024-03-28T18:09:47.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.098+0000 7fb3b7e00700 1 -- 172.21.15.37:0/4006963820 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3ac00cbb0 con 0x7fb3c0076fe0 2024-03-28T18:09:47.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.098+0000 7fb3c6600700 1 -- 172.21.15.37:0/4006963820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c01a42c0 con 0x7fb3c0076fe0 2024-03-28T18:09:47.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.098+0000 7fb3b7e00700 1 -- 172.21.15.37:0/4006963820 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac018680 con 0x7fb3c0076fe0 2024-03-28T18:09:47.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.098+0000 7fb3c6600700 1 -- 172.21.15.37:0/4006963820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c01abb40 con 0x7fb3c0076fe0 2024-03-28T18:09:47.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.099+0000 7fb3c6600700 1 -- 172.21.15.37:0/4006963820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c0068e50 con 0x7fb3c0076fe0 2024-03-28T18:09:47.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.101+0000 7fb3b7e00700 1 -- 172.21.15.37:0/4006963820 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb3ac010430 con 0x7fb3c0076fe0 2024-03-28T18:09:47.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.102+0000 7fb3b7e00700 1 --2- 172.21.15.37:0/4006963820 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb3a40a2890 0x7fb3a40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:47.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.102+0000 7fb3b7e00700 1 -- 172.21.15.37:0/4006963820 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb3ac0cecf0 con 0x7fb3c0076fe0 2024-03-28T18:09:47.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.103+0000 7fb3bf400700 1 --2- 172.21.15.37:0/4006963820 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb3a40a2890 0x7fb3a40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:47.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.103+0000 7fb3bf400700 1 --2- 172.21.15.37:0/4006963820 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb3a40a2890 0x7fb3a40a4d50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0006010 tx=0x7fb3b000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:47.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.106+0000 7fb3b7e00700 1 -- 172.21.15.37:0/4006963820 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb3ac09bd40 con 0x7fb3c0076fe0 2024-03-28T18:09:47.159 INFO:teuthology.orchestra.run.smithi037.stdout:343597383711 2024-03-28T18:09:47.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.344+0000 7fb3c6600700 1 -- 172.21.15.37:0/4006963820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7fb3c002da30 con 0x7fb3c0076fe0 2024-03-28T18:09:47.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.345+0000 7fb3b7e00700 1 -- 172.21.15.37:0/4006963820 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fb3ac097030 con 0x7fb3c0076fe0 2024-03-28T18:09:47.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.348+0000 7fb3c6600700 1 -- 172.21.15.37:0/4006963820 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb3a40a2890 msgr2=0x7fb3a40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:47.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.348+0000 7fb3c6600700 1 --2- 172.21.15.37:0/4006963820 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb3a40a2890 0x7fb3a40a4d50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0006010 tx=0x7fb3b000b260 comp rx=0 tx=0).stop 2024-03-28T18:09:47.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.348+0000 7fb3c6600700 1 -- 172.21.15.37:0/4006963820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c0076fe0 msgr2=0x7fb3c019f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:47.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.348+0000 7fb3c6600700 1 --2- 172.21.15.37:0/4006963820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c0076fe0 0x7fb3c019f500 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac00e9a0 tx=0x7fb3ac00ecb0 comp rx=0 tx=0).stop 2024-03-28T18:09:47.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.350+0000 7fb3c6600700 1 -- 172.21.15.37:0/4006963820 shutdown_connections 2024-03-28T18:09:47.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.350+0000 7fb3c6600700 1 --2- 172.21.15.37:0/4006963820 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb3a40a2890 0x7fb3a40a4d50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:47.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.350+0000 7fb3c6600700 1 --2- 172.21.15.37:0/4006963820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3c0113020 0x7fb3c01a3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:47.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.350+0000 7fb3c6600700 1 --2- 172.21.15.37:0/4006963820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c00766c0 0x7fb3c019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:47.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.350+0000 7fb3c6600700 1 --2- 172.21.15.37:0/4006963820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c0076fe0 0x7fb3c019f500 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:47.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.350+0000 7fb3c6600700 1 -- 172.21.15.37:0/4006963820 >> 172.21.15.37:0/4006963820 conn(0x7fb3c00ffe90 msgr2=0x7fb3c0102280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:47.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.350+0000 7fb3c6600700 1 -- 172.21.15.37:0/4006963820 shutdown_connections 2024-03-28T18:09:47.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.350+0000 7fb3c6600700 1 -- 172.21.15.37:0/4006963820 wait complete. 2024-03-28T18:09:47.212 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:47 smithi037 ceph-mon[31460]: pgmap v363: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2024-03-28T18:09:47.212 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:47 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/4006963820' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T18:09:47.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:47 smithi059 ceph-mon[36190]: pgmap v363: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2024-03-28T18:09:47.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:47 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/4006963820' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T18:09:47.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:47 smithi170 ceph-mon[36280]: pgmap v363: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2024-03-28T18:09:47.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:47 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/4006963820' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T18:09:48.113 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383704 got 343597383711 for osd.11 2024-03-28T18:09:48.113 DEBUG:teuthology.parallel:result is None 2024-03-28T18:09:48.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.911+0000 7fa321200700 1 -- 172.21.15.37:0/1069628552 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa31c104fb0 msgr2=0x7fa31c109000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:48.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.911+0000 7fa321200700 1 --2- 172.21.15.37:0/1069628552 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa31c104fb0 0x7fa31c109000 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa3080099c0 tx=0x7fa308009cd0 comp rx=0 tx=0).stop 2024-03-28T18:09:48.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.912+0000 7fa321200700 1 -- 172.21.15.37:0/1069628552 shutdown_connections 2024-03-28T18:09:48.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.912+0000 7fa321200700 1 --2- 172.21.15.37:0/1069628552 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa31c1096e0 0x7fa31c1112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:48.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.912+0000 7fa321200700 1 --2- 172.21.15.37:0/1069628552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa31c104600 0x7fa31c1049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:48.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.912+0000 7fa321200700 1 --2- 172.21.15.37:0/1069628552 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa31c104fb0 0x7fa31c109000 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:48.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.912+0000 7fa321200700 1 -- 172.21.15.37:0/1069628552 >> 172.21.15.37:0/1069628552 conn(0x7fa31c0ffe90 msgr2=0x7fa31c1022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:48.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.912+0000 7fa321200700 1 -- 172.21.15.37:0/1069628552 shutdown_connections 2024-03-28T18:09:48.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.912+0000 7fa321200700 1 -- 172.21.15.37:0/1069628552 wait complete. 2024-03-28T18:09:48.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.914+0000 7fa321200700 1 Processor -- start 2024-03-28T18:09:48.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.914+0000 7fa321200700 1 -- start start 2024-03-28T18:09:48.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.915+0000 7fa321200700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa31c104600 0x7fa31c1a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:48.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.915+0000 7fa321200700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa31c104fb0 0x7fa31c1a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:48.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.915+0000 7fa321200700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa31c1096e0 0x7fa31c19a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:48.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.915+0000 7fa321200700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c1146a0 con 0x7fa31c104600 2024-03-28T18:09:48.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.915+0000 7fa321200700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c114520 con 0x7fa31c1096e0 2024-03-28T18:09:48.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.915+0000 7fa321200700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c114820 con 0x7fa31c104fb0 2024-03-28T18:09:48.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.915+0000 7fa319400700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa31c104fb0 0x7fa31c1a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:48.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.915+0000 7fa31a800700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa31c1096e0 0x7fa31c19a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:48.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.915+0000 7fa319400700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa31c104fb0 0x7fa31c1a0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:43308/0 (socket says 172.21.15.37:43308) 2024-03-28T18:09:48.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.915+0000 7fa319400700 1 -- 172.21.15.37:0/4033063191 learned_addr learned my addr 172.21.15.37:0/4033063191 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:48.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.915+0000 7fa319e00700 1 --2- 172.21.15.37:0/4033063191 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa31c104600 0x7fa31c1a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:48.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.915+0000 7fa31a800700 1 -- 172.21.15.37:0/4033063191 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa31c104fb0 msgr2=0x7fa31c1a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:48.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.915+0000 7fa31a800700 1 --2- 172.21.15.37:0/4033063191 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa31c104fb0 0x7fa31c1a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:48.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.915+0000 7fa31a800700 1 -- 172.21.15.37:0/4033063191 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa31c104600 msgr2=0x7fa31c1a07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:48.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.915+0000 7fa31a800700 1 --2- 172.21.15.37:0/4033063191 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa31c104600 0x7fa31c1a07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:48.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.915+0000 7fa31a800700 1 -- 172.21.15.37:0/4033063191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa31c19b0e0 con 0x7fa31c1096e0 2024-03-28T18:09:48.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.915+0000 7fa319400700 1 --2- 172.21.15.37:0/4033063191 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa31c104fb0 0x7fa31c1a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:09:48.206 INFO:teuthology.orchestra.run.smithi037.stdout:309237645349 2024-03-28T18:09:48.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.916+0000 7fa31a800700 1 --2- 172.21.15.37:0/4033063191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa31c1096e0 0x7fa31c19a980 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa31400efc0 tx=0x7fa31400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:48.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.916+0000 7fa312a00700 1 -- 172.21.15.37:0/4033063191 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3140092e0 con 0x7fa31c1096e0 2024-03-28T18:09:48.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.916+0000 7fa312a00700 1 -- 172.21.15.37:0/4033063191 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa314004bb0 con 0x7fa31c1096e0 2024-03-28T18:09:48.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.916+0000 7fa312a00700 1 -- 172.21.15.37:0/4033063191 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa314004280 con 0x7fa31c1096e0 2024-03-28T18:09:48.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.916+0000 7fa321200700 1 -- 172.21.15.37:0/4033063191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa31c19b370 con 0x7fa31c1096e0 2024-03-28T18:09:48.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.916+0000 7fa321200700 1 -- 172.21.15.37:0/4033063191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa31c1a7690 con 0x7fa31c1096e0 2024-03-28T18:09:48.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.918+0000 7fa321200700 1 -- 172.21.15.37:0/4033063191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa31c068e50 con 0x7fa31c1096e0 2024-03-28T18:09:48.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.919+0000 7fa312a00700 1 -- 172.21.15.37:0/4033063191 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa31401b030 con 0x7fa31c1096e0 2024-03-28T18:09:48.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.921+0000 7fa312a00700 1 --2- 172.21.15.37:0/4033063191 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa3000a2910 0x7fa3000a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:48.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.921+0000 7fa312a00700 1 -- 172.21.15.37:0/4033063191 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa3140ce5a0 con 0x7fa31c1096e0 2024-03-28T18:09:48.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.921+0000 7fa319e00700 1 --2- 172.21.15.37:0/4033063191 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa3000a2910 0x7fa3000a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:48.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.922+0000 7fa319e00700 1 --2- 172.21.15.37:0/4033063191 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa3000a2910 0x7fa3000a4dd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa31c19bac0 tx=0x7fa30c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:48.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:46.925+0000 7fa312a00700 1 -- 172.21.15.37:0/4033063191 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa314022020 con 0x7fa31c1096e0 2024-03-28T18:09:48.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:47.212+0000 7fa321200700 1 -- 172.21.15.37:0/4033063191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7fa31c02da30 con 0x7fa31c1096e0 2024-03-28T18:09:48.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:47.212+0000 7fa312a00700 1 -- 172.21.15.37:0/4033063191 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fa314010040 con 0x7fa31c1096e0 2024-03-28T18:09:48.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:47.215+0000 7fa321200700 1 -- 172.21.15.37:0/4033063191 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa3000a2910 msgr2=0x7fa3000a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:48.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:47.215+0000 7fa321200700 1 --2- 172.21.15.37:0/4033063191 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa3000a2910 0x7fa3000a4dd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa31c19bac0 tx=0x7fa30c009260 comp rx=0 tx=0).stop 2024-03-28T18:09:48.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:47.215+0000 7fa321200700 1 -- 172.21.15.37:0/4033063191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa31c1096e0 msgr2=0x7fa31c19a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:48.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:47.215+0000 7fa321200700 1 --2- 172.21.15.37:0/4033063191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa31c1096e0 0x7fa31c19a980 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa31400efc0 tx=0x7fa31400c3a0 comp rx=0 tx=0).stop 2024-03-28T18:09:48.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:47.215+0000 7fa321200700 1 -- 172.21.15.37:0/4033063191 shutdown_connections 2024-03-28T18:09:48.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:47.215+0000 7fa321200700 1 --2- 172.21.15.37:0/4033063191 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa3000a2910 0x7fa3000a4dd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:48.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:47.216+0000 7fa321200700 1 --2- 172.21.15.37:0/4033063191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa31c1096e0 0x7fa31c19a980 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:48.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:47.216+0000 7fa321200700 1 --2- 172.21.15.37:0/4033063191 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa31c104600 0x7fa31c1a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:48.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:47.216+0000 7fa321200700 1 --2- 172.21.15.37:0/4033063191 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa31c104fb0 0x7fa31c1a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:48.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:47.216+0000 7fa321200700 1 -- 172.21.15.37:0/4033063191 >> 172.21.15.37:0/4033063191 conn(0x7fa31c0ffe90 msgr2=0x7fa31c102160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:48.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:47.216+0000 7fa321200700 1 -- 172.21.15.37:0/4033063191 shutdown_connections 2024-03-28T18:09:48.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:47.216+0000 7fa321200700 1 -- 172.21.15.37:0/4033063191 wait complete. 2024-03-28T18:09:48.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:48 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/4033063191' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T18:09:48.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:48 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/4033063191' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T18:09:48.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:48 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/4033063191' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T18:09:48.965 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645342 got 309237645349 for osd.10 2024-03-28T18:09:48.966 DEBUG:teuthology.parallel:result is None 2024-03-28T18:09:48.966 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-28T18:09:48.966 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T18:09:49.437 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:09:49.841 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:49 smithi037 ceph-mon[31460]: pgmap v364: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-28T18:09:49.841 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:49 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:49.841 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:49 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:50.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:49 smithi059 ceph-mon[36190]: pgmap v364: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-28T18:09:50.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:49 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:50.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:49 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:49 smithi170 ceph-mon[36280]: pgmap v364: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-28T18:09:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:49 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:49 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:50.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:50 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:09:50.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:50 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:09:50.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:50 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:51.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:50 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:09:51.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:50 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:09:51.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:50 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:51.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:50 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:09:51.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:50 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:09:51.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:50 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:09:51.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.783+0000 7f9ee7400700 1 -- 172.21.15.37:0/2243920644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ee0107080 msgr2=0x7f9ee0069e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:51.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.783+0000 7f9ee7400700 1 --2- 172.21.15.37:0/2243920644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ee0107080 0x7f9ee0069e40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009990 tx=0x7f9ed4009ca0 comp rx=0 tx=0).stop 2024-03-28T18:09:51.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.783+0000 7f9ee7400700 1 -- 172.21.15.37:0/2243920644 shutdown_connections 2024-03-28T18:09:51.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.783+0000 7f9ee7400700 1 --2- 172.21.15.37:0/2243920644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ee0104b90 0x7f9ee0105020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:51.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.783+0000 7f9ee7400700 1 --2- 172.21.15.37:0/2243920644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ee0107080 0x7f9ee0069e40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:51.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.783+0000 7f9ee7400700 1 --2- 172.21.15.37:0/2243920644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ee006a380 0x7f9ee006a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:51.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.783+0000 7f9ee7400700 1 -- 172.21.15.37:0/2243920644 >> 172.21.15.37:0/2243920644 conn(0x7f9ee0077fc0 msgr2=0x7f9ee00783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:51.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.783+0000 7f9ee7400700 1 -- 172.21.15.37:0/2243920644 shutdown_connections 2024-03-28T18:09:51.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.783+0000 7f9ee7400700 1 -- 172.21.15.37:0/2243920644 wait complete. 2024-03-28T18:09:51.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.785+0000 7f9ee7400700 1 Processor -- start 2024-03-28T18:09:51.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.785+0000 7f9ee7400700 1 -- start start 2024-03-28T18:09:51.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.785+0000 7f9ee7400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ee006a380 0x7f9ee019ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:51.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.785+0000 7f9ee7400700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ee0104b90 0x7f9ee019f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:51.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.786+0000 7f9ee7400700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ee0107080 0x7f9ee01a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:51.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.786+0000 7f9ee7400700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee0118a10 con 0x7f9ee0104b90 2024-03-28T18:09:51.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.786+0000 7f9ee7400700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee0118890 con 0x7f9ee006a380 2024-03-28T18:09:51.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.786+0000 7f9ee7400700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee0118b90 con 0x7f9ee0107080 2024-03-28T18:09:51.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.786+0000 7f9ee4a00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ee006a380 0x7f9ee019ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:51.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.786+0000 7f9ee4a00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ee006a380 0x7f9ee019ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:55980/0 (socket says 172.21.15.37:55980) 2024-03-28T18:09:51.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.786+0000 7f9ee5400700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ee0107080 0x7f9ee01a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:51.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.786+0000 7f9ee4a00700 1 -- 172.21.15.37:0/311092817 learned_addr learned my addr 172.21.15.37:0/311092817 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:51.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.786+0000 7f9edfe00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ee0104b90 0x7f9ee019f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:51.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.786+0000 7f9ee4a00700 1 -- 172.21.15.37:0/311092817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ee0107080 msgr2=0x7f9ee01a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:51.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.786+0000 7f9ee4a00700 1 --2- 172.21.15.37:0/311092817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ee0107080 0x7f9ee01a37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:51.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.786+0000 7f9ee4a00700 1 -- 172.21.15.37:0/311092817 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ee0104b90 msgr2=0x7f9ee019f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:51.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.786+0000 7f9ee4a00700 1 --2- 172.21.15.37:0/311092817 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ee0104b90 0x7f9ee019f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:51.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.786+0000 7f9ee4a00700 1 -- 172.21.15.37:0/311092817 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee01a3f30 con 0x7f9ee006a380 2024-03-28T18:09:51.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.786+0000 7f9ee5400700 1 --2- 172.21.15.37:0/311092817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ee0107080 0x7f9ee01a37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:09:51.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.787+0000 7f9ee4a00700 1 --2- 172.21.15.37:0/311092817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ee006a380 0x7f9ee019ef20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4000c00 tx=0x7f9ed400bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.787+0000 7f9edd600700 1 -- 172.21.15.37:0/311092817 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4027070 con 0x7f9ee006a380 2024-03-28T18:09:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.787+0000 7f9edd600700 1 -- 172.21.15.37:0/311092817 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ed4003d70 con 0x7f9ee006a380 2024-03-28T18:09:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.787+0000 7f9edd600700 1 -- 172.21.15.37:0/311092817 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed400f460 con 0x7f9ee006a380 2024-03-28T18:09:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.787+0000 7f9ee7400700 1 -- 172.21.15.37:0/311092817 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee01a41c0 con 0x7f9ee006a380 2024-03-28T18:09:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.787+0000 7f9ee7400700 1 -- 172.21.15.37:0/311092817 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee01abaa0 con 0x7f9ee006a380 2024-03-28T18:09:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.789+0000 7f9ee7400700 1 -- 172.21.15.37:0/311092817 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee0068e50 con 0x7f9ee006a380 2024-03-28T18:09:51.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.790+0000 7f9edd600700 1 -- 172.21.15.37:0/311092817 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9ed402a070 con 0x7f9ee006a380 2024-03-28T18:09:51.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.791+0000 7f9edd600700 1 --2- 172.21.15.37:0/311092817 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9ec80a2910 0x7f9ec80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:51.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.792+0000 7f9edd600700 1 -- 172.21.15.37:0/311092817 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9ed40cf830 con 0x7f9ee006a380 2024-03-28T18:09:51.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.792+0000 7f9edfe00700 1 --2- 172.21.15.37:0/311092817 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9ec80a2910 0x7f9ec80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:51.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.792+0000 7f9edfe00700 1 --2- 172.21.15.37:0/311092817 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9ec80a2910 0x7f9ec80a4dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0078850 tx=0x7f9ecc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:51.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:51.795+0000 7f9edd600700 1 -- 172.21.15.37:0/311092817 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9ed4013070 con 0x7f9ee006a380 2024-03-28T18:09:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:52.109+0000 7f9ee7400700 1 -- 172.21.15.37:0/311092817 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9ee0103990 con 0x7f9ec80a2910 2024-03-28T18:09:52.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:52.118+0000 7f9edd600700 1 -- 172.21.15.37:0/311092817 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259081 (secure 0 0 0) 0x7f9ee0103990 con 0x7f9ec80a2910 2024-03-28T18:09:52.121 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:09:52.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:52.124+0000 7f9ee7400700 1 -- 172.21.15.37:0/311092817 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9ec80a2910 msgr2=0x7f9ec80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:52.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:52.124+0000 7f9ee7400700 1 --2- 172.21.15.37:0/311092817 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9ec80a2910 0x7f9ec80a4dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0078850 tx=0x7f9ecc009260 comp rx=0 tx=0).stop 2024-03-28T18:09:52.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:52.124+0000 7f9ee7400700 1 -- 172.21.15.37:0/311092817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ee006a380 msgr2=0x7f9ee019ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:52.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:52.124+0000 7f9ee7400700 1 --2- 172.21.15.37:0/311092817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ee006a380 0x7f9ee019ef20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4000c00 tx=0x7f9ed400bab0 comp rx=0 tx=0).stop 2024-03-28T18:09:52.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:52.124+0000 7f9ee7400700 1 -- 172.21.15.37:0/311092817 shutdown_connections 2024-03-28T18:09:52.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:52.124+0000 7f9ee7400700 1 --2- 172.21.15.37:0/311092817 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f9ec80a2910 0x7f9ec80a4dd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:52.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:52.124+0000 7f9ee7400700 1 --2- 172.21.15.37:0/311092817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ee0107080 0x7f9ee01a37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:52.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:52.124+0000 7f9ee7400700 1 --2- 172.21.15.37:0/311092817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ee006a380 0x7f9ee019ef20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:52.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:52.124+0000 7f9ee7400700 1 --2- 172.21.15.37:0/311092817 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ee0104b90 0x7f9ee019f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:52.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:52.124+0000 7f9ee7400700 1 -- 172.21.15.37:0/311092817 >> 172.21.15.37:0/311092817 conn(0x7f9ee0077fc0 msgr2=0x7f9ee0103280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:52.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:52.124+0000 7f9ee7400700 1 -- 172.21.15.37:0/311092817 shutdown_connections 2024-03-28T18:09:52.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:52.124+0000 7f9ee7400700 1 -- 172.21.15.37:0/311092817 wait complete. 2024-03-28T18:09:52.127 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-03-28T18:09:52.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:51 smithi059 ceph-mon[36190]: pgmap v365: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-28T18:09:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:51 smithi170 ceph-mon[36280]: pgmap v365: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-28T18:09:52.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:51 smithi037 ceph-mon[31460]: pgmap v365: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-28T18:09:53.011 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-03-28T18:09:51.988984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656889,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":675,"num_read_kb":459,"num_write":349,"num_write_kb":3055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1181430,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":782,"ondisk_log_size":782,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377360,"kb_used_data":11568,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564464,"statfs":{"total":1151940427776,"available":1151554011136,"internally_reserved":0,"allocated":11845632,"data_stored":7438047,"data_compressed":113751,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":72,"num_read_kb":37,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006013"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.005823+0000","last_change":"2024-03-28T18:08:20.025576+0000","last_active":"2024-03-28T18:09:44.005823+0000","last_peered":"2024-03-28T18:09:44.005823+0000","last_clean":"2024-03-28T18:09:44.005823+0000","last_became_active":"2024-03-28T18:08:20.025177+0000","last_became_peered":"2024-03-28T18:08:20.025177+0000","last_unstale":"2024-03-28T18:09:44.005823+0000","last_undegraded":"2024-03-28T18:09:44.005823+0000","last_fullsized":"2024-03-28T18:09:44.005823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:14.521170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.016245+0000","last_change":"2024-03-28T18:08:20.027418+0000","last_active":"2024-03-28T18:09:44.016245+0000","last_peered":"2024-03-28T18:09:44.016245+0000","last_clean":"2024-03-28T18:09:44.016245+0000","last_became_active":"2024-03-28T18:08:20.026731+0000","last_became_peered":"2024-03-28T18:08:20.026731+0000","last_unstale":"2024-03-28T18:09:44.016245+0000","last_undegraded":"2024-03-28T18:09:44.016245+0000","last_fullsized":"2024-03-28T18:09:44.016245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:52:58.328738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.001275+0000","last_change":"2024-03-28T18:08:20.029821+0000","last_active":"2024-03-28T18:09:44.001275+0000","last_peered":"2024-03-28T18:09:44.001275+0000","last_clean":"2024-03-28T18:09:44.001275+0000","last_became_active":"2024-03-28T18:08:20.028245+0000","last_became_peered":"2024-03-28T18:08:20.028245+0000","last_unstale":"2024-03-28T18:09:44.001275+0000","last_undegraded":"2024-03-28T18:09:44.001275+0000","last_fullsized":"2024-03-28T18:09:44.001275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:44.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.966292+0000","last_change":"2024-03-28T18:08:20.015682+0000","last_active":"2024-03-28T18:09:43.966292+0000","last_peered":"2024-03-28T18:09:43.966292+0000","last_clean":"2024-03-28T18:09:43.966292+0000","last_became_active":"2024-03-28T18:08:20.015303+0000","last_became_peered":"2024-03-28T18:08:20.015303+0000","last_unstale":"2024-03-28T18:09:43.966292+0000","last_undegraded":"2024-03-28T18:09:43.966292+0000","last_fullsized":"2024-03-28T18:09:43.966292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:57.456052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.998334+0000","last_change":"2024-03-28T18:08:20.020970+0000","last_active":"2024-03-28T18:09:43.998334+0000","last_peered":"2024-03-28T18:09:43.998334+0000","last_clean":"2024-03-28T18:09:43.998334+0000","last_became_active":"2024-03-28T18:08:20.020646+0000","last_became_peered":"2024-03-28T18:08:20.020646+0000","last_unstale":"2024-03-28T18:09:43.998334+0000","last_undegraded":"2024-03-28T18:09:43.998334+0000","last_fullsized":"2024-03-28T18:09:43.998334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:57:20.422546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.970396+0000","last_change":"2024-03-28T18:08:20.022395+0000","last_active":"2024-03-28T18:09:43.970396+0000","last_peered":"2024-03-28T18:09:43.970396+0000","last_clean":"2024-03-28T18:09:43.970396+0000","last_became_active":"2024-03-28T18:08:20.022022+0000","last_became_peered":"2024-03-28T18:08:20.022022+0000","last_unstale":"2024-03-28T18:09:43.970396+0000","last_undegraded":"2024-03-28T18:09:43.970396+0000","last_fullsized":"2024-03-28T18:09:43.970396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:32.590751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.966769+0000","last_change":"2024-03-28T18:08:20.019990+0000","last_active":"2024-03-28T18:09:43.966769+0000","last_peered":"2024-03-28T18:09:43.966769+0000","last_clean":"2024-03-28T18:09:43.966769+0000","last_became_active":"2024-03-28T18:08:20.019618+0000","last_became_peered":"2024-03-28T18:08:20.019618+0000","last_unstale":"2024-03-28T18:09:43.966769+0000","last_undegraded":"2024-03-28T18:09:43.966769+0000","last_fullsized":"2024-03-28T18:09:43.966769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:48.442579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.968407+0000","last_change":"2024-03-28T18:09:15.889503+0000","last_active":"2024-03-28T18:09:43.968407+0000","last_peered":"2024-03-28T18:09:43.968407+0000","last_clean":"2024-03-28T18:09:43.968407+0000","last_became_active":"2024-03-28T18:09:15.888272+0000","last_became_peered":"2024-03-28T18:09:15.888272+0000","last_unstale":"2024-03-28T18:09:43.968407+0000","last_undegraded":"2024-03-28T18:09:43.968407+0000","last_fullsized":"2024-03-28T18:09:43.968407+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:49.222646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.979721+0000","last_change":"2024-03-28T18:08:20.009732+0000","last_active":"2024-03-28T18:09:43.979721+0000","last_peered":"2024-03-28T18:09:43.979721+0000","last_clean":"2024-03-28T18:09:43.979721+0000","last_became_active":"2024-03-28T18:08:20.009288+0000","last_became_peered":"2024-03-28T18:08:20.009288+0000","last_unstale":"2024-03-28T18:09:43.979721+0000","last_undegraded":"2024-03-28T18:09:43.979721+0000","last_fullsized":"2024-03-28T18:09:43.979721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:20:54.917964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.964811+0000","last_change":"2024-03-28T18:08:20.009687+0000","last_active":"2024-03-28T18:09:43.964811+0000","last_peered":"2024-03-28T18:09:43.964811+0000","last_clean":"2024-03-28T18:09:43.964811+0000","last_became_active":"2024-03-28T18:08:20.008364+0000","last_became_peered":"2024-03-28T18:08:20.008364+0000","last_unstale":"2024-03-28T18:09:43.964811+0000","last_undegraded":"2024-03-28T18:09:43.964811+0000","last_fullsized":"2024-03-28T18:09:43.964811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:08.419202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.978189+0000","last_change":"2024-03-28T18:08:20.011508+0000","last_active":"2024-03-28T18:09:43.978189+0000","last_peered":"2024-03-28T18:09:43.978189+0000","last_clean":"2024-03-28T18:09:43.978189+0000","last_became_active":"2024-03-28T18:08:20.011117+0000","last_became_peered":"2024-03-28T18:08:20.011117+0000","last_unstale":"2024-03-28T18:09:43.978189+0000","last_undegraded":"2024-03-28T18:09:43.978189+0000","last_fullsized":"2024-03-28T18:09:43.978189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:06:19.278931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.966130+0000","last_change":"2024-03-28T18:08:20.011403+0000","last_active":"2024-03-28T18:09:43.966130+0000","last_peered":"2024-03-28T18:09:43.966130+0000","last_clean":"2024-03-28T18:09:43.966130+0000","last_became_active":"2024-03-28T18:08:20.011025+0000","last_became_peered":"2024-03-28T18:08:20.011025+0000","last_unstale":"2024-03-28T18:09:43.966130+0000","last_undegraded":"2024-03-28T18:09:43.966130+0000","last_fullsized":"2024-03-28T18:09:43.966130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:13.599432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.005321+0000","last_change":"2024-03-28T18:08:20.001664+0000","last_active":"2024-03-28T18:09:44.005321+0000","last_peered":"2024-03-28T18:09:44.005321+0000","last_clean":"2024-03-28T18:09:44.005321+0000","last_became_active":"2024-03-28T18:08:20.000910+0000","last_became_peered":"2024-03-28T18:08:20.000910+0000","last_unstale":"2024-03-28T18:09:44.005321+0000","last_undegraded":"2024-03-28T18:09:44.005321+0000","last_fullsized":"2024-03-28T18:09:44.005321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:00:23.796302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.008088+0000","last_change":"2024-03-28T18:08:20.027174+0000","last_active":"2024-03-28T18:09:44.008088+0000","last_peered":"2024-03-28T18:09:44.008088+0000","last_clean":"2024-03-28T18:09:44.008088+0000","last_became_active":"2024-03-28T18:08:20.026113+0000","last_became_peered":"2024-03-28T18:08:20.026113+0000","last_unstale":"2024-03-28T18:09:44.008088+0000","last_undegraded":"2024-03-28T18:09:44.008088+0000","last_fullsized":"2024-03-28T18:09:44.008088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:55:00.061696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.010388+0000","last_change":"2024-03-28T18:08:20.020060+0000","last_active":"2024-03-28T18:09:44.010388+0000","last_peered":"2024-03-28T18:09:44.010388+0000","last_clean":"2024-03-28T18:09:44.010388+0000","last_became_active":"2024-03-28T18:08:20.019691+0000","last_became_peered":"2024-03-28T18:08:20.019691+0000","last_unstale":"2024-03-28T18:09:44.010388+0000","last_undegraded":"2024-03-28T18:09:44.010388+0000","last_fullsized":"2024-03-28T18:09:44.010388+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:27.889737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.003240+0000","last_change":"2024-03-28T18:08:20.024239+0000","last_active":"2024-03-28T18:09:44.003240+0000","last_peered":"2024-03-28T18:09:44.003240+0000","last_clean":"2024-03-28T18:09:44.003240+0000","last_became_active":"2024-03-28T18:08:20.023874+0000","last_became_peered":"2024-03-28T18:08:20.023874+0000","last_unstale":"2024-03-28T18:09:44.003240+0000","last_undegraded":"2024-03-28T18:09:44.003240+0000","last_fullsized":"2024-03-28T18:09:44.003240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:05:32.903195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.008554+0000","last_change":"2024-03-28T18:08:20.033322+0000","last_active":"2024-03-28T18:09:44.008554+0000","last_peered":"2024-03-28T18:09:44.008554+0000","last_clean":"2024-03-28T18:09:44.008554+0000","last_became_active":"2024-03-28T18:08:20.033048+0000","last_became_peered":"2024-03-28T18:08:20.033048+0000","last_unstale":"2024-03-28T18:09:44.008554+0000","last_undegraded":"2024-03-28T18:09:44.008554+0000","last_fullsized":"2024-03-28T18:09:44.008554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:18:59.234141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.986315+0000","last_change":"2024-03-28T18:08:20.022093+0000","last_active":"2024-03-28T18:09:43.986315+0000","last_peered":"2024-03-28T18:09:43.986315+0000","last_clean":"2024-03-28T18:09:43.986315+0000","last_became_active":"2024-03-28T18:08:20.021411+0000","last_became_peered":"2024-03-28T18:08:20.021411+0000","last_unstale":"2024-03-28T18:09:43.986315+0000","last_undegraded":"2024-03-28T18:09:43.986315+0000","last_fullsized":"2024-03-28T18:09:43.986315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:09:40.366628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.001943+0000","last_change":"2024-03-28T18:08:20.028893+0000","last_active":"2024-03-28T18:09:44.001943+0000","last_peered":"2024-03-28T18:09:44.001943+0000","last_clean":"2024-03-28T18:09:44.001943+0000","last_became_active":"2024-03-28T18:08:20.028634+0000","last_became_peered":"2024-03-28T18:08:20.028634+0000","last_unstale":"2024-03-28T18:09:44.001943+0000","last_undegraded":"2024-03-28T18:09:44.001943+0000","last_fullsized":"2024-03-28T18:09:44.001943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:32:27.454406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.734404+0000","last_change":"2024-03-28T18:09:15.734404+0000","last_active":"2024-03-28T18:09:15.734404+0000","last_peered":"2024-03-28T18:09:15.734404+0000","last_clean":"2024-03-28T18:09:15.734404+0000","last_became_active":"2024-03-28T18:09:15.723870+0000","last_became_peered":"2024-03-28T18:09:15.723870+0000","last_unstale":"2024-03-28T18:09:15.734404+0000","last_undegraded":"2024-03-28T18:09:15.734404+0000","last_fullsized":"2024-03-28T18:09:15.734404+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:32.736966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.994923+0000","last_change":"2024-03-28T18:08:20.027855+0000","last_active":"2024-03-28T18:09:43.994923+0000","last_peered":"2024-03-28T18:09:43.994923+0000","last_clean":"2024-03-28T18:09:43.994923+0000","last_became_active":"2024-03-28T18:08:20.027565+0000","last_became_peered":"2024-03-28T18:08:20.027565+0000","last_unstale":"2024-03-28T18:09:43.994923+0000","last_undegraded":"2024-03-28T18:09:43.994923+0000","last_fullsized":"2024-03-28T18:09:43.994923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:36:37.596573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.725729+0000","last_change":"2024-03-28T18:08:19.499055+0000","last_active":"2024-03-28T18:09:15.725729+0000","last_peered":"2024-03-28T18:09:15.725729+0000","last_clean":"2024-03-28T18:09:15.725729+0000","last_became_active":"2024-03-28T18:08:19.497964+0000","last_became_peered":"2024-03-28T18:08:19.497964+0000","last_unstale":"2024-03-28T18:09:15.725729+0000","last_undegraded":"2024-03-28T18:09:15.725729+0000","last_fullsized":"2024-03-28T18:09:15.725729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:38:45.620406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.011364+0000","last_change":"2024-03-28T18:08:20.028967+0000","last_active":"2024-03-28T18:09:44.011364+0000","last_peered":"2024-03-28T18:09:44.011364+0000","last_clean":"2024-03-28T18:09:44.011364+0000","last_became_active":"2024-03-28T18:08:20.028607+0000","last_became_peered":"2024-03-28T18:08:20.028607+0000","last_unstale":"2024-03-28T18:09:44.011364+0000","last_undegraded":"2024-03-28T18:09:44.011364+0000","last_fullsized":"2024-03-28T18:09:44.011364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:21:23.994077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.892649+0000","last_change":"2024-03-28T18:09:15.892649+0000","last_active":"2024-03-28T18:09:15.892649+0000","last_peered":"2024-03-28T18:09:15.892649+0000","last_clean":"2024-03-28T18:09:15.892649+0000","last_became_active":"2024-03-28T18:09:15.888520+0000","last_became_peered":"2024-03-28T18:09:15.888520+0000","last_unstale":"2024-03-28T18:09:15.892649+0000","last_undegraded":"2024-03-28T18:09:15.892649+0000","last_fullsized":"2024-03-28T18:09:15.892649+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:34.085738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.009420+0000","last_change":"2024-03-28T18:08:20.009878+0000","last_active":"2024-03-28T18:09:44.009420+0000","last_peered":"2024-03-28T18:09:44.009420+0000","last_clean":"2024-03-28T18:09:44.009420+0000","last_became_active":"2024-03-28T18:08:20.009534+0000","last_became_peered":"2024-03-28T18:08:20.009534+0000","last_unstale":"2024-03-28T18:09:44.009420+0000","last_undegraded":"2024-03-28T18:09:44.009420+0000","last_fullsized":"2024-03-28T18:09:44.009420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:58.975854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.715836+0000","last_change":"2024-03-28T18:08:19.472613+0000","last_active":"2024-03-28T18:09:15.715836+0000","last_peered":"2024-03-28T18:09:15.715836+0000","last_clean":"2024-03-28T18:09:15.715836+0000","last_became_active":"2024-03-28T18:08:19.472186+0000","last_became_peered":"2024-03-28T18:08:19.472186+0000","last_unstale":"2024-03-28T18:09:15.715836+0000","last_undegraded":"2024-03-28T18:09:15.715836+0000","last_fullsized":"2024-03-28T18:09:15.715836+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:50.807609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.972232+0000","last_change":"2024-03-28T18:08:20.008242+0000","last_active":"2024-03-28T18:09:43.972232+0000","last_peered":"2024-03-28T18:09:43.972232+0000","last_clean":"2024-03-28T18:09:43.972232+0000","last_became_active":"2024-03-28T18:08:20.007942+0000","last_became_peered":"2024-03-28T18:08:20.007942+0000","last_unstale":"2024-03-28T18:09:43.972232+0000","last_undegraded":"2024-03-28T18:09:43.972232+0000","last_fullsized":"2024-03-28T18:09:43.972232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:23.288978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'7","reported_seq":116,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:49.514986+0000","last_change":"2024-03-28T18:08:19.480869+0000","last_active":"2024-03-28T18:09:49.514986+0000","last_peered":"2024-03-28T18:09:49.514986+0000","last_clean":"2024-03-28T18:09:49.514986+0000","last_became_active":"2024-03-28T18:08:19.480441+0000","last_became_peered":"2024-03-28T18:08:19.480441+0000","last_unstale":"2024-03-28T18:09:49.514986+0000","last_undegraded":"2024-03-28T18:09:49.514986+0000","last_fullsized":"2024-03-28T18:09:49.514986+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:26.640605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.959483+0000","last_change":"2024-03-28T18:08:20.021530+0000","last_active":"2024-03-28T18:09:43.959483+0000","last_peered":"2024-03-28T18:09:43.959483+0000","last_clean":"2024-03-28T18:09:43.959483+0000","last_became_active":"2024-03-28T18:08:20.021179+0000","last_became_peered":"2024-03-28T18:08:20.021179+0000","last_unstale":"2024-03-28T18:09:43.959483+0000","last_undegraded":"2024-03-28T18:09:43.959483+0000","last_fullsized":"2024-03-28T18:09:43.959483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:43:07.300300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887260+0000","last_change":"2024-03-28T18:08:19.494597+0000","last_active":"2024-03-28T18:09:15.887260+0000","last_peered":"2024-03-28T18:09:15.887260+0000","last_clean":"2024-03-28T18:09:15.887260+0000","last_became_active":"2024-03-28T18:08:19.494178+0000","last_became_peered":"2024-03-28T18:08:19.494178+0000","last_unstale":"2024-03-28T18:09:15.887260+0000","last_undegraded":"2024-03-28T18:09:15.887260+0000","last_fullsized":"2024-03-28T18:09:15.887260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:08:10.257768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.991782+0000","last_change":"2024-03-28T18:08:20.021062+0000","last_active":"2024-03-28T18:09:43.991782+0000","last_peered":"2024-03-28T18:09:43.991782+0000","last_clean":"2024-03-28T18:09:43.991782+0000","last_became_active":"2024-03-28T18:08:20.020758+0000","last_became_peered":"2024-03-28T18:08:20.020758+0000","last_unstale":"2024-03-28T18:09:43.991782+0000","last_undegraded":"2024-03-28T18:09:43.991782+0000","last_fullsized":"2024-03-28T18:09:43.991782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:47:39.656513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.889688+0000","last_change":"2024-03-28T18:08:19.480189+0000","last_active":"2024-03-28T18:09:15.889688+0000","last_peered":"2024-03-28T18:09:15.889688+0000","last_clean":"2024-03-28T18:09:15.889688+0000","last_became_active":"2024-03-28T18:08:19.479812+0000","last_became_peered":"2024-03-28T18:08:19.479812+0000","last_unstale":"2024-03-28T18:09:15.889688+0000","last_undegraded":"2024-03-28T18:09:15.889688+0000","last_fullsized":"2024-03-28T18:09:15.889688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:46.408565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.004426+0000","last_change":"2024-03-28T18:08:20.023635+0000","last_active":"2024-03-28T18:09:44.004426+0000","last_peered":"2024-03-28T18:09:44.004426+0000","last_clean":"2024-03-28T18:09:44.004426+0000","last_became_active":"2024-03-28T18:08:20.023199+0000","last_became_peered":"2024-03-28T18:08:20.023199+0000","last_unstale":"2024-03-28T18:09:44.004426+0000","last_undegraded":"2024-03-28T18:09:44.004426+0000","last_fullsized":"2024-03-28T18:09:44.004426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:12.477899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.886860+0000","last_change":"2024-03-28T18:08:19.480858+0000","last_active":"2024-03-28T18:09:15.886860+0000","last_peered":"2024-03-28T18:09:15.886860+0000","last_clean":"2024-03-28T18:09:15.886860+0000","last_became_active":"2024-03-28T18:08:19.480418+0000","last_became_peered":"2024-03-28T18:08:19.480418+0000","last_unstale":"2024-03-28T18:09:15.886860+0000","last_undegraded":"2024-03-28T18:09:15.886860+0000","last_fullsized":"2024-03-28T18:09:15.886860+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:59:57.788915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.019764+0000","last_change":"2024-03-28T18:08:20.031719+0000","last_active":"2024-03-28T18:09:44.019764+0000","last_peered":"2024-03-28T18:09:44.019764+0000","last_clean":"2024-03-28T18:09:44.019764+0000","last_became_active":"2024-03-28T18:08:20.029482+0000","last_became_peered":"2024-03-28T18:08:20.029482+0000","last_unstale":"2024-03-28T18:09:44.019764+0000","last_undegraded":"2024-03-28T18:09:44.019764+0000","last_fullsized":"2024-03-28T18:09:44.019764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:13.226596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'4","reported_seq":108,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:48.513630+0000","last_change":"2024-03-28T18:08:19.483346+0000","last_active":"2024-03-28T18:09:48.513630+0000","last_peered":"2024-03-28T18:09:48.513630+0000","last_clean":"2024-03-28T18:09:48.513630+0000","last_became_active":"2024-03-28T18:08:19.483016+0000","last_became_peered":"2024-03-28T18:08:19.483016+0000","last_unstale":"2024-03-28T18:09:48.513630+0000","last_undegraded":"2024-03-28T18:09:48.513630+0000","last_fullsized":"2024-03-28T18:09:48.513630+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:34:33.008477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":85,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.982043+0000","last_change":"2024-03-28T18:08:20.020478+0000","last_active":"2024-03-28T18:09:43.982043+0000","last_peered":"2024-03-28T18:09:43.982043+0000","last_clean":"2024-03-28T18:09:43.982043+0000","last_became_active":"2024-03-28T18:08:20.020054+0000","last_became_peered":"2024-03-28T18:08:20.020054+0000","last_unstale":"2024-03-28T18:09:43.982043+0000","last_undegraded":"2024-03-28T18:09:43.982043+0000","last_fullsized":"2024-03-28T18:09:43.982043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:13:43.655549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716140+0000","last_change":"2024-03-28T18:08:19.479386+0000","last_active":"2024-03-28T18:09:15.716140+0000","last_peered":"2024-03-28T18:09:15.716140+0000","last_clean":"2024-03-28T18:09:15.716140+0000","last_became_active":"2024-03-28T18:08:19.478985+0000","last_became_peered":"2024-03-28T18:08:19.478985+0000","last_unstale":"2024-03-28T18:09:15.716140+0000","last_undegraded":"2024-03-28T18:09:15.716140+0000","last_fullsized":"2024-03-28T18:09:15.716140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:37:22.359475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.007627+0000","last_change":"2024-03-28T18:08:20.021790+0000","last_active":"2024-03-28T18:09:44.007627+0000","last_peered":"2024-03-28T18:09:44.007627+0000","last_clean":"2024-03-28T18:09:44.007627+0000","last_became_active":"2024-03-28T18:08:20.021031+0000","last_became_peered":"2024-03-28T18:08:20.021031+0000","last_unstale":"2024-03-28T18:09:44.007627+0000","last_undegraded":"2024-03-28T18:09:44.007627+0000","last_fullsized":"2024-03-28T18:09:44.007627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:59:50.897692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887205+0000","last_change":"2024-03-28T18:08:19.481576+0000","last_active":"2024-03-28T18:09:15.887205+0000","last_peered":"2024-03-28T18:09:15.887205+0000","last_clean":"2024-03-28T18:09:15.887205+0000","last_became_active":"2024-03-28T18:08:19.481226+0000","last_became_peered":"2024-03-28T18:08:19.481226+0000","last_unstale":"2024-03-28T18:09:15.887205+0000","last_undegraded":"2024-03-28T18:09:15.887205+0000","last_fullsized":"2024-03-28T18:09:15.887205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:54:44.677343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.013687+0000","last_change":"2024-03-28T18:09:15.890672+0000","last_active":"2024-03-28T18:09:44.013687+0000","last_peered":"2024-03-28T18:09:44.013687+0000","last_clean":"2024-03-28T18:09:44.013687+0000","last_became_active":"2024-03-28T18:09:15.890342+0000","last_became_peered":"2024-03-28T18:09:15.890342+0000","last_unstale":"2024-03-28T18:09:44.013687+0000","last_undegraded":"2024-03-28T18:09:44.013687+0000","last_fullsized":"2024-03-28T18:09:44.013687+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:08.268067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'32","reported_seq":230,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:49.514617+0000","last_change":"2024-03-28T18:08:19.477559+0000","last_active":"2024-03-28T18:09:49.514617+0000","last_peered":"2024-03-28T18:09:49.514617+0000","last_clean":"2024-03-28T18:09:49.514617+0000","last_became_active":"2024-03-28T18:08:19.477217+0000","last_became_peered":"2024-03-28T18:08:19.477217+0000","last_unstale":"2024-03-28T18:09:49.514617+0000","last_undegraded":"2024-03-28T18:09:49.514617+0000","last_fullsized":"2024-03-28T18:09:49.514617+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:37.071671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62145,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":179,"num_read_kb":150,"num_write":33,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.951533+0000","last_change":"2024-03-28T18:08:20.011303+0000","last_active":"2024-03-28T18:09:43.951533+0000","last_peered":"2024-03-28T18:09:43.951533+0000","last_clean":"2024-03-28T18:09:43.951533+0000","last_became_active":"2024-03-28T18:08:20.010997+0000","last_became_peered":"2024-03-28T18:08:20.010997+0000","last_unstale":"2024-03-28T18:09:43.951533+0000","last_undegraded":"2024-03-28T18:09:43.951533+0000","last_fullsized":"2024-03-28T18:09:43.951533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:10:56.036132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.718010+0000","last_change":"2024-03-28T18:08:19.477342+0000","last_active":"2024-03-28T18:09:15.718010+0000","last_peered":"2024-03-28T18:09:15.718010+0000","last_clean":"2024-03-28T18:09:15.718010+0000","last_became_active":"2024-03-28T18:08:19.476984+0000","last_became_peered":"2024-03-28T18:08:19.476984+0000","last_unstale":"2024-03-28T18:09:15.718010+0000","last_undegraded":"2024-03-28T18:09:15.718010+0000","last_fullsized":"2024-03-28T18:09:15.718010+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:15:42.606213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.017662+0000","last_change":"2024-03-28T18:08:20.030524+0000","last_active":"2024-03-28T18:09:44.017662+0000","last_peered":"2024-03-28T18:09:44.017662+0000","last_clean":"2024-03-28T18:09:44.017662+0000","last_became_active":"2024-03-28T18:08:20.030227+0000","last_became_peered":"2024-03-28T18:08:20.030227+0000","last_unstale":"2024-03-28T18:09:44.017662+0000","last_undegraded":"2024-03-28T18:09:44.017662+0000","last_fullsized":"2024-03-28T18:09:44.017662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:07:41.763739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.886681+0000","last_change":"2024-03-28T18:08:19.481169+0000","last_active":"2024-03-28T18:09:15.886681+0000","last_peered":"2024-03-28T18:09:15.886681+0000","last_clean":"2024-03-28T18:09:15.886681+0000","last_became_active":"2024-03-28T18:08:19.480634+0000","last_became_peered":"2024-03-28T18:08:19.480634+0000","last_unstale":"2024-03-28T18:09:15.886681+0000","last_undegraded":"2024-03-28T18:09:15.886681+0000","last_fullsized":"2024-03-28T18:09:15.886681+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:39.265215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.006895+0000","last_change":"2024-03-28T18:08:20.015163+0000","last_active":"2024-03-28T18:09:44.006895+0000","last_peered":"2024-03-28T18:09:44.006895+0000","last_clean":"2024-03-28T18:09:44.006895+0000","last_became_active":"2024-03-28T18:08:20.014765+0000","last_became_peered":"2024-03-28T18:08:20.014765+0000","last_unstale":"2024-03-28T18:09:44.006895+0000","last_undegraded":"2024-03-28T18:09:44.006895+0000","last_fullsized":"2024-03-28T18:09:44.006895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:51.035532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'3","reported_seq":108,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:47.541844+0000","last_change":"2024-03-28T18:08:19.479086+0000","last_active":"2024-03-28T18:09:47.541844+0000","last_peered":"2024-03-28T18:09:47.541844+0000","last_clean":"2024-03-28T18:09:47.541844+0000","last_became_active":"2024-03-28T18:08:19.478748+0000","last_became_peered":"2024-03-28T18:08:19.478748+0000","last_unstale":"2024-03-28T18:09:47.541844+0000","last_undegraded":"2024-03-28T18:09:47.541844+0000","last_fullsized":"2024-03-28T18:09:47.541844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:46:59.566998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.973553+0000","last_change":"2024-03-28T18:09:15.888751+0000","last_active":"2024-03-28T18:09:43.973553+0000","last_peered":"2024-03-28T18:09:43.973553+0000","last_clean":"2024-03-28T18:09:43.973553+0000","last_became_active":"2024-03-28T18:09:15.888350+0000","last_became_peered":"2024-03-28T18:09:15.888350+0000","last_unstale":"2024-03-28T18:09:43.973553+0000","last_undegraded":"2024-03-28T18:09:43.973553+0000","last_fullsized":"2024-03-28T18:09:43.973553+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:38:05.822205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":59,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.733434+0000","last_change":"2024-03-28T18:09:15.733434+0000","last_active":"2024-03-28T18:09:15.733434+0000","last_peered":"2024-03-28T18:09:15.733434+0000","last_clean":"2024-03-28T18:09:15.733434+0000","last_became_active":"2024-03-28T18:09:15.722437+0000","last_became_peered":"2024-03-28T18:09:15.722437+0000","last_unstale":"2024-03-28T18:09:15.733434+0000","last_undegraded":"2024-03-28T18:09:15.733434+0000","last_fullsized":"2024-03-28T18:09:15.733434+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:35:18.313676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.971525+0000","last_change":"2024-03-28T18:08:20.007457+0000","last_active":"2024-03-28T18:09:43.971525+0000","last_peered":"2024-03-28T18:09:43.971525+0000","last_clean":"2024-03-28T18:09:43.971525+0000","last_became_active":"2024-03-28T18:08:20.007090+0000","last_became_peered":"2024-03-28T18:08:20.007090+0000","last_unstale":"2024-03-28T18:09:43.971525+0000","last_undegraded":"2024-03-28T18:09:43.971525+0000","last_fullsized":"2024-03-28T18:09:43.971525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:22:45.702706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716349+0000","last_change":"2024-03-28T18:08:19.475104+0000","last_active":"2024-03-28T18:09:15.716349+0000","last_peered":"2024-03-28T18:09:15.716349+0000","last_clean":"2024-03-28T18:09:15.716349+0000","last_became_active":"2024-03-28T18:08:19.474754+0000","last_became_peered":"2024-03-28T18:08:19.474754+0000","last_unstale":"2024-03-28T18:09:15.716349+0000","last_undegraded":"2024-03-28T18:09:15.716349+0000","last_fullsized":"2024-03-28T18:09:15.716349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:19.085631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.963148+0000","last_change":"2024-03-28T18:08:20.009347+0000","last_active":"2024-03-28T18:09:43.963148+0000","last_peered":"2024-03-28T18:09:43.963148+0000","last_clean":"2024-03-28T18:09:43.963148+0000","last_became_active":"2024-03-28T18:08:20.009012+0000","last_became_peered":"2024-03-28T18:08:20.009012+0000","last_unstale":"2024-03-28T18:09:43.963148+0000","last_undegraded":"2024-03-28T18:09:43.963148+0000","last_fullsized":"2024-03-28T18:09:43.963148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:04:26.625830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.721904+0000","last_change":"2024-03-28T18:08:19.482414+0000","last_active":"2024-03-28T18:09:15.721904+0000","last_peered":"2024-03-28T18:09:15.721904+0000","last_clean":"2024-03-28T18:09:15.721904+0000","last_became_active":"2024-03-28T18:08:19.481830+0000","last_became_peered":"2024-03-28T18:08:19.481830+0000","last_unstale":"2024-03-28T18:09:15.721904+0000","last_undegraded":"2024-03-28T18:09:15.721904+0000","last_fullsized":"2024-03-28T18:09:15.721904+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:27:14.259355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.001697+0000","last_change":"2024-03-28T18:08:20.030687+0000","last_active":"2024-03-28T18:09:44.001697+0000","last_peered":"2024-03-28T18:09:44.001697+0000","last_clean":"2024-03-28T18:09:44.001697+0000","last_became_active":"2024-03-28T18:08:20.030308+0000","last_became_peered":"2024-03-28T18:08:20.030308+0000","last_unstale":"2024-03-28T18:09:44.001697+0000","last_undegraded":"2024-03-28T18:09:44.001697+0000","last_fullsized":"2024-03-28T18:09:44.001697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:31.745952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.715942+0000","last_change":"2024-03-28T18:08:19.494446+0000","last_active":"2024-03-28T18:09:15.715942+0000","last_peered":"2024-03-28T18:09:15.715942+0000","last_clean":"2024-03-28T18:09:15.715942+0000","last_became_active":"2024-03-28T18:08:19.494036+0000","last_became_peered":"2024-03-28T18:08:19.494036+0000","last_unstale":"2024-03-28T18:09:15.715942+0000","last_undegraded":"2024-03-28T18:09:15.715942+0000","last_fullsized":"2024-03-28T18:09:15.715942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:42:53.660741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.009319+0000","last_change":"2024-03-28T18:08:20.025706+0000","last_active":"2024-03-28T18:09:44.009319+0000","last_peered":"2024-03-28T18:09:44.009319+0000","last_clean":"2024-03-28T18:09:44.009319+0000","last_became_active":"2024-03-28T18:08:20.025343+0000","last_became_peered":"2024-03-28T18:08:20.025343+0000","last_unstale":"2024-03-28T18:09:44.009319+0000","last_undegraded":"2024-03-28T18:09:44.009319+0000","last_fullsized":"2024-03-28T18:09:44.009319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:32:27.883940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.963857+0000","last_change":"2024-03-28T18:08:20.009636+0000","last_active":"2024-03-28T18:09:43.963857+0000","last_peered":"2024-03-28T18:09:43.963857+0000","last_clean":"2024-03-28T18:09:43.963857+0000","last_became_active":"2024-03-28T18:08:20.009178+0000","last_became_peered":"2024-03-28T18:08:20.009178+0000","last_unstale":"2024-03-28T18:09:43.963857+0000","last_undegraded":"2024-03-28T18:09:43.963857+0000","last_fullsized":"2024-03-28T18:09:43.963857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:28.461411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":108,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:48.543486+0000","last_change":"2024-03-28T18:08:19.501287+0000","last_active":"2024-03-28T18:09:48.543486+0000","last_peered":"2024-03-28T18:09:48.543486+0000","last_clean":"2024-03-28T18:09:48.543486+0000","last_became_active":"2024-03-28T18:08:19.500920+0000","last_became_peered":"2024-03-28T18:08:19.500920+0000","last_unstale":"2024-03-28T18:09:48.543486+0000","last_undegraded":"2024-03-28T18:09:48.543486+0000","last_fullsized":"2024-03-28T18:09:48.543486+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:29.882164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":85,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.010657+0000","last_change":"2024-03-28T18:08:20.020811+0000","last_active":"2024-03-28T18:09:44.010657+0000","last_peered":"2024-03-28T18:09:44.010657+0000","last_clean":"2024-03-28T18:09:44.010657+0000","last_became_active":"2024-03-28T18:08:20.020381+0000","last_became_peered":"2024-03-28T18:08:20.020381+0000","last_unstale":"2024-03-28T18:09:44.010657+0000","last_undegraded":"2024-03-28T18:09:44.010657+0000","last_fullsized":"2024-03-28T18:09:44.010657+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:10:35.999858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887706+0000","last_change":"2024-03-28T18:08:19.478020+0000","last_active":"2024-03-28T18:09:15.887706+0000","last_peered":"2024-03-28T18:09:15.887706+0000","last_clean":"2024-03-28T18:09:15.887706+0000","last_became_active":"2024-03-28T18:08:19.477627+0000","last_became_peered":"2024-03-28T18:08:19.477627+0000","last_unstale":"2024-03-28T18:09:15.887706+0000","last_undegraded":"2024-03-28T18:09:15.887706+0000","last_fullsized":"2024-03-28T18:09:15.887706+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:00:01.649974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":354,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.715885+0000","last_change":"2024-03-28T18:06:58.750932+0000","last_active":"2024-03-28T18:09:15.715885+0000","last_peered":"2024-03-28T18:09:15.715885+0000","last_clean":"2024-03-28T18:09:15.715885+0000","last_became_active":"2024-03-28T18:06:58.750309+0000","last_became_peered":"2024-03-28T18:06:58.750309+0000","last_unstale":"2024-03-28T18:09:15.715885+0000","last_undegraded":"2024-03-28T18:09:15.715885+0000","last_fullsized":"2024-03-28T18:09:15.715885+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:03:00.832683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:03:00.832683+0000","last_clean_scrub_stamp":"2024-03-28T18:03:00.832683+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:46.001886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.001884+0000","last_change":"2024-03-28T18:08:20.009573+0000","last_active":"2024-03-28T18:09:44.001884+0000","last_peered":"2024-03-28T18:09:44.001884+0000","last_clean":"2024-03-28T18:09:44.001884+0000","last_became_active":"2024-03-28T18:08:20.009143+0000","last_became_peered":"2024-03-28T18:08:20.009143+0000","last_unstale":"2024-03-28T18:09:44.001884+0000","last_undegraded":"2024-03-28T18:09:44.001884+0000","last_fullsized":"2024-03-28T18:09:44.001884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:37:50.643356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716024+0000","last_change":"2024-03-28T18:08:19.483733+0000","last_active":"2024-03-28T18:09:15.716024+0000","last_peered":"2024-03-28T18:09:15.716024+0000","last_clean":"2024-03-28T18:09:15.716024+0000","last_became_active":"2024-03-28T18:08:19.483366+0000","last_became_peered":"2024-03-28T18:08:19.483366+0000","last_unstale":"2024-03-28T18:09:15.716024+0000","last_undegraded":"2024-03-28T18:09:15.716024+0000","last_fullsized":"2024-03-28T18:09:15.716024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:29.192442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.715991+0000","last_change":"2024-03-28T18:08:19.481844+0000","last_active":"2024-03-28T18:09:15.715991+0000","last_peered":"2024-03-28T18:09:15.715991+0000","last_clean":"2024-03-28T18:09:15.715991+0000","last_became_active":"2024-03-28T18:08:19.481430+0000","last_became_peered":"2024-03-28T18:08:19.481430+0000","last_unstale":"2024-03-28T18:09:15.715991+0000","last_undegraded":"2024-03-28T18:09:15.715991+0000","last_fullsized":"2024-03-28T18:09:15.715991+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:21:28.789883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.009465+0000","last_change":"2024-03-28T18:08:20.021037+0000","last_active":"2024-03-28T18:09:44.009465+0000","last_peered":"2024-03-28T18:09:44.009465+0000","last_clean":"2024-03-28T18:09:44.009465+0000","last_became_active":"2024-03-28T18:08:20.020680+0000","last_became_peered":"2024-03-28T18:08:20.020680+0000","last_unstale":"2024-03-28T18:09:44.009465+0000","last_undegraded":"2024-03-28T18:09:44.009465+0000","last_fullsized":"2024-03-28T18:09:44.009465+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:08:28.825774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'10","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.738136+0000","last_change":"2024-03-28T18:09:15.737583+0000","last_active":"2024-03-28T18:09:15.738136+0000","last_peered":"2024-03-28T18:09:15.738136+0000","last_clean":"2024-03-28T18:09:15.738136+0000","last_became_active":"2024-03-28T18:09:15.726579+0000","last_became_peered":"2024-03-28T18:09:15.726579+0000","last_unstale":"2024-03-28T18:09:15.738136+0000","last_undegraded":"2024-03-28T18:09:15.738136+0000","last_fullsized":"2024-03-28T18:09:15.738136+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:03.380037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.008936+0000","last_change":"2024-03-28T18:08:20.024938+0000","last_active":"2024-03-28T18:09:44.008936+0000","last_peered":"2024-03-28T18:09:44.008936+0000","last_clean":"2024-03-28T18:09:44.008936+0000","last_became_active":"2024-03-28T18:08:20.024536+0000","last_became_peered":"2024-03-28T18:08:20.024536+0000","last_unstale":"2024-03-28T18:09:44.008936+0000","last_undegraded":"2024-03-28T18:09:44.008936+0000","last_fullsized":"2024-03-28T18:09:44.008936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:46:27.080005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887972+0000","last_change":"2024-03-28T18:08:19.480310+0000","last_active":"2024-03-28T18:09:15.887972+0000","last_peered":"2024-03-28T18:09:15.887972+0000","last_clean":"2024-03-28T18:09:15.887972+0000","last_became_active":"2024-03-28T18:08:19.479919+0000","last_became_peered":"2024-03-28T18:08:19.479919+0000","last_unstale":"2024-03-28T18:09:15.887972+0000","last_undegraded":"2024-03-28T18:09:15.887972+0000","last_fullsized":"2024-03-28T18:09:15.887972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:36:26.239515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.018790+0000","last_change":"2024-03-28T18:08:20.033974+0000","last_active":"2024-03-28T18:09:44.018790+0000","last_peered":"2024-03-28T18:09:44.018790+0000","last_clean":"2024-03-28T18:09:44.018790+0000","last_became_active":"2024-03-28T18:08:20.033544+0000","last_became_peered":"2024-03-28T18:08:20.033544+0000","last_unstale":"2024-03-28T18:09:44.018790+0000","last_undegraded":"2024-03-28T18:09:44.018790+0000","last_fullsized":"2024-03-28T18:09:44.018790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:45:09.617988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.718345+0000","last_change":"2024-03-28T18:08:19.482604+0000","last_active":"2024-03-28T18:09:15.718345+0000","last_peered":"2024-03-28T18:09:15.718345+0000","last_clean":"2024-03-28T18:09:15.718345+0000","last_became_active":"2024-03-28T18:08:19.482248+0000","last_became_peered":"2024-03-28T18:08:19.482248+0000","last_unstale":"2024-03-28T18:09:15.718345+0000","last_undegraded":"2024-03-28T18:09:15.718345+0000","last_fullsized":"2024-03-28T18:09:15.718345+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:23.199196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.929801+0000","last_change":"2024-03-28T18:08:20.011198+0000","last_active":"2024-03-28T18:09:43.929801+0000","last_peered":"2024-03-28T18:09:43.929801+0000","last_clean":"2024-03-28T18:09:43.929801+0000","last_became_active":"2024-03-28T18:08:20.010866+0000","last_became_peered":"2024-03-28T18:08:20.010866+0000","last_unstale":"2024-03-28T18:09:43.929801+0000","last_undegraded":"2024-03-28T18:09:43.929801+0000","last_fullsized":"2024-03-28T18:09:43.929801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:26.342858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.724808+0000","last_change":"2024-03-28T18:08:19.493830+0000","last_active":"2024-03-28T18:09:15.724808+0000","last_peered":"2024-03-28T18:09:15.724808+0000","last_clean":"2024-03-28T18:09:15.724808+0000","last_became_active":"2024-03-28T18:08:19.493447+0000","last_became_peered":"2024-03-28T18:08:19.493447+0000","last_unstale":"2024-03-28T18:09:15.724808+0000","last_undegraded":"2024-03-28T18:09:15.724808+0000","last_fullsized":"2024-03-28T18:09:15.724808+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:33:29.797186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.993414+0000","last_change":"2024-03-28T18:08:20.025620+0000","last_active":"2024-03-28T18:09:43.993414+0000","last_peered":"2024-03-28T18:09:43.993414+0000","last_clean":"2024-03-28T18:09:43.993414+0000","last_became_active":"2024-03-28T18:08:20.025276+0000","last_became_peered":"2024-03-28T18:08:20.025276+0000","last_unstale":"2024-03-28T18:09:43.993414+0000","last_undegraded":"2024-03-28T18:09:43.993414+0000","last_fullsized":"2024-03-28T18:09:43.993414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:30:06.667707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.718293+0000","last_change":"2024-03-28T18:08:19.474204+0000","last_active":"2024-03-28T18:09:15.718293+0000","last_peered":"2024-03-28T18:09:15.718293+0000","last_clean":"2024-03-28T18:09:15.718293+0000","last_became_active":"2024-03-28T18:08:19.473688+0000","last_became_peered":"2024-03-28T18:08:19.473688+0000","last_unstale":"2024-03-28T18:09:15.718293+0000","last_undegraded":"2024-03-28T18:09:15.718293+0000","last_fullsized":"2024-03-28T18:09:15.718293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:15:09.880974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.015079+0000","last_change":"2024-03-28T18:08:20.035920+0000","last_active":"2024-03-28T18:09:44.015079+0000","last_peered":"2024-03-28T18:09:44.015079+0000","last_clean":"2024-03-28T18:09:44.015079+0000","last_became_active":"2024-03-28T18:08:20.035600+0000","last_became_peered":"2024-03-28T18:08:20.035600+0000","last_unstale":"2024-03-28T18:09:44.015079+0000","last_undegraded":"2024-03-28T18:09:44.015079+0000","last_fullsized":"2024-03-28T18:09:44.015079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:41.280448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.715528+0000","last_change":"2024-03-28T18:08:19.481305+0000","last_active":"2024-03-28T18:09:15.715528+0000","last_peered":"2024-03-28T18:09:15.715528+0000","last_clean":"2024-03-28T18:09:15.715528+0000","last_became_active":"2024-03-28T18:08:19.480978+0000","last_became_peered":"2024-03-28T18:08:19.480978+0000","last_unstale":"2024-03-28T18:09:15.715528+0000","last_undegraded":"2024-03-28T18:09:15.715528+0000","last_fullsized":"2024-03-28T18:09:15.715528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:14:06.464967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.002187+0000","last_change":"2024-03-28T18:08:20.017660+0000","last_active":"2024-03-28T18:09:44.002187+0000","last_peered":"2024-03-28T18:09:44.002187+0000","last_clean":"2024-03-28T18:09:44.002187+0000","last_became_active":"2024-03-28T18:08:20.016083+0000","last_became_peered":"2024-03-28T18:08:20.016083+0000","last_unstale":"2024-03-28T18:09:44.002187+0000","last_undegraded":"2024-03-28T18:09:44.002187+0000","last_fullsized":"2024-03-28T18:09:44.002187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:33:12.104274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.718265+0000","last_change":"2024-03-28T18:08:19.479855+0000","last_active":"2024-03-28T18:09:15.718265+0000","last_peered":"2024-03-28T18:09:15.718265+0000","last_clean":"2024-03-28T18:09:15.718265+0000","last_became_active":"2024-03-28T18:08:19.479429+0000","last_became_peered":"2024-03-28T18:08:19.479429+0000","last_unstale":"2024-03-28T18:09:15.718265+0000","last_undegraded":"2024-03-28T18:09:15.718265+0000","last_fullsized":"2024-03-28T18:09:15.718265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:06:23.286970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.978410+0000","last_change":"2024-03-28T18:08:20.015419+0000","last_active":"2024-03-28T18:09:43.978410+0000","last_peered":"2024-03-28T18:09:43.978410+0000","last_clean":"2024-03-28T18:09:43.978410+0000","last_became_active":"2024-03-28T18:08:20.015011+0000","last_became_peered":"2024-03-28T18:08:20.015011+0000","last_unstale":"2024-03-28T18:09:43.978410+0000","last_undegraded":"2024-03-28T18:09:43.978410+0000","last_fullsized":"2024-03-28T18:09:43.978410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:44:46.858352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887075+0000","last_change":"2024-03-28T18:08:19.495541+0000","last_active":"2024-03-28T18:09:15.887075+0000","last_peered":"2024-03-28T18:09:15.887075+0000","last_clean":"2024-03-28T18:09:15.887075+0000","last_became_active":"2024-03-28T18:08:19.495173+0000","last_became_peered":"2024-03-28T18:08:19.495173+0000","last_unstale":"2024-03-28T18:09:15.887075+0000","last_undegraded":"2024-03-28T18:09:15.887075+0000","last_fullsized":"2024-03-28T18:09:15.887075+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:42:06.423107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.981064+0000","last_change":"2024-03-28T18:08:20.018582+0000","last_active":"2024-03-28T18:09:43.981064+0000","last_peered":"2024-03-28T18:09:43.981064+0000","last_clean":"2024-03-28T18:09:43.981064+0000","last_became_active":"2024-03-28T18:08:20.017936+0000","last_became_peered":"2024-03-28T18:08:20.017936+0000","last_unstale":"2024-03-28T18:09:43.981064+0000","last_undegraded":"2024-03-28T18:09:43.981064+0000","last_fullsized":"2024-03-28T18:09:43.981064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:05:49.510246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.888923+0000","last_change":"2024-03-28T18:08:19.497049+0000","last_active":"2024-03-28T18:09:15.888923+0000","last_peered":"2024-03-28T18:09:15.888923+0000","last_clean":"2024-03-28T18:09:15.888923+0000","last_became_active":"2024-03-28T18:08:19.496623+0000","last_became_peered":"2024-03-28T18:08:19.496623+0000","last_unstale":"2024-03-28T18:09:15.888923+0000","last_undegraded":"2024-03-28T18:09:15.888923+0000","last_fullsized":"2024-03-28T18:09:15.888923+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:55:07.842439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.974619+0000","last_change":"2024-03-28T18:08:20.017419+0000","last_active":"2024-03-28T18:09:43.974619+0000","last_peered":"2024-03-28T18:09:43.974619+0000","last_clean":"2024-03-28T18:09:43.974619+0000","last_became_active":"2024-03-28T18:08:20.015916+0000","last_became_peered":"2024-03-28T18:08:20.015916+0000","last_unstale":"2024-03-28T18:09:43.974619+0000","last_undegraded":"2024-03-28T18:09:43.974619+0000","last_fullsized":"2024-03-28T18:09:43.974619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:10:55.927794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.964015+0000","last_change":"2024-03-28T18:08:20.008093+0000","last_active":"2024-03-28T18:09:43.964015+0000","last_peered":"2024-03-28T18:09:43.964015+0000","last_clean":"2024-03-28T18:09:43.964015+0000","last_became_active":"2024-03-28T18:08:20.007801+0000","last_became_peered":"2024-03-28T18:08:20.007801+0000","last_unstale":"2024-03-28T18:09:43.964015+0000","last_undegraded":"2024-03-28T18:09:43.964015+0000","last_fullsized":"2024-03-28T18:09:43.964015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:58.976859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.967721+0000","last_change":"2024-03-28T18:08:20.012771+0000","last_active":"2024-03-28T18:09:43.967721+0000","last_peered":"2024-03-28T18:09:43.967721+0000","last_clean":"2024-03-28T18:09:43.967721+0000","last_became_active":"2024-03-28T18:08:20.012456+0000","last_became_peered":"2024-03-28T18:08:20.012456+0000","last_unstale":"2024-03-28T18:09:43.967721+0000","last_undegraded":"2024-03-28T18:09:43.967721+0000","last_fullsized":"2024-03-28T18:09:43.967721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:56.428332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.000349+0000","last_change":"2024-03-28T18:08:20.034872+0000","last_active":"2024-03-28T18:09:44.000349+0000","last_peered":"2024-03-28T18:09:44.000349+0000","last_clean":"2024-03-28T18:09:44.000349+0000","last_became_active":"2024-03-28T18:08:20.034614+0000","last_became_peered":"2024-03-28T18:08:20.034614+0000","last_unstale":"2024-03-28T18:09:44.000349+0000","last_undegraded":"2024-03-28T18:09:44.000349+0000","last_fullsized":"2024-03-28T18:09:44.000349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:35.658186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.017643+0000","last_change":"2024-03-28T18:08:20.033119+0000","last_active":"2024-03-28T18:09:44.017643+0000","last_peered":"2024-03-28T18:09:44.017643+0000","last_clean":"2024-03-28T18:09:44.017643+0000","last_became_active":"2024-03-28T18:08:20.030694+0000","last_became_peered":"2024-03-28T18:08:20.030694+0000","last_unstale":"2024-03-28T18:09:44.017643+0000","last_undegraded":"2024-03-28T18:09:44.017643+0000","last_fullsized":"2024-03-28T18:09:44.017643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:51:50.082993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.020345+0000","last_change":"2024-03-28T18:08:20.025725+0000","last_active":"2024-03-28T18:09:44.020345+0000","last_peered":"2024-03-28T18:09:44.020345+0000","last_clean":"2024-03-28T18:09:44.020345+0000","last_became_active":"2024-03-28T18:08:20.024929+0000","last_became_peered":"2024-03-28T18:08:20.024929+0000","last_unstale":"2024-03-28T18:09:44.020345+0000","last_undegraded":"2024-03-28T18:09:44.020345+0000","last_fullsized":"2024-03-28T18:09:44.020345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:04.894246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.960815+0000","last_change":"2024-03-28T18:08:20.005293+0000","last_active":"2024-03-28T18:09:43.960815+0000","last_peered":"2024-03-28T18:09:43.960815+0000","last_clean":"2024-03-28T18:09:43.960815+0000","last_became_active":"2024-03-28T18:08:20.004938+0000","last_became_peered":"2024-03-28T18:08:20.004938+0000","last_unstale":"2024-03-28T18:09:43.960815+0000","last_undegraded":"2024-03-28T18:09:43.960815+0000","last_fullsized":"2024-03-28T18:09:43.960815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:36.100810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.971631+0000","last_change":"2024-03-28T18:08:20.008741+0000","last_active":"2024-03-28T18:09:43.971631+0000","last_peered":"2024-03-28T18:09:43.971631+0000","last_clean":"2024-03-28T18:09:43.971631+0000","last_became_active":"2024-03-28T18:08:20.008411+0000","last_became_peered":"2024-03-28T18:08:20.008411+0000","last_unstale":"2024-03-28T18:09:43.971631+0000","last_undegraded":"2024-03-28T18:09:43.971631+0000","last_fullsized":"2024-03-28T18:09:43.971631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:28:52.276278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.996983+0000","last_change":"2024-03-28T18:08:20.028486+0000","last_active":"2024-03-28T18:09:43.996983+0000","last_peered":"2024-03-28T18:09:43.996983+0000","last_clean":"2024-03-28T18:09:43.996983+0000","last_became_active":"2024-03-28T18:08:20.028121+0000","last_became_peered":"2024-03-28T18:08:20.028121+0000","last_unstale":"2024-03-28T18:09:43.996983+0000","last_undegraded":"2024-03-28T18:09:43.996983+0000","last_fullsized":"2024-03-28T18:09:43.996983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:55:26.828305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.966350+0000","last_change":"2024-03-28T18:08:20.028085+0000","last_active":"2024-03-28T18:09:43.966350+0000","last_peered":"2024-03-28T18:09:43.966350+0000","last_clean":"2024-03-28T18:09:43.966350+0000","last_became_active":"2024-03-28T18:08:20.027721+0000","last_became_peered":"2024-03-28T18:08:20.027721+0000","last_unstale":"2024-03-28T18:09:43.966350+0000","last_undegraded":"2024-03-28T18:09:43.966350+0000","last_fullsized":"2024-03-28T18:09:43.966350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:31:36.674002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.011940+0000","last_change":"2024-03-28T18:08:20.023605+0000","last_active":"2024-03-28T18:09:44.011940+0000","last_peered":"2024-03-28T18:09:44.011940+0000","last_clean":"2024-03-28T18:09:44.011940+0000","last_became_active":"2024-03-28T18:08:20.018217+0000","last_became_peered":"2024-03-28T18:08:20.018217+0000","last_unstale":"2024-03-28T18:09:44.011940+0000","last_undegraded":"2024-03-28T18:09:44.011940+0000","last_fullsized":"2024-03-28T18:09:44.011940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:43.717751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.014807+0000","last_change":"2024-03-28T18:08:20.023072+0000","last_active":"2024-03-28T18:09:44.014807+0000","last_peered":"2024-03-28T18:09:44.014807+0000","last_clean":"2024-03-28T18:09:44.014807+0000","last_became_active":"2024-03-28T18:08:20.022690+0000","last_became_peered":"2024-03-28T18:08:20.022690+0000","last_unstale":"2024-03-28T18:09:44.014807+0000","last_undegraded":"2024-03-28T18:09:44.014807+0000","last_fullsized":"2024-03-28T18:09:44.014807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:21.391415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.018858+0000","last_change":"2024-03-28T18:08:20.024261+0000","last_active":"2024-03-28T18:09:44.018858+0000","last_peered":"2024-03-28T18:09:44.018858+0000","last_clean":"2024-03-28T18:09:44.018858+0000","last_became_active":"2024-03-28T18:08:20.023792+0000","last_became_peered":"2024-03-28T18:08:20.023792+0000","last_unstale":"2024-03-28T18:09:44.018858+0000","last_undegraded":"2024-03-28T18:09:44.018858+0000","last_fullsized":"2024-03-28T18:09:44.018858+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:23:08.030982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.961667+0000","last_change":"2024-03-28T18:09:15.720113+0000","last_active":"2024-03-28T18:09:43.961667+0000","last_peered":"2024-03-28T18:09:43.961667+0000","last_clean":"2024-03-28T18:09:43.961667+0000","last_became_active":"2024-03-28T18:09:15.719739+0000","last_became_peered":"2024-03-28T18:09:15.719739+0000","last_unstale":"2024-03-28T18:09:43.961667+0000","last_undegraded":"2024-03-28T18:09:43.961667+0000","last_fullsized":"2024-03-28T18:09:43.961667+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:09:38.179780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":248983,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66287,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":539,"num_read_kb":340,"num_write":166,"num_write_kb":115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":694,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":198861,"data_compressed":5634,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383712,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":456779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645350,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":1109823,"data_compressed":37917,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808875,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070513,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332152,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":1047842,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495677,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":455822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":517872,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":457093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1024000,"data_stored":519530,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":456133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920842,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":456259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757315,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":1048108,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34394,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34394,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62257,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":63915,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63723,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:09:53.016 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T18:09:53.552 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:09:54.023 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:53 smithi037 ceph-mon[31460]: pgmap v366: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T18:09:54.023 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:53 smithi037 ceph-mon[31460]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:09:54.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:53 smithi059 ceph-mon[36190]: pgmap v366: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T18:09:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:53 smithi059 ceph-mon[36190]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:09:54.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:53 smithi170 ceph-mon[36280]: pgmap v366: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T18:09:54.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:53 smithi170 ceph-mon[36280]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:09:55.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.914+0000 7f58ac000700 1 -- 172.21.15.37:0/3801276431 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58a406a930 msgr2=0x7f58a406ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:55.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.914+0000 7f58ac000700 1 --2- 172.21.15.37:0/3801276431 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58a406a930 0x7f58a406ad10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f58980099c0 tx=0x7f5898009cd0 comp rx=0 tx=0).stop 2024-03-28T18:09:55.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.914+0000 7f58ac000700 1 -- 172.21.15.37:0/3801276431 shutdown_connections 2024-03-28T18:09:55.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.914+0000 7f58ac000700 1 --2- 172.21.15.37:0/3801276431 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58a4107510 0x7f58a410f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:55.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.914+0000 7f58ac000700 1 --2- 172.21.15.37:0/3801276431 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58a406a930 0x7f58a406ad10 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:55.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.914+0000 7f58ac000700 1 --2- 172.21.15.37:0/3801276431 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58a406b2e0 0x7f58a4106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:55.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.914+0000 7f58ac000700 1 -- 172.21.15.37:0/3801276431 >> 172.21.15.37:0/3801276431 conn(0x7f58a40770e0 msgr2=0x7f58a40774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:55.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.914+0000 7f58ac000700 1 -- 172.21.15.37:0/3801276431 shutdown_connections 2024-03-28T18:09:55.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.915+0000 7f58ac000700 1 -- 172.21.15.37:0/3801276431 wait complete. 2024-03-28T18:09:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.916+0000 7f58ac000700 1 Processor -- start 2024-03-28T18:09:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.916+0000 7f58ac000700 1 -- start start 2024-03-28T18:09:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.916+0000 7f58ac000700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58a406a930 0x7f58a4109030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.916+0000 7f58ac000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58a406b2e0 0x7f58a4109570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.916+0000 7f58ac000700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58a4107510 0x7f58a4109ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.916+0000 7f58ac000700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a41124f0 con 0x7f58a406b2e0 2024-03-28T18:09:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.916+0000 7f58ac000700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a4112370 con 0x7f58a4107510 2024-03-28T18:09:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.916+0000 7f58ac000700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a4112670 con 0x7f58a406a930 2024-03-28T18:09:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.917+0000 7f58a9600700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58a406a930 0x7f58a4109030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:55.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.917+0000 7f58a9600700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58a406a930 0x7f58a4109030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:51150/0 (socket says 172.21.15.37:51150) 2024-03-28T18:09:55.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.917+0000 7f58a9600700 1 -- 172.21.15.37:0/2381352647 learned_addr learned my addr 172.21.15.37:0/2381352647 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:55.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.917+0000 7f58aa000700 1 --2- 172.21.15.37:0/2381352647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58a4107510 0x7f58a4109ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:55.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.917+0000 7f58a8c00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58a406b2e0 0x7f58a4109570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:55.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.917+0000 7f58a9600700 1 -- 172.21.15.37:0/2381352647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58a4107510 msgr2=0x7f58a4109ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:55.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.917+0000 7f58a9600700 1 --2- 172.21.15.37:0/2381352647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58a4107510 0x7f58a4109ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:55.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.917+0000 7f58a9600700 1 -- 172.21.15.37:0/2381352647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58a406b2e0 msgr2=0x7f58a4109570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:55.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.917+0000 7f58a9600700 1 --2- 172.21.15.37:0/2381352647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58a406b2e0 0x7f58a4109570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:55.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.917+0000 7f58a9600700 1 -- 172.21.15.37:0/2381352647 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a41ab210 con 0x7f58a406a930 2024-03-28T18:09:55.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.917+0000 7f58a9600700 1 --2- 172.21.15.37:0/2381352647 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58a406a930 0x7f58a4109030 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5898005910 tx=0x7f5898009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:55.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.918+0000 7f58a2000700 1 -- 172.21.15.37:0/2381352647 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5898027070 con 0x7f58a406a930 2024-03-28T18:09:55.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.918+0000 7f58a2000700 1 -- 172.21.15.37:0/2381352647 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5898003780 con 0x7f58a406a930 2024-03-28T18:09:55.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.918+0000 7f58ac000700 1 -- 172.21.15.37:0/2381352647 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a41ab4a0 con 0x7f58a406a930 2024-03-28T18:09:55.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.918+0000 7f58a2000700 1 -- 172.21.15.37:0/2381352647 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5898022c90 con 0x7f58a406a930 2024-03-28T18:09:55.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.918+0000 7f58ac000700 1 -- 172.21.15.37:0/2381352647 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a41ab950 con 0x7f58a406a930 2024-03-28T18:09:55.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.919+0000 7f58ac000700 1 -- 172.21.15.37:0/2381352647 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a41038c0 con 0x7f58a406a930 2024-03-28T18:09:55.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.920+0000 7f58a2000700 1 -- 172.21.15.37:0/2381352647 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f589802a070 con 0x7f58a406a930 2024-03-28T18:09:55.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.922+0000 7f58a2000700 1 --2- 172.21.15.37:0/2381352647 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f588c0a2910 0x7f588c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:55.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.922+0000 7f58a2000700 1 -- 172.21.15.37:0/2381352647 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f58980cf990 con 0x7f58a406a930 2024-03-28T18:09:55.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.922+0000 7f58a8c00700 1 --2- 172.21.15.37:0/2381352647 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f588c0a2910 0x7f588c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:55.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.923+0000 7f58a8c00700 1 --2- 172.21.15.37:0/2381352647 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f588c0a2910 0x7f588c0a4dd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5890005ea0 tx=0x7f5890005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:55.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:55.926+0000 7f58a2000700 1 -- 172.21.15.37:0/2381352647 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f589805e740 con 0x7f58a406a930 2024-03-28T18:09:56.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:55 smithi037 ceph-mon[31460]: pgmap v367: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-28T18:09:56.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:56.158+0000 7f58ac000700 1 -- 172.21.15.37:0/2381352647 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f58a40625d0 con 0x7f588c0a2910 2024-03-28T18:09:56.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:56.170+0000 7f58a2000700 1 -- 172.21.15.37:0/2381352647 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259029 (secure 0 0 0) 0x7f58a40625d0 con 0x7f588c0a2910 2024-03-28T18:09:56.176 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:09:56.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:56.177+0000 7f58ac000700 1 -- 172.21.15.37:0/2381352647 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f588c0a2910 msgr2=0x7f588c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:56.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:56.177+0000 7f58ac000700 1 --2- 172.21.15.37:0/2381352647 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f588c0a2910 0x7f588c0a4dd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5890005ea0 tx=0x7f5890005e30 comp rx=0 tx=0).stop 2024-03-28T18:09:56.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:56.177+0000 7f58ac000700 1 -- 172.21.15.37:0/2381352647 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58a406a930 msgr2=0x7f58a4109030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:56.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:56.177+0000 7f58ac000700 1 --2- 172.21.15.37:0/2381352647 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58a406a930 0x7f58a4109030 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5898005910 tx=0x7f5898009f80 comp rx=0 tx=0).stop 2024-03-28T18:09:56.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:56.178+0000 7f58ac000700 1 -- 172.21.15.37:0/2381352647 shutdown_connections 2024-03-28T18:09:56.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:56.178+0000 7f58ac000700 1 --2- 172.21.15.37:0/2381352647 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f588c0a2910 0x7f588c0a4dd0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:56.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:56.178+0000 7f58ac000700 1 --2- 172.21.15.37:0/2381352647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58a4107510 0x7f58a4109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:56.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:56.178+0000 7f58ac000700 1 --2- 172.21.15.37:0/2381352647 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58a406a930 0x7f58a4109030 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:56.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:56.178+0000 7f58ac000700 1 --2- 172.21.15.37:0/2381352647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58a406b2e0 0x7f58a4109570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:56.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:56.178+0000 7f58ac000700 1 -- 172.21.15.37:0/2381352647 >> 172.21.15.37:0/2381352647 conn(0x7f58a40770e0 msgr2=0x7f58a410d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:56.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:56.178+0000 7f58ac000700 1 -- 172.21.15.37:0/2381352647 shutdown_connections 2024-03-28T18:09:56.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:56.178+0000 7f58ac000700 1 -- 172.21.15.37:0/2381352647 wait complete. 2024-03-28T18:09:56.180 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-03-28T18:09:56.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:55 smithi059 ceph-mon[36190]: pgmap v367: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-28T18:09:56.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:55 smithi170 ceph-mon[36280]: pgmap v367: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-28T18:09:57.132 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-03-28T18:09:55.990515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656889,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":705,"num_read_kb":474,"num_write":349,"num_write_kb":3055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1181430,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":782,"ondisk_log_size":782,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377360,"kb_used_data":11568,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564464,"statfs":{"total":1151940427776,"available":1151554011136,"internally_reserved":0,"allocated":11845632,"data_stored":7438047,"data_compressed":113751,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":92,"num_read_kb":47,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004548"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.005823+0000","last_change":"2024-03-28T18:08:20.025576+0000","last_active":"2024-03-28T18:09:44.005823+0000","last_peered":"2024-03-28T18:09:44.005823+0000","last_clean":"2024-03-28T18:09:44.005823+0000","last_became_active":"2024-03-28T18:08:20.025177+0000","last_became_peered":"2024-03-28T18:08:20.025177+0000","last_unstale":"2024-03-28T18:09:44.005823+0000","last_undegraded":"2024-03-28T18:09:44.005823+0000","last_fullsized":"2024-03-28T18:09:44.005823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:14.521170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.016245+0000","last_change":"2024-03-28T18:08:20.027418+0000","last_active":"2024-03-28T18:09:44.016245+0000","last_peered":"2024-03-28T18:09:44.016245+0000","last_clean":"2024-03-28T18:09:44.016245+0000","last_became_active":"2024-03-28T18:08:20.026731+0000","last_became_peered":"2024-03-28T18:08:20.026731+0000","last_unstale":"2024-03-28T18:09:44.016245+0000","last_undegraded":"2024-03-28T18:09:44.016245+0000","last_fullsized":"2024-03-28T18:09:44.016245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:52:58.328738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.001275+0000","last_change":"2024-03-28T18:08:20.029821+0000","last_active":"2024-03-28T18:09:44.001275+0000","last_peered":"2024-03-28T18:09:44.001275+0000","last_clean":"2024-03-28T18:09:44.001275+0000","last_became_active":"2024-03-28T18:08:20.028245+0000","last_became_peered":"2024-03-28T18:08:20.028245+0000","last_unstale":"2024-03-28T18:09:44.001275+0000","last_undegraded":"2024-03-28T18:09:44.001275+0000","last_fullsized":"2024-03-28T18:09:44.001275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:44.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.966292+0000","last_change":"2024-03-28T18:08:20.015682+0000","last_active":"2024-03-28T18:09:43.966292+0000","last_peered":"2024-03-28T18:09:43.966292+0000","last_clean":"2024-03-28T18:09:43.966292+0000","last_became_active":"2024-03-28T18:08:20.015303+0000","last_became_peered":"2024-03-28T18:08:20.015303+0000","last_unstale":"2024-03-28T18:09:43.966292+0000","last_undegraded":"2024-03-28T18:09:43.966292+0000","last_fullsized":"2024-03-28T18:09:43.966292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:57.456052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.998334+0000","last_change":"2024-03-28T18:08:20.020970+0000","last_active":"2024-03-28T18:09:43.998334+0000","last_peered":"2024-03-28T18:09:43.998334+0000","last_clean":"2024-03-28T18:09:43.998334+0000","last_became_active":"2024-03-28T18:08:20.020646+0000","last_became_peered":"2024-03-28T18:08:20.020646+0000","last_unstale":"2024-03-28T18:09:43.998334+0000","last_undegraded":"2024-03-28T18:09:43.998334+0000","last_fullsized":"2024-03-28T18:09:43.998334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:57:20.422546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.970396+0000","last_change":"2024-03-28T18:08:20.022395+0000","last_active":"2024-03-28T18:09:43.970396+0000","last_peered":"2024-03-28T18:09:43.970396+0000","last_clean":"2024-03-28T18:09:43.970396+0000","last_became_active":"2024-03-28T18:08:20.022022+0000","last_became_peered":"2024-03-28T18:08:20.022022+0000","last_unstale":"2024-03-28T18:09:43.970396+0000","last_undegraded":"2024-03-28T18:09:43.970396+0000","last_fullsized":"2024-03-28T18:09:43.970396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:32.590751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.966769+0000","last_change":"2024-03-28T18:08:20.019990+0000","last_active":"2024-03-28T18:09:43.966769+0000","last_peered":"2024-03-28T18:09:43.966769+0000","last_clean":"2024-03-28T18:09:43.966769+0000","last_became_active":"2024-03-28T18:08:20.019618+0000","last_became_peered":"2024-03-28T18:08:20.019618+0000","last_unstale":"2024-03-28T18:09:43.966769+0000","last_undegraded":"2024-03-28T18:09:43.966769+0000","last_fullsized":"2024-03-28T18:09:43.966769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:48.442579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.968407+0000","last_change":"2024-03-28T18:09:15.889503+0000","last_active":"2024-03-28T18:09:43.968407+0000","last_peered":"2024-03-28T18:09:43.968407+0000","last_clean":"2024-03-28T18:09:43.968407+0000","last_became_active":"2024-03-28T18:09:15.888272+0000","last_became_peered":"2024-03-28T18:09:15.888272+0000","last_unstale":"2024-03-28T18:09:43.968407+0000","last_undegraded":"2024-03-28T18:09:43.968407+0000","last_fullsized":"2024-03-28T18:09:43.968407+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:49.222646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.979721+0000","last_change":"2024-03-28T18:08:20.009732+0000","last_active":"2024-03-28T18:09:43.979721+0000","last_peered":"2024-03-28T18:09:43.979721+0000","last_clean":"2024-03-28T18:09:43.979721+0000","last_became_active":"2024-03-28T18:08:20.009288+0000","last_became_peered":"2024-03-28T18:08:20.009288+0000","last_unstale":"2024-03-28T18:09:43.979721+0000","last_undegraded":"2024-03-28T18:09:43.979721+0000","last_fullsized":"2024-03-28T18:09:43.979721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:20:54.917964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.964811+0000","last_change":"2024-03-28T18:08:20.009687+0000","last_active":"2024-03-28T18:09:43.964811+0000","last_peered":"2024-03-28T18:09:43.964811+0000","last_clean":"2024-03-28T18:09:43.964811+0000","last_became_active":"2024-03-28T18:08:20.008364+0000","last_became_peered":"2024-03-28T18:08:20.008364+0000","last_unstale":"2024-03-28T18:09:43.964811+0000","last_undegraded":"2024-03-28T18:09:43.964811+0000","last_fullsized":"2024-03-28T18:09:43.964811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:08.419202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.978189+0000","last_change":"2024-03-28T18:08:20.011508+0000","last_active":"2024-03-28T18:09:43.978189+0000","last_peered":"2024-03-28T18:09:43.978189+0000","last_clean":"2024-03-28T18:09:43.978189+0000","last_became_active":"2024-03-28T18:08:20.011117+0000","last_became_peered":"2024-03-28T18:08:20.011117+0000","last_unstale":"2024-03-28T18:09:43.978189+0000","last_undegraded":"2024-03-28T18:09:43.978189+0000","last_fullsized":"2024-03-28T18:09:43.978189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:06:19.278931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.966130+0000","last_change":"2024-03-28T18:08:20.011403+0000","last_active":"2024-03-28T18:09:43.966130+0000","last_peered":"2024-03-28T18:09:43.966130+0000","last_clean":"2024-03-28T18:09:43.966130+0000","last_became_active":"2024-03-28T18:08:20.011025+0000","last_became_peered":"2024-03-28T18:08:20.011025+0000","last_unstale":"2024-03-28T18:09:43.966130+0000","last_undegraded":"2024-03-28T18:09:43.966130+0000","last_fullsized":"2024-03-28T18:09:43.966130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:13.599432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.005321+0000","last_change":"2024-03-28T18:08:20.001664+0000","last_active":"2024-03-28T18:09:44.005321+0000","last_peered":"2024-03-28T18:09:44.005321+0000","last_clean":"2024-03-28T18:09:44.005321+0000","last_became_active":"2024-03-28T18:08:20.000910+0000","last_became_peered":"2024-03-28T18:08:20.000910+0000","last_unstale":"2024-03-28T18:09:44.005321+0000","last_undegraded":"2024-03-28T18:09:44.005321+0000","last_fullsized":"2024-03-28T18:09:44.005321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:00:23.796302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.008088+0000","last_change":"2024-03-28T18:08:20.027174+0000","last_active":"2024-03-28T18:09:44.008088+0000","last_peered":"2024-03-28T18:09:44.008088+0000","last_clean":"2024-03-28T18:09:44.008088+0000","last_became_active":"2024-03-28T18:08:20.026113+0000","last_became_peered":"2024-03-28T18:08:20.026113+0000","last_unstale":"2024-03-28T18:09:44.008088+0000","last_undegraded":"2024-03-28T18:09:44.008088+0000","last_fullsized":"2024-03-28T18:09:44.008088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:55:00.061696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.010388+0000","last_change":"2024-03-28T18:08:20.020060+0000","last_active":"2024-03-28T18:09:44.010388+0000","last_peered":"2024-03-28T18:09:44.010388+0000","last_clean":"2024-03-28T18:09:44.010388+0000","last_became_active":"2024-03-28T18:08:20.019691+0000","last_became_peered":"2024-03-28T18:08:20.019691+0000","last_unstale":"2024-03-28T18:09:44.010388+0000","last_undegraded":"2024-03-28T18:09:44.010388+0000","last_fullsized":"2024-03-28T18:09:44.010388+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:27.889737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.003240+0000","last_change":"2024-03-28T18:08:20.024239+0000","last_active":"2024-03-28T18:09:44.003240+0000","last_peered":"2024-03-28T18:09:44.003240+0000","last_clean":"2024-03-28T18:09:44.003240+0000","last_became_active":"2024-03-28T18:08:20.023874+0000","last_became_peered":"2024-03-28T18:08:20.023874+0000","last_unstale":"2024-03-28T18:09:44.003240+0000","last_undegraded":"2024-03-28T18:09:44.003240+0000","last_fullsized":"2024-03-28T18:09:44.003240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:05:32.903195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.008554+0000","last_change":"2024-03-28T18:08:20.033322+0000","last_active":"2024-03-28T18:09:44.008554+0000","last_peered":"2024-03-28T18:09:44.008554+0000","last_clean":"2024-03-28T18:09:44.008554+0000","last_became_active":"2024-03-28T18:08:20.033048+0000","last_became_peered":"2024-03-28T18:08:20.033048+0000","last_unstale":"2024-03-28T18:09:44.008554+0000","last_undegraded":"2024-03-28T18:09:44.008554+0000","last_fullsized":"2024-03-28T18:09:44.008554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:18:59.234141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.986315+0000","last_change":"2024-03-28T18:08:20.022093+0000","last_active":"2024-03-28T18:09:43.986315+0000","last_peered":"2024-03-28T18:09:43.986315+0000","last_clean":"2024-03-28T18:09:43.986315+0000","last_became_active":"2024-03-28T18:08:20.021411+0000","last_became_peered":"2024-03-28T18:08:20.021411+0000","last_unstale":"2024-03-28T18:09:43.986315+0000","last_undegraded":"2024-03-28T18:09:43.986315+0000","last_fullsized":"2024-03-28T18:09:43.986315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:09:40.366628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.001943+0000","last_change":"2024-03-28T18:08:20.028893+0000","last_active":"2024-03-28T18:09:44.001943+0000","last_peered":"2024-03-28T18:09:44.001943+0000","last_clean":"2024-03-28T18:09:44.001943+0000","last_became_active":"2024-03-28T18:08:20.028634+0000","last_became_peered":"2024-03-28T18:08:20.028634+0000","last_unstale":"2024-03-28T18:09:44.001943+0000","last_undegraded":"2024-03-28T18:09:44.001943+0000","last_fullsized":"2024-03-28T18:09:44.001943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:32:27.454406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.734404+0000","last_change":"2024-03-28T18:09:15.734404+0000","last_active":"2024-03-28T18:09:15.734404+0000","last_peered":"2024-03-28T18:09:15.734404+0000","last_clean":"2024-03-28T18:09:15.734404+0000","last_became_active":"2024-03-28T18:09:15.723870+0000","last_became_peered":"2024-03-28T18:09:15.723870+0000","last_unstale":"2024-03-28T18:09:15.734404+0000","last_undegraded":"2024-03-28T18:09:15.734404+0000","last_fullsized":"2024-03-28T18:09:15.734404+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:32.736966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.994923+0000","last_change":"2024-03-28T18:08:20.027855+0000","last_active":"2024-03-28T18:09:43.994923+0000","last_peered":"2024-03-28T18:09:43.994923+0000","last_clean":"2024-03-28T18:09:43.994923+0000","last_became_active":"2024-03-28T18:08:20.027565+0000","last_became_peered":"2024-03-28T18:08:20.027565+0000","last_unstale":"2024-03-28T18:09:43.994923+0000","last_undegraded":"2024-03-28T18:09:43.994923+0000","last_fullsized":"2024-03-28T18:09:43.994923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:36:37.596573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.725729+0000","last_change":"2024-03-28T18:08:19.499055+0000","last_active":"2024-03-28T18:09:15.725729+0000","last_peered":"2024-03-28T18:09:15.725729+0000","last_clean":"2024-03-28T18:09:15.725729+0000","last_became_active":"2024-03-28T18:08:19.497964+0000","last_became_peered":"2024-03-28T18:08:19.497964+0000","last_unstale":"2024-03-28T18:09:15.725729+0000","last_undegraded":"2024-03-28T18:09:15.725729+0000","last_fullsized":"2024-03-28T18:09:15.725729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:38:45.620406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.011364+0000","last_change":"2024-03-28T18:08:20.028967+0000","last_active":"2024-03-28T18:09:44.011364+0000","last_peered":"2024-03-28T18:09:44.011364+0000","last_clean":"2024-03-28T18:09:44.011364+0000","last_became_active":"2024-03-28T18:08:20.028607+0000","last_became_peered":"2024-03-28T18:08:20.028607+0000","last_unstale":"2024-03-28T18:09:44.011364+0000","last_undegraded":"2024-03-28T18:09:44.011364+0000","last_fullsized":"2024-03-28T18:09:44.011364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:21:23.994077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.892649+0000","last_change":"2024-03-28T18:09:15.892649+0000","last_active":"2024-03-28T18:09:15.892649+0000","last_peered":"2024-03-28T18:09:15.892649+0000","last_clean":"2024-03-28T18:09:15.892649+0000","last_became_active":"2024-03-28T18:09:15.888520+0000","last_became_peered":"2024-03-28T18:09:15.888520+0000","last_unstale":"2024-03-28T18:09:15.892649+0000","last_undegraded":"2024-03-28T18:09:15.892649+0000","last_fullsized":"2024-03-28T18:09:15.892649+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:34.085738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.009420+0000","last_change":"2024-03-28T18:08:20.009878+0000","last_active":"2024-03-28T18:09:44.009420+0000","last_peered":"2024-03-28T18:09:44.009420+0000","last_clean":"2024-03-28T18:09:44.009420+0000","last_became_active":"2024-03-28T18:08:20.009534+0000","last_became_peered":"2024-03-28T18:08:20.009534+0000","last_unstale":"2024-03-28T18:09:44.009420+0000","last_undegraded":"2024-03-28T18:09:44.009420+0000","last_fullsized":"2024-03-28T18:09:44.009420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:58.975854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.715836+0000","last_change":"2024-03-28T18:08:19.472613+0000","last_active":"2024-03-28T18:09:15.715836+0000","last_peered":"2024-03-28T18:09:15.715836+0000","last_clean":"2024-03-28T18:09:15.715836+0000","last_became_active":"2024-03-28T18:08:19.472186+0000","last_became_peered":"2024-03-28T18:08:19.472186+0000","last_unstale":"2024-03-28T18:09:15.715836+0000","last_undegraded":"2024-03-28T18:09:15.715836+0000","last_fullsized":"2024-03-28T18:09:15.715836+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:50.807609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.972232+0000","last_change":"2024-03-28T18:08:20.008242+0000","last_active":"2024-03-28T18:09:43.972232+0000","last_peered":"2024-03-28T18:09:43.972232+0000","last_clean":"2024-03-28T18:09:43.972232+0000","last_became_active":"2024-03-28T18:08:20.007942+0000","last_became_peered":"2024-03-28T18:08:20.007942+0000","last_unstale":"2024-03-28T18:09:43.972232+0000","last_undegraded":"2024-03-28T18:09:43.972232+0000","last_fullsized":"2024-03-28T18:09:43.972232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:23.288978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'7","reported_seq":121,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:54.520875+0000","last_change":"2024-03-28T18:08:19.480869+0000","last_active":"2024-03-28T18:09:54.520875+0000","last_peered":"2024-03-28T18:09:54.520875+0000","last_clean":"2024-03-28T18:09:54.520875+0000","last_became_active":"2024-03-28T18:08:19.480441+0000","last_became_peered":"2024-03-28T18:08:19.480441+0000","last_unstale":"2024-03-28T18:09:54.520875+0000","last_undegraded":"2024-03-28T18:09:54.520875+0000","last_fullsized":"2024-03-28T18:09:54.520875+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:26.640605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":95,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.959483+0000","last_change":"2024-03-28T18:08:20.021530+0000","last_active":"2024-03-28T18:09:43.959483+0000","last_peered":"2024-03-28T18:09:43.959483+0000","last_clean":"2024-03-28T18:09:43.959483+0000","last_became_active":"2024-03-28T18:08:20.021179+0000","last_became_peered":"2024-03-28T18:08:20.021179+0000","last_unstale":"2024-03-28T18:09:43.959483+0000","last_undegraded":"2024-03-28T18:09:43.959483+0000","last_fullsized":"2024-03-28T18:09:43.959483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:43:07.300300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887260+0000","last_change":"2024-03-28T18:08:19.494597+0000","last_active":"2024-03-28T18:09:15.887260+0000","last_peered":"2024-03-28T18:09:15.887260+0000","last_clean":"2024-03-28T18:09:15.887260+0000","last_became_active":"2024-03-28T18:08:19.494178+0000","last_became_peered":"2024-03-28T18:08:19.494178+0000","last_unstale":"2024-03-28T18:09:15.887260+0000","last_undegraded":"2024-03-28T18:09:15.887260+0000","last_fullsized":"2024-03-28T18:09:15.887260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:08:10.257768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.991782+0000","last_change":"2024-03-28T18:08:20.021062+0000","last_active":"2024-03-28T18:09:43.991782+0000","last_peered":"2024-03-28T18:09:43.991782+0000","last_clean":"2024-03-28T18:09:43.991782+0000","last_became_active":"2024-03-28T18:08:20.020758+0000","last_became_peered":"2024-03-28T18:08:20.020758+0000","last_unstale":"2024-03-28T18:09:43.991782+0000","last_undegraded":"2024-03-28T18:09:43.991782+0000","last_fullsized":"2024-03-28T18:09:43.991782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:47:39.656513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.889688+0000","last_change":"2024-03-28T18:08:19.480189+0000","last_active":"2024-03-28T18:09:15.889688+0000","last_peered":"2024-03-28T18:09:15.889688+0000","last_clean":"2024-03-28T18:09:15.889688+0000","last_became_active":"2024-03-28T18:08:19.479812+0000","last_became_peered":"2024-03-28T18:08:19.479812+0000","last_unstale":"2024-03-28T18:09:15.889688+0000","last_undegraded":"2024-03-28T18:09:15.889688+0000","last_fullsized":"2024-03-28T18:09:15.889688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:46.408565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.004426+0000","last_change":"2024-03-28T18:08:20.023635+0000","last_active":"2024-03-28T18:09:44.004426+0000","last_peered":"2024-03-28T18:09:44.004426+0000","last_clean":"2024-03-28T18:09:44.004426+0000","last_became_active":"2024-03-28T18:08:20.023199+0000","last_became_peered":"2024-03-28T18:08:20.023199+0000","last_unstale":"2024-03-28T18:09:44.004426+0000","last_undegraded":"2024-03-28T18:09:44.004426+0000","last_fullsized":"2024-03-28T18:09:44.004426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:12.477899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.886860+0000","last_change":"2024-03-28T18:08:19.480858+0000","last_active":"2024-03-28T18:09:15.886860+0000","last_peered":"2024-03-28T18:09:15.886860+0000","last_clean":"2024-03-28T18:09:15.886860+0000","last_became_active":"2024-03-28T18:08:19.480418+0000","last_became_peered":"2024-03-28T18:08:19.480418+0000","last_unstale":"2024-03-28T18:09:15.886860+0000","last_undegraded":"2024-03-28T18:09:15.886860+0000","last_fullsized":"2024-03-28T18:09:15.886860+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:59:57.788915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.019764+0000","last_change":"2024-03-28T18:08:20.031719+0000","last_active":"2024-03-28T18:09:44.019764+0000","last_peered":"2024-03-28T18:09:44.019764+0000","last_clean":"2024-03-28T18:09:44.019764+0000","last_became_active":"2024-03-28T18:08:20.029482+0000","last_became_peered":"2024-03-28T18:08:20.029482+0000","last_unstale":"2024-03-28T18:09:44.019764+0000","last_undegraded":"2024-03-28T18:09:44.019764+0000","last_fullsized":"2024-03-28T18:09:44.019764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:13.226596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'4","reported_seq":113,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:53.519329+0000","last_change":"2024-03-28T18:08:19.483346+0000","last_active":"2024-03-28T18:09:53.519329+0000","last_peered":"2024-03-28T18:09:53.519329+0000","last_clean":"2024-03-28T18:09:53.519329+0000","last_became_active":"2024-03-28T18:08:19.483016+0000","last_became_peered":"2024-03-28T18:08:19.483016+0000","last_unstale":"2024-03-28T18:09:53.519329+0000","last_undegraded":"2024-03-28T18:09:53.519329+0000","last_fullsized":"2024-03-28T18:09:53.519329+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:34:33.008477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.982043+0000","last_change":"2024-03-28T18:08:20.020478+0000","last_active":"2024-03-28T18:09:43.982043+0000","last_peered":"2024-03-28T18:09:43.982043+0000","last_clean":"2024-03-28T18:09:43.982043+0000","last_became_active":"2024-03-28T18:08:20.020054+0000","last_became_peered":"2024-03-28T18:08:20.020054+0000","last_unstale":"2024-03-28T18:09:43.982043+0000","last_undegraded":"2024-03-28T18:09:43.982043+0000","last_fullsized":"2024-03-28T18:09:43.982043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:13:43.655549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716140+0000","last_change":"2024-03-28T18:08:19.479386+0000","last_active":"2024-03-28T18:09:15.716140+0000","last_peered":"2024-03-28T18:09:15.716140+0000","last_clean":"2024-03-28T18:09:15.716140+0000","last_became_active":"2024-03-28T18:08:19.478985+0000","last_became_peered":"2024-03-28T18:08:19.478985+0000","last_unstale":"2024-03-28T18:09:15.716140+0000","last_undegraded":"2024-03-28T18:09:15.716140+0000","last_fullsized":"2024-03-28T18:09:15.716140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:37:22.359475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.007627+0000","last_change":"2024-03-28T18:08:20.021790+0000","last_active":"2024-03-28T18:09:44.007627+0000","last_peered":"2024-03-28T18:09:44.007627+0000","last_clean":"2024-03-28T18:09:44.007627+0000","last_became_active":"2024-03-28T18:08:20.021031+0000","last_became_peered":"2024-03-28T18:08:20.021031+0000","last_unstale":"2024-03-28T18:09:44.007627+0000","last_undegraded":"2024-03-28T18:09:44.007627+0000","last_fullsized":"2024-03-28T18:09:44.007627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:59:50.897692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887205+0000","last_change":"2024-03-28T18:08:19.481576+0000","last_active":"2024-03-28T18:09:15.887205+0000","last_peered":"2024-03-28T18:09:15.887205+0000","last_clean":"2024-03-28T18:09:15.887205+0000","last_became_active":"2024-03-28T18:08:19.481226+0000","last_became_peered":"2024-03-28T18:08:19.481226+0000","last_unstale":"2024-03-28T18:09:15.887205+0000","last_undegraded":"2024-03-28T18:09:15.887205+0000","last_fullsized":"2024-03-28T18:09:15.887205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:54:44.677343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.013687+0000","last_change":"2024-03-28T18:09:15.890672+0000","last_active":"2024-03-28T18:09:44.013687+0000","last_peered":"2024-03-28T18:09:44.013687+0000","last_clean":"2024-03-28T18:09:44.013687+0000","last_became_active":"2024-03-28T18:09:15.890342+0000","last_became_peered":"2024-03-28T18:09:15.890342+0000","last_unstale":"2024-03-28T18:09:44.013687+0000","last_undegraded":"2024-03-28T18:09:44.013687+0000","last_fullsized":"2024-03-28T18:09:44.013687+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:08.268067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'32","reported_seq":240,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:54.520487+0000","last_change":"2024-03-28T18:08:19.477559+0000","last_active":"2024-03-28T18:09:54.520487+0000","last_peered":"2024-03-28T18:09:54.520487+0000","last_clean":"2024-03-28T18:09:54.520487+0000","last_became_active":"2024-03-28T18:08:19.477217+0000","last_became_peered":"2024-03-28T18:08:19.477217+0000","last_unstale":"2024-03-28T18:09:54.520487+0000","last_undegraded":"2024-03-28T18:09:54.520487+0000","last_fullsized":"2024-03-28T18:09:54.520487+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:37.071671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62145,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":189,"num_read_kb":155,"num_write":33,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.951533+0000","last_change":"2024-03-28T18:08:20.011303+0000","last_active":"2024-03-28T18:09:43.951533+0000","last_peered":"2024-03-28T18:09:43.951533+0000","last_clean":"2024-03-28T18:09:43.951533+0000","last_became_active":"2024-03-28T18:08:20.010997+0000","last_became_peered":"2024-03-28T18:08:20.010997+0000","last_unstale":"2024-03-28T18:09:43.951533+0000","last_undegraded":"2024-03-28T18:09:43.951533+0000","last_fullsized":"2024-03-28T18:09:43.951533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:10:56.036132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.718010+0000","last_change":"2024-03-28T18:08:19.477342+0000","last_active":"2024-03-28T18:09:15.718010+0000","last_peered":"2024-03-28T18:09:15.718010+0000","last_clean":"2024-03-28T18:09:15.718010+0000","last_became_active":"2024-03-28T18:08:19.476984+0000","last_became_peered":"2024-03-28T18:08:19.476984+0000","last_unstale":"2024-03-28T18:09:15.718010+0000","last_undegraded":"2024-03-28T18:09:15.718010+0000","last_fullsized":"2024-03-28T18:09:15.718010+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:15:42.606213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.017662+0000","last_change":"2024-03-28T18:08:20.030524+0000","last_active":"2024-03-28T18:09:44.017662+0000","last_peered":"2024-03-28T18:09:44.017662+0000","last_clean":"2024-03-28T18:09:44.017662+0000","last_became_active":"2024-03-28T18:08:20.030227+0000","last_became_peered":"2024-03-28T18:08:20.030227+0000","last_unstale":"2024-03-28T18:09:44.017662+0000","last_undegraded":"2024-03-28T18:09:44.017662+0000","last_fullsized":"2024-03-28T18:09:44.017662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:07:41.763739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.886681+0000","last_change":"2024-03-28T18:08:19.481169+0000","last_active":"2024-03-28T18:09:15.886681+0000","last_peered":"2024-03-28T18:09:15.886681+0000","last_clean":"2024-03-28T18:09:15.886681+0000","last_became_active":"2024-03-28T18:08:19.480634+0000","last_became_peered":"2024-03-28T18:08:19.480634+0000","last_unstale":"2024-03-28T18:09:15.886681+0000","last_undegraded":"2024-03-28T18:09:15.886681+0000","last_fullsized":"2024-03-28T18:09:15.886681+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:39.265215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.006895+0000","last_change":"2024-03-28T18:08:20.015163+0000","last_active":"2024-03-28T18:09:44.006895+0000","last_peered":"2024-03-28T18:09:44.006895+0000","last_clean":"2024-03-28T18:09:44.006895+0000","last_became_active":"2024-03-28T18:08:20.014765+0000","last_became_peered":"2024-03-28T18:08:20.014765+0000","last_unstale":"2024-03-28T18:09:44.006895+0000","last_undegraded":"2024-03-28T18:09:44.006895+0000","last_fullsized":"2024-03-28T18:09:44.006895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:51.035532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'3","reported_seq":113,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:52.549579+0000","last_change":"2024-03-28T18:08:19.479086+0000","last_active":"2024-03-28T18:09:52.549579+0000","last_peered":"2024-03-28T18:09:52.549579+0000","last_clean":"2024-03-28T18:09:52.549579+0000","last_became_active":"2024-03-28T18:08:19.478748+0000","last_became_peered":"2024-03-28T18:08:19.478748+0000","last_unstale":"2024-03-28T18:09:52.549579+0000","last_undegraded":"2024-03-28T18:09:52.549579+0000","last_fullsized":"2024-03-28T18:09:52.549579+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:46:59.566998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.973553+0000","last_change":"2024-03-28T18:09:15.888751+0000","last_active":"2024-03-28T18:09:43.973553+0000","last_peered":"2024-03-28T18:09:43.973553+0000","last_clean":"2024-03-28T18:09:43.973553+0000","last_became_active":"2024-03-28T18:09:15.888350+0000","last_became_peered":"2024-03-28T18:09:15.888350+0000","last_unstale":"2024-03-28T18:09:43.973553+0000","last_undegraded":"2024-03-28T18:09:43.973553+0000","last_fullsized":"2024-03-28T18:09:43.973553+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:38:05.822205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":59,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.733434+0000","last_change":"2024-03-28T18:09:15.733434+0000","last_active":"2024-03-28T18:09:15.733434+0000","last_peered":"2024-03-28T18:09:15.733434+0000","last_clean":"2024-03-28T18:09:15.733434+0000","last_became_active":"2024-03-28T18:09:15.722437+0000","last_became_peered":"2024-03-28T18:09:15.722437+0000","last_unstale":"2024-03-28T18:09:15.733434+0000","last_undegraded":"2024-03-28T18:09:15.733434+0000","last_fullsized":"2024-03-28T18:09:15.733434+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:35:18.313676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.971525+0000","last_change":"2024-03-28T18:08:20.007457+0000","last_active":"2024-03-28T18:09:43.971525+0000","last_peered":"2024-03-28T18:09:43.971525+0000","last_clean":"2024-03-28T18:09:43.971525+0000","last_became_active":"2024-03-28T18:08:20.007090+0000","last_became_peered":"2024-03-28T18:08:20.007090+0000","last_unstale":"2024-03-28T18:09:43.971525+0000","last_undegraded":"2024-03-28T18:09:43.971525+0000","last_fullsized":"2024-03-28T18:09:43.971525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:22:45.702706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716349+0000","last_change":"2024-03-28T18:08:19.475104+0000","last_active":"2024-03-28T18:09:15.716349+0000","last_peered":"2024-03-28T18:09:15.716349+0000","last_clean":"2024-03-28T18:09:15.716349+0000","last_became_active":"2024-03-28T18:08:19.474754+0000","last_became_peered":"2024-03-28T18:08:19.474754+0000","last_unstale":"2024-03-28T18:09:15.716349+0000","last_undegraded":"2024-03-28T18:09:15.716349+0000","last_fullsized":"2024-03-28T18:09:15.716349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:19.085631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.963148+0000","last_change":"2024-03-28T18:08:20.009347+0000","last_active":"2024-03-28T18:09:43.963148+0000","last_peered":"2024-03-28T18:09:43.963148+0000","last_clean":"2024-03-28T18:09:43.963148+0000","last_became_active":"2024-03-28T18:08:20.009012+0000","last_became_peered":"2024-03-28T18:08:20.009012+0000","last_unstale":"2024-03-28T18:09:43.963148+0000","last_undegraded":"2024-03-28T18:09:43.963148+0000","last_fullsized":"2024-03-28T18:09:43.963148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:04:26.625830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.721904+0000","last_change":"2024-03-28T18:08:19.482414+0000","last_active":"2024-03-28T18:09:15.721904+0000","last_peered":"2024-03-28T18:09:15.721904+0000","last_clean":"2024-03-28T18:09:15.721904+0000","last_became_active":"2024-03-28T18:08:19.481830+0000","last_became_peered":"2024-03-28T18:08:19.481830+0000","last_unstale":"2024-03-28T18:09:15.721904+0000","last_undegraded":"2024-03-28T18:09:15.721904+0000","last_fullsized":"2024-03-28T18:09:15.721904+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:27:14.259355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.001697+0000","last_change":"2024-03-28T18:08:20.030687+0000","last_active":"2024-03-28T18:09:44.001697+0000","last_peered":"2024-03-28T18:09:44.001697+0000","last_clean":"2024-03-28T18:09:44.001697+0000","last_became_active":"2024-03-28T18:08:20.030308+0000","last_became_peered":"2024-03-28T18:08:20.030308+0000","last_unstale":"2024-03-28T18:09:44.001697+0000","last_undegraded":"2024-03-28T18:09:44.001697+0000","last_fullsized":"2024-03-28T18:09:44.001697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:31.745952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.715942+0000","last_change":"2024-03-28T18:08:19.494446+0000","last_active":"2024-03-28T18:09:15.715942+0000","last_peered":"2024-03-28T18:09:15.715942+0000","last_clean":"2024-03-28T18:09:15.715942+0000","last_became_active":"2024-03-28T18:08:19.494036+0000","last_became_peered":"2024-03-28T18:08:19.494036+0000","last_unstale":"2024-03-28T18:09:15.715942+0000","last_undegraded":"2024-03-28T18:09:15.715942+0000","last_fullsized":"2024-03-28T18:09:15.715942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:42:53.660741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.009319+0000","last_change":"2024-03-28T18:08:20.025706+0000","last_active":"2024-03-28T18:09:44.009319+0000","last_peered":"2024-03-28T18:09:44.009319+0000","last_clean":"2024-03-28T18:09:44.009319+0000","last_became_active":"2024-03-28T18:08:20.025343+0000","last_became_peered":"2024-03-28T18:08:20.025343+0000","last_unstale":"2024-03-28T18:09:44.009319+0000","last_undegraded":"2024-03-28T18:09:44.009319+0000","last_fullsized":"2024-03-28T18:09:44.009319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:32:27.883940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.963857+0000","last_change":"2024-03-28T18:08:20.009636+0000","last_active":"2024-03-28T18:09:43.963857+0000","last_peered":"2024-03-28T18:09:43.963857+0000","last_clean":"2024-03-28T18:09:43.963857+0000","last_became_active":"2024-03-28T18:08:20.009178+0000","last_became_peered":"2024-03-28T18:08:20.009178+0000","last_unstale":"2024-03-28T18:09:43.963857+0000","last_undegraded":"2024-03-28T18:09:43.963857+0000","last_fullsized":"2024-03-28T18:09:43.963857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:28.461411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":113,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:53.550182+0000","last_change":"2024-03-28T18:08:19.501287+0000","last_active":"2024-03-28T18:09:53.550182+0000","last_peered":"2024-03-28T18:09:53.550182+0000","last_clean":"2024-03-28T18:09:53.550182+0000","last_became_active":"2024-03-28T18:08:19.500920+0000","last_became_peered":"2024-03-28T18:08:19.500920+0000","last_unstale":"2024-03-28T18:09:53.550182+0000","last_undegraded":"2024-03-28T18:09:53.550182+0000","last_fullsized":"2024-03-28T18:09:53.550182+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:29.882164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.010657+0000","last_change":"2024-03-28T18:08:20.020811+0000","last_active":"2024-03-28T18:09:44.010657+0000","last_peered":"2024-03-28T18:09:44.010657+0000","last_clean":"2024-03-28T18:09:44.010657+0000","last_became_active":"2024-03-28T18:08:20.020381+0000","last_became_peered":"2024-03-28T18:08:20.020381+0000","last_unstale":"2024-03-28T18:09:44.010657+0000","last_undegraded":"2024-03-28T18:09:44.010657+0000","last_fullsized":"2024-03-28T18:09:44.010657+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:10:35.999858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887706+0000","last_change":"2024-03-28T18:08:19.478020+0000","last_active":"2024-03-28T18:09:15.887706+0000","last_peered":"2024-03-28T18:09:15.887706+0000","last_clean":"2024-03-28T18:09:15.887706+0000","last_became_active":"2024-03-28T18:08:19.477627+0000","last_became_peered":"2024-03-28T18:08:19.477627+0000","last_unstale":"2024-03-28T18:09:15.887706+0000","last_undegraded":"2024-03-28T18:09:15.887706+0000","last_fullsized":"2024-03-28T18:09:15.887706+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:00:01.649974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":354,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.715885+0000","last_change":"2024-03-28T18:06:58.750932+0000","last_active":"2024-03-28T18:09:15.715885+0000","last_peered":"2024-03-28T18:09:15.715885+0000","last_clean":"2024-03-28T18:09:15.715885+0000","last_became_active":"2024-03-28T18:06:58.750309+0000","last_became_peered":"2024-03-28T18:06:58.750309+0000","last_unstale":"2024-03-28T18:09:15.715885+0000","last_undegraded":"2024-03-28T18:09:15.715885+0000","last_fullsized":"2024-03-28T18:09:15.715885+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:03:00.832683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:03:00.832683+0000","last_clean_scrub_stamp":"2024-03-28T18:03:00.832683+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:46.001886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.001884+0000","last_change":"2024-03-28T18:08:20.009573+0000","last_active":"2024-03-28T18:09:44.001884+0000","last_peered":"2024-03-28T18:09:44.001884+0000","last_clean":"2024-03-28T18:09:44.001884+0000","last_became_active":"2024-03-28T18:08:20.009143+0000","last_became_peered":"2024-03-28T18:08:20.009143+0000","last_unstale":"2024-03-28T18:09:44.001884+0000","last_undegraded":"2024-03-28T18:09:44.001884+0000","last_fullsized":"2024-03-28T18:09:44.001884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:37:50.643356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716024+0000","last_change":"2024-03-28T18:08:19.483733+0000","last_active":"2024-03-28T18:09:15.716024+0000","last_peered":"2024-03-28T18:09:15.716024+0000","last_clean":"2024-03-28T18:09:15.716024+0000","last_became_active":"2024-03-28T18:08:19.483366+0000","last_became_peered":"2024-03-28T18:08:19.483366+0000","last_unstale":"2024-03-28T18:09:15.716024+0000","last_undegraded":"2024-03-28T18:09:15.716024+0000","last_fullsized":"2024-03-28T18:09:15.716024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:29.192442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.715991+0000","last_change":"2024-03-28T18:08:19.481844+0000","last_active":"2024-03-28T18:09:15.715991+0000","last_peered":"2024-03-28T18:09:15.715991+0000","last_clean":"2024-03-28T18:09:15.715991+0000","last_became_active":"2024-03-28T18:08:19.481430+0000","last_became_peered":"2024-03-28T18:08:19.481430+0000","last_unstale":"2024-03-28T18:09:15.715991+0000","last_undegraded":"2024-03-28T18:09:15.715991+0000","last_fullsized":"2024-03-28T18:09:15.715991+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:21:28.789883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.009465+0000","last_change":"2024-03-28T18:08:20.021037+0000","last_active":"2024-03-28T18:09:44.009465+0000","last_peered":"2024-03-28T18:09:44.009465+0000","last_clean":"2024-03-28T18:09:44.009465+0000","last_became_active":"2024-03-28T18:08:20.020680+0000","last_became_peered":"2024-03-28T18:08:20.020680+0000","last_unstale":"2024-03-28T18:09:44.009465+0000","last_undegraded":"2024-03-28T18:09:44.009465+0000","last_fullsized":"2024-03-28T18:09:44.009465+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:08:28.825774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'10","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.738136+0000","last_change":"2024-03-28T18:09:15.737583+0000","last_active":"2024-03-28T18:09:15.738136+0000","last_peered":"2024-03-28T18:09:15.738136+0000","last_clean":"2024-03-28T18:09:15.738136+0000","last_became_active":"2024-03-28T18:09:15.726579+0000","last_became_peered":"2024-03-28T18:09:15.726579+0000","last_unstale":"2024-03-28T18:09:15.738136+0000","last_undegraded":"2024-03-28T18:09:15.738136+0000","last_fullsized":"2024-03-28T18:09:15.738136+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:03.380037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.008936+0000","last_change":"2024-03-28T18:08:20.024938+0000","last_active":"2024-03-28T18:09:44.008936+0000","last_peered":"2024-03-28T18:09:44.008936+0000","last_clean":"2024-03-28T18:09:44.008936+0000","last_became_active":"2024-03-28T18:08:20.024536+0000","last_became_peered":"2024-03-28T18:08:20.024536+0000","last_unstale":"2024-03-28T18:09:44.008936+0000","last_undegraded":"2024-03-28T18:09:44.008936+0000","last_fullsized":"2024-03-28T18:09:44.008936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:46:27.080005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887972+0000","last_change":"2024-03-28T18:08:19.480310+0000","last_active":"2024-03-28T18:09:15.887972+0000","last_peered":"2024-03-28T18:09:15.887972+0000","last_clean":"2024-03-28T18:09:15.887972+0000","last_became_active":"2024-03-28T18:08:19.479919+0000","last_became_peered":"2024-03-28T18:08:19.479919+0000","last_unstale":"2024-03-28T18:09:15.887972+0000","last_undegraded":"2024-03-28T18:09:15.887972+0000","last_fullsized":"2024-03-28T18:09:15.887972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:36:26.239515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.018790+0000","last_change":"2024-03-28T18:08:20.033974+0000","last_active":"2024-03-28T18:09:44.018790+0000","last_peered":"2024-03-28T18:09:44.018790+0000","last_clean":"2024-03-28T18:09:44.018790+0000","last_became_active":"2024-03-28T18:08:20.033544+0000","last_became_peered":"2024-03-28T18:08:20.033544+0000","last_unstale":"2024-03-28T18:09:44.018790+0000","last_undegraded":"2024-03-28T18:09:44.018790+0000","last_fullsized":"2024-03-28T18:09:44.018790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:45:09.617988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.718345+0000","last_change":"2024-03-28T18:08:19.482604+0000","last_active":"2024-03-28T18:09:15.718345+0000","last_peered":"2024-03-28T18:09:15.718345+0000","last_clean":"2024-03-28T18:09:15.718345+0000","last_became_active":"2024-03-28T18:08:19.482248+0000","last_became_peered":"2024-03-28T18:08:19.482248+0000","last_unstale":"2024-03-28T18:09:15.718345+0000","last_undegraded":"2024-03-28T18:09:15.718345+0000","last_fullsized":"2024-03-28T18:09:15.718345+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:23.199196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.929801+0000","last_change":"2024-03-28T18:08:20.011198+0000","last_active":"2024-03-28T18:09:43.929801+0000","last_peered":"2024-03-28T18:09:43.929801+0000","last_clean":"2024-03-28T18:09:43.929801+0000","last_became_active":"2024-03-28T18:08:20.010866+0000","last_became_peered":"2024-03-28T18:08:20.010866+0000","last_unstale":"2024-03-28T18:09:43.929801+0000","last_undegraded":"2024-03-28T18:09:43.929801+0000","last_fullsized":"2024-03-28T18:09:43.929801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:26.342858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.724808+0000","last_change":"2024-03-28T18:08:19.493830+0000","last_active":"2024-03-28T18:09:15.724808+0000","last_peered":"2024-03-28T18:09:15.724808+0000","last_clean":"2024-03-28T18:09:15.724808+0000","last_became_active":"2024-03-28T18:08:19.493447+0000","last_became_peered":"2024-03-28T18:08:19.493447+0000","last_unstale":"2024-03-28T18:09:15.724808+0000","last_undegraded":"2024-03-28T18:09:15.724808+0000","last_fullsized":"2024-03-28T18:09:15.724808+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:33:29.797186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.993414+0000","last_change":"2024-03-28T18:08:20.025620+0000","last_active":"2024-03-28T18:09:43.993414+0000","last_peered":"2024-03-28T18:09:43.993414+0000","last_clean":"2024-03-28T18:09:43.993414+0000","last_became_active":"2024-03-28T18:08:20.025276+0000","last_became_peered":"2024-03-28T18:08:20.025276+0000","last_unstale":"2024-03-28T18:09:43.993414+0000","last_undegraded":"2024-03-28T18:09:43.993414+0000","last_fullsized":"2024-03-28T18:09:43.993414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:30:06.667707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.718293+0000","last_change":"2024-03-28T18:08:19.474204+0000","last_active":"2024-03-28T18:09:15.718293+0000","last_peered":"2024-03-28T18:09:15.718293+0000","last_clean":"2024-03-28T18:09:15.718293+0000","last_became_active":"2024-03-28T18:08:19.473688+0000","last_became_peered":"2024-03-28T18:08:19.473688+0000","last_unstale":"2024-03-28T18:09:15.718293+0000","last_undegraded":"2024-03-28T18:09:15.718293+0000","last_fullsized":"2024-03-28T18:09:15.718293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:15:09.880974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.015079+0000","last_change":"2024-03-28T18:08:20.035920+0000","last_active":"2024-03-28T18:09:44.015079+0000","last_peered":"2024-03-28T18:09:44.015079+0000","last_clean":"2024-03-28T18:09:44.015079+0000","last_became_active":"2024-03-28T18:08:20.035600+0000","last_became_peered":"2024-03-28T18:08:20.035600+0000","last_unstale":"2024-03-28T18:09:44.015079+0000","last_undegraded":"2024-03-28T18:09:44.015079+0000","last_fullsized":"2024-03-28T18:09:44.015079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:41.280448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.715528+0000","last_change":"2024-03-28T18:08:19.481305+0000","last_active":"2024-03-28T18:09:15.715528+0000","last_peered":"2024-03-28T18:09:15.715528+0000","last_clean":"2024-03-28T18:09:15.715528+0000","last_became_active":"2024-03-28T18:08:19.480978+0000","last_became_peered":"2024-03-28T18:08:19.480978+0000","last_unstale":"2024-03-28T18:09:15.715528+0000","last_undegraded":"2024-03-28T18:09:15.715528+0000","last_fullsized":"2024-03-28T18:09:15.715528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:14:06.464967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.002187+0000","last_change":"2024-03-28T18:08:20.017660+0000","last_active":"2024-03-28T18:09:44.002187+0000","last_peered":"2024-03-28T18:09:44.002187+0000","last_clean":"2024-03-28T18:09:44.002187+0000","last_became_active":"2024-03-28T18:08:20.016083+0000","last_became_peered":"2024-03-28T18:08:20.016083+0000","last_unstale":"2024-03-28T18:09:44.002187+0000","last_undegraded":"2024-03-28T18:09:44.002187+0000","last_fullsized":"2024-03-28T18:09:44.002187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:33:12.104274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.718265+0000","last_change":"2024-03-28T18:08:19.479855+0000","last_active":"2024-03-28T18:09:15.718265+0000","last_peered":"2024-03-28T18:09:15.718265+0000","last_clean":"2024-03-28T18:09:15.718265+0000","last_became_active":"2024-03-28T18:08:19.479429+0000","last_became_peered":"2024-03-28T18:08:19.479429+0000","last_unstale":"2024-03-28T18:09:15.718265+0000","last_undegraded":"2024-03-28T18:09:15.718265+0000","last_fullsized":"2024-03-28T18:09:15.718265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:06:23.286970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.978410+0000","last_change":"2024-03-28T18:08:20.015419+0000","last_active":"2024-03-28T18:09:43.978410+0000","last_peered":"2024-03-28T18:09:43.978410+0000","last_clean":"2024-03-28T18:09:43.978410+0000","last_became_active":"2024-03-28T18:08:20.015011+0000","last_became_peered":"2024-03-28T18:08:20.015011+0000","last_unstale":"2024-03-28T18:09:43.978410+0000","last_undegraded":"2024-03-28T18:09:43.978410+0000","last_fullsized":"2024-03-28T18:09:43.978410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:44:46.858352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887075+0000","last_change":"2024-03-28T18:08:19.495541+0000","last_active":"2024-03-28T18:09:15.887075+0000","last_peered":"2024-03-28T18:09:15.887075+0000","last_clean":"2024-03-28T18:09:15.887075+0000","last_became_active":"2024-03-28T18:08:19.495173+0000","last_became_peered":"2024-03-28T18:08:19.495173+0000","last_unstale":"2024-03-28T18:09:15.887075+0000","last_undegraded":"2024-03-28T18:09:15.887075+0000","last_fullsized":"2024-03-28T18:09:15.887075+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:42:06.423107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.981064+0000","last_change":"2024-03-28T18:08:20.018582+0000","last_active":"2024-03-28T18:09:43.981064+0000","last_peered":"2024-03-28T18:09:43.981064+0000","last_clean":"2024-03-28T18:09:43.981064+0000","last_became_active":"2024-03-28T18:08:20.017936+0000","last_became_peered":"2024-03-28T18:08:20.017936+0000","last_unstale":"2024-03-28T18:09:43.981064+0000","last_undegraded":"2024-03-28T18:09:43.981064+0000","last_fullsized":"2024-03-28T18:09:43.981064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:05:49.510246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.888923+0000","last_change":"2024-03-28T18:08:19.497049+0000","last_active":"2024-03-28T18:09:15.888923+0000","last_peered":"2024-03-28T18:09:15.888923+0000","last_clean":"2024-03-28T18:09:15.888923+0000","last_became_active":"2024-03-28T18:08:19.496623+0000","last_became_peered":"2024-03-28T18:08:19.496623+0000","last_unstale":"2024-03-28T18:09:15.888923+0000","last_undegraded":"2024-03-28T18:09:15.888923+0000","last_fullsized":"2024-03-28T18:09:15.888923+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:55:07.842439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.974619+0000","last_change":"2024-03-28T18:08:20.017419+0000","last_active":"2024-03-28T18:09:43.974619+0000","last_peered":"2024-03-28T18:09:43.974619+0000","last_clean":"2024-03-28T18:09:43.974619+0000","last_became_active":"2024-03-28T18:08:20.015916+0000","last_became_peered":"2024-03-28T18:08:20.015916+0000","last_unstale":"2024-03-28T18:09:43.974619+0000","last_undegraded":"2024-03-28T18:09:43.974619+0000","last_fullsized":"2024-03-28T18:09:43.974619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:10:55.927794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.964015+0000","last_change":"2024-03-28T18:08:20.008093+0000","last_active":"2024-03-28T18:09:43.964015+0000","last_peered":"2024-03-28T18:09:43.964015+0000","last_clean":"2024-03-28T18:09:43.964015+0000","last_became_active":"2024-03-28T18:08:20.007801+0000","last_became_peered":"2024-03-28T18:08:20.007801+0000","last_unstale":"2024-03-28T18:09:43.964015+0000","last_undegraded":"2024-03-28T18:09:43.964015+0000","last_fullsized":"2024-03-28T18:09:43.964015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:58.976859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.967721+0000","last_change":"2024-03-28T18:08:20.012771+0000","last_active":"2024-03-28T18:09:43.967721+0000","last_peered":"2024-03-28T18:09:43.967721+0000","last_clean":"2024-03-28T18:09:43.967721+0000","last_became_active":"2024-03-28T18:08:20.012456+0000","last_became_peered":"2024-03-28T18:08:20.012456+0000","last_unstale":"2024-03-28T18:09:43.967721+0000","last_undegraded":"2024-03-28T18:09:43.967721+0000","last_fullsized":"2024-03-28T18:09:43.967721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:56.428332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.000349+0000","last_change":"2024-03-28T18:08:20.034872+0000","last_active":"2024-03-28T18:09:44.000349+0000","last_peered":"2024-03-28T18:09:44.000349+0000","last_clean":"2024-03-28T18:09:44.000349+0000","last_became_active":"2024-03-28T18:08:20.034614+0000","last_became_peered":"2024-03-28T18:08:20.034614+0000","last_unstale":"2024-03-28T18:09:44.000349+0000","last_undegraded":"2024-03-28T18:09:44.000349+0000","last_fullsized":"2024-03-28T18:09:44.000349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:35.658186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.017643+0000","last_change":"2024-03-28T18:08:20.033119+0000","last_active":"2024-03-28T18:09:44.017643+0000","last_peered":"2024-03-28T18:09:44.017643+0000","last_clean":"2024-03-28T18:09:44.017643+0000","last_became_active":"2024-03-28T18:08:20.030694+0000","last_became_peered":"2024-03-28T18:08:20.030694+0000","last_unstale":"2024-03-28T18:09:44.017643+0000","last_undegraded":"2024-03-28T18:09:44.017643+0000","last_fullsized":"2024-03-28T18:09:44.017643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:51:50.082993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.020345+0000","last_change":"2024-03-28T18:08:20.025725+0000","last_active":"2024-03-28T18:09:44.020345+0000","last_peered":"2024-03-28T18:09:44.020345+0000","last_clean":"2024-03-28T18:09:44.020345+0000","last_became_active":"2024-03-28T18:08:20.024929+0000","last_became_peered":"2024-03-28T18:08:20.024929+0000","last_unstale":"2024-03-28T18:09:44.020345+0000","last_undegraded":"2024-03-28T18:09:44.020345+0000","last_fullsized":"2024-03-28T18:09:44.020345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:04.894246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.960815+0000","last_change":"2024-03-28T18:08:20.005293+0000","last_active":"2024-03-28T18:09:43.960815+0000","last_peered":"2024-03-28T18:09:43.960815+0000","last_clean":"2024-03-28T18:09:43.960815+0000","last_became_active":"2024-03-28T18:08:20.004938+0000","last_became_peered":"2024-03-28T18:08:20.004938+0000","last_unstale":"2024-03-28T18:09:43.960815+0000","last_undegraded":"2024-03-28T18:09:43.960815+0000","last_fullsized":"2024-03-28T18:09:43.960815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:36.100810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.971631+0000","last_change":"2024-03-28T18:08:20.008741+0000","last_active":"2024-03-28T18:09:43.971631+0000","last_peered":"2024-03-28T18:09:43.971631+0000","last_clean":"2024-03-28T18:09:43.971631+0000","last_became_active":"2024-03-28T18:08:20.008411+0000","last_became_peered":"2024-03-28T18:08:20.008411+0000","last_unstale":"2024-03-28T18:09:43.971631+0000","last_undegraded":"2024-03-28T18:09:43.971631+0000","last_fullsized":"2024-03-28T18:09:43.971631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:28:52.276278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.996983+0000","last_change":"2024-03-28T18:08:20.028486+0000","last_active":"2024-03-28T18:09:43.996983+0000","last_peered":"2024-03-28T18:09:43.996983+0000","last_clean":"2024-03-28T18:09:43.996983+0000","last_became_active":"2024-03-28T18:08:20.028121+0000","last_became_peered":"2024-03-28T18:08:20.028121+0000","last_unstale":"2024-03-28T18:09:43.996983+0000","last_undegraded":"2024-03-28T18:09:43.996983+0000","last_fullsized":"2024-03-28T18:09:43.996983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:55:26.828305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.966350+0000","last_change":"2024-03-28T18:08:20.028085+0000","last_active":"2024-03-28T18:09:43.966350+0000","last_peered":"2024-03-28T18:09:43.966350+0000","last_clean":"2024-03-28T18:09:43.966350+0000","last_became_active":"2024-03-28T18:08:20.027721+0000","last_became_peered":"2024-03-28T18:08:20.027721+0000","last_unstale":"2024-03-28T18:09:43.966350+0000","last_undegraded":"2024-03-28T18:09:43.966350+0000","last_fullsized":"2024-03-28T18:09:43.966350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:31:36.674002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.011940+0000","last_change":"2024-03-28T18:08:20.023605+0000","last_active":"2024-03-28T18:09:44.011940+0000","last_peered":"2024-03-28T18:09:44.011940+0000","last_clean":"2024-03-28T18:09:44.011940+0000","last_became_active":"2024-03-28T18:08:20.018217+0000","last_became_peered":"2024-03-28T18:08:20.018217+0000","last_unstale":"2024-03-28T18:09:44.011940+0000","last_undegraded":"2024-03-28T18:09:44.011940+0000","last_fullsized":"2024-03-28T18:09:44.011940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:43.717751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.014807+0000","last_change":"2024-03-28T18:08:20.023072+0000","last_active":"2024-03-28T18:09:44.014807+0000","last_peered":"2024-03-28T18:09:44.014807+0000","last_clean":"2024-03-28T18:09:44.014807+0000","last_became_active":"2024-03-28T18:08:20.022690+0000","last_became_peered":"2024-03-28T18:08:20.022690+0000","last_unstale":"2024-03-28T18:09:44.014807+0000","last_undegraded":"2024-03-28T18:09:44.014807+0000","last_fullsized":"2024-03-28T18:09:44.014807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:21.391415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.018858+0000","last_change":"2024-03-28T18:08:20.024261+0000","last_active":"2024-03-28T18:09:44.018858+0000","last_peered":"2024-03-28T18:09:44.018858+0000","last_clean":"2024-03-28T18:09:44.018858+0000","last_became_active":"2024-03-28T18:08:20.023792+0000","last_became_peered":"2024-03-28T18:08:20.023792+0000","last_unstale":"2024-03-28T18:09:44.018858+0000","last_undegraded":"2024-03-28T18:09:44.018858+0000","last_fullsized":"2024-03-28T18:09:44.018858+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:23:08.030982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.961667+0000","last_change":"2024-03-28T18:09:15.720113+0000","last_active":"2024-03-28T18:09:43.961667+0000","last_peered":"2024-03-28T18:09:43.961667+0000","last_clean":"2024-03-28T18:09:43.961667+0000","last_became_active":"2024-03-28T18:09:15.719739+0000","last_became_peered":"2024-03-28T18:09:15.719739+0000","last_unstale":"2024-03-28T18:09:43.961667+0000","last_undegraded":"2024-03-28T18:09:43.961667+0000","last_fullsized":"2024-03-28T18:09:43.961667+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:09:38.179780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":248983,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66287,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":569,"num_read_kb":355,"num_write":166,"num_write_kb":115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":694,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":198861,"data_compressed":5634,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383713,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":456779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645351,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":1109823,"data_compressed":37917,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808876,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070514,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332152,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":1047842,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495678,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":455822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":517872,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":457093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1024000,"data_stored":519530,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182479,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":456133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920843,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":456259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757316,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":1048108,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34394,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34394,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62257,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":63915,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63723,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:09:57.137 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-28T18:09:57.137 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-28T18:09:57.137 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-28T18:09:57.137 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-28T18:09:57.271 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:56 smithi037 ceph-mon[31460]: pgmap v368: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2024-03-28T18:09:57.271 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:56 smithi037 ceph-mon[31460]: from='client.24913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:09:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:56 smithi059 ceph-mon[36190]: pgmap v368: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2024-03-28T18:09:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:56 smithi059 ceph-mon[36190]: from='client.24913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:09:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:56 smithi170 ceph-mon[36280]: pgmap v368: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2024-03-28T18:09:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:56 smithi170 ceph-mon[36280]: from='client.24913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:09:57.645 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:09:59.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:09:59 smithi037 ceph-mon[31460]: pgmap v369: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T18:09:59.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:09:59 smithi059 ceph-mon[36190]: pgmap v369: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T18:09:59.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:09:59 smithi170 ceph-mon[36280]: pgmap v369: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T18:09:59.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.967+0000 7f2908a00700 1 -- 172.21.15.37:0/1127998633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f290410c670 msgr2=0x7f290410eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:59.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.967+0000 7f2908a00700 1 --2- 172.21.15.37:0/1127998633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f290410c670 0x7f290410eb10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f28fc00b0a0 tx=0x7f28fc00b3b0 comp rx=0 tx=0).stop 2024-03-28T18:09:59.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.967+0000 7f2908a00700 1 -- 172.21.15.37:0/1127998633 shutdown_connections 2024-03-28T18:09:59.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.967+0000 7f2908a00700 1 --2- 172.21.15.37:0/1127998633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f290410c670 0x7f290410eb10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:59.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.967+0000 7f2908a00700 1 --2- 172.21.15.37:0/1127998633 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2904103290 0x7f2904103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:59.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.967+0000 7f2908a00700 1 --2- 172.21.15.37:0/1127998633 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2904103bb0 0x7f290410c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:59.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.967+0000 7f2908a00700 1 -- 172.21.15.37:0/1127998633 >> 172.21.15.37:0/1127998633 conn(0x7f29040fca60 msgr2=0x7f29040fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:09:59.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.967+0000 7f2908a00700 1 -- 172.21.15.37:0/1127998633 shutdown_connections 2024-03-28T18:09:59.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.967+0000 7f2908a00700 1 -- 172.21.15.37:0/1127998633 wait complete. 2024-03-28T18:09:59.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.968+0000 7f2908a00700 1 Processor -- start 2024-03-28T18:09:59.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.968+0000 7f2908a00700 1 -- start start 2024-03-28T18:09:59.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.969+0000 7f2908a00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2904103290 0x7f29041a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:59.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.969+0000 7f2908a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2904103bb0 0x7f29041a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:59.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.969+0000 7f2908a00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f290410c670 0x7f290419a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:59.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.969+0000 7f2908a00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29041123f0 con 0x7f2904103bb0 2024-03-28T18:09:59.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.969+0000 7f2908a00700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2904112270 con 0x7f290410c670 2024-03-28T18:09:59.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.969+0000 7f2908a00700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2904112570 con 0x7f2904103290 2024-03-28T18:09:59.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.969+0000 7f2902800700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2904103290 0x7f29041a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:59.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.969+0000 7f2902800700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2904103290 0x7f29041a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:51166/0 (socket says 172.21.15.37:51166) 2024-03-28T18:09:59.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.969+0000 7f2902800700 1 -- 172.21.15.37:0/4273256466 learned_addr learned my addr 172.21.15.37:0/4273256466 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:09:59.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.969+0000 7f28fbe00700 1 --2- 172.21.15.37:0/4273256466 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2904103bb0 0x7f29041a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:59.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.969+0000 7f2903200700 1 --2- 172.21.15.37:0/4273256466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f290410c670 0x7f290419a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:59.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.969+0000 7f2902800700 1 -- 172.21.15.37:0/4273256466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f290410c670 msgr2=0x7f290419a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:59.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.969+0000 7f2902800700 1 --2- 172.21.15.37:0/4273256466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f290410c670 0x7f290419a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:59.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.969+0000 7f2902800700 1 -- 172.21.15.37:0/4273256466 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2904103bb0 msgr2=0x7f29041a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:09:59.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.969+0000 7f2902800700 1 --2- 172.21.15.37:0/4273256466 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2904103bb0 0x7f29041a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:09:59.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.969+0000 7f2902800700 1 -- 172.21.15.37:0/4273256466 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290419b070 con 0x7f2904103290 2024-03-28T18:09:59.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.969+0000 7f2903200700 1 --2- 172.21.15.37:0/4273256466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f290410c670 0x7f290419a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:09:59.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.970+0000 7f2902800700 1 --2- 172.21.15.37:0/4273256466 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2904103290 0x7f29041a06c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f28f400e9a0 tx=0x7f28f400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:09:59.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.970+0000 7f28fb400700 1 -- 172.21.15.37:0/4273256466 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f400cbd0 con 0x7f2904103290 2024-03-28T18:09:59.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.970+0000 7f28fb400700 1 -- 172.21.15.37:0/4273256466 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28f400cd70 con 0x7f2904103290 2024-03-28T18:09:59.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.970+0000 7f28fb400700 1 -- 172.21.15.37:0/4273256466 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f4018750 con 0x7f2904103290 2024-03-28T18:09:59.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.970+0000 7f2908a00700 1 -- 172.21.15.37:0/4273256466 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f290419b360 con 0x7f2904103290 2024-03-28T18:09:59.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.970+0000 7f2908a00700 1 -- 172.21.15.37:0/4273256466 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29041a7520 con 0x7f2904103290 2024-03-28T18:09:59.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.971+0000 7f2908a00700 1 -- 172.21.15.37:0/4273256466 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f290406a6c0 con 0x7f2904103290 2024-03-28T18:09:59.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.976+0000 7f28fb400700 1 -- 172.21.15.37:0/4273256466 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f28f4010430 con 0x7f2904103290 2024-03-28T18:09:59.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.976+0000 7f28fb400700 1 --2- 172.21.15.37:0/4273256466 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f28e80a2910 0x7f28e80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:09:59.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.977+0000 7f28fb400700 1 -- 172.21.15.37:0/4273256466 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f28f40cf1b0 con 0x7f2904103290 2024-03-28T18:09:59.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.977+0000 7f28fbe00700 1 --2- 172.21.15.37:0/4273256466 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f28e80a2910 0x7f28e80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:09:59.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.977+0000 7f28fb400700 1 -- 172.21.15.37:0/4273256466 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f28f409c200 con 0x7f2904103290 2024-03-28T18:09:59.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:09:59.978+0000 7f28fbe00700 1 --2- 172.21.15.37:0/4273256466 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f28e80a2910 0x7f28e80a4dd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f290410d920 tx=0x7f28ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:00.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:00.289+0000 7f2908a00700 1 -- 172.21.15.37:0/4273256466 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2904068e50 con 0x7f2904103290 2024-03-28T18:10:00.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:00.290+0000 7f28fb400700 1 -- 172.21.15.37:0/4273256466 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f28f409bb00 con 0x7f2904103290 2024-03-28T18:10:00.290 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:00.290 INFO:teuthology.orchestra.run.smithi037.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-28T18:10:00.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:00.294+0000 7f2908a00700 1 -- 172.21.15.37:0/4273256466 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f28e80a2910 msgr2=0x7f28e80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:00.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:00.294+0000 7f2908a00700 1 --2- 172.21.15.37:0/4273256466 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f28e80a2910 0x7f28e80a4dd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f290410d920 tx=0x7f28ec009260 comp rx=0 tx=0).stop 2024-03-28T18:10:00.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:00.294+0000 7f2908a00700 1 -- 172.21.15.37:0/4273256466 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2904103290 msgr2=0x7f29041a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:00.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:00.294+0000 7f2908a00700 1 --2- 172.21.15.37:0/4273256466 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2904103290 0x7f29041a06c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f28f400e9a0 tx=0x7f28f400ecb0 comp rx=0 tx=0).stop 2024-03-28T18:10:00.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:00.294+0000 7f2908a00700 1 -- 172.21.15.37:0/4273256466 shutdown_connections 2024-03-28T18:10:00.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:00.294+0000 7f2908a00700 1 --2- 172.21.15.37:0/4273256466 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f28e80a2910 0x7f28e80a4dd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:00.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:00.294+0000 7f2908a00700 1 --2- 172.21.15.37:0/4273256466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f290410c670 0x7f290419a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:00.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:00.294+0000 7f2908a00700 1 --2- 172.21.15.37:0/4273256466 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2904103290 0x7f29041a06c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:00.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:00.294+0000 7f2908a00700 1 --2- 172.21.15.37:0/4273256466 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2904103bb0 0x7f29041a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:00.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:00.294+0000 7f2908a00700 1 -- 172.21.15.37:0/4273256466 >> 172.21.15.37:0/4273256466 conn(0x7f29040fca60 msgr2=0x7f290410e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:00.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:00.294+0000 7f2908a00700 1 -- 172.21.15.37:0/4273256466 shutdown_connections 2024-03-28T18:10:00.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:00.294+0000 7f2908a00700 1 -- 172.21.15.37:0/4273256466 wait complete. 2024-03-28T18:10:00.312 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:00 smithi037 ceph-mon[31460]: overall HEALTH_OK 2024-03-28T18:10:00.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:00 smithi059 ceph-mon[36190]: overall HEALTH_OK 2024-03-28T18:10:00.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:00 smithi170 ceph-mon[36280]: overall HEALTH_OK 2024-03-28T18:10:01.232 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-28T18:10:01.233 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:10:01.322 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:01 smithi037 ceph-mon[31460]: pgmap v370: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:10:01.322 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:01 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/4273256466' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T18:10:01.436 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:01 smithi059 ceph-mon[36190]: pgmap v370: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:10:01.436 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:01 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/4273256466' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T18:10:01.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.461+0000 7f6925800640 1 -- 172.21.15.37:0/1214292774 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69200756e0 msgr2=0x7f6920073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:01.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.461+0000 7f6925800640 1 --2- 172.21.15.37:0/1214292774 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69200756e0 0x7f6920073b30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6910009900 tx=0x7f691002bc40 comp rx=0 tx=0).stop 2024-03-28T18:10:01.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.461+0000 7f6925800640 1 -- 172.21.15.37:0/1214292774 shutdown_connections 2024-03-28T18:10:01.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.461+0000 7f6925800640 1 --2- 172.21.15.37:0/1214292774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69201093c0 0x7f692010b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:01.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.461+0000 7f6925800640 1 --2- 172.21.15.37:0/1214292774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6920074100 0x7f6920074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:01.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.461+0000 7f6925800640 1 --2- 172.21.15.37:0/1214292774 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69200756e0 0x7f6920073b30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:01.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.461+0000 7f6925800640 1 -- 172.21.15.37:0/1214292774 >> 172.21.15.37:0/1214292774 conn(0x7f69200fca70 msgr2=0x7f69200feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:01.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.461+0000 7f6925800640 1 -- 172.21.15.37:0/1214292774 shutdown_connections 2024-03-28T18:10:01.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.461+0000 7f6925800640 1 -- 172.21.15.37:0/1214292774 wait complete. 2024-03-28T18:10:01.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.463+0000 7f6925800640 1 Processor -- start 2024-03-28T18:10:01.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.463+0000 7f6925800640 1 -- start start 2024-03-28T18:10:01.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.463+0000 7f6925800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6920074100 0x7f69201999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:01.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.463+0000 7f6925800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69200756e0 0x7f6920199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:01.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.463+0000 7f6925800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69201093c0 0x7f69201a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:01.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.463+0000 7f6925800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692010f390 con 0x7f69201093c0 2024-03-28T18:10:01.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.463+0000 7f6925800640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692010f210 con 0x7f6920074100 2024-03-28T18:10:01.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.463+0000 7f6925800640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692010f510 con 0x7f69200756e0 2024-03-28T18:10:01.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.463+0000 7f691fe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6920074100 0x7f69201999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:01.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.464+0000 7f691fe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6920074100 0x7f69201999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:38842/0 (socket says 172.21.15.37:38842) 2024-03-28T18:10:01.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.464+0000 7f691fe00640 1 -- 172.21.15.37:0/2012026326 learned_addr learned my addr 172.21.15.37:0/2012026326 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:01.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.464+0000 7f6917400640 1 --2- 172.21.15.37:0/2012026326 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69200756e0 0x7f6920199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:01.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.464+0000 7f6924e00640 1 --2- 172.21.15.37:0/2012026326 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69201093c0 0x7f69201a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:01.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.464+0000 7f691fe00640 1 -- 172.21.15.37:0/2012026326 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69200756e0 msgr2=0x7f6920199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:01.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.464+0000 7f691fe00640 1 --2- 172.21.15.37:0/2012026326 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69200756e0 0x7f6920199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:01.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.464+0000 7f691fe00640 1 -- 172.21.15.37:0/2012026326 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69201093c0 msgr2=0x7f69201a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:01.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.464+0000 7f691fe00640 1 --2- 172.21.15.37:0/2012026326 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69201093c0 0x7f69201a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:01.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.464+0000 7f691fe00640 1 -- 172.21.15.37:0/2012026326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69201a15d0 con 0x7f6920074100 2024-03-28T18:10:01.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.464+0000 7f6917400640 1 --2- 172.21.15.37:0/2012026326 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69200756e0 0x7f6920199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:01.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.464+0000 7f6924e00640 1 --2- 172.21.15.37:0/2012026326 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69201093c0 0x7f69201a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:01.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.464+0000 7f691fe00640 1 --2- 172.21.15.37:0/2012026326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6920074100 0x7f69201999b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f691002c8f0 tx=0x7f6910005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:01.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.464+0000 7f691d600640 1 -- 172.21.15.37:0/2012026326 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6910041070 con 0x7f6920074100 2024-03-28T18:10:01.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.465+0000 7f691d600640 1 -- 172.21.15.37:0/2012026326 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69100044d0 con 0x7f6920074100 2024-03-28T18:10:01.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.465+0000 7f691d600640 1 -- 172.21.15.37:0/2012026326 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6910034440 con 0x7f6920074100 2024-03-28T18:10:01.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.465+0000 7f6925800640 1 -- 172.21.15.37:0/2012026326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69201a1860 con 0x7f6920074100 2024-03-28T18:10:01.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.465+0000 7f6925800640 1 -- 172.21.15.37:0/2012026326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69201a1d00 con 0x7f6920074100 2024-03-28T18:10:01.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.466+0000 7f6925800640 1 -- 172.21.15.37:0/2012026326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69201080f0 con 0x7f6920074100 2024-03-28T18:10:01.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.468+0000 7f691d600640 1 -- 172.21.15.37:0/2012026326 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6910044070 con 0x7f6920074100 2024-03-28T18:10:01.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.470+0000 7f691d600640 1 --2- 172.21.15.37:0/2012026326 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f69040a2880 0x7f69040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:01.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.471+0000 7f691d600640 1 -- 172.21.15.37:0/2012026326 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f69100f0690 con 0x7f6920074100 2024-03-28T18:10:01.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.471+0000 7f6917400640 1 --2- 172.21.15.37:0/2012026326 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f69040a2880 0x7f69040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:01.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.471+0000 7f6917400640 1 --2- 172.21.15.37:0/2012026326 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f69040a2880 0x7f69040a4d40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f692019aec0 tx=0x7f690800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:01.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.473+0000 7f691d600640 1 -- 172.21.15.37:0/2012026326 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f69100bd6e0 con 0x7f6920074100 2024-03-28T18:10:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:01 smithi170 ceph-mon[36280]: pgmap v370: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:10:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:01 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/4273256466' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T18:10:01.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.680+0000 7f6925800640 1 -- 172.21.15.37:0/2012026326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69200622a0 con 0x7f6920074100 2024-03-28T18:10:01.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.682+0000 7f691d600640 1 -- 172.21.15.37:0/2012026326 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22575 (secure 0 0 0) 0x7f69100bcfe0 con 0x7f6920074100 2024-03-28T18:10:01.683 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:01.683 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":87,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","created":"2024-03-28T17:59:05.949847+0000","modified":"2024-03-28T18:09:15.709901+0000","last_up_change":"2024-03-28T18:07:21.021030+0000","last_in_change":"2024-03-28T18:07:01.416192+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:03:00.517967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:08:18.365827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:08:19.446282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db60618d-56f7-4130-8a90-84cbc480965b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6803","nonce":3562602035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6805","nonce":3562602035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6809","nonce":3562602035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6807","nonce":3562602035}]},"public_addr":"172.21.15.37:6803/3562602035","cluster_addr":"172.21.15.37:6805/3562602035","heartbeat_back_addr":"172.21.15.37:6809/3562602035","heartbeat_front_addr":"172.21.15.37:6807/3562602035","state":["exists","up"]},{"osd":1,"uuid":"7a4b7ad6-905e-4258-9574-693d4277dfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6801","nonce":3326537211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6803","nonce":3326537211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6807","nonce":3326537211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6805","nonce":3326537211}]},"public_addr":"172.21.15.59:6801/3326537211","cluster_addr":"172.21.15.59:6803/3326537211","heartbeat_back_addr":"172.21.15.59:6807/3326537211","heartbeat_front_addr":"172.21.15.59:6805/3326537211","state":["exists","up"]},{"osd":2,"uuid":"8b76ec1c-46bf-4336-89b5-12d82906e8b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6801","nonce":1269398427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6803","nonce":1269398427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6807","nonce":1269398427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6805","nonce":1269398427}]},"public_addr":"172.21.15.170:6801/1269398427","cluster_addr":"172.21.15.170:6803/1269398427","heartbeat_back_addr":"172.21.15.170:6807/1269398427","heartbeat_front_addr":"172.21.15.170:6805/1269398427","state":["exists","up"]},{"osd":3,"uuid":"66eca1da-2fbd-44e1-8bbf-5635a2c70ad5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6811","nonce":317805240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6813","nonce":317805240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6817","nonce":317805240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6815","nonce":317805240}]},"public_addr":"172.21.15.37:6811/317805240","cluster_addr":"172.21.15.37:6813/317805240","heartbeat_back_addr":"172.21.15.37:6817/317805240","heartbeat_front_addr":"172.21.15.37:6815/317805240","state":["exists","up"]},{"osd":4,"uuid":"4c877335-e42d-4d8e-9f48-a1d0a89ae438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6809","nonce":2410632623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6811","nonce":2410632623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6815","nonce":2410632623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6813","nonce":2410632623}]},"public_addr":"172.21.15.59:6809/2410632623","cluster_addr":"172.21.15.59:6811/2410632623","heartbeat_back_addr":"172.21.15.59:6815/2410632623","heartbeat_front_addr":"172.21.15.59:6813/2410632623","state":["exists","up"]},{"osd":5,"uuid":"b884842a-86e2-455b-9570-ff163b3bc0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6809","nonce":1218943899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6811","nonce":1218943899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6815","nonce":1218943899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6813","nonce":1218943899}]},"public_addr":"172.21.15.170:6809/1218943899","cluster_addr":"172.21.15.170:6811/1218943899","heartbeat_back_addr":"172.21.15.170:6815/1218943899","heartbeat_front_addr":"172.21.15.170:6813/1218943899","state":["exists","up"]},{"osd":6,"uuid":"249a1200-77f4-44ad-981f-1d352f732b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6819","nonce":1899652046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6821","nonce":1899652046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6825","nonce":1899652046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6823","nonce":1899652046}]},"public_addr":"172.21.15.37:6819/1899652046","cluster_addr":"172.21.15.37:6821/1899652046","heartbeat_back_addr":"172.21.15.37:6825/1899652046","heartbeat_front_addr":"172.21.15.37:6823/1899652046","state":["exists","up"]},{"osd":7,"uuid":"c73338e6-f505-4517-a84b-f35e04750a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6817","nonce":1343675690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6819","nonce":1343675690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6823","nonce":1343675690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6821","nonce":1343675690}]},"public_addr":"172.21.15.59:6817/1343675690","cluster_addr":"172.21.15.59:6819/1343675690","heartbeat_back_addr":"172.21.15.59:6823/1343675690","heartbeat_front_addr":"172.21.15.59:6821/1343675690","state":["exists","up"]},{"osd":8,"uuid":"149b8297-ff2d-4a88-9b1e-0def021cc516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6817","nonce":3235794556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6819","nonce":3235794556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6823","nonce":3235794556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6821","nonce":3235794556}]},"public_addr":"172.21.15.170:6817/3235794556","cluster_addr":"172.21.15.170:6819/3235794556","heartbeat_back_addr":"172.21.15.170:6823/3235794556","heartbeat_front_addr":"172.21.15.170:6821/3235794556","state":["exists","up"]},{"osd":9,"uuid":"55c27cc2-c771-44b7-90fe-8f0feca44162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6827","nonce":2307153008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6829","nonce":2307153008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6833","nonce":2307153008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6831","nonce":2307153008}]},"public_addr":"172.21.15.37:6827/2307153008","cluster_addr":"172.21.15.37:6829/2307153008","heartbeat_back_addr":"172.21.15.37:6833/2307153008","heartbeat_front_addr":"172.21.15.37:6831/2307153008","state":["exists","up"]},{"osd":10,"uuid":"c0595135-ab81-49d3-8c23-19f960c03e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6825","nonce":2113830275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6827","nonce":2113830275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6831","nonce":2113830275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6829","nonce":2113830275}]},"public_addr":"172.21.15.59:6825/2113830275","cluster_addr":"172.21.15.59:6827/2113830275","heartbeat_back_addr":"172.21.15.59:6831/2113830275","heartbeat_front_addr":"172.21.15.59:6829/2113830275","state":["exists","up"]},{"osd":11,"uuid":"5871e177-c17b-4637-a797-4fa5125920dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6825","nonce":3082439360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6827","nonce":3082439360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6831","nonce":3082439360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6829","nonce":3082439360}]},"public_addr":"172.21.15.170:6825/3082439360","cluster_addr":"172.21.15.170:6827/3082439360","heartbeat_back_addr":"172.21.15.170:6831/3082439360","heartbeat_front_addr":"172.21.15.170:6829/3082439360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:07.690545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:33.183396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:57.400263+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:30.597211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:57.259044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:22.627823+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:57.585863+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:25.000476+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:50.876688+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:25.859761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:53.563146+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:07:19.304115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3023379428":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1443710231":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6800/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6801/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6801/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6800/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1007625490":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/2895698584":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/497313034":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/3293943475":"2024-03-29T17:59:39.162038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:10:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.685+0000 7f6925800640 1 -- 172.21.15.37:0/2012026326 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f69040a2880 msgr2=0x7f69040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.685+0000 7f6925800640 1 --2- 172.21.15.37:0/2012026326 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f69040a2880 0x7f69040a4d40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f692019aec0 tx=0x7f690800b410 comp rx=0 tx=0).stop 2024-03-28T18:10:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.685+0000 7f6925800640 1 -- 172.21.15.37:0/2012026326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6920074100 msgr2=0x7f69201999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.685+0000 7f6925800640 1 --2- 172.21.15.37:0/2012026326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6920074100 0x7f69201999b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f691002c8f0 tx=0x7f6910005d50 comp rx=0 tx=0).stop 2024-03-28T18:10:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.685+0000 7f6925800640 1 -- 172.21.15.37:0/2012026326 shutdown_connections 2024-03-28T18:10:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.685+0000 7f6925800640 1 --2- 172.21.15.37:0/2012026326 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f69040a2880 0x7f69040a4d40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.685+0000 7f6925800640 1 --2- 172.21.15.37:0/2012026326 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69201093c0 0x7f69201a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:01.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.685+0000 7f6925800640 1 --2- 172.21.15.37:0/2012026326 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69200756e0 0x7f6920199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:01.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.685+0000 7f6925800640 1 --2- 172.21.15.37:0/2012026326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6920074100 0x7f69201999b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:01.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.685+0000 7f6925800640 1 -- 172.21.15.37:0/2012026326 >> 172.21.15.37:0/2012026326 conn(0x7f69200fca70 msgr2=0x7f69201052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:01.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.685+0000 7f6925800640 1 -- 172.21.15.37:0/2012026326 shutdown_connections 2024-03-28T18:10:01.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.685+0000 7f6925800640 1 -- 172.21.15.37:0/2012026326 wait complete. 2024-03-28T18:10:01.703 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:03:00.517967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:08:18.365827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:08:19.446282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:10:01.704 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:10:01.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.938+0000 7f765f200640 1 -- 172.21.15.37:0/102853998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7658107390 msgr2=0x7f76581097e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:01.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.938+0000 7f765f200640 1 --2- 172.21.15.37:0/102853998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7658107390 0x7f76581097e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7650009ca0 tx=0x7f765002c100 comp rx=0 tx=0).stop 2024-03-28T18:10:01.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.938+0000 7f765f200640 1 -- 172.21.15.37:0/102853998 shutdown_connections 2024-03-28T18:10:01.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.938+0000 7f765f200640 1 --2- 172.21.15.37:0/102853998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7658104a60 0x7f7658106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:01.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.938+0000 7f765f200640 1 --2- 172.21.15.37:0/102853998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7658107390 0x7f76581097e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:01.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.938+0000 7f765f200640 1 --2- 172.21.15.37:0/102853998 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7658102130 0x7f7658104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:01.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.938+0000 7f765f200640 1 -- 172.21.15.37:0/102853998 >> 172.21.15.37:0/102853998 conn(0x7f76580fbb40 msgr2=0x7f76580fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:01.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.938+0000 7f765f200640 1 -- 172.21.15.37:0/102853998 shutdown_connections 2024-03-28T18:10:01.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.938+0000 7f765f200640 1 -- 172.21.15.37:0/102853998 wait complete. 2024-03-28T18:10:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.939+0000 7f765f200640 1 Processor -- start 2024-03-28T18:10:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.939+0000 7f765f200640 1 -- start start 2024-03-28T18:10:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.939+0000 7f765f200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7658102130 0x7f7658101aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.939+0000 7f765f200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7658104a60 0x7f76581000e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.939+0000 7f765f200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7658107390 0x7f7658100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.939+0000 7f765f200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765810d350 con 0x7f7658107390 2024-03-28T18:10:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.939+0000 7f765f200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765810d1d0 con 0x7f7658102130 2024-03-28T18:10:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.939+0000 7f765f200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765810d4d0 con 0x7f7658104a60 2024-03-28T18:10:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.940+0000 7f7657e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7658102130 0x7f7658101aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.940+0000 7f7657e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7658102130 0x7f7658101aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:38852/0 (socket says 172.21.15.37:38852) 2024-03-28T18:10:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.940+0000 7f7657400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7658104a60 0x7f76581000e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:01.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.940+0000 7f7657e00640 1 -- 172.21.15.37:0/1641118550 learned_addr learned my addr 172.21.15.37:0/1641118550 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:01.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.940+0000 7f765d200640 1 --2- 172.21.15.37:0/1641118550 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7658107390 0x7f7658100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:01.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.940+0000 7f7657400640 1 -- 172.21.15.37:0/1641118550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7658102130 msgr2=0x7f7658101aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:01.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.940+0000 7f7657400640 1 --2- 172.21.15.37:0/1641118550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7658102130 0x7f7658101aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:01.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.940+0000 7f7657400640 1 -- 172.21.15.37:0/1641118550 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7658107390 msgr2=0x7f7658100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:01.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.940+0000 7f7657400640 1 --2- 172.21.15.37:0/1641118550 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7658107390 0x7f7658100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:01.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.940+0000 7f7657400640 1 -- 172.21.15.37:0/1641118550 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7658100eb0 con 0x7f7658104a60 2024-03-28T18:10:01.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.940+0000 7f7657e00640 1 --2- 172.21.15.37:0/1641118550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7658102130 0x7f7658101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:01.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.940+0000 7f7657400640 1 --2- 172.21.15.37:0/1641118550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7658104a60 0x7f76581000e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f764400e9c0 tx=0x7f764400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:01.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.941+0000 7f7654c00640 1 -- 172.21.15.37:0/1641118550 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764400ccc0 con 0x7f7658104a60 2024-03-28T18:10:01.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.941+0000 7f7654c00640 1 -- 172.21.15.37:0/1641118550 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7644005020 con 0x7f7658104a60 2024-03-28T18:10:01.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.941+0000 7f7654c00640 1 -- 172.21.15.37:0/1641118550 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644004a00 con 0x7f7658104a60 2024-03-28T18:10:01.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.941+0000 7f765f200640 1 -- 172.21.15.37:0/1641118550 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76581a1ae0 con 0x7f7658104a60 2024-03-28T18:10:01.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.941+0000 7f765f200640 1 -- 172.21.15.37:0/1641118550 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76581a1fe0 con 0x7f7658104a60 2024-03-28T18:10:01.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.943+0000 7f765f200640 1 -- 172.21.15.37:0/1641118550 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76580fd630 con 0x7f7658104a60 2024-03-28T18:10:01.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.946+0000 7f7654c00640 1 -- 172.21.15.37:0/1641118550 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f76440051d0 con 0x7f7658104a60 2024-03-28T18:10:01.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.948+0000 7f7654c00640 1 --2- 172.21.15.37:0/1641118550 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f763c0a2880 0x7f763c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:01.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.948+0000 7f7654c00640 1 -- 172.21.15.37:0/1641118550 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f76440ce950 con 0x7f7658104a60 2024-03-28T18:10:01.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.948+0000 7f7657e00640 1 --2- 172.21.15.37:0/1641118550 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f763c0a2880 0x7f763c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:01.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.949+0000 7f7657e00640 1 --2- 172.21.15.37:0/1641118550 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f763c0a2880 0x7f763c0a4d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7648005fd0 tx=0x7f764800b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:01.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:01.951+0000 7f7654c00640 1 -- 172.21.15.37:0/1641118550 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f764409b9a0 con 0x7f7658104a60 2024-03-28T18:10:02.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.158+0000 7f765f200640 1 -- 172.21.15.37:0/1641118550 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f76580622a0 con 0x7f7658104a60 2024-03-28T18:10:02.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.159+0000 7f7654c00640 1 -- 172.21.15.37:0/1641118550 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f764409b2a0 con 0x7f7658104a60 2024-03-28T18:10:02.159 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 1 2024-03-28T18:10:02.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.161+0000 7f765f200640 1 -- 172.21.15.37:0/1641118550 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f763c0a2880 msgr2=0x7f763c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:02.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.161+0000 7f765f200640 1 --2- 172.21.15.37:0/1641118550 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f763c0a2880 0x7f763c0a4d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7648005fd0 tx=0x7f764800b4a0 comp rx=0 tx=0).stop 2024-03-28T18:10:02.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.161+0000 7f765f200640 1 -- 172.21.15.37:0/1641118550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7658104a60 msgr2=0x7f76581000e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:02.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.161+0000 7f765f200640 1 --2- 172.21.15.37:0/1641118550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7658104a60 0x7f76581000e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f764400e9c0 tx=0x7f764400ee80 comp rx=0 tx=0).stop 2024-03-28T18:10:02.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.162+0000 7f765f200640 1 -- 172.21.15.37:0/1641118550 shutdown_connections 2024-03-28T18:10:02.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.162+0000 7f765f200640 1 --2- 172.21.15.37:0/1641118550 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f763c0a2880 0x7f763c0a4d40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:02.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.162+0000 7f765f200640 1 --2- 172.21.15.37:0/1641118550 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7658107390 0x7f7658100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:02.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.162+0000 7f765f200640 1 --2- 172.21.15.37:0/1641118550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7658104a60 0x7f76581000e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:02.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.162+0000 7f765f200640 1 --2- 172.21.15.37:0/1641118550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7658102130 0x7f7658101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:02.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.162+0000 7f765f200640 1 -- 172.21.15.37:0/1641118550 >> 172.21.15.37:0/1641118550 conn(0x7f76580fbb40 msgr2=0x7f7658102a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:02.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.162+0000 7f765f200640 1 -- 172.21.15.37:0/1641118550 shutdown_connections 2024-03-28T18:10:02.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.162+0000 7f765f200640 1 -- 172.21.15.37:0/1641118550 wait complete. 2024-03-28T18:10:02.183 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:10:02.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:02 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2012026326' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:02.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.398+0000 7fdebca00640 1 -- 172.21.15.37:0/3945070575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdeb8107cc0 msgr2=0x7fdeb81080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:02.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.398+0000 7fdebca00640 1 --2- 172.21.15.37:0/3945070575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdeb8107cc0 0x7fdeb81080e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdea8009900 tx=0x7fdea802bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:02.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.398+0000 7fdebca00640 1 -- 172.21.15.37:0/3945070575 shutdown_connections 2024-03-28T18:10:02.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.398+0000 7fdebca00640 1 --2- 172.21.15.37:0/3945070575 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeb8109870 0x7fdeb8110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:02.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.398+0000 7fdebca00640 1 --2- 172.21.15.37:0/3945070575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdeb8108eb0 0x7fdeb8109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:02.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.398+0000 7fdebca00640 1 --2- 172.21.15.37:0/3945070575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdeb8107cc0 0x7fdeb81080e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:02.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.398+0000 7fdebca00640 1 -- 172.21.15.37:0/3945070575 >> 172.21.15.37:0/3945070575 conn(0x7fdeb8076c60 msgr2=0x7fdeb80790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:02.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.398+0000 7fdebca00640 1 -- 172.21.15.37:0/3945070575 shutdown_connections 2024-03-28T18:10:02.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.398+0000 7fdebca00640 1 -- 172.21.15.37:0/3945070575 wait complete. 2024-03-28T18:10:02.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.399+0000 7fdebca00640 1 Processor -- start 2024-03-28T18:10:02.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.399+0000 7fdebca00640 1 -- start start 2024-03-28T18:10:02.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.399+0000 7fdebca00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdeb8107cc0 0x7fdeb819dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:02.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.399+0000 7fdebca00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdeb8108eb0 0x7fdeb819e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:02.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.400+0000 7fdebca00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeb8109870 0x7fdeb81a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:02.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.400+0000 7fdebca00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb8113990 con 0x7fdeb8107cc0 2024-03-28T18:10:02.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.400+0000 7fdebca00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb8113810 con 0x7fdeb8109870 2024-03-28T18:10:02.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.400+0000 7fdebca00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb8113b10 con 0x7fdeb8108eb0 2024-03-28T18:10:02.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.400+0000 7fdeb5400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdeb8108eb0 0x7fdeb819e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:02.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.400+0000 7fdeb5e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdeb8107cc0 0x7fdeb819dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:02.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.400+0000 7fdeb5400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdeb8108eb0 0x7fdeb819e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:51208/0 (socket says 172.21.15.37:51208) 2024-03-28T18:10:02.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.400+0000 7fdeb5400640 1 -- 172.21.15.37:0/1799067602 learned_addr learned my addr 172.21.15.37:0/1799067602 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:02.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.400+0000 7fdeb6800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeb8109870 0x7fdeb81a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:02.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.400+0000 7fdeb5400640 1 -- 172.21.15.37:0/1799067602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeb8109870 msgr2=0x7fdeb81a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:02.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.400+0000 7fdeb5400640 1 --2- 172.21.15.37:0/1799067602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeb8109870 0x7fdeb81a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:02.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.400+0000 7fdeb5400640 1 -- 172.21.15.37:0/1799067602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdeb8107cc0 msgr2=0x7fdeb819dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:02.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.400+0000 7fdeb5400640 1 --2- 172.21.15.37:0/1799067602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdeb8107cc0 0x7fdeb819dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:02.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.400+0000 7fdeb5400640 1 -- 172.21.15.37:0/1799067602 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb81a5bc0 con 0x7fdeb8108eb0 2024-03-28T18:10:02.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.400+0000 7fdeb5400640 1 --2- 172.21.15.37:0/1799067602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdeb8108eb0 0x7fdeb819e4e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdea400e9f0 tx=0x7fdea400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:02.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.401+0000 7fdeaea00640 1 -- 172.21.15.37:0/1799067602 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea400cd80 con 0x7fdeb8108eb0 2024-03-28T18:10:02.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.401+0000 7fdeaea00640 1 -- 172.21.15.37:0/1799067602 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdea4010910 con 0x7fdeb8108eb0 2024-03-28T18:10:02.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.401+0000 7fdeaea00640 1 -- 172.21.15.37:0/1799067602 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea4018990 con 0x7fdeb8108eb0 2024-03-28T18:10:02.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.401+0000 7fdebca00640 1 -- 172.21.15.37:0/1799067602 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb81a5eb0 con 0x7fdeb8108eb0 2024-03-28T18:10:02.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.401+0000 7fdebca00640 1 -- 172.21.15.37:0/1799067602 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb81a63c0 con 0x7fdeb8108eb0 2024-03-28T18:10:02.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.402+0000 7fdebca00640 1 -- 172.21.15.37:0/1799067602 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb810efe0 con 0x7fdeb8108eb0 2024-03-28T18:10:02.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.404+0000 7fdeaea00640 1 -- 172.21.15.37:0/1799067602 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdea4010ac0 con 0x7fdeb8108eb0 2024-03-28T18:10:02.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.406+0000 7fdeaea00640 1 --2- 172.21.15.37:0/1799067602 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fde9c0a2880 0x7fde9c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:02.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.406+0000 7fdeaea00640 1 -- 172.21.15.37:0/1799067602 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fdea40ced30 con 0x7fdeb8108eb0 2024-03-28T18:10:02.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.406+0000 7fdeb5e00640 1 --2- 172.21.15.37:0/1799067602 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fde9c0a2880 0x7fde9c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:02.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.407+0000 7fdeb5e00640 1 --2- 172.21.15.37:0/1799067602 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fde9c0a2880 0x7fde9c0a4d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdea802c170 tx=0x7fdea802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:02.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.409+0000 7fdeaea00640 1 -- 172.21.15.37:0/1799067602 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdea409b730 con 0x7fdeb8108eb0 2024-03-28T18:10:02.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:02 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2012026326' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:02.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:02 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2012026326' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:02.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.613+0000 7fdebca00640 1 -- 172.21.15.37:0/1799067602 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdeb8062340 con 0x7fdeb8108eb0 2024-03-28T18:10:02.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.613+0000 7fdeaea00640 1 -- 172.21.15.37:0/1799067602 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdea409f7d0 con 0x7fdeb8108eb0 2024-03-28T18:10:02.613 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 32 2024-03-28T18:10:02.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.616+0000 7fdebca00640 1 -- 172.21.15.37:0/1799067602 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fde9c0a2880 msgr2=0x7fde9c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:02.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.616+0000 7fdebca00640 1 --2- 172.21.15.37:0/1799067602 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fde9c0a2880 0x7fde9c0a4d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdea802c170 tx=0x7fdea802df90 comp rx=0 tx=0).stop 2024-03-28T18:10:02.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.616+0000 7fdebca00640 1 -- 172.21.15.37:0/1799067602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdeb8108eb0 msgr2=0x7fdeb819e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:02.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.616+0000 7fdebca00640 1 --2- 172.21.15.37:0/1799067602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdeb8108eb0 0x7fdeb819e4e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdea400e9f0 tx=0x7fdea400eeb0 comp rx=0 tx=0).stop 2024-03-28T18:10:02.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.616+0000 7fdebca00640 1 -- 172.21.15.37:0/1799067602 shutdown_connections 2024-03-28T18:10:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.616+0000 7fdebca00640 1 --2- 172.21.15.37:0/1799067602 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fde9c0a2880 0x7fde9c0a4d40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.616+0000 7fdebca00640 1 --2- 172.21.15.37:0/1799067602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeb8109870 0x7fdeb81a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.616+0000 7fdebca00640 1 --2- 172.21.15.37:0/1799067602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdeb8108eb0 0x7fdeb819e4e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.616+0000 7fdebca00640 1 --2- 172.21.15.37:0/1799067602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdeb8107cc0 0x7fdeb819dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.616+0000 7fdebca00640 1 -- 172.21.15.37:0/1799067602 >> 172.21.15.37:0/1799067602 conn(0x7fdeb8076c60 msgr2=0x7fdeb810c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.616+0000 7fdebca00640 1 -- 172.21.15.37:0/1799067602 shutdown_connections 2024-03-28T18:10:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.616+0000 7fdebca00640 1 -- 172.21.15.37:0/1799067602 wait complete. 2024-03-28T18:10:02.637 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:10:02.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.846+0000 7f5633400640 1 -- 172.21.15.37:0/3741591557 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f562c1038e0 msgr2=0x7f562c103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:02.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.846+0000 7f5633400640 1 --2- 172.21.15.37:0/3741591557 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f562c1038e0 0x7f562c103d00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5620009900 tx=0x7f562002bc40 comp rx=0 tx=0).stop 2024-03-28T18:10:02.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.846+0000 7f5633400640 1 -- 172.21.15.37:0/3741591557 shutdown_connections 2024-03-28T18:10:02.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.847+0000 7f5633400640 1 --2- 172.21.15.37:0/3741591557 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f562c105490 0x7f562c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:02.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.847+0000 7f5633400640 1 --2- 172.21.15.37:0/3741591557 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f562c104ad0 0x7f562c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:02.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.847+0000 7f5633400640 1 --2- 172.21.15.37:0/3741591557 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f562c1038e0 0x7f562c103d00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:02.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.847+0000 7f5633400640 1 -- 172.21.15.37:0/3741591557 >> 172.21.15.37:0/3741591557 conn(0x7f562c0fee50 msgr2=0x7f562c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:02.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.847+0000 7f5633400640 1 -- 172.21.15.37:0/3741591557 shutdown_connections 2024-03-28T18:10:02.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.847+0000 7f5633400640 1 -- 172.21.15.37:0/3741591557 wait complete. 2024-03-28T18:10:02.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.848+0000 7f5633400640 1 Processor -- start 2024-03-28T18:10:02.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.848+0000 7f5633400640 1 -- start start 2024-03-28T18:10:02.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.848+0000 7f5633400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f562c1038e0 0x7f562c199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:02.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.848+0000 7f5633400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f562c104ad0 0x7f562c19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:02.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.848+0000 7f5633400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f562c105490 0x7f562c1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:02.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.848+0000 7f5633400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c10f5b0 con 0x7f562c105490 2024-03-28T18:10:02.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.848+0000 7f5633400640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c10f430 con 0x7f562c104ad0 2024-03-28T18:10:02.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.848+0000 7f5633400640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c10f730 con 0x7f562c1038e0 2024-03-28T18:10:02.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.848+0000 7f5630a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f562c1038e0 0x7f562c199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:02.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.848+0000 7f562be00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f562c104ad0 0x7f562c19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:02.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.848+0000 7f5631400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f562c105490 0x7f562c1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:02.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.848+0000 7f5630a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f562c1038e0 0x7f562c199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:51228/0 (socket says 172.21.15.37:51228) 2024-03-28T18:10:02.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.849+0000 7f5630a00640 1 -- 172.21.15.37:0/1893945647 learned_addr learned my addr 172.21.15.37:0/1893945647 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:02.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.849+0000 7f562be00640 1 -- 172.21.15.37:0/1893945647 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f562c1038e0 msgr2=0x7f562c199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:02.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.849+0000 7f562be00640 1 --2- 172.21.15.37:0/1893945647 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f562c1038e0 0x7f562c199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:02.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.849+0000 7f562be00640 1 -- 172.21.15.37:0/1893945647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f562c105490 msgr2=0x7f562c1a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:02.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.849+0000 7f562be00640 1 --2- 172.21.15.37:0/1893945647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f562c105490 0x7f562c1a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:02.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.849+0000 7f562be00640 1 -- 172.21.15.37:0/1893945647 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562c1a1810 con 0x7f562c104ad0 2024-03-28T18:10:02.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.849+0000 7f5630a00640 1 --2- 172.21.15.37:0/1893945647 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f562c1038e0 0x7f562c199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:02.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.849+0000 7f562be00640 1 --2- 172.21.15.37:0/1893945647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f562c104ad0 0x7f562c19a130 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5618005fd0 tx=0x7f561800dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:02.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.850+0000 7f5629600640 1 -- 172.21.15.37:0/1893945647 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5618009c30 con 0x7f562c104ad0 2024-03-28T18:10:02.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.850+0000 7f5629600640 1 -- 172.21.15.37:0/1893945647 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5618009dd0 con 0x7f562c104ad0 2024-03-28T18:10:02.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.850+0000 7f5629600640 1 -- 172.21.15.37:0/1893945647 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5618010460 con 0x7f562c104ad0 2024-03-28T18:10:02.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.850+0000 7f5633400640 1 -- 172.21.15.37:0/1893945647 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f562c1a1b00 con 0x7f562c104ad0 2024-03-28T18:10:02.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.850+0000 7f5633400640 1 -- 172.21.15.37:0/1893945647 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f562c1a20a0 con 0x7f562c104ad0 2024-03-28T18:10:02.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.851+0000 7f5633400640 1 -- 172.21.15.37:0/1893945647 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f562c10aba0 con 0x7f562c104ad0 2024-03-28T18:10:02.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.853+0000 7f5629600640 1 -- 172.21.15.37:0/1893945647 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5618003ef0 con 0x7f562c104ad0 2024-03-28T18:10:02.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.856+0000 7f5629600640 1 --2- 172.21.15.37:0/1893945647 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56140a2880 0x7f56140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:02.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.856+0000 7f5629600640 1 -- 172.21.15.37:0/1893945647 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f56180ce560 con 0x7f562c104ad0 2024-03-28T18:10:02.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.856+0000 7f5630a00640 1 --2- 172.21.15.37:0/1893945647 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56140a2880 0x7f56140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:02.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.857+0000 7f5630a00640 1 --2- 172.21.15.37:0/1893945647 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56140a2880 0x7f56140a4d40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5620005ab0 tx=0x7f562002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:02.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:02.859+0000 7f5629600640 1 -- 172.21.15.37:0/1893945647 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5618017020 con 0x7f562c104ad0 2024-03-28T18:10:03.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.064+0000 7f5633400640 1 -- 172.21.15.37:0/1893945647 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f562c0622d0 con 0x7f562c104ad0 2024-03-28T18:10:03.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.065+0000 7f5629600640 1 -- 172.21.15.37:0/1893945647 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f561809af60 con 0x7f562c104ad0 2024-03-28T18:10:03.065 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 64 2024-03-28T18:10:03.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.068+0000 7f5633400640 1 -- 172.21.15.37:0/1893945647 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56140a2880 msgr2=0x7f56140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:03.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.068+0000 7f5633400640 1 --2- 172.21.15.37:0/1893945647 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56140a2880 0x7f56140a4d40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5620005ab0 tx=0x7f562002cf90 comp rx=0 tx=0).stop 2024-03-28T18:10:03.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.068+0000 7f5633400640 1 -- 172.21.15.37:0/1893945647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f562c104ad0 msgr2=0x7f562c19a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:03.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.068+0000 7f5633400640 1 --2- 172.21.15.37:0/1893945647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f562c104ad0 0x7f562c19a130 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5618005fd0 tx=0x7f561800dde0 comp rx=0 tx=0).stop 2024-03-28T18:10:03.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.068+0000 7f5633400640 1 -- 172.21.15.37:0/1893945647 shutdown_connections 2024-03-28T18:10:03.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.068+0000 7f5633400640 1 --2- 172.21.15.37:0/1893945647 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56140a2880 0x7f56140a4d40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:03.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.068+0000 7f5633400640 1 --2- 172.21.15.37:0/1893945647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f562c104ad0 0x7f562c19a130 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:03.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.068+0000 7f5633400640 1 --2- 172.21.15.37:0/1893945647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f562c105490 0x7f562c1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:03.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.068+0000 7f5633400640 1 --2- 172.21.15.37:0/1893945647 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f562c1038e0 0x7f562c199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:03.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.068+0000 7f5633400640 1 -- 172.21.15.37:0/1893945647 >> 172.21.15.37:0/1893945647 conn(0x7f562c0fee50 msgr2=0x7f562c107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:03.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.068+0000 7f5633400640 1 -- 172.21.15.37:0/1893945647 shutdown_connections 2024-03-28T18:10:03.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.069+0000 7f5633400640 1 -- 172.21.15.37:0/1893945647 wait complete. 2024-03-28T18:10:03.088 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:10:03.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.312+0000 7f22f4a00640 1 -- 172.21.15.37:0/3939948607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22f01014e0 msgr2=0x7f22f0101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:03.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.312+0000 7f22f4a00640 1 --2- 172.21.15.37:0/3939948607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22f01014e0 0x7f22f0101900 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f22e0009900 tx=0x7f22e002bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:03.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.312+0000 7f22f4a00640 1 -- 172.21.15.37:0/3939948607 shutdown_connections 2024-03-28T18:10:03.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.312+0000 7f22f4a00640 1 --2- 172.21.15.37:0/3939948607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22f0103090 0x7f22f0109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:03.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.312+0000 7f22f4a00640 1 --2- 172.21.15.37:0/3939948607 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22f01026d0 0x7f22f0102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:03.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.312+0000 7f22f4a00640 1 --2- 172.21.15.37:0/3939948607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22f01014e0 0x7f22f0101900 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:03.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.312+0000 7f22f4a00640 1 -- 172.21.15.37:0/3939948607 >> 172.21.15.37:0/3939948607 conn(0x7f22f00fca30 msgr2=0x7f22f00fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:03.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.312+0000 7f22f4a00640 1 -- 172.21.15.37:0/3939948607 shutdown_connections 2024-03-28T18:10:03.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.312+0000 7f22f4a00640 1 -- 172.21.15.37:0/3939948607 wait complete. 2024-03-28T18:10:03.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.314+0000 7f22f4a00640 1 Processor -- start 2024-03-28T18:10:03.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.314+0000 7f22f4a00640 1 -- start start 2024-03-28T18:10:03.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.315+0000 7f22f4a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22f01014e0 0x7f22f0199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:03.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.315+0000 7f22f4a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22f01026d0 0x7f22f0199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:03.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.315+0000 7f22f4a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22f0103090 0x7f22f01a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:03.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.315+0000 7f22f4a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f010d1c0 con 0x7f22f0103090 2024-03-28T18:10:03.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.315+0000 7f22f4a00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f010d040 con 0x7f22f01014e0 2024-03-28T18:10:03.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.315+0000 7f22f4a00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f010d340 con 0x7f22f01026d0 2024-03-28T18:10:03.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.315+0000 7f22ef400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22f01014e0 0x7f22f0199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:03.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.315+0000 7f22ef400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22f01014e0 0x7f22f0199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:38892/0 (socket says 172.21.15.37:38892) 2024-03-28T18:10:03.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.315+0000 7f22ef400640 1 -- 172.21.15.37:0/3295146564 learned_addr learned my addr 172.21.15.37:0/3295146564 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:03.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.315+0000 7f22e6a00640 1 --2- 172.21.15.37:0/3295146564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22f01026d0 0x7f22f0199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:03.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.315+0000 7f22efe00640 1 --2- 172.21.15.37:0/3295146564 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22f0103090 0x7f22f01a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:03.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.315+0000 7f22ef400640 1 -- 172.21.15.37:0/3295146564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22f01026d0 msgr2=0x7f22f0199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:03.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.315+0000 7f22ef400640 1 --2- 172.21.15.37:0/3295146564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22f01026d0 0x7f22f0199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:03.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.315+0000 7f22ef400640 1 -- 172.21.15.37:0/3295146564 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22f0103090 msgr2=0x7f22f01a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:03.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.315+0000 7f22ef400640 1 --2- 172.21.15.37:0/3295146564 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22f0103090 0x7f22f01a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:03.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.315+0000 7f22ef400640 1 -- 172.21.15.37:0/3295146564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f01a1660 con 0x7f22f01014e0 2024-03-28T18:10:03.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.315+0000 7f22efe00640 1 --2- 172.21.15.37:0/3295146564 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22f0103090 0x7f22f01a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:03.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.316+0000 7f22ef400640 1 --2- 172.21.15.37:0/3295146564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22f01014e0 0x7f22f0199a40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f22e0000c00 tx=0x7f22e002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:03.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.316+0000 7f22ecc00640 1 -- 172.21.15.37:0/3295146564 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e0043020 con 0x7f22f01014e0 2024-03-28T18:10:03.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.316+0000 7f22ecc00640 1 -- 172.21.15.37:0/3295146564 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22e00054e0 con 0x7f22f01014e0 2024-03-28T18:10:03.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.316+0000 7f22ecc00640 1 -- 172.21.15.37:0/3295146564 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e0003c40 con 0x7f22f01014e0 2024-03-28T18:10:03.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.316+0000 7f22f4a00640 1 -- 172.21.15.37:0/3295146564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f01a18f0 con 0x7f22f01014e0 2024-03-28T18:10:03.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.316+0000 7f22f4a00640 1 -- 172.21.15.37:0/3295146564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f01a1e90 con 0x7f22f01014e0 2024-03-28T18:10:03.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.317+0000 7f22f4a00640 1 -- 172.21.15.37:0/3295146564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f01a2060 con 0x7f22f01014e0 2024-03-28T18:10:03.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.319+0000 7f22ecc00640 1 -- 172.21.15.37:0/3295146564 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f22e0046020 con 0x7f22f01014e0 2024-03-28T18:10:03.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.320+0000 7f22ecc00640 1 --2- 172.21.15.37:0/3295146564 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f22d00a2880 0x7f22d00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:03.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.320+0000 7f22ecc00640 1 -- 172.21.15.37:0/3295146564 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f22e00f0ac0 con 0x7f22f01014e0 2024-03-28T18:10:03.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.320+0000 7f22e6a00640 1 --2- 172.21.15.37:0/3295146564 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f22d00a2880 0x7f22d00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:03.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.321+0000 7f22e6a00640 1 --2- 172.21.15.37:0/3295146564 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f22d00a2880 0x7f22d00a4d40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f22f019af50 tx=0x7f22d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:03.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.324+0000 7f22ecc00640 1 -- 172.21.15.37:0/3295146564 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f22e004d020 con 0x7f22f01014e0 2024-03-28T18:10:03.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:03 smithi037 ceph-mon[31460]: pgmap v371: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:10:03.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:03 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1641118550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:03.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:03 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1799067602' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:03.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:03 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1893945647' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:03.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:03 smithi059 ceph-mon[36190]: pgmap v371: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:10:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:03 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1641118550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:03 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1799067602' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:03 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1893945647' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:03.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:03 smithi170 ceph-mon[36280]: pgmap v371: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:10:03.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:03 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1641118550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:03.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:03 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1799067602' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:03.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:03 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1893945647' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.573+0000 7f22f4a00640 1 -- 172.21.15.37:0/3295146564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22f01a2230 con 0x7f22f01014e0 2024-03-28T18:10:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.574+0000 7f22ecc00640 1 -- 172.21.15.37:0/3295146564 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11383 (secure 0 0 0) 0x7f22e00bdb10 con 0x7f22f01014e0 2024-03-28T18:10:03.574 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:03.574 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1926270371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6838","nonce":1926270371},{"type":"v1","addr":"172.21.15.37:6839","nonce":1926270371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:08:19.968894+0000","modified":"2024-03-28T18:08:25.415653+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24580,"mds_1":24569,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6837/4034607523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":4034607523},{"type":"v1","addr":"172.21.15.59:6837","nonce":4034607523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6839/1575518279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":1575518279},{"type":"v1","addr":"172.21.15.170:6839","nonce":1575518279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6839/2091629078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":2091629078},{"type":"v1","addr":"172.21.15.59:6839","nonce":2091629078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24580":{"gid":24580,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.37:6841/1923263622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6840","nonce":1923263622},{"type":"v1","addr":"172.21.15.37:6841","nonce":1923263622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24580,"qdb_cluster":[24550,24569,24580]},"id":1}]} 2024-03-28T18:10:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 22 2024-03-28T18:10:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.577+0000 7f22f4a00640 1 -- 172.21.15.37:0/3295146564 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f22d00a2880 msgr2=0x7f22d00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.577+0000 7f22f4a00640 1 --2- 172.21.15.37:0/3295146564 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f22d00a2880 0x7f22d00a4d40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f22f019af50 tx=0x7f22d8005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.577+0000 7f22f4a00640 1 -- 172.21.15.37:0/3295146564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22f01014e0 msgr2=0x7f22f0199a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.577+0000 7f22f4a00640 1 --2- 172.21.15.37:0/3295146564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22f01014e0 0x7f22f0199a40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f22e0000c00 tx=0x7f22e002c650 comp rx=0 tx=0).stop 2024-03-28T18:10:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.577+0000 7f22f4a00640 1 -- 172.21.15.37:0/3295146564 shutdown_connections 2024-03-28T18:10:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.577+0000 7f22f4a00640 1 --2- 172.21.15.37:0/3295146564 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f22d00a2880 0x7f22d00a4d40 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.577+0000 7f22f4a00640 1 --2- 172.21.15.37:0/3295146564 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22f0103090 0x7f22f01a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.577+0000 7f22f4a00640 1 --2- 172.21.15.37:0/3295146564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22f01026d0 0x7f22f0199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.577+0000 7f22f4a00640 1 --2- 172.21.15.37:0/3295146564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22f01014e0 0x7f22f0199a40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.577+0000 7f22f4a00640 1 -- 172.21.15.37:0/3295146564 >> 172.21.15.37:0/3295146564 conn(0x7f22f00fca30 msgr2=0x7f22f00738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:03.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.577+0000 7f22f4a00640 1 -- 172.21.15.37:0/3295146564 shutdown_connections 2024-03-28T18:10:03.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.577+0000 7f22f4a00640 1 -- 172.21.15.37:0/3295146564 wait complete. 2024-03-28T18:10:03.598 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:10:03.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.828+0000 7f23ab000640 1 -- 172.21.15.37:0/2441118653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a41038e0 msgr2=0x7f23a4103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:03.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.828+0000 7f23ab000640 1 --2- 172.21.15.37:0/2441118653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a41038e0 0x7f23a4103d00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2394009900 tx=0x7f239402bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:03.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.828+0000 7f23ab000640 1 -- 172.21.15.37:0/2441118653 shutdown_connections 2024-03-28T18:10:03.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.828+0000 7f23ab000640 1 --2- 172.21.15.37:0/2441118653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23a4105490 0x7f23a410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:03.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.828+0000 7f23ab000640 1 --2- 172.21.15.37:0/2441118653 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a4104ad0 0x7f23a4104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:03.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.828+0000 7f23ab000640 1 --2- 172.21.15.37:0/2441118653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a41038e0 0x7f23a4103d00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:03.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.828+0000 7f23ab000640 1 -- 172.21.15.37:0/2441118653 >> 172.21.15.37:0/2441118653 conn(0x7f23a40fee90 msgr2=0x7f23a41012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:03.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.828+0000 7f23ab000640 1 -- 172.21.15.37:0/2441118653 shutdown_connections 2024-03-28T18:10:03.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.828+0000 7f23ab000640 1 -- 172.21.15.37:0/2441118653 wait complete. 2024-03-28T18:10:03.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.829+0000 7f23ab000640 1 Processor -- start 2024-03-28T18:10:03.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.830+0000 7f23ab000640 1 -- start start 2024-03-28T18:10:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.830+0000 7f23ab000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a41038e0 0x7f23a4199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.830+0000 7f23ab000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a4104ad0 0x7f23a419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.830+0000 7f23ab000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23a4105490 0x7f23a41a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.830+0000 7f23ab000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a410f580 con 0x7f23a4104ad0 2024-03-28T18:10:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.830+0000 7f23ab000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a410f400 con 0x7f23a41038e0 2024-03-28T18:10:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.830+0000 7f23ab000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a410f700 con 0x7f23a4105490 2024-03-28T18:10:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.830+0000 7f23a3e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a41038e0 0x7f23a4199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.830+0000 7f23a3e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a41038e0 0x7f23a4199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:55060/0 (socket says 172.21.15.37:55060) 2024-03-28T18:10:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.830+0000 7f23a3e00640 1 -- 172.21.15.37:0/2253901235 learned_addr learned my addr 172.21.15.37:0/2253901235 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.830+0000 7f239b400640 1 --2- 172.21.15.37:0/2253901235 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a4104ad0 0x7f23a419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.831+0000 7f23a9000640 1 --2- 172.21.15.37:0/2253901235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23a4105490 0x7f23a41a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.831+0000 7f23a3e00640 1 -- 172.21.15.37:0/2253901235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23a4105490 msgr2=0x7f23a41a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.831+0000 7f23a3e00640 1 --2- 172.21.15.37:0/2253901235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23a4105490 0x7f23a41a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.831+0000 7f23a3e00640 1 -- 172.21.15.37:0/2253901235 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a4104ad0 msgr2=0x7f23a419a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.831+0000 7f23a3e00640 1 --2- 172.21.15.37:0/2253901235 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a4104ad0 0x7f23a419a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.831+0000 7f23a3e00640 1 -- 172.21.15.37:0/2253901235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a41a17f0 con 0x7f23a41038e0 2024-03-28T18:10:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.831+0000 7f23a3e00640 1 --2- 172.21.15.37:0/2253901235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a41038e0 0x7f23a4199bd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2394000c00 tx=0x7f239402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:03.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.831+0000 7f23a1600640 1 -- 172.21.15.37:0/2253901235 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394043020 con 0x7f23a41038e0 2024-03-28T18:10:03.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.832+0000 7f23a1600640 1 -- 172.21.15.37:0/2253901235 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23940054e0 con 0x7f23a41038e0 2024-03-28T18:10:03.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.832+0000 7f23ab000640 1 -- 172.21.15.37:0/2253901235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a41a1a80 con 0x7f23a41038e0 2024-03-28T18:10:03.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.832+0000 7f23a1600640 1 -- 172.21.15.37:0/2253901235 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394003c40 con 0x7f23a41038e0 2024-03-28T18:10:03.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.832+0000 7f23ab000640 1 -- 172.21.15.37:0/2253901235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a41a1fd0 con 0x7f23a41038e0 2024-03-28T18:10:03.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.833+0000 7f23ab000640 1 -- 172.21.15.37:0/2253901235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a410aba0 con 0x7f23a41038e0 2024-03-28T18:10:03.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.835+0000 7f23a1600640 1 -- 172.21.15.37:0/2253901235 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2394046020 con 0x7f23a41038e0 2024-03-28T18:10:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.836+0000 7f23a1600640 1 --2- 172.21.15.37:0/2253901235 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f23880a2800 0x7f23880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.836+0000 7f23a1600640 1 -- 172.21.15.37:0/2253901235 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f23940f0a70 con 0x7f23a41038e0 2024-03-28T18:10:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.836+0000 7f239b400640 1 --2- 172.21.15.37:0/2253901235 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f23880a2800 0x7f23880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.837+0000 7f239b400640 1 --2- 172.21.15.37:0/2253901235 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f23880a2800 0x7f23880a4cc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f23a419b0e0 tx=0x7f238c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:03.845+0000 7f23a1600640 1 -- 172.21.15.37:0/2253901235 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f239404d020 con 0x7f23a41038e0 2024-03-28T18:10:04.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.065+0000 7f23ab000640 1 -- 172.21.15.37:0/2253901235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23a40622a0 con 0x7f23a41038e0 2024-03-28T18:10:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.066+0000 7f23a1600640 1 -- 172.21.15.37:0/2253901235 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22575 (secure 0 0 0) 0x7f23940bdac0 con 0x7f23a41038e0 2024-03-28T18:10:04.067 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:04.067 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":87,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","created":"2024-03-28T17:59:05.949847+0000","modified":"2024-03-28T18:09:15.709901+0000","last_up_change":"2024-03-28T18:07:21.021030+0000","last_in_change":"2024-03-28T18:07:01.416192+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:03:00.517967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:08:18.365827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:08:19.446282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db60618d-56f7-4130-8a90-84cbc480965b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6803","nonce":3562602035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6805","nonce":3562602035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6809","nonce":3562602035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6807","nonce":3562602035}]},"public_addr":"172.21.15.37:6803/3562602035","cluster_addr":"172.21.15.37:6805/3562602035","heartbeat_back_addr":"172.21.15.37:6809/3562602035","heartbeat_front_addr":"172.21.15.37:6807/3562602035","state":["exists","up"]},{"osd":1,"uuid":"7a4b7ad6-905e-4258-9574-693d4277dfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6801","nonce":3326537211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6803","nonce":3326537211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6807","nonce":3326537211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6805","nonce":3326537211}]},"public_addr":"172.21.15.59:6801/3326537211","cluster_addr":"172.21.15.59:6803/3326537211","heartbeat_back_addr":"172.21.15.59:6807/3326537211","heartbeat_front_addr":"172.21.15.59:6805/3326537211","state":["exists","up"]},{"osd":2,"uuid":"8b76ec1c-46bf-4336-89b5-12d82906e8b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6801","nonce":1269398427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6803","nonce":1269398427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6807","nonce":1269398427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6805","nonce":1269398427}]},"public_addr":"172.21.15.170:6801/1269398427","cluster_addr":"172.21.15.170:6803/1269398427","heartbeat_back_addr":"172.21.15.170:6807/1269398427","heartbeat_front_addr":"172.21.15.170:6805/1269398427","state":["exists","up"]},{"osd":3,"uuid":"66eca1da-2fbd-44e1-8bbf-5635a2c70ad5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6811","nonce":317805240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6813","nonce":317805240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6817","nonce":317805240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6815","nonce":317805240}]},"public_addr":"172.21.15.37:6811/317805240","cluster_addr":"172.21.15.37:6813/317805240","heartbeat_back_addr":"172.21.15.37:6817/317805240","heartbeat_front_addr":"172.21.15.37:6815/317805240","state":["exists","up"]},{"osd":4,"uuid":"4c877335-e42d-4d8e-9f48-a1d0a89ae438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6809","nonce":2410632623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6811","nonce":2410632623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6815","nonce":2410632623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6813","nonce":2410632623}]},"public_addr":"172.21.15.59:6809/2410632623","cluster_addr":"172.21.15.59:6811/2410632623","heartbeat_back_addr":"172.21.15.59:6815/2410632623","heartbeat_front_addr":"172.21.15.59:6813/2410632623","state":["exists","up"]},{"osd":5,"uuid":"b884842a-86e2-455b-9570-ff163b3bc0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6809","nonce":1218943899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6811","nonce":1218943899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6815","nonce":1218943899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6813","nonce":1218943899}]},"public_addr":"172.21.15.170:6809/1218943899","cluster_addr":"172.21.15.170:6811/1218943899","heartbeat_back_addr":"172.21.15.170:6815/1218943899","heartbeat_front_addr":"172.21.15.170:6813/1218943899","state":["exists","up"]},{"osd":6,"uuid":"249a1200-77f4-44ad-981f-1d352f732b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6819","nonce":1899652046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6821","nonce":1899652046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6825","nonce":1899652046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6823","nonce":1899652046}]},"public_addr":"172.21.15.37:6819/1899652046","cluster_addr":"172.21.15.37:6821/1899652046","heartbeat_back_addr":"172.21.15.37:6825/1899652046","heartbeat_front_addr":"172.21.15.37:6823/1899652046","state":["exists","up"]},{"osd":7,"uuid":"c73338e6-f505-4517-a84b-f35e04750a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6817","nonce":1343675690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6819","nonce":1343675690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6823","nonce":1343675690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6821","nonce":1343675690}]},"public_addr":"172.21.15.59:6817/1343675690","cluster_addr":"172.21.15.59:6819/1343675690","heartbeat_back_addr":"172.21.15.59:6823/1343675690","heartbeat_front_addr":"172.21.15.59:6821/1343675690","state":["exists","up"]},{"osd":8,"uuid":"149b8297-ff2d-4a88-9b1e-0def021cc516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6817","nonce":3235794556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6819","nonce":3235794556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6823","nonce":3235794556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6821","nonce":3235794556}]},"public_addr":"172.21.15.170:6817/3235794556","cluster_addr":"172.21.15.170:6819/3235794556","heartbeat_back_addr":"172.21.15.170:6823/3235794556","heartbeat_front_addr":"172.21.15.170:6821/3235794556","state":["exists","up"]},{"osd":9,"uuid":"55c27cc2-c771-44b7-90fe-8f0feca44162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6827","nonce":2307153008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6829","nonce":2307153008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6833","nonce":2307153008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6831","nonce":2307153008}]},"public_addr":"172.21.15.37:6827/2307153008","cluster_addr":"172.21.15.37:6829/2307153008","heartbeat_back_addr":"172.21.15.37:6833/2307153008","heartbeat_front_addr":"172.21.15.37:6831/2307153008","state":["exists","up"]},{"osd":10,"uuid":"c0595135-ab81-49d3-8c23-19f960c03e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6825","nonce":2113830275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6827","nonce":2113830275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6831","nonce":2113830275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6829","nonce":2113830275}]},"public_addr":"172.21.15.59:6825/2113830275","cluster_addr":"172.21.15.59:6827/2113830275","heartbeat_back_addr":"172.21.15.59:6831/2113830275","heartbeat_front_addr":"172.21.15.59:6829/2113830275","state":["exists","up"]},{"osd":11,"uuid":"5871e177-c17b-4637-a797-4fa5125920dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6825","nonce":3082439360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6827","nonce":3082439360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6831","nonce":3082439360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6829","nonce":3082439360}]},"public_addr":"172.21.15.170:6825/3082439360","cluster_addr":"172.21.15.170:6827/3082439360","heartbeat_back_addr":"172.21.15.170:6831/3082439360","heartbeat_front_addr":"172.21.15.170:6829/3082439360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:07.690545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:33.183396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:57.400263+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:30.597211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:57.259044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:22.627823+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:57.585863+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:25.000476+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:50.876688+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:25.859761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:53.563146+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:07:19.304115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3023379428":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1443710231":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6800/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6801/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6801/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6800/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1007625490":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/2895698584":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/497313034":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/3293943475":"2024-03-29T17:59:39.162038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:10:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.069+0000 7f23ab000640 1 -- 172.21.15.37:0/2253901235 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f23880a2800 msgr2=0x7f23880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.069+0000 7f23ab000640 1 --2- 172.21.15.37:0/2253901235 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f23880a2800 0x7f23880a4cc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f23a419b0e0 tx=0x7f238c005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.069+0000 7f23ab000640 1 -- 172.21.15.37:0/2253901235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a41038e0 msgr2=0x7f23a4199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.069+0000 7f23ab000640 1 --2- 172.21.15.37:0/2253901235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a41038e0 0x7f23a4199bd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2394000c00 tx=0x7f239402c650 comp rx=0 tx=0).stop 2024-03-28T18:10:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.069+0000 7f23ab000640 1 -- 172.21.15.37:0/2253901235 shutdown_connections 2024-03-28T18:10:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.069+0000 7f23ab000640 1 --2- 172.21.15.37:0/2253901235 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f23880a2800 0x7f23880a4cc0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.069+0000 7f23ab000640 1 --2- 172.21.15.37:0/2253901235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23a4105490 0x7f23a41a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.069+0000 7f23ab000640 1 --2- 172.21.15.37:0/2253901235 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a4104ad0 0x7f23a419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.069+0000 7f23ab000640 1 --2- 172.21.15.37:0/2253901235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a41038e0 0x7f23a4199bd0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.069+0000 7f23ab000640 1 -- 172.21.15.37:0/2253901235 >> 172.21.15.37:0/2253901235 conn(0x7f23a40fee90 msgr2=0x7f23a4107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.069+0000 7f23ab000640 1 -- 172.21.15.37:0/2253901235 shutdown_connections 2024-03-28T18:10:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.069+0000 7f23ab000640 1 -- 172.21.15.37:0/2253901235 wait complete. 2024-03-28T18:10:04.089 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:03:00.517967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:08:18.365827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:08:19.446282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:10:04.090 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:10:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.317+0000 7f7963200640 1 -- 172.21.15.37:0/3245059382 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7964102000 msgr2=0x7f79641043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.317+0000 7f7963200640 1 --2- 172.21.15.37:0/3245059382 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7964102000 0x7f79641043f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7954009900 tx=0x7f795402bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.318+0000 7f7963200640 1 -- 172.21.15.37:0/3245059382 shutdown_connections 2024-03-28T18:10:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.318+0000 7f7963200640 1 --2- 172.21.15.37:0/3245059382 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7964107260 0x7f79641096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.318+0000 7f7963200640 1 --2- 172.21.15.37:0/3245059382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7964104930 0x7f7964106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.318+0000 7f7963200640 1 --2- 172.21.15.37:0/3245059382 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7964102000 0x7f79641043f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.318+0000 7f7963200640 1 -- 172.21.15.37:0/3245059382 >> 172.21.15.37:0/3245059382 conn(0x7f79640fba10 msgr2=0x7f79640fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.318+0000 7f7963200640 1 -- 172.21.15.37:0/3245059382 shutdown_connections 2024-03-28T18:10:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.318+0000 7f7963200640 1 -- 172.21.15.37:0/3245059382 wait complete. 2024-03-28T18:10:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.319+0000 7f7963200640 1 Processor -- start 2024-03-28T18:10:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.319+0000 7f7963200640 1 -- start start 2024-03-28T18:10:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.319+0000 7f7963200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7964104930 0x7f7964199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.319+0000 7f7963200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7964107260 0x7f796419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.319+0000 7f7963200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f796419f270 0x7f79641a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.319+0000 7f7963200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796410d260 con 0x7f7964104930 2024-03-28T18:10:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.319+0000 7f7963200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796410d0e0 con 0x7f796419f270 2024-03-28T18:10:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.319+0000 7f7963200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796410d3e0 con 0x7f7964107260 2024-03-28T18:10:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.320+0000 7f7962800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f796419f270 0x7f79641a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.320+0000 7f7961400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7964107260 0x7f796419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:04.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.320+0000 7f7961e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7964104930 0x7f7964199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:04.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.320+0000 7f7962800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f796419f270 0x7f79641a1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:55072/0 (socket says 172.21.15.37:55072) 2024-03-28T18:10:04.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.320+0000 7f7962800640 1 -- 172.21.15.37:0/787943468 learned_addr learned my addr 172.21.15.37:0/787943468 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:04.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.320+0000 7f7961400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7964107260 0x7f796419a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:43736/0 (socket says 172.21.15.37:43736) 2024-03-28T18:10:04.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.320+0000 7f7961e00640 1 -- 172.21.15.37:0/787943468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7964107260 msgr2=0x7f796419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:04.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.320+0000 7f7961e00640 1 --2- 172.21.15.37:0/787943468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7964107260 0x7f796419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:04.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.320+0000 7f7961e00640 1 -- 172.21.15.37:0/787943468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f796419f270 msgr2=0x7f79641a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:04.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.320+0000 7f7961e00640 1 --2- 172.21.15.37:0/787943468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f796419f270 0x7f79641a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:04.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.320+0000 7f7961e00640 1 -- 172.21.15.37:0/787943468 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79641a1d00 con 0x7f7964104930 2024-03-28T18:10:04.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.320+0000 7f7962800640 1 --2- 172.21.15.37:0/787943468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f796419f270 0x7f79641a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:04.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.320+0000 7f7961400640 1 --2- 172.21.15.37:0/787943468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7964107260 0x7f796419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:04.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.321+0000 7f7961e00640 1 --2- 172.21.15.37:0/787943468 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7964104930 0x7f7964199d10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7954008000 tx=0x7f7954005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:04.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.321+0000 7f795aa00640 1 -- 172.21.15.37:0/787943468 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7954043020 con 0x7f7964104930 2024-03-28T18:10:04.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.321+0000 7f7963200640 1 -- 172.21.15.37:0/787943468 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79641a1f90 con 0x7f7964104930 2024-03-28T18:10:04.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.321+0000 7f795aa00640 1 -- 172.21.15.37:0/787943468 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79540054e0 con 0x7f7964104930 2024-03-28T18:10:04.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.321+0000 7f795aa00640 1 -- 172.21.15.37:0/787943468 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7954003ca0 con 0x7f7964104930 2024-03-28T18:10:04.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.321+0000 7f7963200640 1 -- 172.21.15.37:0/787943468 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79641a23a0 con 0x7f7964104930 2024-03-28T18:10:04.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.322+0000 7f7963200640 1 -- 172.21.15.37:0/787943468 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7964108db0 con 0x7f7964104930 2024-03-28T18:10:04.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.324+0000 7f795aa00640 1 -- 172.21.15.37:0/787943468 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7954046020 con 0x7f7964104930 2024-03-28T18:10:04.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.326+0000 7f795aa00640 1 --2- 172.21.15.37:0/787943468 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f79480a2800 0x7f79480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:04.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.326+0000 7f795aa00640 1 -- 172.21.15.37:0/787943468 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f79540f08c0 con 0x7f7964104930 2024-03-28T18:10:04.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.326+0000 7f7961400640 1 --2- 172.21.15.37:0/787943468 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f79480a2800 0x7f79480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:04.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.327+0000 7f7961400640 1 --2- 172.21.15.37:0/787943468 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f79480a2800 0x7f79480a4cc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7964069360 tx=0x7f7950005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:04.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.334+0000 7f795aa00640 1 -- 172.21.15.37:0/787943468 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f795404d030 con 0x7f7964104930 2024-03-28T18:10:04.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:04 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3295146564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:04.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:04 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2253901235' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:04.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:04 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3295146564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:04.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:04 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2253901235' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:04.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:04 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3295146564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:04.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:04 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2253901235' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:04.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.539+0000 7f7963200640 1 -- 172.21.15.37:0/787943468 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f79640622a0 con 0x7f7964104930 2024-03-28T18:10:04.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.539+0000 7f795aa00640 1 -- 172.21.15.37:0/787943468 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f79540bd910 con 0x7f7964104930 2024-03-28T18:10:04.539 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 1 2024-03-28T18:10:04.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.542+0000 7f7963200640 1 -- 172.21.15.37:0/787943468 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f79480a2800 msgr2=0x7f79480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:04.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.542+0000 7f7963200640 1 --2- 172.21.15.37:0/787943468 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f79480a2800 0x7f79480a4cc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7964069360 tx=0x7f7950005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:04.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.542+0000 7f7963200640 1 -- 172.21.15.37:0/787943468 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7964104930 msgr2=0x7f7964199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:04.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.542+0000 7f7963200640 1 --2- 172.21.15.37:0/787943468 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7964104930 0x7f7964199d10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7954008000 tx=0x7f7954005d50 comp rx=0 tx=0).stop 2024-03-28T18:10:04.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.542+0000 7f7963200640 1 -- 172.21.15.37:0/787943468 shutdown_connections 2024-03-28T18:10:04.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.542+0000 7f7963200640 1 --2- 172.21.15.37:0/787943468 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f79480a2800 0x7f79480a4cc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:04.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.542+0000 7f7963200640 1 --2- 172.21.15.37:0/787943468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f796419f270 0x7f79641a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:04.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.542+0000 7f7963200640 1 --2- 172.21.15.37:0/787943468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7964107260 0x7f796419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:04.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.542+0000 7f7963200640 1 --2- 172.21.15.37:0/787943468 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7964104930 0x7f7964199d10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:04.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.542+0000 7f7963200640 1 -- 172.21.15.37:0/787943468 >> 172.21.15.37:0/787943468 conn(0x7f79640fba10 msgr2=0x7f79641017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:04.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.542+0000 7f7963200640 1 -- 172.21.15.37:0/787943468 shutdown_connections 2024-03-28T18:10:04.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.542+0000 7f7963200640 1 -- 172.21.15.37:0/787943468 wait complete. 2024-03-28T18:10:04.561 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:10:04.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.787+0000 7f738fe00640 1 -- 172.21.15.37:0/2646702695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7390104930 msgr2=0x7f7390106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:04.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.787+0000 7f738fe00640 1 --2- 172.21.15.37:0/2646702695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7390104930 0x7f7390106d20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f737c009900 tx=0x7f737c02bc50 comp rx=0 tx=0).stop 2024-03-28T18:10:04.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.787+0000 7f738fe00640 1 -- 172.21.15.37:0/2646702695 shutdown_connections 2024-03-28T18:10:04.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.787+0000 7f738fe00640 1 --2- 172.21.15.37:0/2646702695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7390104930 0x7f7390106d20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:04.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.787+0000 7f738fe00640 1 --2- 172.21.15.37:0/2646702695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7390107260 0x7f73901096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:04.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.787+0000 7f738fe00640 1 --2- 172.21.15.37:0/2646702695 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7390102000 0x7f73901043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:04.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.787+0000 7f738fe00640 1 -- 172.21.15.37:0/2646702695 >> 172.21.15.37:0/2646702695 conn(0x7f73900fba10 msgr2=0x7f73900fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:04.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.788+0000 7f738fe00640 1 -- 172.21.15.37:0/2646702695 shutdown_connections 2024-03-28T18:10:04.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.788+0000 7f738fe00640 1 -- 172.21.15.37:0/2646702695 wait complete. 2024-03-28T18:10:04.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.789+0000 7f738fe00640 1 Processor -- start 2024-03-28T18:10:04.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.789+0000 7f738fe00640 1 -- start start 2024-03-28T18:10:04.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.789+0000 7f738fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7390102000 0x7f73901978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:04.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.790+0000 7f738fe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7390104930 0x7f7390197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:04.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.790+0000 7f738fe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7390107260 0x7f739019ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:04.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.790+0000 7f738fe00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739010d220 con 0x7f7390102000 2024-03-28T18:10:04.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.790+0000 7f738fe00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739010d0a0 con 0x7f7390104930 2024-03-28T18:10:04.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.790+0000 7f738fe00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739010d3a0 con 0x7f7390107260 2024-03-28T18:10:04.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.790+0000 7f738e000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7390104930 0x7f7390197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:04.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.790+0000 7f738e000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7390104930 0x7f7390197e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:55098/0 (socket says 172.21.15.37:55098) 2024-03-28T18:10:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.790+0000 7f738e000640 1 -- 172.21.15.37:0/1213532664 learned_addr learned my addr 172.21.15.37:0/1213532664 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.790+0000 7f738ea00640 1 --2- 172.21.15.37:0/1213532664 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7390102000 0x7f73901978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.790+0000 7f738f400640 1 --2- 172.21.15.37:0/1213532664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7390107260 0x7f739019ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.790+0000 7f738e000640 1 -- 172.21.15.37:0/1213532664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7390107260 msgr2=0x7f739019ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.790+0000 7f738e000640 1 --2- 172.21.15.37:0/1213532664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7390107260 0x7f739019ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.790+0000 7f738e000640 1 -- 172.21.15.37:0/1213532664 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7390102000 msgr2=0x7f73901978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.790+0000 7f738e000640 1 --2- 172.21.15.37:0/1213532664 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7390102000 0x7f73901978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.790+0000 7f738e000640 1 -- 172.21.15.37:0/1213532664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739019f4f0 con 0x7f7390104930 2024-03-28T18:10:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.791+0000 7f738e000640 1 --2- 172.21.15.37:0/1213532664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7390104930 0x7f7390197e10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f737c02c260 tx=0x7f737c02c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.791+0000 7f7394a00640 1 -- 172.21.15.37:0/1213532664 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c041070 con 0x7f7390104930 2024-03-28T18:10:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.791+0000 7f7394a00640 1 -- 172.21.15.37:0/1213532664 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f737c004450 con 0x7f7390104930 2024-03-28T18:10:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.791+0000 7f738fe00640 1 -- 172.21.15.37:0/1213532664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739019f780 con 0x7f7390104930 2024-03-28T18:10:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.791+0000 7f7394a00640 1 -- 172.21.15.37:0/1213532664 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c034440 con 0x7f7390104930 2024-03-28T18:10:04.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.791+0000 7f738fe00640 1 -- 172.21.15.37:0/1213532664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739019fd20 con 0x7f7390104930 2024-03-28T18:10:04.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.792+0000 7f738fe00640 1 -- 172.21.15.37:0/1213532664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73900fd500 con 0x7f7390104930 2024-03-28T18:10:04.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.796+0000 7f7394a00640 1 -- 172.21.15.37:0/1213532664 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f737c044070 con 0x7f7390104930 2024-03-28T18:10:04.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.797+0000 7f7394a00640 1 --2- 172.21.15.37:0/1213532664 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f73740a2880 0x7f73740a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:04.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.797+0000 7f7394a00640 1 -- 172.21.15.37:0/1213532664 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f737c0f0470 con 0x7f7390104930 2024-03-28T18:10:04.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.798+0000 7f738ea00640 1 --2- 172.21.15.37:0/1213532664 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f73740a2880 0x7f73740a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:04.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.798+0000 7f738ea00640 1 --2- 172.21.15.37:0/1213532664 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f73740a2880 0x7f73740a4d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f73800095f0 tx=0x7f7380005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:04.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:04.801+0000 7f7394a00640 1 -- 172.21.15.37:0/1213532664 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f737c0bced0 con 0x7f7390104930 2024-03-28T18:10:05.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.015+0000 7f738fe00640 1 -- 172.21.15.37:0/1213532664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f73900622a0 con 0x7f7390104930 2024-03-28T18:10:05.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.016+0000 7f7394a00640 1 -- 172.21.15.37:0/1213532664 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f737c0c0f70 con 0x7f7390104930 2024-03-28T18:10:05.016 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 32 2024-03-28T18:10:05.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.019+0000 7f738fe00640 1 -- 172.21.15.37:0/1213532664 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f73740a2880 msgr2=0x7f73740a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:05.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.019+0000 7f738fe00640 1 --2- 172.21.15.37:0/1213532664 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f73740a2880 0x7f73740a4d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f73800095f0 tx=0x7f7380005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:05.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.019+0000 7f738fe00640 1 -- 172.21.15.37:0/1213532664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7390104930 msgr2=0x7f7390197e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:05.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.019+0000 7f738fe00640 1 --2- 172.21.15.37:0/1213532664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7390104930 0x7f7390197e10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f737c02c260 tx=0x7f737c02c670 comp rx=0 tx=0).stop 2024-03-28T18:10:05.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.019+0000 7f738fe00640 1 -- 172.21.15.37:0/1213532664 shutdown_connections 2024-03-28T18:10:05.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.019+0000 7f738fe00640 1 --2- 172.21.15.37:0/1213532664 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f73740a2880 0x7f73740a4d40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:05.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.019+0000 7f738fe00640 1 --2- 172.21.15.37:0/1213532664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7390104930 0x7f7390197e10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:05.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.019+0000 7f738fe00640 1 --2- 172.21.15.37:0/1213532664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7390107260 0x7f739019ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:05.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.019+0000 7f738fe00640 1 --2- 172.21.15.37:0/1213532664 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7390102000 0x7f73901978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:05.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.019+0000 7f738fe00640 1 -- 172.21.15.37:0/1213532664 >> 172.21.15.37:0/1213532664 conn(0x7f73900fba10 msgr2=0x7f73901028a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:05.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.019+0000 7f738fe00640 1 -- 172.21.15.37:0/1213532664 shutdown_connections 2024-03-28T18:10:05.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.019+0000 7f738fe00640 1 -- 172.21.15.37:0/1213532664 wait complete. 2024-03-28T18:10:05.042 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:10:05.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.265+0000 7f150d600640 1 -- 172.21.15.37:0/3691567996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1508109460 msgr2=0x7f150810b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:05.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.265+0000 7f150d600640 1 --2- 172.21.15.37:0/3691567996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1508109460 0x7f150810b8f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f14f8009900 tx=0x7f14f802bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:05.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.265+0000 7f150d600640 1 -- 172.21.15.37:0/3691567996 shutdown_connections 2024-03-28T18:10:05.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.265+0000 7f150d600640 1 --2- 172.21.15.37:0/3691567996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1508074100 0x7f1508074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:05.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.265+0000 7f150d600640 1 --2- 172.21.15.37:0/3691567996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1508109460 0x7f150810b8f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:05.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.265+0000 7f150d600640 1 --2- 172.21.15.37:0/3691567996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15080756e0 0x7f1508073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:05.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.265+0000 7f150d600640 1 -- 172.21.15.37:0/3691567996 >> 172.21.15.37:0/3691567996 conn(0x7f15080fcb10 msgr2=0x7f15080fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:05.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.265+0000 7f150d600640 1 -- 172.21.15.37:0/3691567996 shutdown_connections 2024-03-28T18:10:05.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.265+0000 7f150d600640 1 -- 172.21.15.37:0/3691567996 wait complete. 2024-03-28T18:10:05.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.266+0000 7f150d600640 1 Processor -- start 2024-03-28T18:10:05.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.267+0000 7f150d600640 1 -- start start 2024-03-28T18:10:05.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.267+0000 7f150d600640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1508074100 0x7f1508199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:05.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.267+0000 7f150d600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15080756e0 0x7f1508199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.267+0000 7f150d600640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1508109460 0x7f15081a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.267+0000 7f150d600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150810f430 con 0x7f15080756e0 2024-03-28T18:10:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.267+0000 7f150d600640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150810f2b0 con 0x7f1508109460 2024-03-28T18:10:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.267+0000 7f150d600640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150810f5b0 con 0x7f1508074100 2024-03-28T18:10:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.267+0000 7f1507e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1508074100 0x7f1508199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.267+0000 7f1507e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1508074100 0x7f1508199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:43764/0 (socket says 172.21.15.37:43764) 2024-03-28T18:10:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.267+0000 7f1507e00640 1 -- 172.21.15.37:0/2147552044 learned_addr learned my addr 172.21.15.37:0/2147552044 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.267+0000 7f150cc00640 1 --2- 172.21.15.37:0/2147552044 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1508109460 0x7f15081a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.267+0000 7f1507400640 1 --2- 172.21.15.37:0/2147552044 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15080756e0 0x7f1508199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.268+0000 7f1507e00640 1 -- 172.21.15.37:0/2147552044 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1508109460 msgr2=0x7f15081a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.268+0000 7f1507e00640 1 --2- 172.21.15.37:0/2147552044 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1508109460 0x7f15081a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.268+0000 7f1507e00640 1 -- 172.21.15.37:0/2147552044 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15080756e0 msgr2=0x7f1508199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.268+0000 7f1507e00640 1 --2- 172.21.15.37:0/2147552044 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15080756e0 0x7f1508199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.268+0000 7f1507e00640 1 -- 172.21.15.37:0/2147552044 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15081a1630 con 0x7f1508074100 2024-03-28T18:10:05.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.268+0000 7f1507e00640 1 --2- 172.21.15.37:0/2147552044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1508074100 0x7f1508199a10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f150000ebc0 tx=0x7f150000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:05.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.268+0000 7f1504c00640 1 -- 172.21.15.37:0/2147552044 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1500010630 con 0x7f1508074100 2024-03-28T18:10:05.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.268+0000 7f1504c00640 1 -- 172.21.15.37:0/2147552044 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1500010cb0 con 0x7f1508074100 2024-03-28T18:10:05.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.268+0000 7f1504c00640 1 -- 172.21.15.37:0/2147552044 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1500018bd0 con 0x7f1508074100 2024-03-28T18:10:05.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.268+0000 7f150d600640 1 -- 172.21.15.37:0/2147552044 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15081a1920 con 0x7f1508074100 2024-03-28T18:10:05.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.268+0000 7f150d600640 1 -- 172.21.15.37:0/2147552044 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15081a1e70 con 0x7f1508074100 2024-03-28T18:10:05.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.269+0000 7f150d600640 1 -- 172.21.15.37:0/2147552044 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15080683b0 con 0x7f1508074100 2024-03-28T18:10:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.271+0000 7f1504c00640 1 -- 172.21.15.37:0/2147552044 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f15000107d0 con 0x7f1508074100 2024-03-28T18:10:05.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.272+0000 7f1504c00640 1 --2- 172.21.15.37:0/2147552044 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f14ec0a2800 0x7f14ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:05.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.273+0000 7f1504c00640 1 -- 172.21.15.37:0/2147552044 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f15000cf090 con 0x7f1508074100 2024-03-28T18:10:05.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.273+0000 7f1507400640 1 --2- 172.21.15.37:0/2147552044 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f14ec0a2800 0x7f14ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:05.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.274+0000 7f1507400640 1 --2- 172.21.15.37:0/2147552044 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f14ec0a2800 0x7f14ec0a4cc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f150819af20 tx=0x7f14f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.276+0000 7f1504c00640 1 -- 172.21.15.37:0/2147552044 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f150009ba90 con 0x7f1508074100 2024-03-28T18:10:05.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:05 smithi037 ceph-mon[31460]: pgmap v372: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:05.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:05 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/787943468' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:05.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:05 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1213532664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:05.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:05 smithi059 ceph-mon[36190]: pgmap v372: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:05.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:05 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/787943468' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:05.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:05 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1213532664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:05.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:05 smithi170 ceph-mon[36280]: pgmap v372: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:05.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:05 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/787943468' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:05.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:05 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1213532664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:05.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.598+0000 7f150d600640 1 -- 172.21.15.37:0/2147552044 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f15080622a0 con 0x7f1508074100 2024-03-28T18:10:05.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.598+0000 7f1504c00640 1 -- 172.21.15.37:0/2147552044 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f150009fb30 con 0x7f1508074100 2024-03-28T18:10:05.598 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 64 2024-03-28T18:10:05.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.601+0000 7f150d600640 1 -- 172.21.15.37:0/2147552044 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f14ec0a2800 msgr2=0x7f14ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:05.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.601+0000 7f150d600640 1 --2- 172.21.15.37:0/2147552044 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f14ec0a2800 0x7f14ec0a4cc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f150819af20 tx=0x7f14f4005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:05.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.601+0000 7f150d600640 1 -- 172.21.15.37:0/2147552044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1508074100 msgr2=0x7f1508199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:05.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.602+0000 7f150d600640 1 --2- 172.21.15.37:0/2147552044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1508074100 0x7f1508199a10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f150000ebc0 tx=0x7f150000c6a0 comp rx=0 tx=0).stop 2024-03-28T18:10:05.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.602+0000 7f150d600640 1 -- 172.21.15.37:0/2147552044 shutdown_connections 2024-03-28T18:10:05.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.602+0000 7f150d600640 1 --2- 172.21.15.37:0/2147552044 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f14ec0a2800 0x7f14ec0a4cc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:05.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.602+0000 7f150d600640 1 --2- 172.21.15.37:0/2147552044 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1508109460 0x7f15081a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:05.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.602+0000 7f150d600640 1 --2- 172.21.15.37:0/2147552044 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15080756e0 0x7f1508199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:05.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.602+0000 7f150d600640 1 --2- 172.21.15.37:0/2147552044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1508074100 0x7f1508199a10 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:05.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.602+0000 7f150d600640 1 -- 172.21.15.37:0/2147552044 >> 172.21.15.37:0/2147552044 conn(0x7f15080fcb10 msgr2=0x7f1508105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:05.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.602+0000 7f150d600640 1 -- 172.21.15.37:0/2147552044 shutdown_connections 2024-03-28T18:10:05.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.602+0000 7f150d600640 1 -- 172.21.15.37:0/2147552044 wait complete. 2024-03-28T18:10:05.622 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:10:05.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.818+0000 7fe364200640 1 -- 172.21.15.37:0/54490613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe35c1038e0 msgr2=0x7fe35c103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:05.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.818+0000 7fe364200640 1 --2- 172.21.15.37:0/54490613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe35c1038e0 0x7fe35c103d00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe350009900 tx=0x7fe35002bc40 comp rx=0 tx=0).stop 2024-03-28T18:10:05.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.818+0000 7fe364200640 1 -- 172.21.15.37:0/54490613 shutdown_connections 2024-03-28T18:10:05.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.818+0000 7fe364200640 1 --2- 172.21.15.37:0/54490613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe35c105490 0x7fe35c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:05.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.818+0000 7fe364200640 1 --2- 172.21.15.37:0/54490613 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe35c104ad0 0x7fe35c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:05.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.818+0000 7fe364200640 1 --2- 172.21.15.37:0/54490613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe35c1038e0 0x7fe35c103d00 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:05.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.818+0000 7fe364200640 1 -- 172.21.15.37:0/54490613 >> 172.21.15.37:0/54490613 conn(0x7fe35c0fee50 msgr2=0x7fe35c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:05.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.818+0000 7fe364200640 1 -- 172.21.15.37:0/54490613 shutdown_connections 2024-03-28T18:10:05.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.818+0000 7fe364200640 1 -- 172.21.15.37:0/54490613 wait complete. 2024-03-28T18:10:05.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.819+0000 7fe364200640 1 Processor -- start 2024-03-28T18:10:05.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.819+0000 7fe364200640 1 -- start start 2024-03-28T18:10:05.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.820+0000 7fe364200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe35c1038e0 0x7fe35c199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:05.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.820+0000 7fe364200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe35c104ad0 0x7fe35c19a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:05.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.820+0000 7fe364200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe35c105490 0x7fe35c1a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:05.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.820+0000 7fe364200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c10f5b0 con 0x7fe35c105490 2024-03-28T18:10:05.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.820+0000 7fe364200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c10f430 con 0x7fe35c104ad0 2024-03-28T18:10:05.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.820+0000 7fe364200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c10f730 con 0x7fe35c1038e0 2024-03-28T18:10:05.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.820+0000 7fe361800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe35c1038e0 0x7fe35c199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:05.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.820+0000 7fe360e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe35c104ad0 0x7fe35c19a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:05.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.820+0000 7fe361800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe35c1038e0 0x7fe35c199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:43784/0 (socket says 172.21.15.37:43784) 2024-03-28T18:10:05.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.820+0000 7fe361800640 1 -- 172.21.15.37:0/1891527920 learned_addr learned my addr 172.21.15.37:0/1891527920 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:05.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.820+0000 7fe362200640 1 --2- 172.21.15.37:0/1891527920 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe35c105490 0x7fe35c1a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:05.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.820+0000 7fe361800640 1 -- 172.21.15.37:0/1891527920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe35c104ad0 msgr2=0x7fe35c19a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:05.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.820+0000 7fe361800640 1 --2- 172.21.15.37:0/1891527920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe35c104ad0 0x7fe35c19a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:05.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.820+0000 7fe361800640 1 -- 172.21.15.37:0/1891527920 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe35c105490 msgr2=0x7fe35c1a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:05.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.820+0000 7fe361800640 1 --2- 172.21.15.37:0/1891527920 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe35c105490 0x7fe35c1a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:05.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.820+0000 7fe361800640 1 -- 172.21.15.37:0/1891527920 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35c1a17e0 con 0x7fe35c1038e0 2024-03-28T18:10:05.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.820+0000 7fe360e00640 1 --2- 172.21.15.37:0/1891527920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe35c104ad0 0x7fe35c19a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:05.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.821+0000 7fe361800640 1 --2- 172.21.15.37:0/1891527920 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe35c1038e0 0x7fe35c199bc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe35002efd0 tx=0x7fe35002c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:05.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.821+0000 7fe35a000640 1 -- 172.21.15.37:0/1891527920 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe350041070 con 0x7fe35c1038e0 2024-03-28T18:10:05.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.821+0000 7fe35a000640 1 -- 172.21.15.37:0/1891527920 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe350004450 con 0x7fe35c1038e0 2024-03-28T18:10:05.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.821+0000 7fe364200640 1 -- 172.21.15.37:0/1891527920 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35c1a1a70 con 0x7fe35c1038e0 2024-03-28T18:10:05.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.821+0000 7fe35a000640 1 -- 172.21.15.37:0/1891527920 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe350003e50 con 0x7fe35c1038e0 2024-03-28T18:10:05.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.821+0000 7fe364200640 1 -- 172.21.15.37:0/1891527920 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe35c1a1f80 con 0x7fe35c1038e0 2024-03-28T18:10:05.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.822+0000 7fe364200640 1 -- 172.21.15.37:0/1891527920 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe35c10aba0 con 0x7fe35c1038e0 2024-03-28T18:10:05.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.826+0000 7fe35a000640 1 -- 172.21.15.37:0/1891527920 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe350044070 con 0x7fe35c1038e0 2024-03-28T18:10:05.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.827+0000 7fe35a000640 1 --2- 172.21.15.37:0/1891527920 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe3440a2a20 0x7fe3440a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:05.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.828+0000 7fe35a000640 1 -- 172.21.15.37:0/1891527920 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe3500f0570 con 0x7fe35c1038e0 2024-03-28T18:10:05.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.828+0000 7fe360e00640 1 --2- 172.21.15.37:0/1891527920 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe3440a2a20 0x7fe3440a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:05.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.828+0000 7fe360e00640 1 --2- 172.21.15.37:0/1891527920 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe3440a2a20 0x7fe3440a4ee0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe3480095f0 tx=0x7fe348005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:05.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:05.831+0000 7fe35a000640 1 -- 172.21.15.37:0/1891527920 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe3500bcf70 con 0x7fe35c1038e0 2024-03-28T18:10:06.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.074+0000 7fe364200640 1 -- 172.21.15.37:0/1891527920 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe35c02d070 con 0x7fe35c1038e0 2024-03-28T18:10:06.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.075+0000 7fe35a000640 1 -- 172.21.15.37:0/1891527920 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11383 (secure 0 0 0) 0x7fe3500c1010 con 0x7fe35c1038e0 2024-03-28T18:10:06.075 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:06.075 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1926270371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6838","nonce":1926270371},{"type":"v1","addr":"172.21.15.37:6839","nonce":1926270371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:08:19.968894+0000","modified":"2024-03-28T18:08:25.415653+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24580,"mds_1":24569,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6837/4034607523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":4034607523},{"type":"v1","addr":"172.21.15.59:6837","nonce":4034607523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6839/1575518279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":1575518279},{"type":"v1","addr":"172.21.15.170:6839","nonce":1575518279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6839/2091629078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":2091629078},{"type":"v1","addr":"172.21.15.59:6839","nonce":2091629078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24580":{"gid":24580,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.37:6841/1923263622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6840","nonce":1923263622},{"type":"v1","addr":"172.21.15.37:6841","nonce":1923263622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24580,"qdb_cluster":[24550,24569,24580]},"id":1}]} 2024-03-28T18:10:06.075 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 22 2024-03-28T18:10:06.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.078+0000 7fe364200640 1 -- 172.21.15.37:0/1891527920 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe3440a2a20 msgr2=0x7fe3440a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:06.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.078+0000 7fe364200640 1 --2- 172.21.15.37:0/1891527920 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe3440a2a20 0x7fe3440a4ee0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe3480095f0 tx=0x7fe348005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:06.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.078+0000 7fe364200640 1 -- 172.21.15.37:0/1891527920 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe35c1038e0 msgr2=0x7fe35c199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:06.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.078+0000 7fe364200640 1 --2- 172.21.15.37:0/1891527920 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe35c1038e0 0x7fe35c199bc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe35002efd0 tx=0x7fe35002c660 comp rx=0 tx=0).stop 2024-03-28T18:10:06.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.078+0000 7fe364200640 1 -- 172.21.15.37:0/1891527920 shutdown_connections 2024-03-28T18:10:06.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.078+0000 7fe364200640 1 --2- 172.21.15.37:0/1891527920 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe3440a2a20 0x7fe3440a4ee0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:06.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.078+0000 7fe364200640 1 --2- 172.21.15.37:0/1891527920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe35c104ad0 0x7fe35c19a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:06.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.078+0000 7fe364200640 1 --2- 172.21.15.37:0/1891527920 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe35c105490 0x7fe35c1a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:06.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.078+0000 7fe364200640 1 --2- 172.21.15.37:0/1891527920 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe35c1038e0 0x7fe35c199bc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:06.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.078+0000 7fe364200640 1 -- 172.21.15.37:0/1891527920 >> 172.21.15.37:0/1891527920 conn(0x7fe35c0fee50 msgr2=0x7fe35c107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:06.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.078+0000 7fe364200640 1 -- 172.21.15.37:0/1891527920 shutdown_connections 2024-03-28T18:10:06.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.078+0000 7fe364200640 1 -- 172.21.15.37:0/1891527920 wait complete. 2024-03-28T18:10:06.097 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:10:06.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.305+0000 7f8cdb200640 1 -- 172.21.15.37:0/3271213018 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cd4104ad0 msgr2=0x7f8cd4104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:06.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.305+0000 7f8cdb200640 1 --2- 172.21.15.37:0/3271213018 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cd4104ad0 0x7f8cd4104f50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0009900 tx=0x7f8cc002bc50 comp rx=0 tx=0).stop 2024-03-28T18:10:06.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.305+0000 7f8cdb200640 1 -- 172.21.15.37:0/3271213018 shutdown_connections 2024-03-28T18:10:06.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.305+0000 7f8cdb200640 1 --2- 172.21.15.37:0/3271213018 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd4105490 0x7f8cd410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:06.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.305+0000 7f8cdb200640 1 --2- 172.21.15.37:0/3271213018 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cd4104ad0 0x7f8cd4104f50 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:06.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.305+0000 7f8cdb200640 1 --2- 172.21.15.37:0/3271213018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cd41038e0 0x7f8cd4103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:06.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.305+0000 7f8cdb200640 1 -- 172.21.15.37:0/3271213018 >> 172.21.15.37:0/3271213018 conn(0x7f8cd40fee50 msgr2=0x7f8cd4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:06.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.305+0000 7f8cdb200640 1 -- 172.21.15.37:0/3271213018 shutdown_connections 2024-03-28T18:10:06.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.305+0000 7f8cdb200640 1 -- 172.21.15.37:0/3271213018 wait complete. 2024-03-28T18:10:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.306+0000 7f8cdb200640 1 Processor -- start 2024-03-28T18:10:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.306+0000 7f8cdb200640 1 -- start start 2024-03-28T18:10:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.307+0000 7f8cdb200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd41038e0 0x7f8cd4199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.307+0000 7f8cdb200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cd4104ad0 0x7f8cd419a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.307+0000 7f8cdb200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cd4105490 0x7f8cd41a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.307+0000 7f8cdb200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd410f590 con 0x7f8cd41038e0 2024-03-28T18:10:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.307+0000 7f8cdb200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd410f410 con 0x7f8cd4105490 2024-03-28T18:10:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.307+0000 7f8cdb200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd410f710 con 0x7f8cd4104ad0 2024-03-28T18:10:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.307+0000 7f8cd3e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd41038e0 0x7f8cd4199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.307+0000 7f8cd3e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd41038e0 0x7f8cd4199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54196/0 (socket says 172.21.15.37:54196) 2024-03-28T18:10:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.307+0000 7f8cd3e00640 1 -- 172.21.15.37:0/1934550118 learned_addr learned my addr 172.21.15.37:0/1934550118 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.307+0000 7f8cd3400640 1 --2- 172.21.15.37:0/1934550118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cd4104ad0 0x7f8cd419a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.307+0000 7f8cd9200640 1 --2- 172.21.15.37:0/1934550118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cd4105490 0x7f8cd41a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.307+0000 7f8cd3e00640 1 -- 172.21.15.37:0/1934550118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cd4104ad0 msgr2=0x7f8cd419a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.307+0000 7f8cd3e00640 1 --2- 172.21.15.37:0/1934550118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cd4104ad0 0x7f8cd419a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.307+0000 7f8cd3e00640 1 -- 172.21.15.37:0/1934550118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cd4105490 msgr2=0x7f8cd41a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:06.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.307+0000 7f8cd3e00640 1 --2- 172.21.15.37:0/1934550118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cd4105490 0x7f8cd41a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:06.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.307+0000 7f8cd3e00640 1 -- 172.21.15.37:0/1934550118 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd41a17a0 con 0x7f8cd41038e0 2024-03-28T18:10:06.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.307+0000 7f8cd3400640 1 --2- 172.21.15.37:0/1934550118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cd4104ad0 0x7f8cd419a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:06.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.307+0000 7f8cd9200640 1 --2- 172.21.15.37:0/1934550118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cd4105490 0x7f8cd41a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:06.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.308+0000 7f8cd3e00640 1 --2- 172.21.15.37:0/1934550118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd41038e0 0x7f8cd4199b80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400e9c0 tx=0x7f8cc400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:06.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.308+0000 7f8cd0c00640 1 -- 172.21.15.37:0/1934550118 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc400cde0 con 0x7f8cd41038e0 2024-03-28T18:10:06.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.308+0000 7f8cdb200640 1 -- 172.21.15.37:0/1934550118 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd41a1a90 con 0x7f8cd41038e0 2024-03-28T18:10:06.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.308+0000 7f8cd0c00640 1 -- 172.21.15.37:0/1934550118 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8cc4010910 con 0x7f8cd41038e0 2024-03-28T18:10:06.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.308+0000 7f8cd0c00640 1 -- 172.21.15.37:0/1934550118 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc4018a10 con 0x7f8cd41038e0 2024-03-28T18:10:06.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.308+0000 7f8cdb200640 1 -- 172.21.15.37:0/1934550118 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd41a1fb0 con 0x7f8cd41038e0 2024-03-28T18:10:06.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.310+0000 7f8cdb200640 1 -- 172.21.15.37:0/1934550118 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd410aba0 con 0x7f8cd41038e0 2024-03-28T18:10:06.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.312+0000 7f8cd0c00640 1 -- 172.21.15.37:0/1934550118 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8cc401f020 con 0x7f8cd41038e0 2024-03-28T18:10:06.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.314+0000 7f8cd0c00640 1 --2- 172.21.15.37:0/1934550118 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8cb80a2800 0x7f8cb80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:06.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.315+0000 7f8cd0c00640 1 -- 172.21.15.37:0/1934550118 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f8cc40cec70 con 0x7f8cd41038e0 2024-03-28T18:10:06.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.315+0000 7f8cd3400640 1 --2- 172.21.15.37:0/1934550118 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8cb80a2800 0x7f8cb80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:06.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.316+0000 7f8cd3400640 1 --2- 172.21.15.37:0/1934550118 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8cb80a2800 0x7f8cb80a4cc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8cc002c260 tx=0x7f8cc0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.318+0000 7f8cd0c00640 1 -- 172.21.15.37:0/1934550118 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8cc409bcc0 con 0x7f8cd41038e0 2024-03-28T18:10:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:06 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2147552044' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:06 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1891527920' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:06.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.524+0000 7f8cdb200640 1 -- 172.21.15.37:0/1934550118 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cd40622a0 con 0x7f8cd41038e0 2024-03-28T18:10:06.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.527+0000 7f8cd0c00640 1 -- 172.21.15.37:0/1934550118 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22575 (secure 0 0 0) 0x7f8cc409b5c0 con 0x7f8cd41038e0 2024-03-28T18:10:06.528 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:06.528 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":87,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","created":"2024-03-28T17:59:05.949847+0000","modified":"2024-03-28T18:09:15.709901+0000","last_up_change":"2024-03-28T18:07:21.021030+0000","last_in_change":"2024-03-28T18:07:01.416192+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:03:00.517967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:08:18.365827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:08:19.446282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db60618d-56f7-4130-8a90-84cbc480965b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6803","nonce":3562602035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6805","nonce":3562602035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6809","nonce":3562602035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6807","nonce":3562602035}]},"public_addr":"172.21.15.37:6803/3562602035","cluster_addr":"172.21.15.37:6805/3562602035","heartbeat_back_addr":"172.21.15.37:6809/3562602035","heartbeat_front_addr":"172.21.15.37:6807/3562602035","state":["exists","up"]},{"osd":1,"uuid":"7a4b7ad6-905e-4258-9574-693d4277dfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6801","nonce":3326537211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6803","nonce":3326537211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6807","nonce":3326537211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6805","nonce":3326537211}]},"public_addr":"172.21.15.59:6801/3326537211","cluster_addr":"172.21.15.59:6803/3326537211","heartbeat_back_addr":"172.21.15.59:6807/3326537211","heartbeat_front_addr":"172.21.15.59:6805/3326537211","state":["exists","up"]},{"osd":2,"uuid":"8b76ec1c-46bf-4336-89b5-12d82906e8b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6801","nonce":1269398427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6803","nonce":1269398427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6807","nonce":1269398427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6805","nonce":1269398427}]},"public_addr":"172.21.15.170:6801/1269398427","cluster_addr":"172.21.15.170:6803/1269398427","heartbeat_back_addr":"172.21.15.170:6807/1269398427","heartbeat_front_addr":"172.21.15.170:6805/1269398427","state":["exists","up"]},{"osd":3,"uuid":"66eca1da-2fbd-44e1-8bbf-5635a2c70ad5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6811","nonce":317805240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6813","nonce":317805240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6817","nonce":317805240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6815","nonce":317805240}]},"public_addr":"172.21.15.37:6811/317805240","cluster_addr":"172.21.15.37:6813/317805240","heartbeat_back_addr":"172.21.15.37:6817/317805240","heartbeat_front_addr":"172.21.15.37:6815/317805240","state":["exists","up"]},{"osd":4,"uuid":"4c877335-e42d-4d8e-9f48-a1d0a89ae438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6809","nonce":2410632623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6811","nonce":2410632623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6815","nonce":2410632623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6813","nonce":2410632623}]},"public_addr":"172.21.15.59:6809/2410632623","cluster_addr":"172.21.15.59:6811/2410632623","heartbeat_back_addr":"172.21.15.59:6815/2410632623","heartbeat_front_addr":"172.21.15.59:6813/2410632623","state":["exists","up"]},{"osd":5,"uuid":"b884842a-86e2-455b-9570-ff163b3bc0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6809","nonce":1218943899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6811","nonce":1218943899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6815","nonce":1218943899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6813","nonce":1218943899}]},"public_addr":"172.21.15.170:6809/1218943899","cluster_addr":"172.21.15.170:6811/1218943899","heartbeat_back_addr":"172.21.15.170:6815/1218943899","heartbeat_front_addr":"172.21.15.170:6813/1218943899","state":["exists","up"]},{"osd":6,"uuid":"249a1200-77f4-44ad-981f-1d352f732b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6819","nonce":1899652046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6821","nonce":1899652046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6825","nonce":1899652046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6823","nonce":1899652046}]},"public_addr":"172.21.15.37:6819/1899652046","cluster_addr":"172.21.15.37:6821/1899652046","heartbeat_back_addr":"172.21.15.37:6825/1899652046","heartbeat_front_addr":"172.21.15.37:6823/1899652046","state":["exists","up"]},{"osd":7,"uuid":"c73338e6-f505-4517-a84b-f35e04750a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6817","nonce":1343675690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6819","nonce":1343675690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6823","nonce":1343675690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6821","nonce":1343675690}]},"public_addr":"172.21.15.59:6817/1343675690","cluster_addr":"172.21.15.59:6819/1343675690","heartbeat_back_addr":"172.21.15.59:6823/1343675690","heartbeat_front_addr":"172.21.15.59:6821/1343675690","state":["exists","up"]},{"osd":8,"uuid":"149b8297-ff2d-4a88-9b1e-0def021cc516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6817","nonce":3235794556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6819","nonce":3235794556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6823","nonce":3235794556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6821","nonce":3235794556}]},"public_addr":"172.21.15.170:6817/3235794556","cluster_addr":"172.21.15.170:6819/3235794556","heartbeat_back_addr":"172.21.15.170:6823/3235794556","heartbeat_front_addr":"172.21.15.170:6821/3235794556","state":["exists","up"]},{"osd":9,"uuid":"55c27cc2-c771-44b7-90fe-8f0feca44162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6827","nonce":2307153008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6829","nonce":2307153008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6833","nonce":2307153008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6831","nonce":2307153008}]},"public_addr":"172.21.15.37:6827/2307153008","cluster_addr":"172.21.15.37:6829/2307153008","heartbeat_back_addr":"172.21.15.37:6833/2307153008","heartbeat_front_addr":"172.21.15.37:6831/2307153008","state":["exists","up"]},{"osd":10,"uuid":"c0595135-ab81-49d3-8c23-19f960c03e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6825","nonce":2113830275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6827","nonce":2113830275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6831","nonce":2113830275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6829","nonce":2113830275}]},"public_addr":"172.21.15.59:6825/2113830275","cluster_addr":"172.21.15.59:6827/2113830275","heartbeat_back_addr":"172.21.15.59:6831/2113830275","heartbeat_front_addr":"172.21.15.59:6829/2113830275","state":["exists","up"]},{"osd":11,"uuid":"5871e177-c17b-4637-a797-4fa5125920dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6825","nonce":3082439360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6827","nonce":3082439360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6831","nonce":3082439360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6829","nonce":3082439360}]},"public_addr":"172.21.15.170:6825/3082439360","cluster_addr":"172.21.15.170:6827/3082439360","heartbeat_back_addr":"172.21.15.170:6831/3082439360","heartbeat_front_addr":"172.21.15.170:6829/3082439360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:07.690545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:33.183396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:57.400263+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:30.597211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:57.259044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:22.627823+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:57.585863+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:25.000476+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:50.876688+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:25.859761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:53.563146+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:07:19.304115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3023379428":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1443710231":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6800/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6801/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6801/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6800/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1007625490":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/2895698584":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/497313034":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/3293943475":"2024-03-29T17:59:39.162038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:10:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.531+0000 7f8cdb200640 1 -- 172.21.15.37:0/1934550118 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8cb80a2800 msgr2=0x7f8cb80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.531+0000 7f8cdb200640 1 --2- 172.21.15.37:0/1934550118 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8cb80a2800 0x7f8cb80a4cc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8cc002c260 tx=0x7f8cc0000bc0 comp rx=0 tx=0).stop 2024-03-28T18:10:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.531+0000 7f8cdb200640 1 -- 172.21.15.37:0/1934550118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd41038e0 msgr2=0x7f8cd4199b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.531+0000 7f8cdb200640 1 --2- 172.21.15.37:0/1934550118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd41038e0 0x7f8cd4199b80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400e9c0 tx=0x7f8cc400ee80 comp rx=0 tx=0).stop 2024-03-28T18:10:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.532+0000 7f8cdb200640 1 -- 172.21.15.37:0/1934550118 shutdown_connections 2024-03-28T18:10:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.532+0000 7f8cdb200640 1 --2- 172.21.15.37:0/1934550118 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8cb80a2800 0x7f8cb80a4cc0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.532+0000 7f8cdb200640 1 --2- 172.21.15.37:0/1934550118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cd4105490 0x7f8cd41a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.532+0000 7f8cdb200640 1 --2- 172.21.15.37:0/1934550118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cd4104ad0 0x7f8cd419a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.532+0000 7f8cdb200640 1 --2- 172.21.15.37:0/1934550118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd41038e0 0x7f8cd4199b80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.532+0000 7f8cdb200640 1 -- 172.21.15.37:0/1934550118 >> 172.21.15.37:0/1934550118 conn(0x7f8cd40fee50 msgr2=0x7f8cd4107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.532+0000 7f8cdb200640 1 -- 172.21.15.37:0/1934550118 shutdown_connections 2024-03-28T18:10:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.532+0000 7f8cdb200640 1 -- 172.21.15.37:0/1934550118 wait complete. 2024-03-28T18:10:06.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:06 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2147552044' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:06.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:06 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1891527920' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:06.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:06 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2147552044' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:06.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:06 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1891527920' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:06.551 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:10:06.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.768+0000 7f5e45e00640 1 -- 172.21.15.37:0/4113216963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e401030b0 msgr2=0x7f5e40109940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:06.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.768+0000 7f5e45e00640 1 --2- 172.21.15.37:0/4113216963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e401030b0 0x7f5e40109940 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5e38009900 tx=0x7f5e3802bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:06.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.769+0000 7f5e45e00640 1 -- 172.21.15.37:0/4113216963 shutdown_connections 2024-03-28T18:10:06.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.769+0000 7f5e45e00640 1 --2- 172.21.15.37:0/4113216963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e401030b0 0x7f5e40109940 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:06.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.769+0000 7f5e45e00640 1 --2- 172.21.15.37:0/4113216963 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e401026f0 0x7f5e40102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:06.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.769+0000 7f5e45e00640 1 --2- 172.21.15.37:0/4113216963 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e40101500 0x7f5e40101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:06.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.769+0000 7f5e45e00640 1 -- 172.21.15.37:0/4113216963 >> 172.21.15.37:0/4113216963 conn(0x7f5e400fca70 msgr2=0x7f5e400feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:06.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.769+0000 7f5e45e00640 1 -- 172.21.15.37:0/4113216963 shutdown_connections 2024-03-28T18:10:06.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.769+0000 7f5e45e00640 1 -- 172.21.15.37:0/4113216963 wait complete. 2024-03-28T18:10:06.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.770+0000 7f5e45e00640 1 Processor -- start 2024-03-28T18:10:06.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.770+0000 7f5e45e00640 1 -- start start 2024-03-28T18:10:06.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.770+0000 7f5e45e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e40101500 0x7f5e40199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:06.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.770+0000 7f5e45e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e401026f0 0x7f5e40199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:06.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.770+0000 7f5e45e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e401030b0 0x7f5e401a0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:06.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.770+0000 7f5e45e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4010d1e0 con 0x7f5e401026f0 2024-03-28T18:10:06.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.770+0000 7f5e45e00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4010d060 con 0x7f5e40101500 2024-03-28T18:10:06.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.770+0000 7f5e45e00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4010d360 con 0x7f5e401030b0 2024-03-28T18:10:06.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.770+0000 7f5e44a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e40101500 0x7f5e40199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:06.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.770+0000 7f5e44a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e40101500 0x7f5e40199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:55186/0 (socket says 172.21.15.37:55186) 2024-03-28T18:10:06.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.770+0000 7f5e44a00640 1 -- 172.21.15.37:0/564517843 learned_addr learned my addr 172.21.15.37:0/564517843 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:06.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.770+0000 7f5e45400640 1 --2- 172.21.15.37:0/564517843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e401030b0 0x7f5e401a0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:06.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.770+0000 7f5e3fe00640 1 --2- 172.21.15.37:0/564517843 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e401026f0 0x7f5e40199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:06.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.771+0000 7f5e44a00640 1 -- 172.21.15.37:0/564517843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e401030b0 msgr2=0x7f5e401a0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:06.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.771+0000 7f5e44a00640 1 --2- 172.21.15.37:0/564517843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e401030b0 0x7f5e401a0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:06.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.771+0000 7f5e44a00640 1 -- 172.21.15.37:0/564517843 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e401026f0 msgr2=0x7f5e40199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:06.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.771+0000 7f5e44a00640 1 --2- 172.21.15.37:0/564517843 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e401026f0 0x7f5e40199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:06.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.771+0000 7f5e44a00640 1 -- 172.21.15.37:0/564517843 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e401a1590 con 0x7f5e40101500 2024-03-28T18:10:06.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.771+0000 7f5e44a00640 1 --2- 172.21.15.37:0/564517843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e40101500 0x7f5e40199970 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5e34009fd0 tx=0x7f5e3400c270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:06.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.771+0000 7f5e3d600640 1 -- 172.21.15.37:0/564517843 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e34011470 con 0x7f5e40101500 2024-03-28T18:10:06.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.771+0000 7f5e3d600640 1 -- 172.21.15.37:0/564517843 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e34011af0 con 0x7f5e40101500 2024-03-28T18:10:06.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.771+0000 7f5e3d600640 1 -- 172.21.15.37:0/564517843 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3401ab90 con 0x7f5e40101500 2024-03-28T18:10:06.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.771+0000 7f5e45e00640 1 -- 172.21.15.37:0/564517843 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e401a1880 con 0x7f5e40101500 2024-03-28T18:10:06.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.772+0000 7f5e45e00640 1 -- 172.21.15.37:0/564517843 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e401a1e20 con 0x7f5e40101500 2024-03-28T18:10:06.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.773+0000 7f5e45e00640 1 -- 172.21.15.37:0/564517843 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e401a20b0 con 0x7f5e40101500 2024-03-28T18:10:06.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.774+0000 7f5e3d600640 1 -- 172.21.15.37:0/564517843 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5e34011610 con 0x7f5e40101500 2024-03-28T18:10:06.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.776+0000 7f5e3d600640 1 --2- 172.21.15.37:0/564517843 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5e280a2880 0x7f5e280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:06.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.776+0000 7f5e3d600640 1 -- 172.21.15.37:0/564517843 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5e340cefa0 con 0x7f5e40101500 2024-03-28T18:10:06.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.776+0000 7f5e3fe00640 1 --2- 172.21.15.37:0/564517843 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5e280a2880 0x7f5e280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:06.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.777+0000 7f5e3fe00640 1 --2- 172.21.15.37:0/564517843 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5e280a2880 0x7f5e280a4d40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5e4019ae80 tx=0x7f5e2c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:06.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:06.780+0000 7f5e3d600640 1 -- 172.21.15.37:0/564517843 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5e34016030 con 0x7f5e40101500 2024-03-28T18:10:07.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.034+0000 7f5e45e00640 1 -- 172.21.15.37:0/564517843 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e401a2280 con 0x7f5e40101500 2024-03-28T18:10:07.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.035+0000 7f5e3d600640 1 -- 172.21.15.37:0/564517843 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11383 (secure 0 0 0) 0x7f5e3409bff0 con 0x7f5e40101500 2024-03-28T18:10:07.035 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:07.036 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1926270371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6838","nonce":1926270371},{"type":"v1","addr":"172.21.15.37:6839","nonce":1926270371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:08:19.968894+0000","modified":"2024-03-28T18:08:25.415653+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24580,"mds_1":24569,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6837/4034607523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":4034607523},{"type":"v1","addr":"172.21.15.59:6837","nonce":4034607523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6839/1575518279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":1575518279},{"type":"v1","addr":"172.21.15.170:6839","nonce":1575518279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6839/2091629078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":2091629078},{"type":"v1","addr":"172.21.15.59:6839","nonce":2091629078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24580":{"gid":24580,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.37:6841/1923263622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6840","nonce":1923263622},{"type":"v1","addr":"172.21.15.37:6841","nonce":1923263622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24580,"qdb_cluster":[24550,24569,24580]},"id":1}]} 2024-03-28T18:10:07.036 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 22 2024-03-28T18:10:07.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.039+0000 7f5e45e00640 1 -- 172.21.15.37:0/564517843 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5e280a2880 msgr2=0x7f5e280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.039+0000 7f5e45e00640 1 --2- 172.21.15.37:0/564517843 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5e280a2880 0x7f5e280a4d40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5e4019ae80 tx=0x7f5e2c005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:07.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.039+0000 7f5e45e00640 1 -- 172.21.15.37:0/564517843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e40101500 msgr2=0x7f5e40199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.039+0000 7f5e45e00640 1 --2- 172.21.15.37:0/564517843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e40101500 0x7f5e40199970 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5e34009fd0 tx=0x7f5e3400c270 comp rx=0 tx=0).stop 2024-03-28T18:10:07.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.039+0000 7f5e45e00640 1 -- 172.21.15.37:0/564517843 shutdown_connections 2024-03-28T18:10:07.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.039+0000 7f5e45e00640 1 --2- 172.21.15.37:0/564517843 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f5e280a2880 0x7f5e280a4d40 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.039+0000 7f5e45e00640 1 --2- 172.21.15.37:0/564517843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e401030b0 0x7f5e401a0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.039+0000 7f5e45e00640 1 --2- 172.21.15.37:0/564517843 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e401026f0 0x7f5e40199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.039+0000 7f5e45e00640 1 --2- 172.21.15.37:0/564517843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e40101500 0x7f5e40199970 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.039+0000 7f5e45e00640 1 -- 172.21.15.37:0/564517843 >> 172.21.15.37:0/564517843 conn(0x7f5e400fca70 msgr2=0x7f5e400738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:07.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.039+0000 7f5e45e00640 1 -- 172.21.15.37:0/564517843 shutdown_connections 2024-03-28T18:10:07.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.039+0000 7f5e45e00640 1 -- 172.21.15.37:0/564517843 wait complete. 2024-03-28T18:10:07.065 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T18:08:19.968894+0000', 'modified': '2024-03-28T18:08:25.415653+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24580, 'mds_1': 24569, 'mds_2': 24550}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24512': {'gid': 24512, 'name': 'h', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.59:6837/4034607523', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 4034607523}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 4034607523}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24517': {'gid': 24517, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.170:6835/4133178144', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6834', 'nonce': 4133178144}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 4133178144}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24529': {'gid': 24529, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.170:6837/1308157923', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6836', 'nonce': 1308157923}, {'type': 'v1', 'addr': '172.21.15.170:6837', 'nonce': 1308157923}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24550': {'gid': 24550, 'name': 'i', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.170:6839/1575518279', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6838', 'nonce': 1575518279}, {'type': 'v1', 'addr': '172.21.15.170:6839', 'nonce': 1575518279}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24569': {'gid': 24569, 'name': 'b', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.59:6839/2091629078', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6838', 'nonce': 2091629078}, {'type': 'v1', 'addr': '172.21.15.59:6839', 'nonce': 2091629078}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24580': {'gid': 24580, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.37:6841/1923263622', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.37:6840', 'nonce': 1923263622}, {'type': 'v1', 'addr': '172.21.15.37:6841', 'nonce': 1923263622}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24580, 'qdb_cluster': [24550, 24569, 24580]} 2024-03-28T18:10:07.065 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-28T18:10:07.065 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-28T18:10:07.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.293+0000 7fe818a00640 1 -- 172.21.15.37:0/3158332861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe814105470 msgr2=0x7fe81410bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.293+0000 7fe818a00640 1 --2- 172.21.15.37:0/3158332861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe814105470 0x7fe81410bd00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009ca0 tx=0x7fe80c02c100 comp rx=0 tx=0).stop 2024-03-28T18:10:07.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.293+0000 7fe818a00640 1 -- 172.21.15.37:0/3158332861 shutdown_connections 2024-03-28T18:10:07.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.293+0000 7fe818a00640 1 --2- 172.21.15.37:0/3158332861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe814105470 0x7fe81410bd00 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.293+0000 7fe818a00640 1 --2- 172.21.15.37:0/3158332861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe814104ab0 0x7fe814104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.293+0000 7fe818a00640 1 --2- 172.21.15.37:0/3158332861 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8141038c0 0x7fe814103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.293+0000 7fe818a00640 1 -- 172.21.15.37:0/3158332861 >> 172.21.15.37:0/3158332861 conn(0x7fe8140fee50 msgr2=0x7fe814101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:07.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.293+0000 7fe818a00640 1 -- 172.21.15.37:0/3158332861 shutdown_connections 2024-03-28T18:10:07.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.293+0000 7fe818a00640 1 -- 172.21.15.37:0/3158332861 wait complete. 2024-03-28T18:10:07.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.294+0000 7fe818a00640 1 Processor -- start 2024-03-28T18:10:07.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.294+0000 7fe818a00640 1 -- start start 2024-03-28T18:10:07.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.295+0000 7fe818a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8141038c0 0x7fe814199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.295+0000 7fe818a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe814104ab0 0x7fe81419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.295+0000 7fe818a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe814105470 0x7fe8141a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.295+0000 7fe818a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81410f540 con 0x7fe814104ab0 2024-03-28T18:10:07.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.295+0000 7fe818a00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81410f3c0 con 0x7fe814105470 2024-03-28T18:10:07.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.295+0000 7fe818a00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81410f6c0 con 0x7fe8141038c0 2024-03-28T18:10:07.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.295+0000 7fe811e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8141038c0 0x7fe814199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.295+0000 7fe811e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8141038c0 0x7fe814199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:43816/0 (socket says 172.21.15.37:43816) 2024-03-28T18:10:07.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.295+0000 7fe811e00640 1 -- 172.21.15.37:0/1601655244 learned_addr learned my addr 172.21.15.37:0/1601655244 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:07.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.295+0000 7fe812800640 1 --2- 172.21.15.37:0/1601655244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe814105470 0x7fe8141a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.295+0000 7fe80be00640 1 --2- 172.21.15.37:0/1601655244 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe814104ab0 0x7fe81419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.295+0000 7fe811e00640 1 -- 172.21.15.37:0/1601655244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe814105470 msgr2=0x7fe8141a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.295+0000 7fe811e00640 1 --2- 172.21.15.37:0/1601655244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe814105470 0x7fe8141a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.295+0000 7fe811e00640 1 -- 172.21.15.37:0/1601655244 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe814104ab0 msgr2=0x7fe81419a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.295+0000 7fe811e00640 1 --2- 172.21.15.37:0/1601655244 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe814104ab0 0x7fe81419a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.295+0000 7fe811e00640 1 -- 172.21.15.37:0/1601655244 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8141a1810 con 0x7fe8141038c0 2024-03-28T18:10:07.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.296+0000 7fe811e00640 1 --2- 172.21.15.37:0/1601655244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8141038c0 0x7fe814199bf0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe80400e9c0 tx=0x7fe80400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:07.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.296+0000 7fe80aa00640 1 -- 172.21.15.37:0/1601655244 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80400cd80 con 0x7fe8141038c0 2024-03-28T18:10:07.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.296+0000 7fe80aa00640 1 -- 172.21.15.37:0/1601655244 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe8040035f0 con 0x7fe8141038c0 2024-03-28T18:10:07.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.296+0000 7fe80aa00640 1 -- 172.21.15.37:0/1601655244 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe804010690 con 0x7fe8141038c0 2024-03-28T18:10:07.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.296+0000 7fe818a00640 1 -- 172.21.15.37:0/1601655244 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8141a1b00 con 0x7fe8141038c0 2024-03-28T18:10:07.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.296+0000 7fe818a00640 1 -- 172.21.15.37:0/1601655244 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8141a20a0 con 0x7fe8141038c0 2024-03-28T18:10:07.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.299+0000 7fe80aa00640 1 -- 172.21.15.37:0/1601655244 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe8040037a0 con 0x7fe8141038c0 2024-03-28T18:10:07.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.300+0000 7fe81b817740 1 Processor -- start 2024-03-28T18:10:07.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.300+0000 7fe81b817740 1 -- start start 2024-03-28T18:10:07.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.300+0000 7fe81b817740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f253ba0 0x55822f253fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.300+0000 7fe81b817740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f254590 0x55822f259090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.300+0000 7fe81b817740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f259700 0x55822f25bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.300+0000 7fe81b817740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55822f1b8c90 con 0x55822f254590 2024-03-28T18:10:07.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.300+0000 7fe81b817740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55822f161980 con 0x55822f259700 2024-03-28T18:10:07.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.300+0000 7fe81b817740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55822f1b4d30 con 0x55822f253ba0 2024-03-28T18:10:07.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.300+0000 7fe812800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f253ba0 0x55822f253fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.300+0000 7fe812800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f253ba0 0x55822f253fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:43822/0 (socket says 172.21.15.37:43822) 2024-03-28T18:10:07.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.300+0000 7fe80be00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f259700 0x55822f25bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.300+0000 7fe812800640 1 -- 172.21.15.37:0/3007158703 learned_addr learned my addr 172.21.15.37:0/3007158703 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:07.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.300+0000 7fe811e00640 1 --2- 172.21.15.37:0/3007158703 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f254590 0x55822f259090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.301+0000 7fe812800640 1 -- 172.21.15.37:0/3007158703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f259700 msgr2=0x55822f25bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.301+0000 7fe812800640 1 --2- 172.21.15.37:0/3007158703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f259700 0x55822f25bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.301+0000 7fe812800640 1 -- 172.21.15.37:0/3007158703 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f254590 msgr2=0x55822f259090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.301+0000 7fe812800640 1 --2- 172.21.15.37:0/3007158703 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f254590 0x55822f259090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.301+0000 7fe812800640 1 -- 172.21.15.37:0/3007158703 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55822f179510 con 0x55822f253ba0 2024-03-28T18:10:07.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.301+0000 7fe80be00640 1 --2- 172.21.15.37:0/3007158703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f259700 0x55822f25bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:07.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.301+0000 7fe812800640 1 --2- 172.21.15.37:0/3007158703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f253ba0 0x55822f253fc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009ca0 tx=0x7fe80c0323d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c824a7b71c6e7daf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:07.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.301+0000 7fe803000640 1 -- 172.21.15.37:0/3007158703 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c043070 con 0x55822f253ba0 2024-03-28T18:10:07.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.301+0000 7fe80aa00640 1 --2- 172.21.15.37:0/1601655244 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe7f80a2880 0x7fe7f80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.301+0000 7fe803000640 1 -- 172.21.15.37:0/3007158703 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe80c032c90 con 0x55822f253ba0 2024-03-28T18:10:07.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.301+0000 7fe803000640 1 -- 172.21.15.37:0/3007158703 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c039040 con 0x55822f253ba0 2024-03-28T18:10:07.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.302+0000 7fe80aa00640 1 -- 172.21.15.37:0/1601655244 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe8040cee90 con 0x7fe8141038c0 2024-03-28T18:10:07.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.302+0000 7fe811e00640 1 --2- 172.21.15.37:0/1601655244 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe7f80a2880 0x7fe7f80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.302+0000 7fe81b817740 1 -- 172.21.15.37:0/3007158703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f253ba0 msgr2=0x55822f253fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.302+0000 7fe81b817740 1 --2- 172.21.15.37:0/3007158703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f253ba0 0x55822f253fc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009ca0 tx=0x7fe80c0323d0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.302+0000 7fe81b817740 1 -- 172.21.15.37:0/3007158703 shutdown_connections 2024-03-28T18:10:07.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.302+0000 7fe81b817740 1 --2- 172.21.15.37:0/3007158703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f259700 0x55822f25bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.302+0000 7fe81b817740 1 --2- 172.21.15.37:0/3007158703 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f254590 0x55822f259090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.302+0000 7fe81b817740 1 --2- 172.21.15.37:0/3007158703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f253ba0 0x55822f253fc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.302+0000 7fe81b817740 1 -- 172.21.15.37:0/3007158703 >> 172.21.15.37:0/3007158703 conn(0x55822f24f110 msgr2=0x55822f24f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:07.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.302+0000 7fe81b817740 1 -- 172.21.15.37:0/3007158703 shutdown_connections 2024-03-28T18:10:07.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.302+0000 7fe811e00640 1 --2- 172.21.15.37:0/1601655244 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe7f80a2880 0x7fe7f80a4d40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe804097c40 tx=0x7fe804000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:07.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.303+0000 7fe81b817740 1 -- 172.21.15.37:0/3007158703 wait complete. 2024-03-28T18:10:07.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.303+0000 7fe81b817740 1 -- start start 2024-03-28T18:10:07.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.303+0000 7fe81b817740 1 Processor -- start 2024-03-28T18:10:07.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.304+0000 7fe81b817740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f253ba0 0x55822f25e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.304+0000 7fe81b817740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 0x55822f25e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.304+0000 7fe81b817740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f259700 0x55822f265630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.304+0000 7fe81b817740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55822f25c650 con 0x55822f253ba0 2024-03-28T18:10:07.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.304+0000 7fe81b817740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55822f174dd0 con 0x55822f259700 2024-03-28T18:10:07.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.304+0000 7fe81b817740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55822f25c7d0 con 0x55822f254590 2024-03-28T18:10:07.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.304+0000 7fe81b817740 10 client.0 _ll_register_callbacks cb 0x55822eeccac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:10:07.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.304+0000 7fe80be00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 0x55822f25e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.304+0000 7fe80be00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 0x55822f25e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:43838/0 (socket says 172.21.15.37:43838) 2024-03-28T18:10:07.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.304+0000 7fe80be00640 1 -- 172.21.15.37:0/3542256982 learned_addr learned my addr 172.21.15.37:0/3542256982 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:07.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.304+0000 7fe812800640 1 --2- 172.21.15.37:0/3542256982 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f253ba0 0x55822f25e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.304+0000 7fe812800640 1 --2- 172.21.15.37:0/3542256982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f259700 0x55822f265630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.304+0000 7fe80be00640 1 -- 172.21.15.37:0/3542256982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f259700 msgr2=0x55822f265630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.304+0000 7fe80be00640 1 --2- 172.21.15.37:0/3542256982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f259700 0x55822f265630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.304+0000 7fe80be00640 1 -- 172.21.15.37:0/3542256982 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f253ba0 msgr2=0x55822f25e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.304+0000 7fe80be00640 1 --2- 172.21.15.37:0/3542256982 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f253ba0 0x55822f25e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.304+0000 7fe80be00640 1 -- 172.21.15.37:0/3542256982 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55822eecd2d0 con 0x55822f254590 2024-03-28T18:10:07.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.304+0000 7fe812800640 1 --2- 172.21.15.37:0/3542256982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f259700 0x55822f265630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:07.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.305+0000 7fe80be00640 1 --2- 172.21.15.37:0/3542256982 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 0x55822f25e5f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc012700 tx=0x7fe7fc012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd7e5598ec56a46 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:07.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.305+0000 7fe7f7400640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:10:07.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.305+0000 7fe7f7400640 1 -- 172.21.15.37:0/3542256982 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc011b50 con 0x55822f254590 2024-03-28T18:10:07.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.305+0000 7fe7f7400640 1 -- 172.21.15.37:0/3542256982 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe7fc011cf0 con 0x55822f254590 2024-03-28T18:10:07.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.305+0000 7fe81b817740 1 -- 172.21.15.37:0/3542256982 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe80c032c90 con 0x55822f254590 2024-03-28T18:10:07.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.305+0000 7fe7f7400640 1 -- 172.21.15.37:0/3542256982 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7fc004030 con 0x55822f254590 2024-03-28T18:10:07.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.305+0000 7fe81b817740 10 client.25003 fetch_fsmap learned FSMap version 22 2024-03-28T18:10:07.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.305+0000 7fe81b817740 1 -- 172.21.15.37:0/3542256982 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55822ef976b0 con 0x55822f254590 2024-03-28T18:10:07.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.306+0000 7fe7f7400640 1 -- 172.21.15.37:0/3542256982 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe80c032c90 con 0x55822f254590 2024-03-28T18:10:07.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.306+0000 7fe7f7400640 1 -- 172.21.15.37:0/3542256982 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe7fc01c460 con 0x55822f254590 2024-03-28T18:10:07.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.306+0000 7fe81b817740 10 client.25003 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T18:10:07.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.306+0000 7fe81b817740 10 client.25003 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T18:10:07.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.306+0000 7fe81b817740 20 client.25003 populate_metadata read hostname 'smithi037' 2024-03-28T18:10:07.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.306+0000 7fe81b817740 1 --2- 172.21.15.37:0/3542256982 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x55822f26a980 0x55822f269a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.306+0000 7fe81b817740 4 client.25003 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:10:07.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.306+0000 7fe81b817740 1 -- 172.21.15.37:0/3542256982 --> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55822f15a520 con 0x55822f26a980 2024-03-28T18:10:07.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.306+0000 7fe812800640 1 --2- 172.21.15.37:0/3542256982 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x55822f26a980 0x55822f269a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.307+0000 7fe812800640 1 --2- 172.21.15.37:0/3542256982 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x55822f26a980 0x55822f269a20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3db3df5ee14cf10d server_cookie=f2c9c115ec23dd89 in_seq=0 out_seq=0 2024-03-28T18:10:07.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.307+0000 7fe7f7400640 10 client.25003 ms_handle_connect on v2:172.21.15.170:6838/1575518279 2024-03-28T18:10:07.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.307+0000 7fe7f7400640 1 -- 172.21.15.37:0/3542256982 <== mds.2 v2:172.21.15.170:6838/1575518279 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55822f15a520 con 0x55822f26a980 2024-03-28T18:10:07.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.307+0000 7fe7f7400640 10 client.25003 handle_command_reply: tid=0 2024-03-28T18:10:07.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.308+0000 7fe81b817740 1 client.25003 shutdown 2024-03-28T18:10:07.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.308+0000 7fe81b817740 1 -- 172.21.15.37:0/3542256982 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 msgr2=0x55822f25e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.308+0000 7fe81b817740 1 --2- 172.21.15.37:0/3542256982 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 0x55822f25e5f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc012700 tx=0x7fe7fc012bc0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.308+0000 7fe81b817740 1 -- 172.21.15.37:0/3542256982 shutdown_connections 2024-03-28T18:10:07.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.308+0000 7fe81b817740 1 --2- 172.21.15.37:0/3542256982 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x55822f26a980 0x55822f269a20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.308+0000 7fe81b817740 1 --2- 172.21.15.37:0/3542256982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f259700 0x55822f265630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.308+0000 7fe81b817740 1 --2- 172.21.15.37:0/3542256982 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 0x55822f25e5f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.308+0000 7fe81b817740 1 --2- 172.21.15.37:0/3542256982 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f253ba0 0x55822f25e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.308+0000 7fe7f7400640 0 client.25003 ms_handle_reset on v2:172.21.15.170:6838/1575518279 2024-03-28T18:10:07.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.308+0000 7fe81b817740 1 -- 172.21.15.37:0/3542256982 >> 172.21.15.37:0/3542256982 conn(0x55822f24f110 msgr2=0x55822f24e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:07.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.308+0000 7fe81b817740 1 -- 172.21.15.37:0/3542256982 shutdown_connections 2024-03-28T18:10:07.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.308+0000 7fe81b817740 1 -- 172.21.15.37:0/3542256982 wait complete. 2024-03-28T18:10:07.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.308+0000 7fe81b817740 20 client.25003 trim_cache size 0 max 16384 2024-03-28T18:10:07.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.340+0000 7fe81b817740 1 Processor -- start 2024-03-28T18:10:07.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.340+0000 7fe81b817740 1 -- start start 2024-03-28T18:10:07.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.340+0000 7fe81b817740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f253ba0 0x55822f255030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.340+0000 7fe81b817740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 0x55822f2655f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.340+0000 7fe81b817740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f259700 0x55822f269a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.340+0000 7fe81b817740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55822f26a1a0 con 0x55822f253ba0 2024-03-28T18:10:07.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.340+0000 7fe81b817740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55822f26a800 con 0x55822f259700 2024-03-28T18:10:07.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.340+0000 7fe81b817740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55822f26ae60 con 0x55822f254590 2024-03-28T18:10:07.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.340+0000 7fe80be00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 0x55822f2655f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.340+0000 7fe80be00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 0x55822f2655f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:43842/0 (socket says 172.21.15.37:43842) 2024-03-28T18:10:07.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.340+0000 7fe80be00640 1 -- 172.21.15.37:0/4064067195 learned_addr learned my addr 172.21.15.37:0/4064067195 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:07.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.340+0000 7fe812800640 1 --2- 172.21.15.37:0/4064067195 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f253ba0 0x55822f255030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.341+0000 7fe812800640 1 --2- 172.21.15.37:0/4064067195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f259700 0x55822f269a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.341+0000 7fe80be00640 1 -- 172.21.15.37:0/4064067195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f259700 msgr2=0x55822f269a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.341+0000 7fe80be00640 1 --2- 172.21.15.37:0/4064067195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f259700 0x55822f269a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.341+0000 7fe80be00640 1 -- 172.21.15.37:0/4064067195 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f253ba0 msgr2=0x55822f255030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.341+0000 7fe80be00640 1 --2- 172.21.15.37:0/4064067195 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f253ba0 0x55822f255030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.341+0000 7fe80be00640 1 -- 172.21.15.37:0/4064067195 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55822f25f0a0 con 0x55822f254590 2024-03-28T18:10:07.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.341+0000 7fe812800640 1 --2- 172.21.15.37:0/4064067195 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f253ba0 0x55822f255030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:07.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.341+0000 7fe80be00640 1 --2- 172.21.15.37:0/4064067195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 0x55822f2655f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc01c950 tx=0x7fe7fc0092a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35fc687044687cb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.341+0000 7fe802600640 1 -- 172.21.15.37:0/4064067195 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc02c070 con 0x55822f254590 2024-03-28T18:10:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.342+0000 7fe802600640 1 -- 172.21.15.37:0/4064067195 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe7fc01c460 con 0x55822f254590 2024-03-28T18:10:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.342+0000 7fe802600640 1 -- 172.21.15.37:0/4064067195 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc003650 con 0x55822f254590 2024-03-28T18:10:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.342+0000 7fe81b817740 1 -- 172.21.15.37:0/4064067195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 msgr2=0x55822f2655f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.342+0000 7fe81b817740 1 --2- 172.21.15.37:0/4064067195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 0x55822f2655f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc01c950 tx=0x7fe7fc0092a0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.342+0000 7fe81b817740 1 -- 172.21.15.37:0/4064067195 shutdown_connections 2024-03-28T18:10:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.342+0000 7fe81b817740 1 --2- 172.21.15.37:0/4064067195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f259700 0x55822f269a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.342+0000 7fe81b817740 1 --2- 172.21.15.37:0/4064067195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 0x55822f2655f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.342+0000 7fe81b817740 1 --2- 172.21.15.37:0/4064067195 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f253ba0 0x55822f255030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.342+0000 7fe81b817740 1 -- 172.21.15.37:0/4064067195 >> 172.21.15.37:0/4064067195 conn(0x55822f24f110 msgr2=0x55822f25cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.342+0000 7fe81b817740 1 -- 172.21.15.37:0/4064067195 shutdown_connections 2024-03-28T18:10:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.342+0000 7fe81b817740 1 -- 172.21.15.37:0/4064067195 wait complete. 2024-03-28T18:10:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.343+0000 7fe81b817740 1 -- start start 2024-03-28T18:10:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.343+0000 7fe81b817740 1 Processor -- start 2024-03-28T18:10:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.343+0000 7fe81b817740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f253ba0 0x55822f261160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.343+0000 7fe81b817740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 0x55822f2616a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.343+0000 7fe81b817740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f259700 0x55822f2646c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.343+0000 7fe81b817740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55822f26bb20 con 0x55822f259700 2024-03-28T18:10:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.343+0000 7fe81b817740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55822f26b4c0 con 0x55822f253ba0 2024-03-28T18:10:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.343+0000 7fe81b817740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55822f26c180 con 0x55822f254590 2024-03-28T18:10:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.343+0000 7fe81b817740 10 client.0 _ll_register_callbacks cb 0x7fe7fc0096a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:10:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.343+0000 7fe80be00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 0x55822f2616a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.343+0000 7fe812800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f253ba0 0x55822f261160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.343+0000 7fe80be00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 0x55822f2616a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:43846/0 (socket says 172.21.15.37:43846) 2024-03-28T18:10:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.343+0000 7fe80be00640 1 -- 172.21.15.37:0/2171111723 learned_addr learned my addr 172.21.15.37:0/2171111723 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.344+0000 7fe812800640 1 --2- 172.21.15.37:0/2171111723 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f259700 0x55822f2646c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.344+0000 7fe80be00640 1 -- 172.21.15.37:0/2171111723 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f253ba0 msgr2=0x55822f261160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.344+0000 7fe80be00640 1 --2- 172.21.15.37:0/2171111723 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f253ba0 0x55822f261160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.344+0000 7fe80be00640 1 -- 172.21.15.37:0/2171111723 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f259700 msgr2=0x55822f2646c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.344+0000 7fe80be00640 1 --2- 172.21.15.37:0/2171111723 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f259700 0x55822f2646c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.344+0000 7fe80be00640 1 -- 172.21.15.37:0/2171111723 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55822f264cb0 con 0x55822f254590 2024-03-28T18:10:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.344+0000 7fe812800640 1 --2- 172.21.15.37:0/2171111723 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f259700 0x55822f2646c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.344+0000 7fe80be00640 1 --2- 172.21.15.37:0/2171111723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 0x55822f2616a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc012c00 tx=0x7fe7fc005410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=469d5a9d6edc190f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.344+0000 7fe7f7400640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:10:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.344+0000 7fe7f7400640 1 -- 172.21.15.37:0/2171111723 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc02c040 con 0x55822f254590 2024-03-28T18:10:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.344+0000 7fe7f7400640 1 -- 172.21.15.37:0/2171111723 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe7fc035070 con 0x55822f254590 2024-03-28T18:10:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.344+0000 7fe81b817740 1 -- 172.21.15.37:0/2171111723 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe7fc01c460 con 0x55822f254590 2024-03-28T18:10:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.345+0000 7fe7f7400640 1 -- 172.21.15.37:0/2171111723 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7fc01ca70 con 0x55822f254590 2024-03-28T18:10:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.345+0000 7fe81b817740 10 client.25009 fetch_fsmap learned FSMap version 22 2024-03-28T18:10:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.345+0000 7fe81b817740 1 -- 172.21.15.37:0/2171111723 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55822f2c5e10 con 0x55822f254590 2024-03-28T18:10:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.345+0000 7fe7f7400640 1 -- 172.21.15.37:0/2171111723 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe7fc01c460 con 0x55822f254590 2024-03-28T18:10:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.345+0000 7fe7f7400640 1 -- 172.21.15.37:0/2171111723 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe7fc040030 con 0x55822f254590 2024-03-28T18:10:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.345+0000 7fe81b817740 10 client.25009 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T18:10:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.345+0000 7fe81b817740 10 client.25009 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T18:10:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.345+0000 7fe81b817740 20 client.25009 populate_metadata read hostname 'smithi037' 2024-03-28T18:10:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.345+0000 7fe81b817740 1 --2- 172.21.15.37:0/2171111723 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x55822f2ca110 0x55822f2c5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.345+0000 7fe81b817740 4 client.25009 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:10:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.345+0000 7fe81b817740 1 -- 172.21.15.37:0/2171111723 --> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55822f1c61d0 con 0x55822f2ca110 2024-03-28T18:10:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.345+0000 7fe812800640 1 --2- 172.21.15.37:0/2171111723 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x55822f2ca110 0x55822f2c5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.346+0000 7fe812800640 1 --2- 172.21.15.37:0/2171111723 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x55822f2ca110 0x55822f2c5fe0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2a80e9c140e203af server_cookie=2cdf5f7146691afd in_seq=0 out_seq=0 2024-03-28T18:10:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.346+0000 7fe7f7400640 10 client.25009 ms_handle_connect on v2:172.21.15.170:6838/1575518279 2024-03-28T18:10:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.346+0000 7fe7f7400640 1 -- 172.21.15.37:0/2171111723 <== mds.2 v2:172.21.15.170:6838/1575518279 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55822f1c61d0 con 0x55822f2ca110 2024-03-28T18:10:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.346+0000 7fe7f7400640 10 client.25009 handle_command_reply: tid=0 2024-03-28T18:10:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.346+0000 7fe81b817740 1 client.25009 shutdown 2024-03-28T18:10:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.346+0000 7fe81b817740 1 -- 172.21.15.37:0/2171111723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 msgr2=0x55822f2616a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.346+0000 7fe81b817740 1 --2- 172.21.15.37:0/2171111723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 0x55822f2616a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc012c00 tx=0x7fe7fc005410 comp rx=0 tx=0).stop 2024-03-28T18:10:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.346+0000 7fe81b817740 1 -- 172.21.15.37:0/2171111723 shutdown_connections 2024-03-28T18:10:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.346+0000 7fe81b817740 1 --2- 172.21.15.37:0/2171111723 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x55822f2ca110 0x55822f2c5fe0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.346+0000 7fe81b817740 1 --2- 172.21.15.37:0/2171111723 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55822f259700 0x55822f2646c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.346+0000 7fe81b817740 1 --2- 172.21.15.37:0/2171111723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55822f254590 0x55822f2616a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.346+0000 7fe81b817740 1 --2- 172.21.15.37:0/2171111723 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55822f253ba0 0x55822f261160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.346+0000 7fe7f7400640 0 client.25009 ms_handle_reset on v2:172.21.15.170:6838/1575518279 2024-03-28T18:10:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.346+0000 7fe81b817740 1 -- 172.21.15.37:0/2171111723 >> 172.21.15.37:0/2171111723 conn(0x55822f24f110 msgr2=0x55822f269a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.347+0000 7fe81b817740 1 -- 172.21.15.37:0/2171111723 shutdown_connections 2024-03-28T18:10:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.347+0000 7fe81b817740 1 -- 172.21.15.37:0/2171111723 wait complete. 2024-03-28T18:10:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.347+0000 7fe81b817740 20 client.25009 trim_cache size 0 max 16384 2024-03-28T18:10:07.353 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:10:07.353 INFO:teuthology.orchestra.run.smithi037.stdout: "cluster_fsid": "b5ad257c-ed2c-11ee-95d0-87774f69a715", 2024-03-28T18:10:07.353 INFO:teuthology.orchestra.run.smithi037.stdout: "whoami": 2, 2024-03-28T18:10:07.353 INFO:teuthology.orchestra.run.smithi037.stdout: "id": 24550, 2024-03-28T18:10:07.354 INFO:teuthology.orchestra.run.smithi037.stdout: "want_state": "up:active", 2024-03-28T18:10:07.354 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "up:active", 2024-03-28T18:10:07.354 INFO:teuthology.orchestra.run.smithi037.stdout: "fs_name": "cephfs", 2024-03-28T18:10:07.354 INFO:teuthology.orchestra.run.smithi037.stdout: "rank_uptime": 102.94007010599999, 2024-03-28T18:10:07.354 INFO:teuthology.orchestra.run.smithi037.stdout: "mdsmap_epoch": 22, 2024-03-28T18:10:07.354 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap_epoch": 87, 2024-03-28T18:10:07.354 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap_epoch_barrier": 85, 2024-03-28T18:10:07.354 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 125.02969721700001 2024-03-28T18:10:07.354 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:10:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.347+0000 7fe818a00640 1 -- 172.21.15.37:0/1601655244 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe7f80a2880 msgr2=0x7fe7f80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.347+0000 7fe818a00640 1 --2- 172.21.15.37:0/1601655244 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe7f80a2880 0x7fe7f80a4d40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe804097c40 tx=0x7fe804000bc0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.347+0000 7fe818a00640 1 -- 172.21.15.37:0/1601655244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8141038c0 msgr2=0x7fe814199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.347+0000 7fe818a00640 1 --2- 172.21.15.37:0/1601655244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8141038c0 0x7fe814199bf0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe80400e9c0 tx=0x7fe80400ee80 comp rx=0 tx=0).stop 2024-03-28T18:10:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.347+0000 7fe818a00640 1 -- 172.21.15.37:0/1601655244 shutdown_connections 2024-03-28T18:10:07.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.347+0000 7fe818a00640 1 --2- 172.21.15.37:0/1601655244 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe7f80a2880 0x7fe7f80a4d40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.347+0000 7fe818a00640 1 --2- 172.21.15.37:0/1601655244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe814105470 0x7fe8141a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.347+0000 7fe818a00640 1 --2- 172.21.15.37:0/1601655244 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe814104ab0 0x7fe81419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.347+0000 7fe818a00640 1 --2- 172.21.15.37:0/1601655244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8141038c0 0x7fe814199bf0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.347+0000 7fe818a00640 1 -- 172.21.15.37:0/1601655244 >> 172.21.15.37:0/1601655244 conn(0x7fe8140fee50 msgr2=0x7fe814107d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:07.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.348+0000 7fe818a00640 1 -- 172.21.15.37:0/1601655244 shutdown_connections 2024-03-28T18:10:07.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.348+0000 7fe818a00640 1 -- 172.21.15.37:0/1601655244 wait complete. 2024-03-28T18:10:07.362 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:07 smithi037 ceph-mon[31460]: pgmap v373: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:10:07.362 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:07 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1934550118' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:07.362 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:07 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/564517843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:07.365 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T18:10:07.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:07 smithi059 ceph-mon[36190]: pgmap v373: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:10:07.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:07 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1934550118' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:07.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:07 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/564517843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:07.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:07 smithi170 ceph-mon[36280]: pgmap v373: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:10:07.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:07 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1934550118' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:07.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:07 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/564517843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:07.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.590+0000 7f7d12200640 1 -- 172.21.15.37:0/3125995278 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d0c102760 msgr2=0x7f7d0c104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.590+0000 7f7d12200640 1 --2- 172.21.15.37:0/3125995278 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d0c102760 0x7f7d0c104bd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8009900 tx=0x7f7cf802bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:07.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.591+0000 7f7d12200640 1 -- 172.21.15.37:0/3125995278 shutdown_connections 2024-03-28T18:10:07.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.591+0000 7f7d12200640 1 --2- 172.21.15.37:0/3125995278 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d0c102760 0x7f7d0c104bd0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.591+0000 7f7d12200640 1 --2- 172.21.15.37:0/3125995278 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d0c105110 0x7f7d0c105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.591+0000 7f7d12200640 1 --2- 172.21.15.37:0/3125995278 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d0c0ffdf0 0x7f7d0c102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.591+0000 7f7d12200640 1 -- 172.21.15.37:0/3125995278 >> 172.21.15.37:0/3125995278 conn(0x7f7d0c0fba10 msgr2=0x7f7d0c0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:07.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.591+0000 7f7d12200640 1 -- 172.21.15.37:0/3125995278 shutdown_connections 2024-03-28T18:10:07.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.591+0000 7f7d12200640 1 -- 172.21.15.37:0/3125995278 wait complete. 2024-03-28T18:10:07.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.592+0000 7f7d12200640 1 Processor -- start 2024-03-28T18:10:07.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.593+0000 7f7d12200640 1 -- start start 2024-03-28T18:10:07.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.593+0000 7f7d12200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d0c0ffdf0 0x7f7d0c19de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.593+0000 7f7d12200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d0c102760 0x7f7d0c19e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.593+0000 7f7d12200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d0c105110 0x7f7d0c1a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.593+0000 7f7d12200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0c113840 con 0x7f7d0c105110 2024-03-28T18:10:07.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.593+0000 7f7d12200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0c1136c0 con 0x7f7d0c102760 2024-03-28T18:10:07.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.593+0000 7f7d12200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0c1139c0 con 0x7f7d0c0ffdf0 2024-03-28T18:10:07.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.593+0000 7f7d10e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d0c0ffdf0 0x7f7d0c19de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.593+0000 7f7d10e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d0c0ffdf0 0x7f7d0c19de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:43864/0 (socket says 172.21.15.37:43864) 2024-03-28T18:10:07.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.593+0000 7f7d10e00640 1 -- 172.21.15.37:0/4180133823 learned_addr learned my addr 172.21.15.37:0/4180133823 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:07.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.593+0000 7f7d0be00640 1 --2- 172.21.15.37:0/4180133823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d0c102760 0x7f7d0c19e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.593+0000 7f7d11800640 1 --2- 172.21.15.37:0/4180133823 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d0c105110 0x7f7d0c1a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.594+0000 7f7d10e00640 1 -- 172.21.15.37:0/4180133823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d0c102760 msgr2=0x7f7d0c19e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.594+0000 7f7d10e00640 1 --2- 172.21.15.37:0/4180133823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d0c102760 0x7f7d0c19e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.594+0000 7f7d10e00640 1 -- 172.21.15.37:0/4180133823 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d0c105110 msgr2=0x7f7d0c1a53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.594+0000 7f7d10e00640 1 --2- 172.21.15.37:0/4180133823 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d0c105110 0x7f7d0c1a53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.594+0000 7f7d10e00640 1 -- 172.21.15.37:0/4180133823 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0c1a5a60 con 0x7f7d0c0ffdf0 2024-03-28T18:10:07.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.594+0000 7f7d11800640 1 --2- 172.21.15.37:0/4180133823 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d0c105110 0x7f7d0c1a53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:10:07.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.594+0000 7f7d10e00640 1 --2- 172.21.15.37:0/4180133823 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d0c0ffdf0 0x7f7d0c19de40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7d0000e9f0 tx=0x7f7d0000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:07.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.594+0000 7f7d09600640 1 -- 172.21.15.37:0/4180133823 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0000cd80 con 0x7f7d0c0ffdf0 2024-03-28T18:10:07.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.594+0000 7f7d09600640 1 -- 172.21.15.37:0/4180133823 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d00005500 con 0x7f7d0c0ffdf0 2024-03-28T18:10:07.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.594+0000 7f7d12200640 1 -- 172.21.15.37:0/4180133823 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d0c1a5d50 con 0x7f7d0c0ffdf0 2024-03-28T18:10:07.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.594+0000 7f7d09600640 1 -- 172.21.15.37:0/4180133823 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d00010690 con 0x7f7d0c0ffdf0 2024-03-28T18:10:07.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.594+0000 7f7d12200640 1 -- 172.21.15.37:0/4180133823 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d0c1a61c0 con 0x7f7d0c0ffdf0 2024-03-28T18:10:07.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.598+0000 7f7d09600640 1 -- 172.21.15.37:0/4180133823 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7d000046c0 con 0x7f7d0c0ffdf0 2024-03-28T18:10:07.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.598+0000 7f7d16563740 1 Processor -- start 2024-03-28T18:10:07.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.598+0000 7f7d16563740 1 -- start start 2024-03-28T18:10:07.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.598+0000 7f7d16563740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad88169d80 0x55ad8816a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.598+0000 7f7d16563740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 0x55ad8816f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.598+0000 7f7d16563740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad8816f8e0 0x55ad88171d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.598+0000 7f7d16563740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad880cec90 con 0x55ad88169d80 2024-03-28T18:10:07.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.598+0000 7f7d16563740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad88077980 con 0x55ad8816f8e0 2024-03-28T18:10:07.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.598+0000 7f7d16563740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad880cad30 con 0x55ad8816a770 2024-03-28T18:10:07.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.599+0000 7f7d10e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 0x55ad8816f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.599+0000 7f7d11800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad88169d80 0x55ad8816a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.599+0000 7f7d0be00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad8816f8e0 0x55ad88171d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.599+0000 7f7d10e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 0x55ad8816f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:43874/0 (socket says 172.21.15.37:43874) 2024-03-28T18:10:07.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.599+0000 7f7d10e00640 1 -- 172.21.15.37:0/3923297674 learned_addr learned my addr 172.21.15.37:0/3923297674 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:07.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.599+0000 7f7d10e00640 1 -- 172.21.15.37:0/3923297674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad8816f8e0 msgr2=0x55ad88171d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.599+0000 7f7d10e00640 1 --2- 172.21.15.37:0/3923297674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad8816f8e0 0x55ad88171d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.599+0000 7f7d10e00640 1 -- 172.21.15.37:0/3923297674 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad88169d80 msgr2=0x55ad8816a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.599+0000 7f7d10e00640 1 --2- 172.21.15.37:0/3923297674 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad88169d80 0x55ad8816a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.599+0000 7f7d10e00640 1 -- 172.21.15.37:0/3923297674 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad8808f510 con 0x55ad8816a770 2024-03-28T18:10:07.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.599+0000 7f7d09600640 1 --2- 172.21.15.37:0/4180133823 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f7cf40a2800 0x7f7cf40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.599+0000 7f7d10e00640 1 --2- 172.21.15.37:0/3923297674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 0x55ad8816f270 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7d000d2e50 tx=0x7f7d000d2f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbb687feaa5ea10b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:07.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.600+0000 7f7d11800640 1 --2- 172.21.15.37:0/4180133823 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f7cf40a2800 0x7f7cf40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.600+0000 7f7d09600640 1 -- 172.21.15.37:0/4180133823 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7d000cef40 con 0x7f7d0c0ffdf0 2024-03-28T18:10:07.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.600+0000 7f7cfe000640 1 -- 172.21.15.37:0/3923297674 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d00097b30 con 0x55ad8816a770 2024-03-28T18:10:07.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.600+0000 7f7cfe000640 1 -- 172.21.15.37:0/3923297674 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d0009bf90 con 0x55ad8816a770 2024-03-28T18:10:07.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.600+0000 7f7cfe000640 1 -- 172.21.15.37:0/3923297674 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0009b1f0 con 0x55ad8816a770 2024-03-28T18:10:07.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.600+0000 7f7d16563740 1 -- 172.21.15.37:0/3923297674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 msgr2=0x55ad8816f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.600+0000 7f7d16563740 1 --2- 172.21.15.37:0/3923297674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 0x55ad8816f270 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7d000d2e50 tx=0x7f7d000d2f40 comp rx=0 tx=0).stop 2024-03-28T18:10:07.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.600+0000 7f7d11800640 1 --2- 172.21.15.37:0/4180133823 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f7cf40a2800 0x7f7cf40a4cc0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x55ad8816afd0 tx=0x7f7d04009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:07.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.601+0000 7f7d16563740 1 -- 172.21.15.37:0/3923297674 shutdown_connections 2024-03-28T18:10:07.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.601+0000 7f7d16563740 1 --2- 172.21.15.37:0/3923297674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad8816f8e0 0x55ad88171d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.601+0000 7f7d16563740 1 --2- 172.21.15.37:0/3923297674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 0x55ad8816f270 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.601+0000 7f7d16563740 1 --2- 172.21.15.37:0/3923297674 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad88169d80 0x55ad8816a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.601+0000 7f7d16563740 1 -- 172.21.15.37:0/3923297674 >> 172.21.15.37:0/3923297674 conn(0x55ad881652f0 msgr2=0x55ad881656f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:07.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.601+0000 7f7d16563740 1 -- 172.21.15.37:0/3923297674 shutdown_connections 2024-03-28T18:10:07.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.601+0000 7f7d16563740 1 -- 172.21.15.37:0/3923297674 wait complete. 2024-03-28T18:10:07.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.601+0000 7f7d16563740 1 -- start start 2024-03-28T18:10:07.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.601+0000 7f7d16563740 1 Processor -- start 2024-03-28T18:10:07.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.602+0000 7f7d16563740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad88169d80 0x55ad88174290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.602+0000 7f7d16563740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 0x55ad881747d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.602+0000 7f7d16563740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad8816f8e0 0x55ad8817b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.602+0000 7f7d16563740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad88172890 con 0x55ad88169d80 2024-03-28T18:10:07.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.602+0000 7f7d16563740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad8808add0 con 0x55ad8816f8e0 2024-03-28T18:10:07.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.602+0000 7f7d16563740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad88172a10 con 0x55ad8816a770 2024-03-28T18:10:07.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.602+0000 7f7d16563740 10 client.0 _ll_register_callbacks cb 0x55ad87de2ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:10:07.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.602+0000 7f7d0be00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 0x55ad881747d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.602+0000 7f7d11800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad8816f8e0 0x55ad8817b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.602+0000 7f7d0be00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 0x55ad881747d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:43876/0 (socket says 172.21.15.37:43876) 2024-03-28T18:10:07.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.602+0000 7f7d0be00640 1 -- 172.21.15.37:0/1714322217 learned_addr learned my addr 172.21.15.37:0/1714322217 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:07.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.602+0000 7f7d10e00640 1 --2- 172.21.15.37:0/1714322217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad88169d80 0x55ad88174290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.603+0000 7f7d0be00640 1 -- 172.21.15.37:0/1714322217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad8816f8e0 msgr2=0x55ad8817b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.603+0000 7f7d0be00640 1 --2- 172.21.15.37:0/1714322217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad8816f8e0 0x55ad8817b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.603+0000 7f7d0be00640 1 -- 172.21.15.37:0/1714322217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad88169d80 msgr2=0x55ad88174290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.603+0000 7f7d0be00640 1 --2- 172.21.15.37:0/1714322217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad88169d80 0x55ad88174290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.603+0000 7f7d0be00640 1 -- 172.21.15.37:0/1714322217 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad87de32d0 con 0x55ad8816a770 2024-03-28T18:10:07.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.603+0000 7f7d10e00640 1 --2- 172.21.15.37:0/1714322217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad88169d80 0x55ad88174290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:10:07.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.603+0000 7f7d11800640 1 --2- 172.21.15.37:0/1714322217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad8816f8e0 0x55ad8817b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:07.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.603+0000 7f7d0be00640 1 --2- 172.21.15.37:0/1714322217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 0x55ad881747d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8005ab0 tx=0x7f7cf8004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70546923aa3650e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:07.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.603+0000 7f7cf2a00640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:10:07.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.603+0000 7f7cf2a00640 1 -- 172.21.15.37:0/1714322217 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8005d50 con 0x55ad8816a770 2024-03-28T18:10:07.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.603+0000 7f7cf2a00640 1 -- 172.21.15.37:0/1714322217 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7cf80046c0 con 0x55ad8816a770 2024-03-28T18:10:07.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.603+0000 7f7d16563740 1 -- 172.21.15.37:0/1714322217 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7d0009bf90 con 0x55ad8816a770 2024-03-28T18:10:07.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.604+0000 7f7cf2a00640 1 -- 172.21.15.37:0/1714322217 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7cf80048c0 con 0x55ad8816a770 2024-03-28T18:10:07.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.604+0000 7f7d16563740 10 client.25021 fetch_fsmap learned FSMap version 22 2024-03-28T18:10:07.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.604+0000 7f7d16563740 1 -- 172.21.15.37:0/1714322217 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55ad87ead6b0 con 0x55ad8816a770 2024-03-28T18:10:07.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.604+0000 7f7cf2a00640 1 -- 172.21.15.37:0/1714322217 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d0009bf90 con 0x55ad8816a770 2024-03-28T18:10:07.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.604+0000 7f7cf2a00640 1 -- 172.21.15.37:0/1714322217 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7cf80038c0 con 0x55ad8816a770 2024-03-28T18:10:07.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.604+0000 7f7d16563740 10 client.25021 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T18:10:07.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.604+0000 7f7d16563740 10 client.25021 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T18:10:07.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.604+0000 7f7d16563740 20 client.25021 populate_metadata read hostname 'smithi037' 2024-03-28T18:10:07.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.605+0000 7f7d16563740 1 --2- 172.21.15.37:0/1714322217 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55ad88180a80 0x55ad8817fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.605+0000 7f7d16563740 4 client.25021 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:10:07.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.605+0000 7f7d16563740 1 -- 172.21.15.37:0/1714322217 --> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ad88070520 con 0x55ad88180a80 2024-03-28T18:10:07.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.605+0000 7f7d10e00640 1 --2- 172.21.15.37:0/1714322217 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55ad88180a80 0x55ad8817fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.605+0000 7f7d10e00640 1 --2- 172.21.15.37:0/1714322217 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55ad88180a80 0x55ad8817fb50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ea85eaacd1cfde0f server_cookie=87a95b06608ab86e in_seq=0 out_seq=0 2024-03-28T18:10:07.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.605+0000 7f7cf2a00640 10 client.25021 ms_handle_connect on v2:172.21.15.59:6838/2091629078 2024-03-28T18:10:07.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.607+0000 7f7cf2a00640 1 -- 172.21.15.37:0/1714322217 <== mds.1 v2:172.21.15.59:6838/2091629078 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ad88070520 con 0x55ad88180a80 2024-03-28T18:10:07.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.607+0000 7f7cf2a00640 10 client.25021 handle_command_reply: tid=0 2024-03-28T18:10:07.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.607+0000 7f7d16563740 1 client.25021 shutdown 2024-03-28T18:10:07.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.607+0000 7f7d16563740 1 -- 172.21.15.37:0/1714322217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 msgr2=0x55ad881747d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.607+0000 7f7d16563740 1 --2- 172.21.15.37:0/1714322217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 0x55ad881747d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8005ab0 tx=0x7f7cf8004270 comp rx=0 tx=0).stop 2024-03-28T18:10:07.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.607+0000 7f7d16563740 1 -- 172.21.15.37:0/1714322217 shutdown_connections 2024-03-28T18:10:07.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.607+0000 7f7d16563740 1 --2- 172.21.15.37:0/1714322217 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55ad88180a80 0x55ad8817fb50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.607+0000 7f7d16563740 1 --2- 172.21.15.37:0/1714322217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad8816f8e0 0x55ad8817b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.607+0000 7f7d16563740 1 --2- 172.21.15.37:0/1714322217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 0x55ad881747d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.607+0000 7f7d16563740 1 --2- 172.21.15.37:0/1714322217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad88169d80 0x55ad88174290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.607+0000 7f7cf2a00640 0 client.25021 ms_handle_reset on v2:172.21.15.59:6838/2091629078 2024-03-28T18:10:07.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.607+0000 7f7d16563740 1 -- 172.21.15.37:0/1714322217 >> 172.21.15.37:0/1714322217 conn(0x55ad881652f0 msgr2=0x55ad881644f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:07.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.607+0000 7f7d16563740 1 -- 172.21.15.37:0/1714322217 shutdown_connections 2024-03-28T18:10:07.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.607+0000 7f7d16563740 1 -- 172.21.15.37:0/1714322217 wait complete. 2024-03-28T18:10:07.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.607+0000 7f7d16563740 20 client.25021 trim_cache size 0 max 16384 2024-03-28T18:10:07.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.627+0000 7f7d16563740 1 Processor -- start 2024-03-28T18:10:07.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.627+0000 7f7d16563740 1 -- start start 2024-03-28T18:10:07.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.627+0000 7f7d16563740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad881652f0 0x55ad881656d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.627+0000 7f7d16563740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad88169d80 0x55ad8817b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.627+0000 7f7d16563740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 0x55ad88173ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.627+0000 7f7d16563740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad881802a0 con 0x55ad881652f0 2024-03-28T18:10:07.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.627+0000 7f7d16563740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad88180900 con 0x55ad88169d80 2024-03-28T18:10:07.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.627+0000 7f7d16563740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad88180f60 con 0x55ad8816a770 2024-03-28T18:10:07.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.627+0000 7f7d10e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad881652f0 0x55ad881656d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.627+0000 7f7d11800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 0x55ad88173ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.627+0000 7f7d0be00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad88169d80 0x55ad8817b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.627+0000 7f7d0be00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad88169d80 0x55ad8817b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:55268/0 (socket says 172.21.15.37:55268) 2024-03-28T18:10:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.627+0000 7f7d11800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 0x55ad88173ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:43884/0 (socket says 172.21.15.37:43884) 2024-03-28T18:10:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.627+0000 7f7d0be00640 1 -- 172.21.15.37:0/3840363987 learned_addr learned my addr 172.21.15.37:0/3840363987 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.628+0000 7f7d11800640 1 -- 172.21.15.37:0/3840363987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad88169d80 msgr2=0x55ad8817b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.628+0000 7f7d11800640 1 --2- 172.21.15.37:0/3840363987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad88169d80 0x55ad8817b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.628+0000 7f7d11800640 1 -- 172.21.15.37:0/3840363987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad881652f0 msgr2=0x55ad881656d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.628+0000 7f7d11800640 1 --2- 172.21.15.37:0/3840363987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad881652f0 0x55ad881656d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.628+0000 7f7d11800640 1 -- 172.21.15.37:0/3840363987 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad88175280 con 0x55ad8816a770 2024-03-28T18:10:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.628+0000 7f7d0be00640 1 --2- 172.21.15.37:0/3840363987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad88169d80 0x55ad8817b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:07.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.628+0000 7f7d11800640 1 --2- 172.21.15.37:0/3840363987 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 0x55ad88173ea0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7d04005ee0 tx=0x7f7d04015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=223262e7ab2db3cb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:07.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.628+0000 7f7cfd600640 1 -- 172.21.15.37:0/3840363987 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0402c070 con 0x55ad8816a770 2024-03-28T18:10:07.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.628+0000 7f7cfd600640 1 -- 172.21.15.37:0/3840363987 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d04010db0 con 0x55ad8816a770 2024-03-28T18:10:07.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.628+0000 7f7cfd600640 1 -- 172.21.15.37:0/3840363987 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0401e740 con 0x55ad8816a770 2024-03-28T18:10:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.629+0000 7f7d16563740 1 -- 172.21.15.37:0/3840363987 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 msgr2=0x55ad88173ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.629+0000 7f7d16563740 1 --2- 172.21.15.37:0/3840363987 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 0x55ad88173ea0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7d04005ee0 tx=0x7f7d04015000 comp rx=0 tx=0).stop 2024-03-28T18:10:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.629+0000 7f7d16563740 1 -- 172.21.15.37:0/3840363987 shutdown_connections 2024-03-28T18:10:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.629+0000 7f7d16563740 1 --2- 172.21.15.37:0/3840363987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad88169d80 0x55ad8817b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.629+0000 7f7d16563740 1 --2- 172.21.15.37:0/3840363987 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad8816a770 0x55ad88173ea0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.629+0000 7f7d16563740 1 --2- 172.21.15.37:0/3840363987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad881652f0 0x55ad881656d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.629+0000 7f7d16563740 1 -- 172.21.15.37:0/3840363987 >> 172.21.15.37:0/3840363987 conn(0x55ad8817fec0 msgr2=0x55ad88172db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.630+0000 7f7d16563740 1 -- 172.21.15.37:0/3840363987 shutdown_connections 2024-03-28T18:10:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.630+0000 7f7d16563740 1 -- 172.21.15.37:0/3840363987 wait complete. 2024-03-28T18:10:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.630+0000 7f7d16563740 1 -- start start 2024-03-28T18:10:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.630+0000 7f7d16563740 1 Processor -- start 2024-03-28T18:10:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.631+0000 7f7d16563740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad881652f0 0x55ad8817b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.631+0000 7f7d16563740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad88169d80 0x55ad8817b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.631+0000 7f7d16563740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad8816a770 0x55ad88177df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.631+0000 7f7d16563740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad88181c20 con 0x55ad88169d80 2024-03-28T18:10:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.631+0000 7f7d16563740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad881815c0 con 0x55ad8816a770 2024-03-28T18:10:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.631+0000 7f7d16563740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad88182280 con 0x55ad881652f0 2024-03-28T18:10:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.631+0000 7f7d16563740 10 client.0 _ll_register_callbacks cb 0x7f7cf8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:10:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.631+0000 7f7d10e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad881652f0 0x55ad8817b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.631+0000 7f7d11800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad8816a770 0x55ad88177df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.631+0000 7f7d10e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad881652f0 0x55ad8817b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:43890/0 (socket says 172.21.15.37:43890) 2024-03-28T18:10:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.631+0000 7f7d0be00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad88169d80 0x55ad8817b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.631+0000 7f7d10e00640 1 -- 172.21.15.37:0/975172629 learned_addr learned my addr 172.21.15.37:0/975172629 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.632+0000 7f7d10e00640 1 -- 172.21.15.37:0/975172629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad8816a770 msgr2=0x55ad88177df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.632+0000 7f7d10e00640 1 --2- 172.21.15.37:0/975172629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad8816a770 0x55ad88177df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.632+0000 7f7d10e00640 1 -- 172.21.15.37:0/975172629 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad88169d80 msgr2=0x55ad8817b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.632+0000 7f7d10e00640 1 --2- 172.21.15.37:0/975172629 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad88169d80 0x55ad8817b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.632+0000 7f7d10e00640 1 -- 172.21.15.37:0/975172629 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad88178620 con 0x55ad881652f0 2024-03-28T18:10:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.632+0000 7f7d11800640 1 --2- 172.21.15.37:0/975172629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad8816a770 0x55ad88177df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.632+0000 7f7d10e00640 1 --2- 172.21.15.37:0/975172629 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad881652f0 0x55ad8817b280 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7d000d2e20 tx=0x7f7d00009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2968a51904bc3200 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.632+0000 7f7cf2a00640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:10:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.632+0000 7f7cf2a00640 1 -- 172.21.15.37:0/975172629 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0009b3d0 con 0x55ad881652f0 2024-03-28T18:10:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.632+0000 7f7cf2a00640 1 -- 172.21.15.37:0/975172629 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7d00099b90 con 0x55ad881652f0 2024-03-28T18:10:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.632+0000 7f7d16563740 1 -- 172.21.15.37:0/975172629 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7d04010db0 con 0x55ad881652f0 2024-03-28T18:10:07.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.633+0000 7f7cf2a00640 1 -- 172.21.15.37:0/975172629 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d0009b660 con 0x55ad881652f0 2024-03-28T18:10:07.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.633+0000 7f7d16563740 10 client.25027 fetch_fsmap learned FSMap version 22 2024-03-28T18:10:07.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.633+0000 7f7d16563740 1 -- 172.21.15.37:0/975172629 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55ad881dc8f0 con 0x55ad881652f0 2024-03-28T18:10:07.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.633+0000 7f7cf2a00640 1 -- 172.21.15.37:0/975172629 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d04010db0 con 0x55ad881652f0 2024-03-28T18:10:07.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.633+0000 7f7cf2a00640 1 -- 172.21.15.37:0/975172629 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7d000d2610 con 0x55ad881652f0 2024-03-28T18:10:07.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.633+0000 7f7d16563740 10 client.25027 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T18:10:07.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.633+0000 7f7d16563740 10 client.25027 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T18:10:07.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.633+0000 7f7d16563740 20 client.25027 populate_metadata read hostname 'smithi037' 2024-03-28T18:10:07.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.633+0000 7f7d16563740 1 --2- 172.21.15.37:0/975172629 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55ad8816f8e0 0x55ad881dcac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.633+0000 7f7d16563740 4 client.25027 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:10:07.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.633+0000 7f7d16563740 1 -- 172.21.15.37:0/975172629 --> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ad880dc1d0 con 0x55ad8816f8e0 2024-03-28T18:10:07.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.633+0000 7f7d0be00640 1 --2- 172.21.15.37:0/975172629 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55ad8816f8e0 0x55ad881dcac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.634+0000 7f7d0be00640 1 --2- 172.21.15.37:0/975172629 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55ad8816f8e0 0x55ad881dcac0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7e6de686c8c0d77 server_cookie=5204bca4a1e80fc7 in_seq=0 out_seq=0 2024-03-28T18:10:07.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.634+0000 7f7cf2a00640 10 client.25027 ms_handle_connect on v2:172.21.15.59:6838/2091629078 2024-03-28T18:10:07.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.634+0000 7f7cf2a00640 1 -- 172.21.15.37:0/975172629 <== mds.1 v2:172.21.15.59:6838/2091629078 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55ad880dc1d0 con 0x55ad8816f8e0 2024-03-28T18:10:07.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.634+0000 7f7cf2a00640 10 client.25027 handle_command_reply: tid=0 2024-03-28T18:10:07.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.634+0000 7f7d16563740 1 client.25027 shutdown 2024-03-28T18:10:07.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.634+0000 7f7d16563740 1 -- 172.21.15.37:0/975172629 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad881652f0 msgr2=0x55ad8817b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.634+0000 7f7d16563740 1 --2- 172.21.15.37:0/975172629 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad881652f0 0x55ad8817b280 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7d000d2e20 tx=0x7f7d00009310 comp rx=0 tx=0).stop 2024-03-28T18:10:07.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.634+0000 7f7d16563740 1 -- 172.21.15.37:0/975172629 shutdown_connections 2024-03-28T18:10:07.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.634+0000 7f7d16563740 1 --2- 172.21.15.37:0/975172629 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55ad8816f8e0 0x55ad881dcac0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.634+0000 7f7d16563740 1 --2- 172.21.15.37:0/975172629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad8816a770 0x55ad88177df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.634+0000 7f7d16563740 1 --2- 172.21.15.37:0/975172629 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ad88169d80 0x55ad8817b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.634+0000 7f7cf2a00640 0 client.25027 ms_handle_reset on v2:172.21.15.59:6838/2091629078 2024-03-28T18:10:07.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.634+0000 7f7d16563740 1 --2- 172.21.15.37:0/975172629 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad881652f0 0x55ad8817b280 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.634+0000 7f7d16563740 1 -- 172.21.15.37:0/975172629 >> 172.21.15.37:0/975172629 conn(0x55ad8817fec0 msgr2=0x55ad881656d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:07.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.635+0000 7f7d16563740 1 -- 172.21.15.37:0/975172629 shutdown_connections 2024-03-28T18:10:07.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.635+0000 7f7d16563740 1 -- 172.21.15.37:0/975172629 wait complete. 2024-03-28T18:10:07.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.635+0000 7f7d16563740 20 client.25027 trim_cache size 0 max 16384 2024-03-28T18:10:07.639 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:10:07.639 INFO:teuthology.orchestra.run.smithi037.stdout: "cluster_fsid": "b5ad257c-ed2c-11ee-95d0-87774f69a715", 2024-03-28T18:10:07.640 INFO:teuthology.orchestra.run.smithi037.stdout: "whoami": 1, 2024-03-28T18:10:07.640 INFO:teuthology.orchestra.run.smithi037.stdout: "id": 24569, 2024-03-28T18:10:07.640 INFO:teuthology.orchestra.run.smithi037.stdout: "want_state": "up:active", 2024-03-28T18:10:07.640 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "up:active", 2024-03-28T18:10:07.640 INFO:teuthology.orchestra.run.smithi037.stdout: "fs_name": "cephfs", 2024-03-28T18:10:07.640 INFO:teuthology.orchestra.run.smithi037.stdout: "rank_uptime": 104.255236932, 2024-03-28T18:10:07.640 INFO:teuthology.orchestra.run.smithi037.stdout: "mdsmap_epoch": 22, 2024-03-28T18:10:07.640 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap_epoch": 87, 2024-03-28T18:10:07.640 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap_epoch_barrier": 85, 2024-03-28T18:10:07.640 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 113.99352116599999 2024-03-28T18:10:07.640 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:10:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.635+0000 7f7d12200640 1 -- 172.21.15.37:0/4180133823 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f7cf40a2800 msgr2=0x7f7cf40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.635+0000 7f7d12200640 1 --2- 172.21.15.37:0/4180133823 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f7cf40a2800 0x7f7cf40a4cc0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x55ad8816afd0 tx=0x7f7d04009500 comp rx=0 tx=0).stop 2024-03-28T18:10:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.635+0000 7f7d12200640 1 -- 172.21.15.37:0/4180133823 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d0c0ffdf0 msgr2=0x7f7d0c19de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.635+0000 7f7d12200640 1 --2- 172.21.15.37:0/4180133823 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d0c0ffdf0 0x7f7d0c19de40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7d0000e9f0 tx=0x7f7d0000edf0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.635+0000 7f7d12200640 1 -- 172.21.15.37:0/4180133823 shutdown_connections 2024-03-28T18:10:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.635+0000 7f7d12200640 1 --2- 172.21.15.37:0/4180133823 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f7cf40a2800 0x7f7cf40a4cc0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.635+0000 7f7d12200640 1 --2- 172.21.15.37:0/4180133823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d0c102760 0x7f7d0c19e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.635+0000 7f7d12200640 1 --2- 172.21.15.37:0/4180133823 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d0c105110 0x7f7d0c1a53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.635+0000 7f7d12200640 1 --2- 172.21.15.37:0/4180133823 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d0c0ffdf0 0x7f7d0c19de40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.635+0000 7f7d12200640 1 -- 172.21.15.37:0/4180133823 >> 172.21.15.37:0/4180133823 conn(0x7f7d0c0fba10 msgr2=0x7f7d0c100770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:07.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.636+0000 7f7d12200640 1 -- 172.21.15.37:0/4180133823 shutdown_connections 2024-03-28T18:10:07.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.636+0000 7f7d12200640 1 -- 172.21.15.37:0/4180133823 wait complete. 2024-03-28T18:10:07.653 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-28T18:10:07.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.875+0000 7f819de00640 1 -- 172.21.15.37:0/761289478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8198102790 msgr2=0x7f8198102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.875+0000 7f819de00640 1 --2- 172.21.15.37:0/761289478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8198102790 0x7f8198102c10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8180009900 tx=0x7f818002bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:07.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.876+0000 7f819de00640 1 -- 172.21.15.37:0/761289478 shutdown_connections 2024-03-28T18:10:07.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.876+0000 7f819de00640 1 --2- 172.21.15.37:0/761289478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8198102790 0x7f8198102c10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.876+0000 7f819de00640 1 --2- 172.21.15.37:0/761289478 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8198103150 0x7f81981099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.876+0000 7f819de00640 1 --2- 172.21.15.37:0/761289478 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81981015a0 0x7f81981019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.876+0000 7f819de00640 1 -- 172.21.15.37:0/761289478 >> 172.21.15.37:0/761289478 conn(0x7f81980fcb10 msgr2=0x7f81980fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:07.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.876+0000 7f819de00640 1 -- 172.21.15.37:0/761289478 shutdown_connections 2024-03-28T18:10:07.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.876+0000 7f819de00640 1 -- 172.21.15.37:0/761289478 wait complete. 2024-03-28T18:10:07.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.877+0000 7f819de00640 1 Processor -- start 2024-03-28T18:10:07.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.877+0000 7f819de00640 1 -- start start 2024-03-28T18:10:07.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.878+0000 7f819de00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81981015a0 0x7f8198075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.878+0000 7f819de00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8198102790 0x7f8198073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.878+0000 7f819de00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8198103150 0x7f8198073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.878+0000 7f819de00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819810d250 con 0x7f8198102790 2024-03-28T18:10:07.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.878+0000 7f819de00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819810d0d0 con 0x7f8198103150 2024-03-28T18:10:07.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.878+0000 7f819de00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819810d3d0 con 0x7f81981015a0 2024-03-28T18:10:07.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.878+0000 7f819d400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8198103150 0x7f8198073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.878+0000 7f819ca00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81981015a0 0x7f8198075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.878+0000 7f819d400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8198103150 0x7f8198073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:55300/0 (socket says 172.21.15.37:55300) 2024-03-28T18:10:07.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.878+0000 7f819ca00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81981015a0 0x7f8198075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:43912/0 (socket says 172.21.15.37:43912) 2024-03-28T18:10:07.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.878+0000 7f819d400640 1 -- 172.21.15.37:0/40032120 learned_addr learned my addr 172.21.15.37:0/40032120 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:07.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.878+0000 7f818fe00640 1 --2- 172.21.15.37:0/40032120 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8198102790 0x7f8198073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.879+0000 7f819d400640 1 -- 172.21.15.37:0/40032120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81981015a0 msgr2=0x7f8198075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.879+0000 7f819d400640 1 --2- 172.21.15.37:0/40032120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81981015a0 0x7f8198075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.879+0000 7f819d400640 1 -- 172.21.15.37:0/40032120 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8198102790 msgr2=0x7f8198073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.879+0000 7f819d400640 1 --2- 172.21.15.37:0/40032120 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8198102790 0x7f8198073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.879+0000 7f819d400640 1 -- 172.21.15.37:0/40032120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8198074860 con 0x7f8198103150 2024-03-28T18:10:07.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.879+0000 7f819d400640 1 --2- 172.21.15.37:0/40032120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8198103150 0x7f8198073fd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f819000eb10 tx=0x7f819000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:07.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.879+0000 7f8196000640 1 -- 172.21.15.37:0/40032120 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8190004b20 con 0x7f8198103150 2024-03-28T18:10:07.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.879+0000 7f8196000640 1 -- 172.21.15.37:0/40032120 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8190004cc0 con 0x7f8198103150 2024-03-28T18:10:07.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.879+0000 7f819de00640 1 -- 172.21.15.37:0/40032120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81981a19e0 con 0x7f8198103150 2024-03-28T18:10:07.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.879+0000 7f8196000640 1 -- 172.21.15.37:0/40032120 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8190010760 con 0x7f8198103150 2024-03-28T18:10:07.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.880+0000 7f819de00640 1 -- 172.21.15.37:0/40032120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81981a1f00 con 0x7f8198103150 2024-03-28T18:10:07.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.883+0000 7f81a21e7740 1 Processor -- start 2024-03-28T18:10:07.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.884+0000 7f8196000640 1 -- 172.21.15.37:0/40032120 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8190005020 con 0x7f8198103150 2024-03-28T18:10:07.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.885+0000 7f81a21e7740 1 -- start start 2024-03-28T18:10:07.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.885+0000 7f81a21e7740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55876172dd80 0x55876172e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.885+0000 7f81a21e7740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55876172e770 0x558761733270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.885+0000 7f81a21e7740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587617338e0 0x558761735d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.885+0000 7f81a21e7740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x558761692c90 con 0x55876172dd80 2024-03-28T18:10:07.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.885+0000 7f81a21e7740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55876163b980 con 0x5587617338e0 2024-03-28T18:10:07.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.885+0000 7f81a21e7740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55876168ed30 con 0x55876172e770 2024-03-28T18:10:07.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.885+0000 7f818fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55876172dd80 0x55876172e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.885+0000 7f819ca00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55876172e770 0x558761733270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.885+0000 7f818fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55876172dd80 0x55876172e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54368/0 (socket says 172.21.15.37:54368) 2024-03-28T18:10:07.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.885+0000 7f818fe00640 1 -- 172.21.15.37:0/483000201 learned_addr learned my addr 172.21.15.37:0/483000201 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:07.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.885+0000 7f818fe00640 1 --2- 172.21.15.37:0/483000201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587617338e0 0x558761735d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.886+0000 7f819ca00640 1 -- 172.21.15.37:0/483000201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587617338e0 msgr2=0x558761735d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.886+0000 7f819ca00640 1 --2- 172.21.15.37:0/483000201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587617338e0 0x558761735d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.886+0000 7f819ca00640 1 -- 172.21.15.37:0/483000201 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55876172dd80 msgr2=0x55876172e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.886+0000 7f819ca00640 1 --2- 172.21.15.37:0/483000201 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55876172dd80 0x55876172e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.886+0000 7f819ca00640 1 -- 172.21.15.37:0/483000201 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558761653510 con 0x55876172e770 2024-03-28T18:10:07.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.886+0000 7f818fe00640 1 --2- 172.21.15.37:0/483000201 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55876172dd80 0x55876172e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:07.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.886+0000 7f819ca00640 1 --2- 172.21.15.37:0/483000201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55876172e770 0x558761733270 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f818800bbc0 tx=0x7f81880107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=900cc5e02cf5320d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:07.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.886+0000 7f818dc00640 1 -- 172.21.15.37:0/483000201 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8188004030 con 0x55876172e770 2024-03-28T18:10:07.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.886+0000 7f818dc00640 1 -- 172.21.15.37:0/483000201 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8188005500 con 0x55876172e770 2024-03-28T18:10:07.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.886+0000 7f818dc00640 1 -- 172.21.15.37:0/483000201 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8188004cb0 con 0x55876172e770 2024-03-28T18:10:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.887+0000 7f8196000640 1 --2- 172.21.15.37:0/40032120 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f817c0a2880 0x7f817c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.887+0000 7f818fe00640 1 --2- 172.21.15.37:0/40032120 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f817c0a2880 0x7f817c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.887+0000 7f8196000640 1 -- 172.21.15.37:0/40032120 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f81900ceff0 con 0x7f8198103150 2024-03-28T18:10:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.887+0000 7f81a21e7740 1 -- 172.21.15.37:0/483000201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55876172e770 msgr2=0x558761733270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.887+0000 7f81a21e7740 1 --2- 172.21.15.37:0/483000201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55876172e770 0x558761733270 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f818800bbc0 tx=0x7f81880107a0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.887+0000 7f81a21e7740 1 -- 172.21.15.37:0/483000201 shutdown_connections 2024-03-28T18:10:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.887+0000 7f81a21e7740 1 --2- 172.21.15.37:0/483000201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587617338e0 0x558761735d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.887+0000 7f81a21e7740 1 --2- 172.21.15.37:0/483000201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55876172e770 0x558761733270 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.887+0000 7f81a21e7740 1 --2- 172.21.15.37:0/483000201 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55876172dd80 0x55876172e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.887+0000 7f81a21e7740 1 -- 172.21.15.37:0/483000201 >> 172.21.15.37:0/483000201 conn(0x5587617292f0 msgr2=0x5587617296f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.888+0000 7f81a21e7740 1 -- 172.21.15.37:0/483000201 shutdown_connections 2024-03-28T18:10:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.888+0000 7f81a21e7740 1 -- 172.21.15.37:0/483000201 wait complete. 2024-03-28T18:10:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.888+0000 7f818fe00640 1 --2- 172.21.15.37:0/40032120 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f817c0a2880 0x7f817c0a4d40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55876172efd0 tx=0x7f8180005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.888+0000 7f81a21e7740 1 -- start start 2024-03-28T18:10:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.889+0000 7f81a21e7740 1 Processor -- start 2024-03-28T18:10:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.889+0000 7f81a21e7740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55876172dd80 0x558761738260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.889+0000 7f81a21e7740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55876172e770 0x5587617387a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.889+0000 7f81a21e7740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5587617338e0 0x55876173f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.889+0000 7f81a21e7740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x558761736890 con 0x5587617338e0 2024-03-28T18:10:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.889+0000 7f81a21e7740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55876164edd0 con 0x55876172e770 2024-03-28T18:10:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.889+0000 7f81a21e7740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558761736a10 con 0x55876172dd80 2024-03-28T18:10:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.889+0000 7f81a21e7740 10 client.0 _ll_register_callbacks cb 0x5587613a6ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:10:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.889+0000 7f819ca00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55876172dd80 0x558761738260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.890+0000 7f819ca00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55876172dd80 0x558761738260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:43930/0 (socket says 172.21.15.37:43930) 2024-03-28T18:10:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.890+0000 7f819ca00640 1 -- 172.21.15.37:0/3472735529 learned_addr learned my addr 172.21.15.37:0/3472735529 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.890+0000 7f819d400640 1 --2- 172.21.15.37:0/3472735529 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5587617338e0 0x55876173f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.890+0000 7f818fe00640 1 --2- 172.21.15.37:0/3472735529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55876172e770 0x5587617387a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.890+0000 7f819ca00640 1 -- 172.21.15.37:0/3472735529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55876172e770 msgr2=0x5587617387a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.890+0000 7f819ca00640 1 --2- 172.21.15.37:0/3472735529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55876172e770 0x5587617387a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.890+0000 7f819ca00640 1 -- 172.21.15.37:0/3472735529 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5587617338e0 msgr2=0x55876173f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.890+0000 7f819ca00640 1 --2- 172.21.15.37:0/3472735529 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5587617338e0 0x55876173f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.890+0000 7f819ca00640 1 -- 172.21.15.37:0/3472735529 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5587613a72d0 con 0x55876172dd80 2024-03-28T18:10:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.890+0000 7f819ca00640 1 --2- 172.21.15.37:0/3472735529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55876172dd80 0x558761738260 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8188010ca0 tx=0x7f818800bfa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2ed2e42173de9cd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.890+0000 7f8186000640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:10:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.891+0000 7f8186000640 1 -- 172.21.15.37:0/3472735529 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818802b060 con 0x55876172dd80 2024-03-28T18:10:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.891+0000 7f8186000640 1 -- 172.21.15.37:0/3472735529 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f8188004bc0 con 0x55876172dd80 2024-03-28T18:10:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.891+0000 7f81a21e7740 1 -- 172.21.15.37:0/3472735529 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8188005500 con 0x55876172dd80 2024-03-28T18:10:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.891+0000 7f8186000640 1 -- 172.21.15.37:0/3472735529 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8188004de0 con 0x55876172dd80 2024-03-28T18:10:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.891+0000 7f81a21e7740 10 client.25039 fetch_fsmap learned FSMap version 22 2024-03-28T18:10:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.891+0000 7f81a21e7740 1 -- 172.21.15.37:0/3472735529 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5587614716b0 con 0x55876172dd80 2024-03-28T18:10:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.891+0000 7f8186000640 1 -- 172.21.15.37:0/3472735529 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8188005500 con 0x55876172dd80 2024-03-28T18:10:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.891+0000 7f8186000640 1 -- 172.21.15.37:0/3472735529 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8188011ad0 con 0x55876172dd80 2024-03-28T18:10:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.892+0000 7f81a21e7740 10 client.25039 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T18:10:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.892+0000 7f81a21e7740 10 client.25039 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T18:10:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.892+0000 7f81a21e7740 20 client.25039 populate_metadata read hostname 'smithi037' 2024-03-28T18:10:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.892+0000 7f81a21e7740 1 --2- 172.21.15.37:0/3472735529 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x558761744600 0x558761748a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.892+0000 7f81a21e7740 4 client.25039 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:10:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.892+0000 7f81a21e7740 1 -- 172.21.15.37:0/3472735529 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558761634520 con 0x558761744600 2024-03-28T18:10:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.892+0000 7f818fe00640 1 --2- 172.21.15.37:0/3472735529 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x558761744600 0x558761748a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.893+0000 7f818fe00640 1 --2- 172.21.15.37:0/3472735529 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x558761744600 0x558761748a80 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=deb97406c7fe93f9 server_cookie=60dc9911e219df28 in_seq=0 out_seq=0 2024-03-28T18:10:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.893+0000 7f8186000640 10 client.25039 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:10:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.894+0000 7f8186000640 1 -- 172.21.15.37:0/3472735529 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558761634520 con 0x558761744600 2024-03-28T18:10:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.894+0000 7f8186000640 10 client.25039 handle_command_reply: tid=0 2024-03-28T18:10:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.894+0000 7f81a21e7740 1 client.25039 shutdown 2024-03-28T18:10:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.894+0000 7f81a21e7740 1 -- 172.21.15.37:0/3472735529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55876172dd80 msgr2=0x558761738260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.894+0000 7f81a21e7740 1 --2- 172.21.15.37:0/3472735529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55876172dd80 0x558761738260 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8188010ca0 tx=0x7f818800bfa0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.894+0000 7f81a21e7740 1 -- 172.21.15.37:0/3472735529 shutdown_connections 2024-03-28T18:10:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.894+0000 7f81a21e7740 1 --2- 172.21.15.37:0/3472735529 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x558761744600 0x558761748a80 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.894+0000 7f81a21e7740 1 --2- 172.21.15.37:0/3472735529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55876172e770 0x5587617387a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.894+0000 7f81a21e7740 1 --2- 172.21.15.37:0/3472735529 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5587617338e0 0x55876173f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.894+0000 7f8186000640 0 client.25039 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:10:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.894+0000 7f81a21e7740 1 --2- 172.21.15.37:0/3472735529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55876172dd80 0x558761738260 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.894+0000 7f81a21e7740 1 -- 172.21.15.37:0/3472735529 >> 172.21.15.37:0/3472735529 conn(0x5587617292f0 msgr2=0x5587617284f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.894+0000 7f81a21e7740 1 -- 172.21.15.37:0/3472735529 shutdown_connections 2024-03-28T18:10:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.894+0000 7f81a21e7740 1 -- 172.21.15.37:0/3472735529 wait complete. 2024-03-28T18:10:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.895+0000 7f81a21e7740 20 client.25039 trim_cache size 0 max 16384 2024-03-28T18:10:07.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.931+0000 7f81a21e7740 1 Processor -- start 2024-03-28T18:10:07.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.931+0000 7f81a21e7740 1 -- start start 2024-03-28T18:10:07.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.931+0000 7f81a21e7740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55876172dd80 0x558761736e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.931+0000 7f81a21e7740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55876172e770 0x55876173f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.931+0000 7f81a21e7740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5587617338e0 0x55876172cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.931+0000 7f81a21e7740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x558761743e20 con 0x55876172e770 2024-03-28T18:10:07.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.931+0000 7f81a21e7740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x558761744480 con 0x55876172dd80 2024-03-28T18:10:07.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.931+0000 7f81a21e7740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558761744ae0 con 0x5587617338e0 2024-03-28T18:10:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.932+0000 7f819ca00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55876172dd80 0x558761736e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.932+0000 7f819d400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5587617338e0 0x55876172cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.932+0000 7f819ca00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55876172dd80 0x558761736e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:55328/0 (socket says 172.21.15.37:55328) 2024-03-28T18:10:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.932+0000 7f819ca00640 1 -- 172.21.15.37:0/2718205071 learned_addr learned my addr 172.21.15.37:0/2718205071 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.932+0000 7f818fe00640 1 --2- 172.21.15.37:0/2718205071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55876172e770 0x55876173f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.932+0000 7f819ca00640 1 -- 172.21.15.37:0/2718205071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5587617338e0 msgr2=0x55876172cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.932+0000 7f819ca00640 1 --2- 172.21.15.37:0/2718205071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5587617338e0 0x55876172cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.932+0000 7f819ca00640 1 -- 172.21.15.37:0/2718205071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55876172e770 msgr2=0x55876173f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.932+0000 7f819ca00640 1 --2- 172.21.15.37:0/2718205071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55876172e770 0x55876173f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.932+0000 7f819ca00640 1 -- 172.21.15.37:0/2718205071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55876172a060 con 0x55876172dd80 2024-03-28T18:10:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.932+0000 7f819d400640 1 --2- 172.21.15.37:0/2718205071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5587617338e0 0x55876172cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.932+0000 7f818fe00640 1 --2- 172.21.15.37:0/2718205071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55876172e770 0x55876173f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:10:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.932+0000 7f819ca00640 1 --2- 172.21.15.37:0/2718205071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55876172dd80 0x558761736e10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f818800b3d0 tx=0x7f818801b6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8009e4d412f6f029 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.933+0000 7f818d200640 1 -- 172.21.15.37:0/2718205071 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8188036030 con 0x55876172dd80 2024-03-28T18:10:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.933+0000 7f818d200640 1 -- 172.21.15.37:0/2718205071 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f818801f070 con 0x55876172dd80 2024-03-28T18:10:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.933+0000 7f818d200640 1 -- 172.21.15.37:0/2718205071 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818802b030 con 0x55876172dd80 2024-03-28T18:10:07.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.934+0000 7f81a21e7740 1 -- 172.21.15.37:0/2718205071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55876172dd80 msgr2=0x558761736e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.934+0000 7f81a21e7740 1 --2- 172.21.15.37:0/2718205071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55876172dd80 0x558761736e10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f818800b3d0 tx=0x7f818801b6e0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.934+0000 7f81a21e7740 1 -- 172.21.15.37:0/2718205071 shutdown_connections 2024-03-28T18:10:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.934+0000 7f81a21e7740 1 --2- 172.21.15.37:0/2718205071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5587617338e0 0x55876172cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.934+0000 7f81a21e7740 1 --2- 172.21.15.37:0/2718205071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55876172e770 0x55876173f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.934+0000 7f81a21e7740 1 --2- 172.21.15.37:0/2718205071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55876172dd80 0x558761736e10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.934+0000 7f81a21e7740 1 -- 172.21.15.37:0/2718205071 >> 172.21.15.37:0/2718205071 conn(0x5587617292f0 msgr2=0x55876172f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.934+0000 7f81a21e7740 1 -- 172.21.15.37:0/2718205071 shutdown_connections 2024-03-28T18:10:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.934+0000 7f81a21e7740 1 -- 172.21.15.37:0/2718205071 wait complete. 2024-03-28T18:10:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.935+0000 7f81a21e7740 1 -- start start 2024-03-28T18:10:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.935+0000 7f81a21e7740 1 Processor -- start 2024-03-28T18:10:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.935+0000 7f81a21e7740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55876172dd80 0x55876173d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.935+0000 7f81a21e7740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55876172e770 0x55876179bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.936+0000 7f81a21e7740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587617338e0 0x55876179e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.936+0000 7f81a21e7740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587617457a0 con 0x55876172dd80 2024-03-28T18:10:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.936+0000 7f819ca00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55876172dd80 0x55876173d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.936+0000 7f818fe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55876172e770 0x55876179bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.936+0000 7f818fe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55876172e770 0x55876179bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:43954/0 (socket says 172.21.15.37:43954) 2024-03-28T18:10:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.936+0000 7f818fe00640 1 -- 172.21.15.37:0/4051557894 learned_addr learned my addr 172.21.15.37:0/4051557894 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.936+0000 7f819d400640 1 --2- 172.21.15.37:0/4051557894 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587617338e0 0x55876179e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.936+0000 7f81a21e7740 1 -- 172.21.15.37:0/4051557894 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x558761745140 con 0x5587617338e0 2024-03-28T18:10:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.936+0000 7f81a21e7740 1 -- 172.21.15.37:0/4051557894 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558761745e00 con 0x55876172e770 2024-03-28T18:10:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.936+0000 7f81a21e7740 10 client.0 _ll_register_callbacks cb 0x7f818800ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:10:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.936+0000 7f818fe00640 1 -- 172.21.15.37:0/4051557894 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587617338e0 msgr2=0x55876179e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.936+0000 7f818fe00640 1 --2- 172.21.15.37:0/4051557894 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587617338e0 0x55876179e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.936+0000 7f818fe00640 1 -- 172.21.15.37:0/4051557894 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55876172dd80 msgr2=0x55876173d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.936+0000 7f818fe00640 1 --2- 172.21.15.37:0/4051557894 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55876172dd80 0x55876173d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.936+0000 7f818fe00640 1 -- 172.21.15.37:0/4051557894 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55876179e840 con 0x55876172e770 2024-03-28T18:10:07.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.936+0000 7f819d400640 1 --2- 172.21.15.37:0/4051557894 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587617338e0 0x55876179e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:07.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.937+0000 7f818fe00640 1 --2- 172.21.15.37:0/4051557894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55876172e770 0x55876179bb80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f818002c8e0 tx=0x7f818002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26c1602328185e2b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:07.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.937+0000 7f8186000640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:10:07.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.937+0000 7f8186000640 1 -- 172.21.15.37:0/4051557894 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8180004a90 con 0x55876172e770 2024-03-28T18:10:07.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.937+0000 7f8186000640 1 -- 172.21.15.37:0/4051557894 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f818003a040 con 0x55876172e770 2024-03-28T18:10:07.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.937+0000 7f81a21e7740 1 -- 172.21.15.37:0/4051557894 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f818801f070 con 0x55876172e770 2024-03-28T18:10:07.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.937+0000 7f8186000640 1 -- 172.21.15.37:0/4051557894 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8180068050 con 0x55876172e770 2024-03-28T18:10:07.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.937+0000 7f81a21e7740 10 client.25045 fetch_fsmap learned FSMap version 22 2024-03-28T18:10:07.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.937+0000 7f81a21e7740 1 -- 172.21.15.37:0/4051557894 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55876179fb50 con 0x55876172e770 2024-03-28T18:10:07.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.937+0000 7f8186000640 1 -- 172.21.15.37:0/4051557894 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f818801f070 con 0x55876172e770 2024-03-28T18:10:07.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.938+0000 7f8186000640 1 -- 172.21.15.37:0/4051557894 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8180003890 con 0x55876172e770 2024-03-28T18:10:07.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.938+0000 7f81a21e7740 10 client.25045 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T18:10:07.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.938+0000 7f81a21e7740 10 client.25045 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T18:10:07.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.938+0000 7f81a21e7740 20 client.25045 populate_metadata read hostname 'smithi037' 2024-03-28T18:10:07.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.938+0000 7f81a21e7740 1 --2- 172.21.15.37:0/4051557894 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x5587617a4180 0x55876179fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:07.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.938+0000 7f81a21e7740 4 client.25045 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:10:07.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.938+0000 7f81a21e7740 1 -- 172.21.15.37:0/4051557894 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5587616a01d0 con 0x5587617a4180 2024-03-28T18:10:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.938+0000 7f819ca00640 1 --2- 172.21.15.37:0/4051557894 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x5587617a4180 0x55876179fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.939+0000 7f819ca00640 1 --2- 172.21.15.37:0/4051557894 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x5587617a4180 0x55876179fde0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffb050b3224706cc server_cookie=c853a2ecb48f7bd6 in_seq=0 out_seq=0 2024-03-28T18:10:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.939+0000 7f8186000640 10 client.25045 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:10:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.939+0000 7f8186000640 1 -- 172.21.15.37:0/4051557894 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x5587616a01d0 con 0x5587617a4180 2024-03-28T18:10:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.939+0000 7f8186000640 10 client.25045 handle_command_reply: tid=0 2024-03-28T18:10:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.939+0000 7f81a21e7740 1 client.25045 shutdown 2024-03-28T18:10:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.939+0000 7f81a21e7740 1 -- 172.21.15.37:0/4051557894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55876172e770 msgr2=0x55876179bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.939+0000 7f81a21e7740 1 --2- 172.21.15.37:0/4051557894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55876172e770 0x55876179bb80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f818002c8e0 tx=0x7f818002bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.939+0000 7f81a21e7740 1 -- 172.21.15.37:0/4051557894 shutdown_connections 2024-03-28T18:10:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.939+0000 7f81a21e7740 1 --2- 172.21.15.37:0/4051557894 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x5587617a4180 0x55876179fde0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.939+0000 7f81a21e7740 1 --2- 172.21.15.37:0/4051557894 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587617338e0 0x55876179e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.939+0000 7f81a21e7740 1 --2- 172.21.15.37:0/4051557894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55876172e770 0x55876179bb80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.939+0000 7f81a21e7740 1 --2- 172.21.15.37:0/4051557894 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55876172dd80 0x55876173d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.939+0000 7f81a21e7740 1 -- 172.21.15.37:0/4051557894 >> 172.21.15.37:0/4051557894 conn(0x5587617292f0 msgr2=0x55876172c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.939+0000 7f8186000640 0 client.25045 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:10:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.940+0000 7f81a21e7740 1 -- 172.21.15.37:0/4051557894 shutdown_connections 2024-03-28T18:10:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.940+0000 7f81a21e7740 1 -- 172.21.15.37:0/4051557894 wait complete. 2024-03-28T18:10:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.940+0000 7f81a21e7740 20 client.25045 trim_cache size 0 max 16384 2024-03-28T18:10:07.945 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:10:07.945 INFO:teuthology.orchestra.run.smithi037.stdout: "cluster_fsid": "b5ad257c-ed2c-11ee-95d0-87774f69a715", 2024-03-28T18:10:07.945 INFO:teuthology.orchestra.run.smithi037.stdout: "whoami": 0, 2024-03-28T18:10:07.945 INFO:teuthology.orchestra.run.smithi037.stdout: "id": 24580, 2024-03-28T18:10:07.945 INFO:teuthology.orchestra.run.smithi037.stdout: "want_state": "up:active", 2024-03-28T18:10:07.945 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "up:active", 2024-03-28T18:10:07.945 INFO:teuthology.orchestra.run.smithi037.stdout: "fs_name": "cephfs", 2024-03-28T18:10:07.946 INFO:teuthology.orchestra.run.smithi037.stdout: "rank_uptime": 107.426983882, 2024-03-28T18:10:07.946 INFO:teuthology.orchestra.run.smithi037.stdout: "mdsmap_epoch": 22, 2024-03-28T18:10:07.946 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap_epoch": 87, 2024-03-28T18:10:07.946 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap_epoch_barrier": 85, 2024-03-28T18:10:07.946 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 117.362145031 2024-03-28T18:10:07.946 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:10:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.941+0000 7f819de00640 1 -- 172.21.15.37:0/40032120 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f817c0a2880 msgr2=0x7f817c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.941+0000 7f819de00640 1 --2- 172.21.15.37:0/40032120 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f817c0a2880 0x7f817c0a4d40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55876172efd0 tx=0x7f8180005fd0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.941+0000 7f819de00640 1 -- 172.21.15.37:0/40032120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8198103150 msgr2=0x7f8198073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.941+0000 7f819de00640 1 --2- 172.21.15.37:0/40032120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8198103150 0x7f8198073fd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f819000eb10 tx=0x7f819000c2d0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.941+0000 7f819de00640 1 -- 172.21.15.37:0/40032120 shutdown_connections 2024-03-28T18:10:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.941+0000 7f819de00640 1 --2- 172.21.15.37:0/40032120 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f817c0a2880 0x7f817c0a4d40 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.941+0000 7f819de00640 1 --2- 172.21.15.37:0/40032120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8198103150 0x7f8198073fd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.941+0000 7f819de00640 1 --2- 172.21.15.37:0/40032120 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8198102790 0x7f8198073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.941+0000 7f819de00640 1 --2- 172.21.15.37:0/40032120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81981015a0 0x7f8198075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.941+0000 7f819de00640 1 -- 172.21.15.37:0/40032120 >> 172.21.15.37:0/40032120 conn(0x7f81980fcb10 msgr2=0x7f8198105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.941+0000 7f819de00640 1 -- 172.21.15.37:0/40032120 shutdown_connections 2024-03-28T18:10:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:07.941+0000 7f819de00640 1 -- 172.21.15.37:0/40032120 wait complete. 2024-03-28T18:10:07.959 INFO:tasks.cephadm:Setup complete, yielding 2024-03-28T18:10:07.959 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T18:10:07.972 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi037.front.sepia.ceph.com 2024-03-28T18:10:07.972 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- bash -c 'ceph orch status' 2024-03-28T18:10:08.453 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:10:09.165 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:09 smithi037 ceph-mon[31460]: pgmap v374: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:09.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:09 smithi059 ceph-mon[36190]: pgmap v374: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:09.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:09 smithi170 ceph-mon[36280]: pgmap v374: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:10.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.854+0000 7f6770e00700 1 -- 172.21.15.37:0/2708723042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f676c10a0a0 msgr2=0x7f676c10c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:10.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.854+0000 7f6770e00700 1 --2- 172.21.15.37:0/2708723042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f676c10a0a0 0x7f676c10c530 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6764009990 tx=0x7f6764009ca0 comp rx=0 tx=0).stop 2024-03-28T18:10:10.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.854+0000 7f6770e00700 1 -- 172.21.15.37:0/2708723042 shutdown_connections 2024-03-28T18:10:10.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.854+0000 7f6770e00700 1 --2- 172.21.15.37:0/2708723042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f676c10a0a0 0x7f676c10c530 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:10.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.854+0000 7f6770e00700 1 --2- 172.21.15.37:0/2708723042 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f676c0761c0 0x7f676c074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:10.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.854+0000 7f6770e00700 1 --2- 172.21.15.37:0/2708723042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f676c074bf0 0x7f676c075070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:10.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.854+0000 7f6770e00700 1 -- 172.21.15.37:0/2708723042 >> 172.21.15.37:0/2708723042 conn(0x7f676c0fd730 msgr2=0x7f676c0ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:10.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.854+0000 7f6770e00700 1 -- 172.21.15.37:0/2708723042 shutdown_connections 2024-03-28T18:10:10.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.854+0000 7f6770e00700 1 -- 172.21.15.37:0/2708723042 wait complete. 2024-03-28T18:10:10.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.856+0000 7f6770e00700 1 Processor -- start 2024-03-28T18:10:10.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.857+0000 7f6770e00700 1 -- start start 2024-03-28T18:10:10.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.857+0000 7f6770e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f676c074bf0 0x7f676c19a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:10.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.857+0000 7f6770e00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f676c0761c0 0x7f676c19ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.857+0000 7f6770e00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f676c10a0a0 0x7f676c1a1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.858+0000 7f6770e00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c110020 con 0x7f676c074bf0 2024-03-28T18:10:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.858+0000 7f6770e00700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c10fea0 con 0x7f676c0761c0 2024-03-28T18:10:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.858+0000 7f6770e00700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c1101a0 con 0x7f676c10a0a0 2024-03-28T18:10:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.858+0000 7f6769400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f676c0761c0 0x7f676c19ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.858+0000 7f6769400700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f676c0761c0 0x7f676c19ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:55354/0 (socket says 172.21.15.37:55354) 2024-03-28T18:10:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.858+0000 7f6769400700 1 -- 172.21.15.37:0/3480436537 learned_addr learned my addr 172.21.15.37:0/3480436537 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.858+0000 7f6769e00700 1 --2- 172.21.15.37:0/3480436537 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f676c074bf0 0x7f676c19a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.858+0000 7f676a800700 1 --2- 172.21.15.37:0/3480436537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f676c10a0a0 0x7f676c1a1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.858+0000 7f676a800700 1 -- 172.21.15.37:0/3480436537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f676c0761c0 msgr2=0x7f676c19ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.858+0000 7f676a800700 1 --2- 172.21.15.37:0/3480436537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f676c0761c0 0x7f676c19ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.858+0000 7f676a800700 1 -- 172.21.15.37:0/3480436537 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f676c074bf0 msgr2=0x7f676c19a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.858+0000 7f676a800700 1 --2- 172.21.15.37:0/3480436537 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f676c074bf0 0x7f676c19a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.858+0000 7f676a800700 1 -- 172.21.15.37:0/3480436537 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676c1a2230 con 0x7f676c10a0a0 2024-03-28T18:10:10.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.859+0000 7f676a800700 1 --2- 172.21.15.37:0/3480436537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f676c10a0a0 0x7f676c1a1b90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f676400b7b0 tx=0x7f676400bbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:10.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.859+0000 7f6762a00700 1 -- 172.21.15.37:0/3480436537 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6764027070 con 0x7f676c10a0a0 2024-03-28T18:10:10.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.859+0000 7f6762a00700 1 -- 172.21.15.37:0/3480436537 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6764003d70 con 0x7f676c10a0a0 2024-03-28T18:10:10.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.859+0000 7f6770e00700 1 -- 172.21.15.37:0/3480436537 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676c1a24c0 con 0x7f676c10a0a0 2024-03-28T18:10:10.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.859+0000 7f6762a00700 1 -- 172.21.15.37:0/3480436537 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676400f6c0 con 0x7f676c10a0a0 2024-03-28T18:10:10.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.859+0000 7f6770e00700 1 -- 172.21.15.37:0/3480436537 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f676c1a2970 con 0x7f676c10a0a0 2024-03-28T18:10:10.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.860+0000 7f6770e00700 1 -- 172.21.15.37:0/3480436537 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f676c068e50 con 0x7f676c10a0a0 2024-03-28T18:10:10.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.864+0000 7f6762a00700 1 -- 172.21.15.37:0/3480436537 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f676402a070 con 0x7f676c10a0a0 2024-03-28T18:10:10.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.866+0000 7f6762a00700 1 --2- 172.21.15.37:0/3480436537 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f67500a2910 0x7f67500a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:10.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.866+0000 7f6762a00700 1 -- 172.21.15.37:0/3480436537 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f67640cffa0 con 0x7f676c10a0a0 2024-03-28T18:10:10.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.866+0000 7f6769e00700 1 --2- 172.21.15.37:0/3480436537 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f67500a2910 0x7f67500a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:10.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.866+0000 7f6762a00700 1 -- 172.21.15.37:0/3480436537 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6764013070 con 0x7f676c10a0a0 2024-03-28T18:10:10.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:10.867+0000 7f6769e00700 1 --2- 172.21.15.37:0/3480436537 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f67500a2910 0x7f67500a4dd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f675c009550 tx=0x7f675c0091f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:11.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:11.177+0000 7f6770e00700 1 -- 172.21.15.37:0/3480436537 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f676c0625d0 con 0x7f67500a2910 2024-03-28T18:10:11.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:11.179+0000 7f6762a00700 1 -- 172.21.15.37:0/3480436537 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f676c0625d0 con 0x7f67500a2910 2024-03-28T18:10:11.179 INFO:teuthology.orchestra.run.smithi037.stdout:Backend: cephadm 2024-03-28T18:10:11.179 INFO:teuthology.orchestra.run.smithi037.stdout:Available: Yes 2024-03-28T18:10:11.179 INFO:teuthology.orchestra.run.smithi037.stdout:Paused: No 2024-03-28T18:10:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:11.182+0000 7f6770e00700 1 -- 172.21.15.37:0/3480436537 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f67500a2910 msgr2=0x7f67500a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:11.182+0000 7f6770e00700 1 --2- 172.21.15.37:0/3480436537 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f67500a2910 0x7f67500a4dd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f675c009550 tx=0x7f675c0091f0 comp rx=0 tx=0).stop 2024-03-28T18:10:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:11.182+0000 7f6770e00700 1 -- 172.21.15.37:0/3480436537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f676c10a0a0 msgr2=0x7f676c1a1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:11.182+0000 7f6770e00700 1 --2- 172.21.15.37:0/3480436537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f676c10a0a0 0x7f676c1a1b90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f676400b7b0 tx=0x7f676400bbb0 comp rx=0 tx=0).stop 2024-03-28T18:10:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:11.182+0000 7f6770e00700 1 -- 172.21.15.37:0/3480436537 shutdown_connections 2024-03-28T18:10:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:11.182+0000 7f6770e00700 1 --2- 172.21.15.37:0/3480436537 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f67500a2910 0x7f67500a4dd0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:11.182+0000 7f6770e00700 1 --2- 172.21.15.37:0/3480436537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f676c10a0a0 0x7f676c1a1b90 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:11.182+0000 7f6770e00700 1 --2- 172.21.15.37:0/3480436537 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f676c074bf0 0x7f676c19a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:11.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:11.182+0000 7f6770e00700 1 --2- 172.21.15.37:0/3480436537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f676c0761c0 0x7f676c19ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:11.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:11.182+0000 7f6770e00700 1 -- 172.21.15.37:0/3480436537 >> 172.21.15.37:0/3480436537 conn(0x7f676c0fd730 msgr2=0x7f676c105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:11.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:11.182+0000 7f6770e00700 1 -- 172.21.15.37:0/3480436537 shutdown_connections 2024-03-28T18:10:11.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:11.182+0000 7f6770e00700 1 -- 172.21.15.37:0/3480436537 wait complete. 2024-03-28T18:10:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:11 smithi037 ceph-mon[31460]: pgmap v375: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:10:11.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:11 smithi059 ceph-mon[36190]: pgmap v375: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:10:11.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:11 smithi170 ceph-mon[36280]: pgmap v375: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:10:12.040 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-28T18:10:12.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:12 smithi037 ceph-mon[31460]: from='client.25051 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:10:12.522 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:10:12.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:12 smithi059 ceph-mon[36190]: from='client.25051 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:10:12.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:12 smithi170 ceph-mon[36280]: from='client.25051 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:10:13.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:13 smithi037 ceph-mon[31460]: pgmap v376: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:10:13.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:13 smithi059 ceph-mon[36190]: pgmap v376: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:10:13.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:13 smithi170 ceph-mon[36280]: pgmap v376: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:10:14.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.840+0000 7fd111a00700 1 -- 172.21.15.37:0/2029849210 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd10c1043f0 msgr2=0x7fd10c104810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:14.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.840+0000 7fd111a00700 1 --2- 172.21.15.37:0/2029849210 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd10c1043f0 0x7fd10c104810 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc009990 tx=0x7fd0fc009ca0 comp rx=0 tx=0).stop 2024-03-28T18:10:14.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.840+0000 7fd111a00700 1 -- 172.21.15.37:0/2029849210 shutdown_connections 2024-03-28T18:10:14.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.840+0000 7fd111a00700 1 --2- 172.21.15.37:0/2029849210 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd10c105f90 0x7fd10c10c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:14.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.840+0000 7fd111a00700 1 --2- 172.21.15.37:0/2029849210 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd10c1043f0 0x7fd10c104810 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:14.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.840+0000 7fd111a00700 1 --2- 172.21.15.37:0/2029849210 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd10c1055f0 0x7fd10c105a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:14.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.840+0000 7fd111a00700 1 -- 172.21.15.37:0/2029849210 >> 172.21.15.37:0/2029849210 conn(0x7fd10c0ff9b0 msgr2=0x7fd10c101dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:14.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.840+0000 7fd111a00700 1 -- 172.21.15.37:0/2029849210 shutdown_connections 2024-03-28T18:10:14.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.840+0000 7fd111a00700 1 -- 172.21.15.37:0/2029849210 wait complete. 2024-03-28T18:10:14.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.841+0000 7fd111a00700 1 Processor -- start 2024-03-28T18:10:14.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.842+0000 7fd111a00700 1 -- start start 2024-03-28T18:10:14.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.842+0000 7fd111a00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd10c1043f0 0x7fd10c19a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:14.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.842+0000 7fd111a00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd10c1055f0 0x7fd10c19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:14.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.842+0000 7fd111a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd10c105f90 0x7fd10c1a1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:14.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.842+0000 7fd111a00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c110010 con 0x7fd10c105f90 2024-03-28T18:10:14.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.842+0000 7fd111a00700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c10fe90 con 0x7fd10c1043f0 2024-03-28T18:10:14.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.842+0000 7fd111a00700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c110190 con 0x7fd10c1055f0 2024-03-28T18:10:14.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.842+0000 7fd10aa00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd10c1043f0 0x7fd10c19a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:14.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.842+0000 7fd10aa00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd10c1043f0 0x7fd10c19a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:57110/0 (socket says 172.21.15.37:57110) 2024-03-28T18:10:14.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.842+0000 7fd10aa00700 1 -- 172.21.15.37:0/245961025 learned_addr learned my addr 172.21.15.37:0/245961025 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:14.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.842+0000 7fd10a000700 1 --2- 172.21.15.37:0/245961025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd10c1055f0 0x7fd10c19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:14.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.843+0000 7fd10aa00700 1 -- 172.21.15.37:0/245961025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd10c1055f0 msgr2=0x7fd10c19aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:14.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.843+0000 7fd10aa00700 1 --2- 172.21.15.37:0/245961025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd10c1055f0 0x7fd10c19aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:14.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.843+0000 7fd10aa00700 1 -- 172.21.15.37:0/245961025 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd10c105f90 msgr2=0x7fd10c1a1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:14.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.843+0000 7fd10aa00700 1 --2- 172.21.15.37:0/245961025 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd10c105f90 0x7fd10c1a1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:14.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.843+0000 7fd10aa00700 1 -- 172.21.15.37:0/245961025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd10c1a21f0 con 0x7fd10c1043f0 2024-03-28T18:10:14.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.843+0000 7fd10aa00700 1 --2- 172.21.15.37:0/245961025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd10c1043f0 0x7fd10c19a590 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc0039c0 tx=0x7fd0fc0057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:14.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.843+0000 7fd103400700 1 -- 172.21.15.37:0/245961025 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0fc026070 con 0x7fd10c1043f0 2024-03-28T18:10:14.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.843+0000 7fd103400700 1 -- 172.21.15.37:0/245961025 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd0fc004150 con 0x7fd10c1043f0 2024-03-28T18:10:14.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.843+0000 7fd103400700 1 -- 172.21.15.37:0/245961025 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0fc021410 con 0x7fd10c1043f0 2024-03-28T18:10:14.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.843+0000 7fd111a00700 1 -- 172.21.15.37:0/245961025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd10c1a2480 con 0x7fd10c1043f0 2024-03-28T18:10:14.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.844+0000 7fd111a00700 1 -- 172.21.15.37:0/245961025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd10c1a2880 con 0x7fd10c1043f0 2024-03-28T18:10:14.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.845+0000 7fd111a00700 1 -- 172.21.15.37:0/245961025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd10c068e90 con 0x7fd10c1043f0 2024-03-28T18:10:14.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.846+0000 7fd103400700 1 -- 172.21.15.37:0/245961025 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd0fc029070 con 0x7fd10c1043f0 2024-03-28T18:10:14.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.848+0000 7fd103400700 1 --2- 172.21.15.37:0/245961025 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd0f00a2910 0x7fd0f00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:14.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.848+0000 7fd103400700 1 -- 172.21.15.37:0/245961025 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd0fc0cf4d0 con 0x7fd10c1043f0 2024-03-28T18:10:14.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.848+0000 7fd10a000700 1 --2- 172.21.15.37:0/245961025 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd0f00a2910 0x7fd0f00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:14.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.849+0000 7fd10a000700 1 --2- 172.21.15.37:0/245961025 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd0f00a2910 0x7fd0f00a4dd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd0f800ac50 tx=0x7fd0f800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:14.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:14.852+0000 7fd103400700 1 -- 172.21.15.37:0/245961025 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd0fc05d740 con 0x7fd10c1043f0 2024-03-28T18:10:15.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:15.098+0000 7fd111a00700 1 -- 172.21.15.37:0/245961025 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd10c0625d0 con 0x7fd0f00a2910 2024-03-28T18:10:15.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:15.127+0000 7fd103400700 1 -- 172.21.15.37:0/245961025 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4123 (secure 0 0 0) 0x7fd10c0625d0 con 0x7fd0f00a2910 2024-03-28T18:10:15.128 INFO:teuthology.orchestra.run.smithi037.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-28T18:10:15.128 INFO:teuthology.orchestra.run.smithi037.stdout:mds.a smithi037 running (2m) 26s ago 2m 24.5M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 38fcb06288bf 2024-03-28T18:10:15.128 INFO:teuthology.orchestra.run.smithi037.stdout:mds.b smithi059 running (2m) 53s ago 2m 17.0M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 014effd69bd2 2024-03-28T18:10:15.128 INFO:teuthology.orchestra.run.smithi037.stdout:mds.c smithi170 running (2m) 53s ago 2m 18.1M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 bd1817055859 2024-03-28T18:10:15.128 INFO:teuthology.orchestra.run.smithi037.stdout:mds.d smithi037 running (2m) 26s ago 2m 12.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 4b0671f3d1a0 2024-03-28T18:10:15.128 INFO:teuthology.orchestra.run.smithi037.stdout:mds.e smithi059 running (2m) 53s ago 2m 13.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 e836f5f0fe33 2024-03-28T18:10:15.128 INFO:teuthology.orchestra.run.smithi037.stdout:mds.f smithi170 running (2m) 53s ago 2m 18.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 ac5c3af1d16c 2024-03-28T18:10:15.128 INFO:teuthology.orchestra.run.smithi037.stdout:mds.g smithi037 running (2m) 26s ago 2m 12.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 5c83e57ee4ad 2024-03-28T18:10:15.129 INFO:teuthology.orchestra.run.smithi037.stdout:mds.h smithi059 running (2m) 53s ago 2m 19.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 1e6766595523 2024-03-28T18:10:15.129 INFO:teuthology.orchestra.run.smithi037.stdout:mds.i smithi170 running (2m) 53s ago 2m 17.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 b46963971f81 2024-03-28T18:10:15.129 INFO:teuthology.orchestra.run.smithi037.stdout:mds.j smithi037 running (2m) 26s ago 2m 12.5M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 feecadfcbd8b 2024-03-28T18:10:15.129 INFO:teuthology.orchestra.run.smithi037.stdout:mds.k smithi059 running (2m) 53s ago 2m 14.0M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 9f1727b54bbe 2024-03-28T18:10:15.129 INFO:teuthology.orchestra.run.smithi037.stdout:mds.l smithi170 running (2m) 53s ago 2m 14.1M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 8a70934209e5 2024-03-28T18:10:15.129 INFO:teuthology.orchestra.run.smithi037.stdout:mgr.x smithi037 *:9283,8765 running (11m) 26s ago 10m 512M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 203afa0bd6e2 2024-03-28T18:10:15.129 INFO:teuthology.orchestra.run.smithi037.stdout:mgr.y smithi059 *:8443,8765 running (8m) 53s ago 8m 452M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 72e8a8cfa8e1 2024-03-28T18:10:15.129 INFO:teuthology.orchestra.run.smithi037.stdout:mgr.z smithi170 *:8443,8765 running (8m) 53s ago 8m 451M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 c20168783afc 2024-03-28T18:10:15.129 INFO:teuthology.orchestra.run.smithi037.stdout:mon.a smithi037 running (11m) 26s ago 11m 48.2M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 10bef11fe84f 2024-03-28T18:10:15.129 INFO:teuthology.orchestra.run.smithi037.stdout:mon.b smithi059 running (8m) 53s ago 8m 40.5M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 1ea8be72aa5a 2024-03-28T18:10:15.129 INFO:teuthology.orchestra.run.smithi037.stdout:mon.c smithi170 running (9m) 53s ago 9m 41.1M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 cc015f7b0d4f 2024-03-28T18:10:15.129 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0 smithi037 running (8m) 26s ago 8m 53.7M 4273M 19.0.0-2145-g3be9a5c9 c664afe56bb0 5b85fdbfbc9b 2024-03-28T18:10:15.129 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1 smithi059 running (7m) 53s ago 7m 48.1M 4271M 19.0.0-2145-g3be9a5c9 c664afe56bb0 69378431f975 2024-03-28T18:10:15.129 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2 smithi170 running (7m) 53s ago 7m 57.3M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 38bf86d6b2a9 2024-03-28T18:10:15.129 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3 smithi037 running (6m) 26s ago 6m 53.1M 4273M 19.0.0-2145-g3be9a5c9 c664afe56bb0 f401527aa834 2024-03-28T18:10:15.130 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4 smithi059 running (6m) 53s ago 6m 54.3M 4271M 19.0.0-2145-g3be9a5c9 c664afe56bb0 ebdd6a61252b 2024-03-28T18:10:15.130 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5 smithi170 running (5m) 53s ago 5m 56.6M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 a8ca131b5bb4 2024-03-28T18:10:15.130 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6 smithi037 running (5m) 26s ago 5m 51.7M 4273M 19.0.0-2145-g3be9a5c9 c664afe56bb0 d400739ca415 2024-03-28T18:10:15.130 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7 smithi059 running (4m) 53s ago 4m 55.7M 4271M 19.0.0-2145-g3be9a5c9 c664afe56bb0 8dce674003ab 2024-03-28T18:10:15.130 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8 smithi170 running (4m) 53s ago 4m 42.7M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 a07c768217c8 2024-03-28T18:10:15.130 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9 smithi037 running (3m) 26s ago 3m 40.6M 4273M 19.0.0-2145-g3be9a5c9 c664afe56bb0 3294c27915b0 2024-03-28T18:10:15.130 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10 smithi059 running (3m) 53s ago 3m 55.4M 4271M 19.0.0-2145-g3be9a5c9 c664afe56bb0 1e83ca9b9388 2024-03-28T18:10:15.130 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11 smithi170 running (3m) 53s ago 3m 40.6M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 e658362d4dea 2024-03-28T18:10:15.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:15.130+0000 7fd111a00700 1 -- 172.21.15.37:0/245961025 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd0f00a2910 msgr2=0x7fd0f00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:15.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:15.130+0000 7fd111a00700 1 --2- 172.21.15.37:0/245961025 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd0f00a2910 0x7fd0f00a4dd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd0f800ac50 tx=0x7fd0f800a380 comp rx=0 tx=0).stop 2024-03-28T18:10:15.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:15.130+0000 7fd111a00700 1 -- 172.21.15.37:0/245961025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd10c1043f0 msgr2=0x7fd10c19a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:15.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:15.130+0000 7fd111a00700 1 --2- 172.21.15.37:0/245961025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd10c1043f0 0x7fd10c19a590 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc0039c0 tx=0x7fd0fc0057c0 comp rx=0 tx=0).stop 2024-03-28T18:10:15.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:15.130+0000 7fd111a00700 1 -- 172.21.15.37:0/245961025 shutdown_connections 2024-03-28T18:10:15.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:15.130+0000 7fd111a00700 1 --2- 172.21.15.37:0/245961025 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd0f00a2910 0x7fd0f00a4dd0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:15.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:15.130+0000 7fd111a00700 1 --2- 172.21.15.37:0/245961025 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd10c105f90 0x7fd10c1a1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:15.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:15.130+0000 7fd111a00700 1 --2- 172.21.15.37:0/245961025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd10c1043f0 0x7fd10c19a590 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:15.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:15.131+0000 7fd111a00700 1 --2- 172.21.15.37:0/245961025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd10c1055f0 0x7fd10c19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:15.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:15.131+0000 7fd111a00700 1 -- 172.21.15.37:0/245961025 >> 172.21.15.37:0/245961025 conn(0x7fd10c0ff9b0 msgr2=0x7fd10c1088f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:15.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:15.131+0000 7fd111a00700 1 -- 172.21.15.37:0/245961025 shutdown_connections 2024-03-28T18:10:15.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:15.131+0000 7fd111a00700 1 -- 172.21.15.37:0/245961025 wait complete. 2024-03-28T18:10:15.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:15 smithi037 ceph-mon[31460]: pgmap v377: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:15.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:15 smithi059 ceph-mon[36190]: pgmap v377: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:15.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:15 smithi170 ceph-mon[36280]: pgmap v377: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:16.051 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-28T18:10:16.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:16 smithi037 ceph-mon[31460]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:10:16.524 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:10:16.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:16 smithi059 ceph-mon[36190]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:10:16.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:16 smithi170 ceph-mon[36280]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:10:17.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:17 smithi037 ceph-mon[31460]: pgmap v378: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:10:17.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:17 smithi059 ceph-mon[36190]: pgmap v378: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:10:17.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:17 smithi170 ceph-mon[36280]: pgmap v378: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:10:18.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.730+0000 7f256a000700 1 -- 172.21.15.37:0/173450632 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2564102b70 msgr2=0x7f2564104f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:18.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.730+0000 7f256a000700 1 --2- 172.21.15.37:0/173450632 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2564102b70 0x7f2564104f60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f255800b0a0 tx=0x7f255800b3b0 comp rx=0 tx=0).stop 2024-03-28T18:10:18.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.730+0000 7f256a000700 1 -- 172.21.15.37:0/173450632 shutdown_connections 2024-03-28T18:10:18.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.730+0000 7f256a000700 1 --2- 172.21.15.37:0/173450632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2564107dd0 0x7f256410a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:18.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.730+0000 7f256a000700 1 --2- 172.21.15.37:0/173450632 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2564102b70 0x7f2564104f60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:18.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.730+0000 7f256a000700 1 --2- 172.21.15.37:0/173450632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25641054a0 0x7f2564107890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:18.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.730+0000 7f256a000700 1 -- 172.21.15.37:0/173450632 >> 172.21.15.37:0/173450632 conn(0x7f25640fc580 msgr2=0x7f25640fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:18.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.730+0000 7f256a000700 1 -- 172.21.15.37:0/173450632 shutdown_connections 2024-03-28T18:10:18.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.731+0000 7f256a000700 1 -- 172.21.15.37:0/173450632 wait complete. 2024-03-28T18:10:18.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.732+0000 7f256a000700 1 Processor -- start 2024-03-28T18:10:18.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.732+0000 7f256a000700 1 -- start start 2024-03-28T18:10:18.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.733+0000 7f256a000700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2564102b70 0x7f256419a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:18.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.733+0000 7f256a000700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25641054a0 0x7f256419aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:18.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.733+0000 7f256a000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2564107dd0 0x7f25641a1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:18.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.733+0000 7f256a000700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256410dd40 con 0x7f2564107dd0 2024-03-28T18:10:18.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.733+0000 7f256a000700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256410dbc0 con 0x7f2564102b70 2024-03-28T18:10:18.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.733+0000 7f256a000700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256410dec0 con 0x7f25641054a0 2024-03-28T18:10:18.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.733+0000 7f2563e00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25641054a0 0x7f256419aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:18.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.733+0000 7f2568c00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2564102b70 0x7f256419a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:18.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.733+0000 7f2563e00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25641054a0 0x7f256419aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56052/0 (socket says 172.21.15.37:56052) 2024-03-28T18:10:18.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.733+0000 7f2563e00700 1 -- 172.21.15.37:0/260358609 learned_addr learned my addr 172.21.15.37:0/260358609 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:18.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.733+0000 7f2569600700 1 --2- 172.21.15.37:0/260358609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2564107dd0 0x7f25641a1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:18.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.733+0000 7f2568c00700 1 -- 172.21.15.37:0/260358609 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25641054a0 msgr2=0x7f256419aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:18.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.733+0000 7f2568c00700 1 --2- 172.21.15.37:0/260358609 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25641054a0 0x7f256419aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:18.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.733+0000 7f2568c00700 1 -- 172.21.15.37:0/260358609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2564107dd0 msgr2=0x7f25641a1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:18.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.733+0000 7f2568c00700 1 --2- 172.21.15.37:0/260358609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2564107dd0 0x7f25641a1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:18.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.733+0000 7f2568c00700 1 -- 172.21.15.37:0/260358609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25641a2140 con 0x7f2564102b70 2024-03-28T18:10:18.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.734+0000 7f2568c00700 1 --2- 172.21.15.37:0/260358609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2564102b70 0x7f256419a4e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2558000f80 tx=0x7f2558004ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:18.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.734+0000 7f2561600700 1 -- 172.21.15.37:0/260358609 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2558021070 con 0x7f2564102b70 2024-03-28T18:10:18.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.734+0000 7f2561600700 1 -- 172.21.15.37:0/260358609 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2558007b80 con 0x7f2564102b70 2024-03-28T18:10:18.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.734+0000 7f2561600700 1 -- 172.21.15.37:0/260358609 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255801d450 con 0x7f2564102b70 2024-03-28T18:10:18.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.734+0000 7f256a000700 1 -- 172.21.15.37:0/260358609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25641a23d0 con 0x7f2564102b70 2024-03-28T18:10:18.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.734+0000 7f256a000700 1 -- 172.21.15.37:0/260358609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25641a28d0 con 0x7f2564102b70 2024-03-28T18:10:18.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.735+0000 7f256a000700 1 -- 172.21.15.37:0/260358609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2564068e50 con 0x7f2564102b70 2024-03-28T18:10:18.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.737+0000 7f2561600700 1 -- 172.21.15.37:0/260358609 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2558004020 con 0x7f2564102b70 2024-03-28T18:10:18.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.738+0000 7f2561600700 1 --2- 172.21.15.37:0/260358609 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f254c0a2910 0x7f254c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:18.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.739+0000 7f2561600700 1 -- 172.21.15.37:0/260358609 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f255800f020 con 0x7f2564102b70 2024-03-28T18:10:18.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.739+0000 7f2563e00700 1 --2- 172.21.15.37:0/260358609 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f254c0a2910 0x7f254c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:18.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.739+0000 7f2563e00700 1 --2- 172.21.15.37:0/260358609 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f254c0a2910 0x7f254c0a4dd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2550009d20 tx=0x7f2550009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:18.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:18.743+0000 7f2561600700 1 -- 172.21.15.37:0/260358609 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f255802a020 con 0x7f2564102b70 2024-03-28T18:10:19.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:19.002+0000 7f256a000700 1 -- 172.21.15.37:0/260358609 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f25640625d0 con 0x7f254c0a2910 2024-03-28T18:10:19.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:19.012+0000 7f2561600700 1 -- 172.21.15.37:0/260358609 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f25640625d0 con 0x7f254c0a2910 2024-03-28T18:10:19.012 INFO:teuthology.orchestra.run.smithi037.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-28T18:10:19.012 INFO:teuthology.orchestra.run.smithi037.stdout:mds.all 12/12 56s ago 2m smithi037=a;smithi037=d;smithi037=g;smithi037=j;smithi059=b;smithi059=e;smithi059=h;smithi059=k;smithi170=c;smithi170=f;smithi170=i;smithi170=l;count:12 2024-03-28T18:10:19.012 INFO:teuthology.orchestra.run.smithi037.stdout:mgr 3/3 56s ago 8m smithi037=x;smithi059=y;smithi170=z;count:3 2024-03-28T18:10:19.013 INFO:teuthology.orchestra.run.smithi037.stdout:mon 3/3 56s ago 9m smithi037:172.21.15.37=a;smithi059:172.21.15.59=b;smithi170:172.21.15.170=c;count:3 2024-03-28T18:10:19.013 INFO:teuthology.orchestra.run.smithi037.stdout:osd 12 56s ago - 2024-03-28T18:10:19.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:19.015+0000 7f256a000700 1 -- 172.21.15.37:0/260358609 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f254c0a2910 msgr2=0x7f254c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:19.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:19.015+0000 7f256a000700 1 --2- 172.21.15.37:0/260358609 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f254c0a2910 0x7f254c0a4dd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2550009d20 tx=0x7f2550009260 comp rx=0 tx=0).stop 2024-03-28T18:10:19.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:19.015+0000 7f256a000700 1 -- 172.21.15.37:0/260358609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2564102b70 msgr2=0x7f256419a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:19.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:19.015+0000 7f256a000700 1 --2- 172.21.15.37:0/260358609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2564102b70 0x7f256419a4e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2558000f80 tx=0x7f2558004ce0 comp rx=0 tx=0).stop 2024-03-28T18:10:19.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:19.015+0000 7f256a000700 1 -- 172.21.15.37:0/260358609 shutdown_connections 2024-03-28T18:10:19.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:19.015+0000 7f256a000700 1 --2- 172.21.15.37:0/260358609 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f254c0a2910 0x7f254c0a4dd0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:19.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:19.015+0000 7f256a000700 1 --2- 172.21.15.37:0/260358609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2564107dd0 0x7f25641a1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:19.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:19.015+0000 7f256a000700 1 --2- 172.21.15.37:0/260358609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2564102b70 0x7f256419a4e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:19.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:19.015+0000 7f256a000700 1 --2- 172.21.15.37:0/260358609 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25641054a0 0x7f256419aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:19.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:19.015+0000 7f256a000700 1 -- 172.21.15.37:0/260358609 >> 172.21.15.37:0/260358609 conn(0x7f25640fc580 msgr2=0x7f25641022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:19.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:19.015+0000 7f256a000700 1 -- 172.21.15.37:0/260358609 shutdown_connections 2024-03-28T18:10:19.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:19.015+0000 7f256a000700 1 -- 172.21.15.37:0/260358609 wait complete. 2024-03-28T18:10:19.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:19 smithi037 ceph-mon[31460]: pgmap v379: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:19.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:19 smithi059 ceph-mon[36190]: pgmap v379: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:19.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:19 smithi170 ceph-mon[36280]: pgmap v379: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:19.850 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-28T18:10:20.330 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:10:20.493 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:20 smithi037 ceph-mon[31460]: from='client.25037 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:10:20.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:20 smithi059 ceph-mon[36190]: from='client.25037 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:10:20.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:20 smithi170 ceph-mon[36280]: from='client.25037 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:10:21.445 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:21 smithi037 ceph-mon[31460]: pgmap v380: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:10:21.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:21 smithi059 ceph-mon[36190]: pgmap v380: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:10:21.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:21 smithi170 ceph-mon[36280]: pgmap v380: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:10:22.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.656+0000 7fa441c00700 1 -- 172.21.15.37:0/121080045 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa43c0761a0 msgr2=0x7fa43c074600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:22.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.656+0000 7fa441c00700 1 --2- 172.21.15.37:0/121080045 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa43c0761a0 0x7fa43c074600 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa42c009990 tx=0x7fa42c009ca0 comp rx=0 tx=0).stop 2024-03-28T18:10:22.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.656+0000 7fa441c00700 1 -- 172.21.15.37:0/121080045 shutdown_connections 2024-03-28T18:10:22.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.656+0000 7fa441c00700 1 --2- 172.21.15.37:0/121080045 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa43c10a0b0 0x7fa43c10c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:22.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.656+0000 7fa441c00700 1 --2- 172.21.15.37:0/121080045 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa43c0761a0 0x7fa43c074600 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:22.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.656+0000 7fa441c00700 1 --2- 172.21.15.37:0/121080045 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa43c074bd0 0x7fa43c075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:22.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.656+0000 7fa441c00700 1 -- 172.21.15.37:0/121080045 >> 172.21.15.37:0/121080045 conn(0x7fa43c0fd7a0 msgr2=0x7fa43c0ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:22.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.656+0000 7fa441c00700 1 -- 172.21.15.37:0/121080045 shutdown_connections 2024-03-28T18:10:22.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.656+0000 7fa441c00700 1 -- 172.21.15.37:0/121080045 wait complete. 2024-03-28T18:10:22.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.658+0000 7fa441c00700 1 Processor -- start 2024-03-28T18:10:22.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.659+0000 7fa441c00700 1 -- start start 2024-03-28T18:10:22.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.659+0000 7fa441c00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa43c074bd0 0x7fa43c19a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:22.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.659+0000 7fa441c00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa43c0761a0 0x7fa43c19ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:22.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.659+0000 7fa441c00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa43c10a0b0 0x7fa43c1a1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.659+0000 7fa441c00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43c10ffe0 con 0x7fa43c074bd0 2024-03-28T18:10:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.659+0000 7fa441c00700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43c10fe60 con 0x7fa43c0761a0 2024-03-28T18:10:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.659+0000 7fa441c00700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43c110160 con 0x7fa43c10a0b0 2024-03-28T18:10:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.660+0000 7fa43aa00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa43c074bd0 0x7fa43c19a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.660+0000 7fa43a000700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa43c0761a0 0x7fa43c19ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.660+0000 7fa43a000700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa43c0761a0 0x7fa43c19ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:57140/0 (socket says 172.21.15.37:57140) 2024-03-28T18:10:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.660+0000 7fa43a000700 1 -- 172.21.15.37:0/3989081659 learned_addr learned my addr 172.21.15.37:0/3989081659 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.660+0000 7fa43b400700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa43c10a0b0 0x7fa43c1a1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:22.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.660+0000 7fa43a000700 1 -- 172.21.15.37:0/3989081659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa43c10a0b0 msgr2=0x7fa43c1a1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:22.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.660+0000 7fa43a000700 1 --2- 172.21.15.37:0/3989081659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa43c10a0b0 0x7fa43c1a1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:22.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.660+0000 7fa43a000700 1 -- 172.21.15.37:0/3989081659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa43c074bd0 msgr2=0x7fa43c19a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:22.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.660+0000 7fa43a000700 1 --2- 172.21.15.37:0/3989081659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa43c074bd0 0x7fa43c19a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:22.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.660+0000 7fa43a000700 1 -- 172.21.15.37:0/3989081659 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa43c1a2250 con 0x7fa43c0761a0 2024-03-28T18:10:22.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.660+0000 7fa43a000700 1 --2- 172.21.15.37:0/3989081659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa43c0761a0 0x7fa43c19ab30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa42800eb30 tx=0x7fa42800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:22.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.660+0000 7fa433400700 1 -- 172.21.15.37:0/3989081659 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42800cbd0 con 0x7fa43c0761a0 2024-03-28T18:10:22.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.661+0000 7fa433400700 1 -- 172.21.15.37:0/3989081659 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa428004500 con 0x7fa43c0761a0 2024-03-28T18:10:22.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.661+0000 7fa433400700 1 -- 172.21.15.37:0/3989081659 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa428010430 con 0x7fa43c0761a0 2024-03-28T18:10:22.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.661+0000 7fa441c00700 1 -- 172.21.15.37:0/3989081659 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa43c1a2540 con 0x7fa43c0761a0 2024-03-28T18:10:22.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.661+0000 7fa441c00700 1 -- 172.21.15.37:0/3989081659 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa43c1a2a30 con 0x7fa43c0761a0 2024-03-28T18:10:22.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.662+0000 7fa441c00700 1 -- 172.21.15.37:0/3989081659 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa43c068e50 con 0x7fa43c0761a0 2024-03-28T18:10:22.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.664+0000 7fa433400700 1 -- 172.21.15.37:0/3989081659 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa428003f30 con 0x7fa43c0761a0 2024-03-28T18:10:22.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.666+0000 7fa433400700 1 --2- 172.21.15.37:0/3989081659 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa4200a2890 0x7fa4200a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:22.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.666+0000 7fa433400700 1 -- 172.21.15.37:0/3989081659 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa4280ceda0 con 0x7fa43c0761a0 2024-03-28T18:10:22.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.667+0000 7fa43aa00700 1 --2- 172.21.15.37:0/3989081659 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa4200a2890 0x7fa4200a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:22.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.667+0000 7fa43aa00700 1 --2- 172.21.15.37:0/3989081659 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa4200a2890 0x7fa4200a4d50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa42c000c00 tx=0x7fa42c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:22.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.669+0000 7fa433400700 1 -- 172.21.15.37:0/3989081659 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa42809bdf0 con 0x7fa43c0761a0 2024-03-28T18:10:22.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.903+0000 7fa441c00700 1 -- 172.21.15.37:0/3989081659 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa43c0625d0 con 0x7fa4200a2890 2024-03-28T18:10:22.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.906+0000 7fa433400700 1 -- 172.21.15.37:0/3989081659 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7fa43c0625d0 con 0x7fa4200a2890 2024-03-28T18:10:22.906 INFO:teuthology.orchestra.run.smithi037.stdout:HOST ADDR LABELS STATUS 2024-03-28T18:10:22.906 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037 172.21.15.37 2024-03-28T18:10:22.907 INFO:teuthology.orchestra.run.smithi037.stdout:smithi059 172.21.15.59 2024-03-28T18:10:22.907 INFO:teuthology.orchestra.run.smithi037.stdout:smithi170 172.21.15.170 2024-03-28T18:10:22.907 INFO:teuthology.orchestra.run.smithi037.stdout:3 hosts in cluster 2024-03-28T18:10:22.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.910+0000 7fa441c00700 1 -- 172.21.15.37:0/3989081659 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa4200a2890 msgr2=0x7fa4200a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:22.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.910+0000 7fa441c00700 1 --2- 172.21.15.37:0/3989081659 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa4200a2890 0x7fa4200a4d50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa42c000c00 tx=0x7fa42c005fb0 comp rx=0 tx=0).stop 2024-03-28T18:10:22.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.910+0000 7fa441c00700 1 -- 172.21.15.37:0/3989081659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa43c0761a0 msgr2=0x7fa43c19ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:22.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.910+0000 7fa441c00700 1 --2- 172.21.15.37:0/3989081659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa43c0761a0 0x7fa43c19ab30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa42800eb30 tx=0x7fa42800ee40 comp rx=0 tx=0).stop 2024-03-28T18:10:22.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.910+0000 7fa441c00700 1 -- 172.21.15.37:0/3989081659 shutdown_connections 2024-03-28T18:10:22.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.910+0000 7fa441c00700 1 --2- 172.21.15.37:0/3989081659 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fa4200a2890 0x7fa4200a4d50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:22.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.910+0000 7fa441c00700 1 --2- 172.21.15.37:0/3989081659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa43c10a0b0 0x7fa43c1a1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:22.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.910+0000 7fa441c00700 1 --2- 172.21.15.37:0/3989081659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa43c074bd0 0x7fa43c19a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:22.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.910+0000 7fa441c00700 1 --2- 172.21.15.37:0/3989081659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa43c0761a0 0x7fa43c19ab30 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:22.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.910+0000 7fa441c00700 1 -- 172.21.15.37:0/3989081659 >> 172.21.15.37:0/3989081659 conn(0x7fa43c0fd7a0 msgr2=0x7fa43c105fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:22.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.911+0000 7fa441c00700 1 -- 172.21.15.37:0/3989081659 shutdown_connections 2024-03-28T18:10:22.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:22.911+0000 7fa441c00700 1 -- 172.21.15.37:0/3989081659 wait complete. 2024-03-28T18:10:23.334 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:23 smithi037 ceph-mon[31460]: pgmap v381: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:10:23.334 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:23 smithi037 ceph-mon[31460]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:10:23.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:23 smithi059 ceph-mon[36190]: pgmap v381: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:10:23.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:23 smithi059 ceph-mon[36190]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:10:23.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:23 smithi170 ceph-mon[36280]: pgmap v381: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:10:23.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:23 smithi170 ceph-mon[36280]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:10:23.739 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-28T18:10:24.203 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:10:25.396 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:25 smithi037 ceph-mon[31460]: pgmap v382: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:25.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:25 smithi059 ceph-mon[36190]: pgmap v382: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:25.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:25 smithi170 ceph-mon[36280]: pgmap v382: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:26.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.463+0000 7f3d15000700 1 -- 172.21.15.37:0/1128242877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d10105fa0 msgr2=0x7f3d1010c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:26.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.463+0000 7f3d15000700 1 --2- 172.21.15.37:0/1128242877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d10105fa0 0x7f3d1010c850 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3d08009f90 tx=0x7f3d0800b330 comp rx=0 tx=0).stop 2024-03-28T18:10:26.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.463+0000 7f3d15000700 1 -- 172.21.15.37:0/1128242877 shutdown_connections 2024-03-28T18:10:26.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.463+0000 7f3d15000700 1 --2- 172.21.15.37:0/1128242877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d10105fa0 0x7f3d1010c850 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:26.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.463+0000 7f3d15000700 1 --2- 172.21.15.37:0/1128242877 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d10104400 0x7f3d10104820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:26.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.464+0000 7f3d15000700 1 --2- 172.21.15.37:0/1128242877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d10105600 0x7f3d10105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:26.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.464+0000 7f3d15000700 1 -- 172.21.15.37:0/1128242877 >> 172.21.15.37:0/1128242877 conn(0x7f3d100ff9e0 msgr2=0x7f3d10101e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:26.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.464+0000 7f3d15000700 1 -- 172.21.15.37:0/1128242877 shutdown_connections 2024-03-28T18:10:26.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.464+0000 7f3d15000700 1 -- 172.21.15.37:0/1128242877 wait complete. 2024-03-28T18:10:26.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.465+0000 7f3d15000700 1 Processor -- start 2024-03-28T18:10:26.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.465+0000 7f3d15000700 1 -- start start 2024-03-28T18:10:26.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.466+0000 7f3d15000700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d10104400 0x7f3d1019a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:26.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.466+0000 7f3d15000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d10105600 0x7f3d1019aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:26.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.466+0000 7f3d15000700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d10105fa0 0x7f3d101a1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:26.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.466+0000 7f3d15000700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d10110020 con 0x7f3d10105600 2024-03-28T18:10:26.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.466+0000 7f3d15000700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1010fea0 con 0x7f3d10104400 2024-03-28T18:10:26.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.466+0000 7f3d15000700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d101101a0 con 0x7f3d10105fa0 2024-03-28T18:10:26.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.466+0000 7f3d0de00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d10104400 0x7f3d1019a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:26.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.466+0000 7f3d0de00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d10104400 0x7f3d1019a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:57786/0 (socket says 172.21.15.37:57786) 2024-03-28T18:10:26.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.466+0000 7f3d0de00700 1 -- 172.21.15.37:0/120697842 learned_addr learned my addr 172.21.15.37:0/120697842 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:26.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.466+0000 7f3d0d400700 1 --2- 172.21.15.37:0/120697842 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d10105600 0x7f3d1019aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:26.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.466+0000 7f3d0e800700 1 --2- 172.21.15.37:0/120697842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d10105fa0 0x7f3d101a1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:26.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.466+0000 7f3d0de00700 1 -- 172.21.15.37:0/120697842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d10105fa0 msgr2=0x7f3d101a1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:26.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.466+0000 7f3d0de00700 1 --2- 172.21.15.37:0/120697842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d10105fa0 0x7f3d101a1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:26.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.467+0000 7f3d0de00700 1 -- 172.21.15.37:0/120697842 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d10105600 msgr2=0x7f3d1019aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:26.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.467+0000 7f3d0de00700 1 --2- 172.21.15.37:0/120697842 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d10105600 0x7f3d1019aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:26.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.467+0000 7f3d0de00700 1 -- 172.21.15.37:0/120697842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d101a2170 con 0x7f3d10104400 2024-03-28T18:10:26.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.467+0000 7f3d0d400700 1 --2- 172.21.15.37:0/120697842 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d10105600 0x7f3d1019aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:26.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.467+0000 7f3d0de00700 1 --2- 172.21.15.37:0/120697842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d10104400 0x7f3d1019a510 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000c990 tx=0x7f3d0000cca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:26.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.467+0000 7f3d06a00700 1 -- 172.21.15.37:0/120697842 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00007b00 con 0x7f3d10104400 2024-03-28T18:10:26.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.467+0000 7f3d06a00700 1 -- 172.21.15.37:0/120697842 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d00007ca0 con 0x7f3d10104400 2024-03-28T18:10:26.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.467+0000 7f3d15000700 1 -- 172.21.15.37:0/120697842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d10076630 con 0x7f3d10104400 2024-03-28T18:10:26.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.467+0000 7f3d06a00700 1 -- 172.21.15.37:0/120697842 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d000186f0 con 0x7f3d10104400 2024-03-28T18:10:26.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.467+0000 7f3d15000700 1 -- 172.21.15.37:0/120697842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d10076bc0 con 0x7f3d10104400 2024-03-28T18:10:26.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.469+0000 7f3d15000700 1 -- 172.21.15.37:0/120697842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d10068e50 con 0x7f3d10104400 2024-03-28T18:10:26.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.470+0000 7f3d06a00700 1 -- 172.21.15.37:0/120697842 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3d00018890 con 0x7f3d10104400 2024-03-28T18:10:26.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.472+0000 7f3d06a00700 1 --2- 172.21.15.37:0/120697842 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f3cf40a2910 0x7f3cf40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:26.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.472+0000 7f3d06a00700 1 -- 172.21.15.37:0/120697842 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3d000ce960 con 0x7f3d10104400 2024-03-28T18:10:26.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.472+0000 7f3d0d400700 1 --2- 172.21.15.37:0/120697842 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f3cf40a2910 0x7f3cf40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:26.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.473+0000 7f3d0d400700 1 --2- 172.21.15.37:0/120697842 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f3cf40a2910 0x7f3cf40a4dd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc009750 tx=0x7f3cfc009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:26.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.476+0000 7f3d06a00700 1 -- 172.21.15.37:0/120697842 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3d00014030 con 0x7f3d10104400 2024-03-28T18:10:26.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.713+0000 7f3d15000700 1 -- 172.21.15.37:0/120697842 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d10062690 con 0x7f3cf40a2910 2024-03-28T18:10:26.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.717+0000 7f3d06a00700 1 -- 172.21.15.37:0/120697842 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7f3d10062690 con 0x7f3cf40a2910 2024-03-28T18:10:26.717 INFO:teuthology.orchestra.run.smithi037.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-28T18:10:26.717 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331005T400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-28T18:10:26.717 INFO:teuthology.orchestra.run.smithi037.stdout:smithi059 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330009L400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-28T18:10:26.717 INFO:teuthology.orchestra.run.smithi037.stdout:smithi170 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7476002R480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-03-28T18:10:26.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.719+0000 7f3d15000700 1 -- 172.21.15.37:0/120697842 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f3cf40a2910 msgr2=0x7f3cf40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:26.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.719+0000 7f3d15000700 1 --2- 172.21.15.37:0/120697842 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f3cf40a2910 0x7f3cf40a4dd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc009750 tx=0x7f3cfc009310 comp rx=0 tx=0).stop 2024-03-28T18:10:26.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.719+0000 7f3d15000700 1 -- 172.21.15.37:0/120697842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d10104400 msgr2=0x7f3d1019a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:26.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.719+0000 7f3d15000700 1 --2- 172.21.15.37:0/120697842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d10104400 0x7f3d1019a510 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000c990 tx=0x7f3d0000cca0 comp rx=0 tx=0).stop 2024-03-28T18:10:26.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.720+0000 7f3d15000700 1 -- 172.21.15.37:0/120697842 shutdown_connections 2024-03-28T18:10:26.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.720+0000 7f3d15000700 1 --2- 172.21.15.37:0/120697842 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f3cf40a2910 0x7f3cf40a4dd0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:26.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.720+0000 7f3d15000700 1 --2- 172.21.15.37:0/120697842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d10105fa0 0x7f3d101a1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:26.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.720+0000 7f3d15000700 1 --2- 172.21.15.37:0/120697842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d10104400 0x7f3d1019a510 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:26.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.720+0000 7f3d15000700 1 --2- 172.21.15.37:0/120697842 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d10105600 0x7f3d1019aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:26.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.720+0000 7f3d15000700 1 -- 172.21.15.37:0/120697842 >> 172.21.15.37:0/120697842 conn(0x7f3d100ff9e0 msgr2=0x7f3d10108830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:26.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.720+0000 7f3d15000700 1 -- 172.21.15.37:0/120697842 shutdown_connections 2024-03-28T18:10:26.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:26.720+0000 7f3d15000700 1 -- 172.21.15.37:0/120697842 wait complete. 2024-03-28T18:10:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:27 smithi037 ceph-mon[31460]: pgmap v383: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:10:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:27 smithi037 ceph-mon[31460]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:10:27.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:27 smithi059 ceph-mon[36190]: pgmap v383: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:10:27.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:27 smithi059 ceph-mon[36190]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:10:27.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:27 smithi170 ceph-mon[36280]: pgmap v383: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:10:27.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:27 smithi170 ceph-mon[36280]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:10:27.583 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T18:10:27.592 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi037.front.sepia.ceph.com 2024-03-28T18:10:27.593 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-28T18:10:28.053 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:10:29.225 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:29 smithi037 ceph-mon[31460]: pgmap v384: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:29.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:29 smithi059 ceph-mon[36190]: pgmap v384: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:29.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:29 smithi170 ceph-mon[36280]: pgmap v384: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:30.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.185+0000 7f027c000700 1 -- 172.21.15.37:0/960387720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0274074b90 msgr2=0x7f0274075010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:30.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.185+0000 7f027c000700 1 --2- 172.21.15.37:0/960387720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0274074b90 0x7f0274075010 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f02600098f0 tx=0x7f0260009c00 comp rx=0 tx=0).stop 2024-03-28T18:10:30.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.186+0000 7f027c000700 1 -- 172.21.15.37:0/960387720 shutdown_connections 2024-03-28T18:10:30.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.186+0000 7f027c000700 1 --2- 172.21.15.37:0/960387720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f027410a0a0 0x7f027410c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:30.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.186+0000 7f027c000700 1 --2- 172.21.15.37:0/960387720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0274076160 0x7f02740745c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:30.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.186+0000 7f027c000700 1 --2- 172.21.15.37:0/960387720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0274074b90 0x7f0274075010 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:30.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.186+0000 7f027c000700 1 -- 172.21.15.37:0/960387720 >> 172.21.15.37:0/960387720 conn(0x7f02740fd7a0 msgr2=0x7f02740ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:30.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.186+0000 7f027c000700 1 -- 172.21.15.37:0/960387720 shutdown_connections 2024-03-28T18:10:30.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.186+0000 7f027c000700 1 -- 172.21.15.37:0/960387720 wait complete. 2024-03-28T18:10:30.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.188+0000 7f027c000700 1 Processor -- start 2024-03-28T18:10:30.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.188+0000 7f027c000700 1 -- start start 2024-03-28T18:10:30.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.188+0000 7f027c000700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0274074b90 0x7f027419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:30.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.188+0000 7f027c000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0274076160 0x7f027419ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:30.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.189+0000 7f0279600700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0274074b90 0x7f027419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:30.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.189+0000 7f027c000700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f027410a0a0 0x7f02741a1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:30.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.189+0000 7f027c000700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027410ffc0 con 0x7f0274076160 2024-03-28T18:10:30.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.189+0000 7f027c000700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027410fe40 con 0x7f0274074b90 2024-03-28T18:10:30.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.189+0000 7f0279600700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0274074b90 0x7f027419a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:57806/0 (socket says 172.21.15.37:57806) 2024-03-28T18:10:30.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.189+0000 7f027c000700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0274110140 con 0x7f027410a0a0 2024-03-28T18:10:30.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.189+0000 7f0279600700 1 -- 172.21.15.37:0/2295920776 learned_addr learned my addr 172.21.15.37:0/2295920776 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:30.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.189+0000 7f027a000700 1 --2- 172.21.15.37:0/2295920776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f027410a0a0 0x7f02741a1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:30.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.189+0000 7f0279600700 1 -- 172.21.15.37:0/2295920776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f027410a0a0 msgr2=0x7f02741a1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:30.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.189+0000 7f0279600700 1 --2- 172.21.15.37:0/2295920776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f027410a0a0 0x7f02741a1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:30.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.189+0000 7f0279600700 1 -- 172.21.15.37:0/2295920776 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0274076160 msgr2=0x7f027419ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:30.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.189+0000 7f0279600700 1 --2- 172.21.15.37:0/2295920776 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0274076160 0x7f027419ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:30.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.189+0000 7f0279600700 1 -- 172.21.15.37:0/2295920776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02741a2290 con 0x7f0274074b90 2024-03-28T18:10:30.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.189+0000 7f027a000700 1 --2- 172.21.15.37:0/2295920776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f027410a0a0 0x7f02741a1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:30.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.190+0000 7f0279600700 1 --2- 172.21.15.37:0/2295920776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0274074b90 0x7f027419a630 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f026800ec10 tx=0x7f026800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:30.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.190+0000 7f0272000700 1 -- 172.21.15.37:0/2295920776 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026800cde0 con 0x7f0274074b90 2024-03-28T18:10:30.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.190+0000 7f0272000700 1 -- 172.21.15.37:0/2295920776 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0268004500 con 0x7f0274074b90 2024-03-28T18:10:30.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.190+0000 7f0272000700 1 -- 172.21.15.37:0/2295920776 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0268010690 con 0x7f0274074b90 2024-03-28T18:10:30.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.190+0000 7f027c000700 1 -- 172.21.15.37:0/2295920776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02741a2580 con 0x7f0274074b90 2024-03-28T18:10:30.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.190+0000 7f027c000700 1 -- 172.21.15.37:0/2295920776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02741a2a70 con 0x7f0274074b90 2024-03-28T18:10:30.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.191+0000 7f027c000700 1 -- 172.21.15.37:0/2295920776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0274068e50 con 0x7f0274074b90 2024-03-28T18:10:30.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.193+0000 7f0272000700 1 -- 172.21.15.37:0/2295920776 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0268003fc0 con 0x7f0274074b90 2024-03-28T18:10:30.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.194+0000 7f0272000700 1 --2- 172.21.15.37:0/2295920776 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f025c0a2910 0x7f025c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:30.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.194+0000 7f0272000700 1 -- 172.21.15.37:0/2295920776 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f02680ceea0 con 0x7f0274074b90 2024-03-28T18:10:30.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.194+0000 7f0278c00700 1 --2- 172.21.15.37:0/2295920776 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f025c0a2910 0x7f025c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:30.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.195+0000 7f0278c00700 1 --2- 172.21.15.37:0/2295920776 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f025c0a2910 0x7f025c0a4dd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0260009230 tx=0x7f02600058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:30.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.198+0000 7f0272000700 1 -- 172.21.15.37:0/2295920776 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f026809bef0 con 0x7f0274074b90 2024-03-28T18:10:30.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.574+0000 7f027c000700 1 -- 172.21.15.37:0/2295920776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0274062da0 con 0x7f0274074b90 2024-03-28T18:10:30.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.575+0000 7f0272000700 1 -- 172.21.15.37:0/2295920776 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+2986 (secure 0 0 0) 0x7f026809b7f0 con 0x7f0274074b90 2024-03-28T18:10:30.576 INFO:teuthology.orchestra.run.smithi037.stdout:e22 2024-03-28T18:10:30.576 INFO:teuthology.orchestra.run.smithi037.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T18:10:30.576 INFO:teuthology.orchestra.run.smithi037.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T18:10:30.576 INFO:teuthology.orchestra.run.smithi037.stdout:legacy client fscid: 1 2024-03-28T18:10:30.576 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:30.576 INFO:teuthology.orchestra.run.smithi037.stdout:Filesystem 'cephfs' (1) 2024-03-28T18:10:30.576 INFO:teuthology.orchestra.run.smithi037.stdout:fs_name cephfs 2024-03-28T18:10:30.576 INFO:teuthology.orchestra.run.smithi037.stdout:epoch 22 2024-03-28T18:10:30.576 INFO:teuthology.orchestra.run.smithi037.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-28T18:10:30.577 INFO:teuthology.orchestra.run.smithi037.stdout:created 2024-03-28T18:08:19.968894+0000 2024-03-28T18:10:30.577 INFO:teuthology.orchestra.run.smithi037.stdout:modified 2024-03-28T18:08:25.415653+0000 2024-03-28T18:10:30.577 INFO:teuthology.orchestra.run.smithi037.stdout:tableserver 0 2024-03-28T18:10:30.577 INFO:teuthology.orchestra.run.smithi037.stdout:root 0 2024-03-28T18:10:30.577 INFO:teuthology.orchestra.run.smithi037.stdout:session_timeout 300 2024-03-28T18:10:30.577 INFO:teuthology.orchestra.run.smithi037.stdout:session_autoclose 300 2024-03-28T18:10:30.577 INFO:teuthology.orchestra.run.smithi037.stdout:max_file_size 1099511627776 2024-03-28T18:10:30.577 INFO:teuthology.orchestra.run.smithi037.stdout:max_xattr_size 65536 2024-03-28T18:10:30.577 INFO:teuthology.orchestra.run.smithi037.stdout:required_client_features {} 2024-03-28T18:10:30.577 INFO:teuthology.orchestra.run.smithi037.stdout:last_failure 0 2024-03-28T18:10:30.577 INFO:teuthology.orchestra.run.smithi037.stdout:last_failure_osd_epoch 0 2024-03-28T18:10:30.577 INFO:teuthology.orchestra.run.smithi037.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T18:10:30.577 INFO:teuthology.orchestra.run.smithi037.stdout:max_mds 3 2024-03-28T18:10:30.577 INFO:teuthology.orchestra.run.smithi037.stdout:in 0,1,2 2024-03-28T18:10:30.577 INFO:teuthology.orchestra.run.smithi037.stdout:up {0=24580,1=24569,2=24550} 2024-03-28T18:10:30.578 INFO:teuthology.orchestra.run.smithi037.stdout:failed 2024-03-28T18:10:30.578 INFO:teuthology.orchestra.run.smithi037.stdout:damaged 2024-03-28T18:10:30.578 INFO:teuthology.orchestra.run.smithi037.stdout:stopped 2024-03-28T18:10:30.578 INFO:teuthology.orchestra.run.smithi037.stdout:data_pools [3] 2024-03-28T18:10:30.578 INFO:teuthology.orchestra.run.smithi037.stdout:metadata_pool 2 2024-03-28T18:10:30.578 INFO:teuthology.orchestra.run.smithi037.stdout:inline_data disabled 2024-03-28T18:10:30.578 INFO:teuthology.orchestra.run.smithi037.stdout:balancer 2024-03-28T18:10:30.578 INFO:teuthology.orchestra.run.smithi037.stdout:bal_rank_mask -1 2024-03-28T18:10:30.578 INFO:teuthology.orchestra.run.smithi037.stdout:standby_count_wanted 0 2024-03-28T18:10:30.578 INFO:teuthology.orchestra.run.smithi037.stdout:qdb_cluster leader: 24580 members: 24550,24569,24580 2024-03-28T18:10:30.578 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.a{0:24580} state up:active seq 4 addr [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:10:30.578 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.c{0:24529} state up:standby-replay seq 1 addr [v2:172.21.15.170:6836/1308157923,v1:172.21.15.170:6837/1308157923] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:10:30.578 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.b{1:24569} state up:active seq 4 addr [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:10:30.578 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.f{1:24517} state up:standby-replay seq 1 addr [v2:172.21.15.170:6834/4133178144,v1:172.21.15.170:6835/4133178144] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:10:30.579 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.i{2:24550} state up:active seq 7 addr [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:10:30.579 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.h{2:24512} state up:standby-replay seq 1 addr [v2:172.21.15.59:6836/4034607523,v1:172.21.15.59:6837/4034607523] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:10:30.579 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:30.579 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:30.579 INFO:teuthology.orchestra.run.smithi037.stdout:Standby daemons: 2024-03-28T18:10:30.579 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:30.579 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.j{-1:14388} state up:standby seq 1 addr [v2:172.21.15.37:6834/967056034,v1:172.21.15.37:6835/967056034] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:10:30.579 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.g{-1:14454} state up:standby seq 1 addr [v2:172.21.15.37:6838/1926270371,v1:172.21.15.37:6839/1926270371] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:10:30.579 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.k{-1:24437} state up:standby seq 1 addr [v2:172.21.15.59:6832/16451159,v1:172.21.15.59:6833/16451159] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:10:30.579 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.l{-1:24448} state up:standby seq 1 addr [v2:172.21.15.170:6832/576965351,v1:172.21.15.170:6833/576965351] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:10:30.579 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.e{-1:24485} state up:standby seq 1 addr [v2:172.21.15.59:6834/2253706559,v1:172.21.15.59:6835/2253706559] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:10:30.579 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.d{-1:24487} state up:standby seq 1 addr [v2:172.21.15.37:6836/3972375740,v1:172.21.15.37:6837/3972375740] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:10:30.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.579+0000 7f027c000700 1 -- 172.21.15.37:0/2295920776 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f025c0a2910 msgr2=0x7f025c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:30.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.579+0000 7f027c000700 1 --2- 172.21.15.37:0/2295920776 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f025c0a2910 0x7f025c0a4dd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0260009230 tx=0x7f02600058e0 comp rx=0 tx=0).stop 2024-03-28T18:10:30.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.579+0000 7f027c000700 1 -- 172.21.15.37:0/2295920776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0274074b90 msgr2=0x7f027419a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:30.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.579+0000 7f027c000700 1 --2- 172.21.15.37:0/2295920776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0274074b90 0x7f027419a630 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f026800ec10 tx=0x7f026800c2d0 comp rx=0 tx=0).stop 2024-03-28T18:10:30.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.579+0000 7f027c000700 1 -- 172.21.15.37:0/2295920776 shutdown_connections 2024-03-28T18:10:30.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.579+0000 7f027c000700 1 --2- 172.21.15.37:0/2295920776 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f025c0a2910 0x7f025c0a4dd0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:30.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.579+0000 7f027c000700 1 --2- 172.21.15.37:0/2295920776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f027410a0a0 0x7f02741a1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:30.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.579+0000 7f027c000700 1 --2- 172.21.15.37:0/2295920776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0274074b90 0x7f027419a630 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:30.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.579+0000 7f027c000700 1 --2- 172.21.15.37:0/2295920776 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0274076160 0x7f027419ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:30.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.579+0000 7f027c000700 1 -- 172.21.15.37:0/2295920776 >> 172.21.15.37:0/2295920776 conn(0x7f02740fd7a0 msgr2=0x7f0274105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:30.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.579+0000 7f027c000700 1 -- 172.21.15.37:0/2295920776 shutdown_connections 2024-03-28T18:10:30.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:30.580+0000 7f027c000700 1 -- 172.21.15.37:0/2295920776 wait complete. 2024-03-28T18:10:30.581 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 22 2024-03-28T18:10:31.350 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:31 smithi037 ceph-mon[31460]: pgmap v385: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:10:31.350 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:31 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2295920776' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T18:10:31.433 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-28T18:10:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:31 smithi059 ceph-mon[36190]: pgmap v385: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:10:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:31 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2295920776' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T18:10:31.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:31 smithi170 ceph-mon[36280]: pgmap v385: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:10:31.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:31 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2295920776' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T18:10:31.936 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:10:33.334 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:33 smithi037 ceph-mon[31460]: pgmap v386: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:10:33.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:33 smithi059 ceph-mon[36190]: pgmap v386: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:10:33.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:33 smithi170 ceph-mon[36280]: pgmap v386: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:10:34.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.227+0000 7f1d32a00700 1 -- 172.21.15.37:0/1084492707 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2c104410 msgr2=0x7f1d2c104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:34.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.227+0000 7f1d32a00700 1 --2- 172.21.15.37:0/1084492707 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2c104410 0x7f1d2c104830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0099c0 tx=0x7f1d1c009cd0 comp rx=0 tx=0).stop 2024-03-28T18:10:34.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.227+0000 7f1d32a00700 1 -- 172.21.15.37:0/1084492707 shutdown_connections 2024-03-28T18:10:34.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.227+0000 7f1d32a00700 1 --2- 172.21.15.37:0/1084492707 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d2c105fb0 0x7f1d2c10c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:34.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.227+0000 7f1d32a00700 1 --2- 172.21.15.37:0/1084492707 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2c104410 0x7f1d2c104830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:34.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.227+0000 7f1d32a00700 1 --2- 172.21.15.37:0/1084492707 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d2c105610 0x7f1d2c105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:34.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.227+0000 7f1d32a00700 1 -- 172.21.15.37:0/1084492707 >> 172.21.15.37:0/1084492707 conn(0x7f1d2c0ff9b0 msgr2=0x7f1d2c101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:34.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.227+0000 7f1d32a00700 1 -- 172.21.15.37:0/1084492707 shutdown_connections 2024-03-28T18:10:34.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.227+0000 7f1d32a00700 1 -- 172.21.15.37:0/1084492707 wait complete. 2024-03-28T18:10:34.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.228+0000 7f1d32a00700 1 Processor -- start 2024-03-28T18:10:34.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.228+0000 7f1d32a00700 1 -- start start 2024-03-28T18:10:34.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.229+0000 7f1d32a00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2c104410 0x7f1d2c19e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:34.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.229+0000 7f1d32a00700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d2c105610 0x7f1d2c19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:34.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.229+0000 7f1d32a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d2c105fb0 0x7f1d2c1a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:34.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.229+0000 7f1d32a00700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c10ffd0 con 0x7f1d2c105fb0 2024-03-28T18:10:34.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.229+0000 7f1d32a00700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c10fe50 con 0x7f1d2c104410 2024-03-28T18:10:34.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.229+0000 7f1d32a00700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c110150 con 0x7f1d2c105610 2024-03-28T18:10:34.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.229+0000 7f1d2b400700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d2c105610 0x7f1d2c19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:34.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.229+0000 7f1d2be00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2c104410 0x7f1d2c19e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:34.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.229+0000 7f1d2b400700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d2c105610 0x7f1d2c19ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55024/0 (socket says 172.21.15.37:55024) 2024-03-28T18:10:34.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.229+0000 7f1d2b400700 1 -- 172.21.15.37:0/1674367503 learned_addr learned my addr 172.21.15.37:0/1674367503 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:34.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.229+0000 7f1d30a00700 1 --2- 172.21.15.37:0/1674367503 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d2c105fb0 0x7f1d2c1a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:34.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.230+0000 7f1d2be00700 1 -- 172.21.15.37:0/1674367503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d2c105610 msgr2=0x7f1d2c19ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:34.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.230+0000 7f1d2be00700 1 --2- 172.21.15.37:0/1674367503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d2c105610 0x7f1d2c19ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:34.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.230+0000 7f1d2be00700 1 -- 172.21.15.37:0/1674367503 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d2c105fb0 msgr2=0x7f1d2c1a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:34.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.230+0000 7f1d2be00700 1 --2- 172.21.15.37:0/1674367503 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d2c105fb0 0x7f1d2c1a5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:34.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.230+0000 7f1d2be00700 1 -- 172.21.15.37:0/1674367503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2c1a6650 con 0x7f1d2c104410 2024-03-28T18:10:34.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.230+0000 7f1d2be00700 1 --2- 172.21.15.37:0/1674367503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2c104410 0x7f1d2c19e9f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0098e0 tx=0x7f1d1c00bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:34.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.230+0000 7f1d28c00700 1 -- 172.21.15.37:0/1674367503 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c009e60 con 0x7f1d2c104410 2024-03-28T18:10:34.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.230+0000 7f1d28c00700 1 -- 172.21.15.37:0/1674367503 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d1c003e30 con 0x7f1d2c104410 2024-03-28T18:10:34.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.230+0000 7f1d28c00700 1 -- 172.21.15.37:0/1674367503 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c00fd90 con 0x7f1d2c104410 2024-03-28T18:10:34.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.230+0000 7f1d32a00700 1 -- 172.21.15.37:0/1674367503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d2c1a68e0 con 0x7f1d2c104410 2024-03-28T18:10:34.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.231+0000 7f1d32a00700 1 -- 172.21.15.37:0/1674367503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d2c1a6e10 con 0x7f1d2c104410 2024-03-28T18:10:34.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.232+0000 7f1d32a00700 1 -- 172.21.15.37:0/1674367503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d2c068e50 con 0x7f1d2c104410 2024-03-28T18:10:34.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.233+0000 7f1d28c00700 1 -- 172.21.15.37:0/1674367503 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1d1c00f4f0 con 0x7f1d2c104410 2024-03-28T18:10:34.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.235+0000 7f1d28c00700 1 --2- 172.21.15.37:0/1674367503 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f1d100a2910 0x7f1d100a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:34.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.235+0000 7f1d28c00700 1 -- 172.21.15.37:0/1674367503 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f1d1c0cf710 con 0x7f1d2c104410 2024-03-28T18:10:34.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.235+0000 7f1d2b400700 1 --2- 172.21.15.37:0/1674367503 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f1d100a2910 0x7f1d100a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:34.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.236+0000 7f1d2b400700 1 --2- 172.21.15.37:0/1674367503 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f1d100a2910 0x7f1d100a4dd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1d18009680 tx=0x7f1d18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:34.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.239+0000 7f1d28c00700 1 -- 172.21.15.37:0/1674367503 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1d1c013070 con 0x7f1d2c104410 2024-03-28T18:10:34.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.469+0000 7f1d32a00700 1 -- 172.21.15.37:0/1674367503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f1d2c1a7710 con 0x7f1d2c104410 2024-03-28T18:10:34.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.470+0000 7f1d28c00700 1 -- 172.21.15.37:0/1674367503 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v87) v1 ==== 56+0+5278 (secure 0 0 0) 0x7f1d1c09c7f0 con 0x7f1d2c104410 2024-03-28T18:10:34.471 INFO:teuthology.orchestra.run.smithi037.stdout:epoch 87 2024-03-28T18:10:34.471 INFO:teuthology.orchestra.run.smithi037.stdout:fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 2024-03-28T18:10:34.471 INFO:teuthology.orchestra.run.smithi037.stdout:created 2024-03-28T17:59:05.949847+0000 2024-03-28T18:10:34.471 INFO:teuthology.orchestra.run.smithi037.stdout:modified 2024-03-28T18:09:15.709901+0000 2024-03-28T18:10:34.471 INFO:teuthology.orchestra.run.smithi037.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-28T18:10:34.471 INFO:teuthology.orchestra.run.smithi037.stdout:crush_version 26 2024-03-28T18:10:34.471 INFO:teuthology.orchestra.run.smithi037.stdout:full_ratio 0.95 2024-03-28T18:10:34.471 INFO:teuthology.orchestra.run.smithi037.stdout:backfillfull_ratio 0.9 2024-03-28T18:10:34.471 INFO:teuthology.orchestra.run.smithi037.stdout:nearfull_ratio 0.85 2024-03-28T18:10:34.471 INFO:teuthology.orchestra.run.smithi037.stdout:require_min_compat_client luminous 2024-03-28T18:10:34.471 INFO:teuthology.orchestra.run.smithi037.stdout:min_compat_client luminous 2024-03-28T18:10:34.472 INFO:teuthology.orchestra.run.smithi037.stdout:require_osd_release squid 2024-03-28T18:10:34.472 INFO:teuthology.orchestra.run.smithi037.stdout:stretch_mode_enabled false 2024-03-28T18:10:34.472 INFO:teuthology.orchestra.run.smithi037.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-28T18:10:34.472 INFO:teuthology.orchestra.run.smithi037.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-03-28T18:10:34.472 INFO:teuthology.orchestra.run.smithi037.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-28T18:10:34.472 INFO:teuthology.orchestra.run.smithi037.stdout:max_osd 12 2024-03-28T18:10:34.472 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0 up in weight 1 up_from 8 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.37:6802/3562602035,v1:172.21.15.37:6803/3562602035] [v2:172.21.15.37:6804/3562602035,v1:172.21.15.37:6805/3562602035] exists,up db60618d-56f7-4130-8a90-84cbc480965b 2024-03-28T18:10:34.472 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1 up in weight 1 up_from 13 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6800/3326537211,v1:172.21.15.59:6801/3326537211] [v2:172.21.15.59:6802/3326537211,v1:172.21.15.59:6803/3326537211] exists,up 7a4b7ad6-905e-4258-9574-693d4277dfc6 2024-03-28T18:10:34.472 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2 up in weight 1 up_from 18 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6800/1269398427,v1:172.21.15.170:6801/1269398427] [v2:172.21.15.170:6802/1269398427,v1:172.21.15.170:6803/1269398427] exists,up 8b76ec1c-46bf-4336-89b5-12d82906e8b7 2024-03-28T18:10:34.472 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3 up in weight 1 up_from 25 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.37:6810/317805240,v1:172.21.15.37:6811/317805240] [v2:172.21.15.37:6812/317805240,v1:172.21.15.37:6813/317805240] exists,up 66eca1da-2fbd-44e1-8bbf-5635a2c70ad5 2024-03-28T18:10:34.472 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4 up in weight 1 up_from 33 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6808/2410632623,v1:172.21.15.59:6809/2410632623] [v2:172.21.15.59:6810/2410632623,v1:172.21.15.59:6811/2410632623] exists,up 4c877335-e42d-4d8e-9f48-a1d0a89ae438 2024-03-28T18:10:34.472 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5 up in weight 1 up_from 38 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6808/1218943899,v1:172.21.15.170:6809/1218943899] [v2:172.21.15.170:6810/1218943899,v1:172.21.15.170:6811/1218943899] exists,up b884842a-86e2-455b-9570-ff163b3bc0cf 2024-03-28T18:10:34.472 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6 up in weight 1 up_from 46 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.37:6818/1899652046,v1:172.21.15.37:6819/1899652046] [v2:172.21.15.37:6820/1899652046,v1:172.21.15.37:6821/1899652046] exists,up 249a1200-77f4-44ad-981f-1d352f732b12 2024-03-28T18:10:34.472 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7 up in weight 1 up_from 51 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6816/1343675690,v1:172.21.15.59:6817/1343675690] [v2:172.21.15.59:6818/1343675690,v1:172.21.15.59:6819/1343675690] exists,up c73338e6-f505-4517-a84b-f35e04750a99 2024-03-28T18:10:34.473 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8 up in weight 1 up_from 59 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6816/3235794556,v1:172.21.15.170:6817/3235794556] [v2:172.21.15.170:6818/3235794556,v1:172.21.15.170:6819/3235794556] exists,up 149b8297-ff2d-4a88-9b1e-0def021cc516 2024-03-28T18:10:34.473 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9 up in weight 1 up_from 67 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.37:6826/2307153008,v1:172.21.15.37:6827/2307153008] [v2:172.21.15.37:6828/2307153008,v1:172.21.15.37:6829/2307153008] exists,up 55c27cc2-c771-44b7-90fe-8f0feca44162 2024-03-28T18:10:34.473 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6824/2113830275,v1:172.21.15.59:6825/2113830275] [v2:172.21.15.59:6826/2113830275,v1:172.21.15.59:6827/2113830275] exists,up c0595135-ab81-49d3-8c23-19f960c03e18 2024-03-28T18:10:34.473 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11 up in weight 1 up_from 80 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6824/3082439360,v1:172.21.15.170:6825/3082439360] [v2:172.21.15.170:6826/3082439360,v1:172.21.15.170:6827/3082439360] exists,up 5871e177-c17b-4637-a797-4fa5125920dd 2024-03-28T18:10:34.473 INFO:teuthology.orchestra.run.smithi037.stdout:pg_upmap_items 2.6 [8,11] 2024-03-28T18:10:34.473 INFO:teuthology.orchestra.run.smithi037.stdout:pg_upmap_items 2.10 [2,1] 2024-03-28T18:10:34.473 INFO:teuthology.orchestra.run.smithi037.stdout:pg_upmap_items 2.1d [2,4] 2024-03-28T18:10:34.473 INFO:teuthology.orchestra.run.smithi037.stdout:pg_upmap_items 2.1f [7,11] 2024-03-28T18:10:34.473 INFO:teuthology.orchestra.run.smithi037.stdout:pg_upmap_items 3.10 [5,9] 2024-03-28T18:10:34.473 INFO:teuthology.orchestra.run.smithi037.stdout:pg_upmap_items 3.14 [4,6] 2024-03-28T18:10:34.473 INFO:teuthology.orchestra.run.smithi037.stdout:pg_upmap_items 3.2c [4,9] 2024-03-28T18:10:34.473 INFO:teuthology.orchestra.run.smithi037.stdout:pg_upmap_items 3.38 [3,9] 2024-03-28T18:10:34.473 INFO:teuthology.orchestra.run.smithi037.stdout:blocklist 172.21.15.37:0/3023379428 expires 2024-03-29T18:00:13.809417+0000 2024-03-28T18:10:34.473 INFO:teuthology.orchestra.run.smithi037.stdout:blocklist 172.21.15.37:0/1443710231 expires 2024-03-29T18:00:13.809417+0000 2024-03-28T18:10:34.473 INFO:teuthology.orchestra.run.smithi037.stdout:blocklist 172.21.15.37:6800/1052839239 expires 2024-03-29T17:59:39.162038+0000 2024-03-28T18:10:34.473 INFO:teuthology.orchestra.run.smithi037.stdout:blocklist 172.21.15.37:6801/3459498509 expires 2024-03-29T18:00:13.809417+0000 2024-03-28T18:10:34.474 INFO:teuthology.orchestra.run.smithi037.stdout:blocklist 172.21.15.37:6801/1052839239 expires 2024-03-29T17:59:39.162038+0000 2024-03-28T18:10:34.474 INFO:teuthology.orchestra.run.smithi037.stdout:blocklist 172.21.15.37:6800/3459498509 expires 2024-03-29T18:00:13.809417+0000 2024-03-28T18:10:34.474 INFO:teuthology.orchestra.run.smithi037.stdout:blocklist 172.21.15.37:0/1007625490 expires 2024-03-29T17:59:39.162038+0000 2024-03-28T18:10:34.474 INFO:teuthology.orchestra.run.smithi037.stdout:blocklist 172.21.15.37:0/2895698584 expires 2024-03-29T17:59:39.162038+0000 2024-03-28T18:10:34.474 INFO:teuthology.orchestra.run.smithi037.stdout:blocklist 172.21.15.37:0/497313034 expires 2024-03-29T18:00:13.809417+0000 2024-03-28T18:10:34.474 INFO:teuthology.orchestra.run.smithi037.stdout:blocklist 172.21.15.37:0/3293943475 expires 2024-03-29T17:59:39.162038+0000 2024-03-28T18:10:34.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.473+0000 7f1d32a00700 1 -- 172.21.15.37:0/1674367503 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f1d100a2910 msgr2=0x7f1d100a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:34.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.473+0000 7f1d32a00700 1 --2- 172.21.15.37:0/1674367503 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f1d100a2910 0x7f1d100a4dd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1d18009680 tx=0x7f1d18009260 comp rx=0 tx=0).stop 2024-03-28T18:10:34.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.473+0000 7f1d32a00700 1 -- 172.21.15.37:0/1674367503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2c104410 msgr2=0x7f1d2c19e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:34.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.473+0000 7f1d32a00700 1 --2- 172.21.15.37:0/1674367503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2c104410 0x7f1d2c19e9f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0098e0 tx=0x7f1d1c00bf60 comp rx=0 tx=0).stop 2024-03-28T18:10:34.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.474+0000 7f1d32a00700 1 -- 172.21.15.37:0/1674367503 shutdown_connections 2024-03-28T18:10:34.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.474+0000 7f1d32a00700 1 --2- 172.21.15.37:0/1674367503 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f1d100a2910 0x7f1d100a4dd0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:34.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.474+0000 7f1d32a00700 1 --2- 172.21.15.37:0/1674367503 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d2c105fb0 0x7f1d2c1a5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:34.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.474+0000 7f1d32a00700 1 --2- 172.21.15.37:0/1674367503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2c104410 0x7f1d2c19e9f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:34.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.474+0000 7f1d32a00700 1 --2- 172.21.15.37:0/1674367503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d2c105610 0x7f1d2c19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:34.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.474+0000 7f1d32a00700 1 -- 172.21.15.37:0/1674367503 >> 172.21.15.37:0/1674367503 conn(0x7f1d2c0ff9b0 msgr2=0x7f1d2c0785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:34.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.474+0000 7f1d32a00700 1 -- 172.21.15.37:0/1674367503 shutdown_connections 2024-03-28T18:10:34.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:34.474+0000 7f1d32a00700 1 -- 172.21.15.37:0/1674367503 wait complete. 2024-03-28T18:10:35.311 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:35 smithi037 ceph-mon[31460]: pgmap v387: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:35.311 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:35 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1674367503' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T18:10:35.384 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-28T18:10:35.395 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:10:35.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:35 smithi059 ceph-mon[36190]: pgmap v387: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:35.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:35 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1674367503' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T18:10:35.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:35 smithi170 ceph-mon[36280]: pgmap v387: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:35.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:35 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1674367503' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T18:10:35.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.666+0000 7f9569000640 1 -- 172.21.15.37:0/731005745 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9564103290 msgr2=0x7f9564109b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:35.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.666+0000 7f9569000640 1 --2- 172.21.15.37:0/731005745 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9564103290 0x7f9564109b20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f955c009ca0 tx=0x7f955c02c100 comp rx=0 tx=0).stop 2024-03-28T18:10:35.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.666+0000 7f9569000640 1 -- 172.21.15.37:0/731005745 shutdown_connections 2024-03-28T18:10:35.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.666+0000 7f9569000640 1 --2- 172.21.15.37:0/731005745 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9564103290 0x7f9564109b20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:35.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.666+0000 7f9569000640 1 --2- 172.21.15.37:0/731005745 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95641028d0 0x7f9564102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:35.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.666+0000 7f9569000640 1 --2- 172.21.15.37:0/731005745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95641016e0 0x7f9564101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:35.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.666+0000 7f9569000640 1 -- 172.21.15.37:0/731005745 >> 172.21.15.37:0/731005745 conn(0x7f95640fcc50 msgr2=0x7f95640ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:35.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.666+0000 7f9569000640 1 -- 172.21.15.37:0/731005745 shutdown_connections 2024-03-28T18:10:35.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.667+0000 7f9569000640 1 -- 172.21.15.37:0/731005745 wait complete. 2024-03-28T18:10:35.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.668+0000 7f9569000640 1 Processor -- start 2024-03-28T18:10:35.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.668+0000 7f9569000640 1 -- start start 2024-03-28T18:10:35.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.668+0000 7f9569000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95641016e0 0x7f9564199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:35.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.668+0000 7f9569000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95641028d0 0x7f956419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:35.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.668+0000 7f9569000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9564103290 0x7f95641a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:35.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.668+0000 7f9569000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956410d360 con 0x7f95641016e0 2024-03-28T18:10:35.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.668+0000 7f9569000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956410d1e0 con 0x7f95641028d0 2024-03-28T18:10:35.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.668+0000 7f9569000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956410d4e0 con 0x7f9564103290 2024-03-28T18:10:35.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.668+0000 7f9561e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95641016e0 0x7f9564199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:35.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.668+0000 7f9561e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95641016e0 0x7f9564199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49014/0 (socket says 172.21.15.37:49014) 2024-03-28T18:10:35.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.668+0000 7f9562800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9564103290 0x7f95641a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:35.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.668+0000 7f9561e00640 1 -- 172.21.15.37:0/4221812538 learned_addr learned my addr 172.21.15.37:0/4221812538 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:35.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.669+0000 7f955be00640 1 --2- 172.21.15.37:0/4221812538 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95641028d0 0x7f956419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:35.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.669+0000 7f9561e00640 1 -- 172.21.15.37:0/4221812538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9564103290 msgr2=0x7f95641a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:35.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.669+0000 7f9561e00640 1 --2- 172.21.15.37:0/4221812538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9564103290 0x7f95641a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:35.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.669+0000 7f9561e00640 1 -- 172.21.15.37:0/4221812538 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95641028d0 msgr2=0x7f956419a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:35.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.669+0000 7f9561e00640 1 --2- 172.21.15.37:0/4221812538 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95641028d0 0x7f956419a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:35.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.669+0000 7f9561e00640 1 -- 172.21.15.37:0/4221812538 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95641a17b0 con 0x7f95641016e0 2024-03-28T18:10:35.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.669+0000 7f9562800640 1 --2- 172.21.15.37:0/4221812538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9564103290 0x7f95641a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:35.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.669+0000 7f9561e00640 1 --2- 172.21.15.37:0/4221812538 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95641016e0 0x7f9564199b90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f955400e9c0 tx=0x7f955400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:35.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.669+0000 7f955aa00640 1 -- 172.21.15.37:0/4221812538 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955400cd80 con 0x7f95641016e0 2024-03-28T18:10:35.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.669+0000 7f955aa00640 1 -- 172.21.15.37:0/4221812538 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95540035f0 con 0x7f95641016e0 2024-03-28T18:10:35.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.669+0000 7f955aa00640 1 -- 172.21.15.37:0/4221812538 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9554010690 con 0x7f95641016e0 2024-03-28T18:10:35.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.670+0000 7f9569000640 1 -- 172.21.15.37:0/4221812538 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95641a1aa0 con 0x7f95641016e0 2024-03-28T18:10:35.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.670+0000 7f9569000640 1 -- 172.21.15.37:0/4221812538 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95641a1fa0 con 0x7f95641016e0 2024-03-28T18:10:35.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.671+0000 7f9569000640 1 -- 172.21.15.37:0/4221812538 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95641a2230 con 0x7f95641016e0 2024-03-28T18:10:35.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.674+0000 7f955aa00640 1 -- 172.21.15.37:0/4221812538 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f95540037a0 con 0x7f95641016e0 2024-03-28T18:10:35.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.677+0000 7f955aa00640 1 --2- 172.21.15.37:0/4221812538 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f95480a2880 0x7f95480a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:35.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.677+0000 7f955aa00640 1 -- 172.21.15.37:0/4221812538 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f95540cede0 con 0x7f95641016e0 2024-03-28T18:10:35.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.677+0000 7f955be00640 1 --2- 172.21.15.37:0/4221812538 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f95480a2880 0x7f95480a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:35.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.678+0000 7f955be00640 1 --2- 172.21.15.37:0/4221812538 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f95480a2880 0x7f95480a4d40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f956419b0a0 tx=0x7f954c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:35.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.682+0000 7f955aa00640 1 -- 172.21.15.37:0/4221812538 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f955409be30 con 0x7f95641016e0 2024-03-28T18:10:35.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.903+0000 7f9569000640 1 -- 172.21.15.37:0/4221812538 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95641a2400 con 0x7f95641016e0 2024-03-28T18:10:35.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.905+0000 7f955aa00640 1 -- 172.21.15.37:0/4221812538 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22575 (secure 0 0 0) 0x7f955409b730 con 0x7f95641016e0 2024-03-28T18:10:35.905 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:35.905 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":87,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","created":"2024-03-28T17:59:05.949847+0000","modified":"2024-03-28T18:09:15.709901+0000","last_up_change":"2024-03-28T18:07:21.021030+0000","last_in_change":"2024-03-28T18:07:01.416192+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:03:00.517967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:08:18.365827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:08:19.446282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db60618d-56f7-4130-8a90-84cbc480965b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6803","nonce":3562602035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6805","nonce":3562602035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6809","nonce":3562602035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6807","nonce":3562602035}]},"public_addr":"172.21.15.37:6803/3562602035","cluster_addr":"172.21.15.37:6805/3562602035","heartbeat_back_addr":"172.21.15.37:6809/3562602035","heartbeat_front_addr":"172.21.15.37:6807/3562602035","state":["exists","up"]},{"osd":1,"uuid":"7a4b7ad6-905e-4258-9574-693d4277dfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6801","nonce":3326537211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6803","nonce":3326537211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6807","nonce":3326537211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6805","nonce":3326537211}]},"public_addr":"172.21.15.59:6801/3326537211","cluster_addr":"172.21.15.59:6803/3326537211","heartbeat_back_addr":"172.21.15.59:6807/3326537211","heartbeat_front_addr":"172.21.15.59:6805/3326537211","state":["exists","up"]},{"osd":2,"uuid":"8b76ec1c-46bf-4336-89b5-12d82906e8b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6801","nonce":1269398427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6803","nonce":1269398427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6807","nonce":1269398427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6805","nonce":1269398427}]},"public_addr":"172.21.15.170:6801/1269398427","cluster_addr":"172.21.15.170:6803/1269398427","heartbeat_back_addr":"172.21.15.170:6807/1269398427","heartbeat_front_addr":"172.21.15.170:6805/1269398427","state":["exists","up"]},{"osd":3,"uuid":"66eca1da-2fbd-44e1-8bbf-5635a2c70ad5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6811","nonce":317805240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6813","nonce":317805240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6817","nonce":317805240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6815","nonce":317805240}]},"public_addr":"172.21.15.37:6811/317805240","cluster_addr":"172.21.15.37:6813/317805240","heartbeat_back_addr":"172.21.15.37:6817/317805240","heartbeat_front_addr":"172.21.15.37:6815/317805240","state":["exists","up"]},{"osd":4,"uuid":"4c877335-e42d-4d8e-9f48-a1d0a89ae438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6809","nonce":2410632623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6811","nonce":2410632623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6815","nonce":2410632623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6813","nonce":2410632623}]},"public_addr":"172.21.15.59:6809/2410632623","cluster_addr":"172.21.15.59:6811/2410632623","heartbeat_back_addr":"172.21.15.59:6815/2410632623","heartbeat_front_addr":"172.21.15.59:6813/2410632623","state":["exists","up"]},{"osd":5,"uuid":"b884842a-86e2-455b-9570-ff163b3bc0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6809","nonce":1218943899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6811","nonce":1218943899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6815","nonce":1218943899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6813","nonce":1218943899}]},"public_addr":"172.21.15.170:6809/1218943899","cluster_addr":"172.21.15.170:6811/1218943899","heartbeat_back_addr":"172.21.15.170:6815/1218943899","heartbeat_front_addr":"172.21.15.170:6813/1218943899","state":["exists","up"]},{"osd":6,"uuid":"249a1200-77f4-44ad-981f-1d352f732b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6819","nonce":1899652046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6821","nonce":1899652046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6825","nonce":1899652046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6823","nonce":1899652046}]},"public_addr":"172.21.15.37:6819/1899652046","cluster_addr":"172.21.15.37:6821/1899652046","heartbeat_back_addr":"172.21.15.37:6825/1899652046","heartbeat_front_addr":"172.21.15.37:6823/1899652046","state":["exists","up"]},{"osd":7,"uuid":"c73338e6-f505-4517-a84b-f35e04750a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6817","nonce":1343675690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6819","nonce":1343675690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6823","nonce":1343675690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6821","nonce":1343675690}]},"public_addr":"172.21.15.59:6817/1343675690","cluster_addr":"172.21.15.59:6819/1343675690","heartbeat_back_addr":"172.21.15.59:6823/1343675690","heartbeat_front_addr":"172.21.15.59:6821/1343675690","state":["exists","up"]},{"osd":8,"uuid":"149b8297-ff2d-4a88-9b1e-0def021cc516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6817","nonce":3235794556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6819","nonce":3235794556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6823","nonce":3235794556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6821","nonce":3235794556}]},"public_addr":"172.21.15.170:6817/3235794556","cluster_addr":"172.21.15.170:6819/3235794556","heartbeat_back_addr":"172.21.15.170:6823/3235794556","heartbeat_front_addr":"172.21.15.170:6821/3235794556","state":["exists","up"]},{"osd":9,"uuid":"55c27cc2-c771-44b7-90fe-8f0feca44162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6827","nonce":2307153008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6829","nonce":2307153008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6833","nonce":2307153008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6831","nonce":2307153008}]},"public_addr":"172.21.15.37:6827/2307153008","cluster_addr":"172.21.15.37:6829/2307153008","heartbeat_back_addr":"172.21.15.37:6833/2307153008","heartbeat_front_addr":"172.21.15.37:6831/2307153008","state":["exists","up"]},{"osd":10,"uuid":"c0595135-ab81-49d3-8c23-19f960c03e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6825","nonce":2113830275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6827","nonce":2113830275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6831","nonce":2113830275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6829","nonce":2113830275}]},"public_addr":"172.21.15.59:6825/2113830275","cluster_addr":"172.21.15.59:6827/2113830275","heartbeat_back_addr":"172.21.15.59:6831/2113830275","heartbeat_front_addr":"172.21.15.59:6829/2113830275","state":["exists","up"]},{"osd":11,"uuid":"5871e177-c17b-4637-a797-4fa5125920dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6825","nonce":3082439360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6827","nonce":3082439360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6831","nonce":3082439360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6829","nonce":3082439360}]},"public_addr":"172.21.15.170:6825/3082439360","cluster_addr":"172.21.15.170:6827/3082439360","heartbeat_back_addr":"172.21.15.170:6831/3082439360","heartbeat_front_addr":"172.21.15.170:6829/3082439360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:07.690545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:33.183396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:57.400263+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:30.597211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:57.259044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:22.627823+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:57.585863+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:25.000476+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:50.876688+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:25.859761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:53.563146+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:07:19.304115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3023379428":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1443710231":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6800/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6801/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6801/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6800/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1007625490":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/2895698584":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/497313034":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/3293943475":"2024-03-29T17:59:39.162038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:10:35.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.907+0000 7f9569000640 1 -- 172.21.15.37:0/4221812538 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f95480a2880 msgr2=0x7f95480a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:35.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.907+0000 7f9569000640 1 --2- 172.21.15.37:0/4221812538 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f95480a2880 0x7f95480a4d40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f956419b0a0 tx=0x7f954c005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:35.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.907+0000 7f9569000640 1 -- 172.21.15.37:0/4221812538 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95641016e0 msgr2=0x7f9564199b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:35.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.907+0000 7f9569000640 1 --2- 172.21.15.37:0/4221812538 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95641016e0 0x7f9564199b90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f955400e9c0 tx=0x7f955400ee80 comp rx=0 tx=0).stop 2024-03-28T18:10:35.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.907+0000 7f9569000640 1 -- 172.21.15.37:0/4221812538 shutdown_connections 2024-03-28T18:10:35.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.907+0000 7f9569000640 1 --2- 172.21.15.37:0/4221812538 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f95480a2880 0x7f95480a4d40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:35.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.907+0000 7f9569000640 1 --2- 172.21.15.37:0/4221812538 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95641028d0 0x7f956419a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:35.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.908+0000 7f9569000640 1 --2- 172.21.15.37:0/4221812538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9564103290 0x7f95641a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:35.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.908+0000 7f9569000640 1 --2- 172.21.15.37:0/4221812538 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95641016e0 0x7f9564199b90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:35.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.908+0000 7f9569000640 1 -- 172.21.15.37:0/4221812538 >> 172.21.15.37:0/4221812538 conn(0x7f95640fcc50 msgr2=0x7f95640738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:35.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.908+0000 7f9569000640 1 -- 172.21.15.37:0/4221812538 shutdown_connections 2024-03-28T18:10:35.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:35.908+0000 7f9569000640 1 -- 172.21.15.37:0/4221812538 wait complete. 2024-03-28T18:10:35.926 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:03:00.517967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:08:18.365827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:08:19.446282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:10:35.927 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:10:36.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.159+0000 7f1117400640 1 -- 172.21.15.37:0/3204302547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11101028a0 msgr2=0x7f1110102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:36.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.159+0000 7f1117400640 1 --2- 172.21.15.37:0/3204302547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11101028a0 0x7f1110102d20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f10fc009900 tx=0x7f10fc02bc40 comp rx=0 tx=0).stop 2024-03-28T18:10:36.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.159+0000 7f1117400640 1 -- 172.21.15.37:0/3204302547 shutdown_connections 2024-03-28T18:10:36.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.159+0000 7f1117400640 1 --2- 172.21.15.37:0/3204302547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1110103260 0x7f1110109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:36.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.159+0000 7f1117400640 1 --2- 172.21.15.37:0/3204302547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11101028a0 0x7f1110102d20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:36.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.159+0000 7f1117400640 1 --2- 172.21.15.37:0/3204302547 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11101016b0 0x7f1110101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:36.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.159+0000 7f1117400640 1 -- 172.21.15.37:0/3204302547 >> 172.21.15.37:0/3204302547 conn(0x7f11100fcc40 msgr2=0x7f11100ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:36.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.159+0000 7f1117400640 1 -- 172.21.15.37:0/3204302547 shutdown_connections 2024-03-28T18:10:36.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.159+0000 7f1117400640 1 -- 172.21.15.37:0/3204302547 wait complete. 2024-03-28T18:10:36.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.160+0000 7f1117400640 1 Processor -- start 2024-03-28T18:10:36.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.161+0000 7f1117400640 1 -- start start 2024-03-28T18:10:36.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.161+0000 7f1117400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11101016b0 0x7f1110199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:36.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.161+0000 7f1117400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11101028a0 0x7f111019a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:36.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.161+0000 7f1117400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1110103260 0x7f11101a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:36.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.161+0000 7f1117400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111010d380 con 0x7f1110103260 2024-03-28T18:10:36.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.161+0000 7f1117400640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111010d200 con 0x7f11101016b0 2024-03-28T18:10:36.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.161+0000 7f1117400640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111010d500 con 0x7f11101028a0 2024-03-28T18:10:36.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.162+0000 7f1114a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11101016b0 0x7f1110199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:36.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.162+0000 7f1114a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11101016b0 0x7f1110199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:37646/0 (socket says 172.21.15.37:37646) 2024-03-28T18:10:36.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.162+0000 7f1114a00640 1 -- 172.21.15.37:0/2367001077 learned_addr learned my addr 172.21.15.37:0/2367001077 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:36.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.162+0000 7f110fe00640 1 --2- 172.21.15.37:0/2367001077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11101028a0 0x7f111019a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:36.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.162+0000 7f1115400640 1 --2- 172.21.15.37:0/2367001077 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1110103260 0x7f11101a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:36.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.162+0000 7f1114a00640 1 -- 172.21.15.37:0/2367001077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11101028a0 msgr2=0x7f111019a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:36.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.162+0000 7f1114a00640 1 --2- 172.21.15.37:0/2367001077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11101028a0 0x7f111019a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:36.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.162+0000 7f1114a00640 1 -- 172.21.15.37:0/2367001077 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1110103260 msgr2=0x7f11101a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:36.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.162+0000 7f1114a00640 1 --2- 172.21.15.37:0/2367001077 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1110103260 0x7f11101a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:36.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.162+0000 7f1114a00640 1 -- 172.21.15.37:0/2367001077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11101a1800 con 0x7f11101016b0 2024-03-28T18:10:36.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.162+0000 7f1115400640 1 --2- 172.21.15.37:0/2367001077 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1110103260 0x7f11101a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:10:36.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.162+0000 7f1114a00640 1 --2- 172.21.15.37:0/2367001077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11101016b0 0x7f1110199be0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f110400e9c0 tx=0x7f110400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:36.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.163+0000 7f110d600640 1 -- 172.21.15.37:0/2367001077 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110400cd80 con 0x7f11101016b0 2024-03-28T18:10:36.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.163+0000 7f1117400640 1 -- 172.21.15.37:0/2367001077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11101a1af0 con 0x7f11101016b0 2024-03-28T18:10:36.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.163+0000 7f110d600640 1 -- 172.21.15.37:0/2367001077 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11040035f0 con 0x7f11101016b0 2024-03-28T18:10:36.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.163+0000 7f110d600640 1 -- 172.21.15.37:0/2367001077 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1104010690 con 0x7f11101016b0 2024-03-28T18:10:36.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.163+0000 7f1117400640 1 -- 172.21.15.37:0/2367001077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11101a2060 con 0x7f11101016b0 2024-03-28T18:10:36.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.164+0000 7f1117400640 1 -- 172.21.15.37:0/2367001077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11101a2230 con 0x7f11101016b0 2024-03-28T18:10:36.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.167+0000 7f110d600640 1 -- 172.21.15.37:0/2367001077 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f11040037a0 con 0x7f11101016b0 2024-03-28T18:10:36.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.170+0000 7f110d600640 1 --2- 172.21.15.37:0/2367001077 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f10f80a2880 0x7f10f80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:36.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.170+0000 7f110d600640 1 -- 172.21.15.37:0/2367001077 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f11040d0130 con 0x7f11101016b0 2024-03-28T18:10:36.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.170+0000 7f110fe00640 1 --2- 172.21.15.37:0/2367001077 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f10f80a2880 0x7f10f80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:36.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.171+0000 7f110fe00640 1 --2- 172.21.15.37:0/2367001077 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f10f80a2880 0x7f10f80a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f10fc02c250 tx=0x7f10fc02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:36.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.173+0000 7f110d600640 1 -- 172.21.15.37:0/2367001077 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f110409bee0 con 0x7f11101016b0 2024-03-28T18:10:36.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:36 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/4221812538' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:36.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.384+0000 7f1117400640 1 -- 172.21.15.37:0/2367001077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f11101a2ab0 con 0x7f11101016b0 2024-03-28T18:10:36.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.384+0000 7f110d600640 1 -- 172.21.15.37:0/2367001077 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f110409b7e0 con 0x7f11101016b0 2024-03-28T18:10:36.384 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 1 2024-03-28T18:10:36.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.388+0000 7f1117400640 1 -- 172.21.15.37:0/2367001077 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f10f80a2880 msgr2=0x7f10f80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:36.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.388+0000 7f1117400640 1 --2- 172.21.15.37:0/2367001077 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f10f80a2880 0x7f10f80a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f10fc02c250 tx=0x7f10fc02cf90 comp rx=0 tx=0).stop 2024-03-28T18:10:36.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.388+0000 7f1117400640 1 -- 172.21.15.37:0/2367001077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11101016b0 msgr2=0x7f1110199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:36.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.388+0000 7f1117400640 1 --2- 172.21.15.37:0/2367001077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11101016b0 0x7f1110199be0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f110400e9c0 tx=0x7f110400ee80 comp rx=0 tx=0).stop 2024-03-28T18:10:36.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.388+0000 7f1117400640 1 -- 172.21.15.37:0/2367001077 shutdown_connections 2024-03-28T18:10:36.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.388+0000 7f1117400640 1 --2- 172.21.15.37:0/2367001077 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f10f80a2880 0x7f10f80a4d40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:36.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.388+0000 7f1117400640 1 --2- 172.21.15.37:0/2367001077 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1110103260 0x7f11101a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:36.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.388+0000 7f1117400640 1 --2- 172.21.15.37:0/2367001077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11101028a0 0x7f111019a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:36.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.388+0000 7f1117400640 1 --2- 172.21.15.37:0/2367001077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11101016b0 0x7f1110199be0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:36.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.388+0000 7f1117400640 1 -- 172.21.15.37:0/2367001077 >> 172.21.15.37:0/2367001077 conn(0x7f11100fcc40 msgr2=0x7f11100738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:36.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.388+0000 7f1117400640 1 -- 172.21.15.37:0/2367001077 shutdown_connections 2024-03-28T18:10:36.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.388+0000 7f1117400640 1 -- 172.21.15.37:0/2367001077 wait complete. 2024-03-28T18:10:36.416 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:10:36.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:36 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/4221812538' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:36.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:36 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/4221812538' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:36.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.640+0000 7f3016a00640 1 -- 172.21.15.37:0/451538531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3010102130 msgr2=0x7f3010104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:36.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.640+0000 7f3016a00640 1 --2- 172.21.15.37:0/451538531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3010102130 0x7f3010104520 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3000009ca0 tx=0x7f300002c100 comp rx=0 tx=0).stop 2024-03-28T18:10:36.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.640+0000 7f3016a00640 1 -- 172.21.15.37:0/451538531 shutdown_connections 2024-03-28T18:10:36.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.640+0000 7f3016a00640 1 --2- 172.21.15.37:0/451538531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3010107390 0x7f30101097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:36.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.640+0000 7f3016a00640 1 --2- 172.21.15.37:0/451538531 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3010104a60 0x7f3010106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:36.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.640+0000 7f3016a00640 1 --2- 172.21.15.37:0/451538531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3010102130 0x7f3010104520 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:36.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.640+0000 7f3016a00640 1 -- 172.21.15.37:0/451538531 >> 172.21.15.37:0/451538531 conn(0x7f30100fbb40 msgr2=0x7f30100fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:36.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.640+0000 7f3016a00640 1 -- 172.21.15.37:0/451538531 shutdown_connections 2024-03-28T18:10:36.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.641+0000 7f3016a00640 1 -- 172.21.15.37:0/451538531 wait complete. 2024-03-28T18:10:36.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.641+0000 7f3016a00640 1 Processor -- start 2024-03-28T18:10:36.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.642+0000 7f3016a00640 1 -- start start 2024-03-28T18:10:36.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.642+0000 7f3016a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3010102130 0x7f3010101aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:36.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.642+0000 7f3016a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3010104a60 0x7f30101000e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:36.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.642+0000 7f3016a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3010107390 0x7f3010100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:36.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.642+0000 7f3016a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301010d340 con 0x7f3010107390 2024-03-28T18:10:36.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.642+0000 7f3016a00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301010d1c0 con 0x7f3010104a60 2024-03-28T18:10:36.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.642+0000 7f3016a00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301010d4c0 con 0x7f3010102130 2024-03-28T18:10:36.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.642+0000 7f300fe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3010102130 0x7f3010101aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:36.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.642+0000 7f3007400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3010104a60 0x7f30101000e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:36.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.642+0000 7f3014a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3010107390 0x7f3010100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:36.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.642+0000 7f300fe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3010102130 0x7f3010101aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55080/0 (socket says 172.21.15.37:55080) 2024-03-28T18:10:36.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.642+0000 7f300fe00640 1 -- 172.21.15.37:0/3577911923 learned_addr learned my addr 172.21.15.37:0/3577911923 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:36.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.643+0000 7f300fe00640 1 -- 172.21.15.37:0/3577911923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3010104a60 msgr2=0x7f30101000e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:36.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.643+0000 7f300fe00640 1 --2- 172.21.15.37:0/3577911923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3010104a60 0x7f30101000e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:36.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.643+0000 7f300fe00640 1 -- 172.21.15.37:0/3577911923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3010107390 msgr2=0x7f3010100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:36.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.643+0000 7f300fe00640 1 --2- 172.21.15.37:0/3577911923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3010107390 0x7f3010100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:36.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.643+0000 7f300fe00640 1 -- 172.21.15.37:0/3577911923 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3010100eb0 con 0x7f3010102130 2024-03-28T18:10:36.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.643+0000 7f300fe00640 1 --2- 172.21.15.37:0/3577911923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3010102130 0x7f3010101aa0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f30000325f0 tx=0x7f3000035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:36.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.643+0000 7f300d600640 1 -- 172.21.15.37:0/3577911923 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3000039040 con 0x7f3010102130 2024-03-28T18:10:36.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.643+0000 7f300d600640 1 -- 172.21.15.37:0/3577911923 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3000044070 con 0x7f3010102130 2024-03-28T18:10:36.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.643+0000 7f3016a00640 1 -- 172.21.15.37:0/3577911923 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30101a1ad0 con 0x7f3010102130 2024-03-28T18:10:36.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.643+0000 7f300d600640 1 -- 172.21.15.37:0/3577911923 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30000044d0 con 0x7f3010102130 2024-03-28T18:10:36.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.643+0000 7f3016a00640 1 -- 172.21.15.37:0/3577911923 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30101a1ff0 con 0x7f3010102130 2024-03-28T18:10:36.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.644+0000 7f3016a00640 1 -- 172.21.15.37:0/3577911923 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30100fd630 con 0x7f3010102130 2024-03-28T18:10:36.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.646+0000 7f300d600640 1 -- 172.21.15.37:0/3577911923 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3000004670 con 0x7f3010102130 2024-03-28T18:10:36.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.647+0000 7f300d600640 1 --2- 172.21.15.37:0/3577911923 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f2ff40a2800 0x7f2ff40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:36.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.648+0000 7f300d600640 1 -- 172.21.15.37:0/3577911923 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f30000f0fe0 con 0x7f3010102130 2024-03-28T18:10:36.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.648+0000 7f3007400640 1 --2- 172.21.15.37:0/3577911923 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f2ff40a2800 0x7f2ff40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:36.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.649+0000 7f3007400640 1 --2- 172.21.15.37:0/3577911923 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f2ff40a2800 0x7f2ff40a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3010101810 tx=0x7f2ff8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:36.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.652+0000 7f300d600640 1 -- 172.21.15.37:0/3577911923 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f30000be050 con 0x7f3010102130 2024-03-28T18:10:36.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.852+0000 7f3016a00640 1 -- 172.21.15.37:0/3577911923 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f30100622a0 con 0x7f3010102130 2024-03-28T18:10:36.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.853+0000 7f300d600640 1 -- 172.21.15.37:0/3577911923 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f30000bd950 con 0x7f3010102130 2024-03-28T18:10:36.853 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 32 2024-03-28T18:10:36.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.855+0000 7f3016a00640 1 -- 172.21.15.37:0/3577911923 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f2ff40a2800 msgr2=0x7f2ff40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:36.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.856+0000 7f3016a00640 1 --2- 172.21.15.37:0/3577911923 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f2ff40a2800 0x7f2ff40a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3010101810 tx=0x7f2ff8005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:36.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.856+0000 7f3016a00640 1 -- 172.21.15.37:0/3577911923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3010102130 msgr2=0x7f3010101aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:36.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.856+0000 7f3016a00640 1 --2- 172.21.15.37:0/3577911923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3010102130 0x7f3010101aa0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f30000325f0 tx=0x7f3000035d10 comp rx=0 tx=0).stop 2024-03-28T18:10:36.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.856+0000 7f3016a00640 1 -- 172.21.15.37:0/3577911923 shutdown_connections 2024-03-28T18:10:36.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.856+0000 7f3016a00640 1 --2- 172.21.15.37:0/3577911923 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f2ff40a2800 0x7f2ff40a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:36.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.856+0000 7f3016a00640 1 --2- 172.21.15.37:0/3577911923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3010104a60 0x7f30101000e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:36.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.856+0000 7f3016a00640 1 --2- 172.21.15.37:0/3577911923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3010107390 0x7f3010100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:36.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.856+0000 7f3016a00640 1 --2- 172.21.15.37:0/3577911923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3010102130 0x7f3010101aa0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:36.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.856+0000 7f3016a00640 1 -- 172.21.15.37:0/3577911923 >> 172.21.15.37:0/3577911923 conn(0x7f30100fbb40 msgr2=0x7f3010102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:36.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.856+0000 7f3016a00640 1 -- 172.21.15.37:0/3577911923 shutdown_connections 2024-03-28T18:10:36.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:36.856+0000 7f3016a00640 1 -- 172.21.15.37:0/3577911923 wait complete. 2024-03-28T18:10:36.875 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:10:37.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.095+0000 7f599b200640 1 -- 172.21.15.37:0/2429157434 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f599c102760 msgr2=0x7f599c104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:37.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.095+0000 7f599b200640 1 --2- 172.21.15.37:0/2429157434 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f599c102760 0x7f599c104bd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5984009900 tx=0x7f598402bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:37.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.095+0000 7f599b200640 1 -- 172.21.15.37:0/2429157434 shutdown_connections 2024-03-28T18:10:37.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.095+0000 7f599b200640 1 --2- 172.21.15.37:0/2429157434 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f599c105110 0x7f599c105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:37.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.095+0000 7f599b200640 1 --2- 172.21.15.37:0/2429157434 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f599c102760 0x7f599c104bd0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:37.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.095+0000 7f599b200640 1 --2- 172.21.15.37:0/2429157434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f599c0ffdf0 0x7f599c102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:37.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.095+0000 7f599b200640 1 -- 172.21.15.37:0/2429157434 >> 172.21.15.37:0/2429157434 conn(0x7f599c0fba10 msgr2=0x7f599c0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:37.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.095+0000 7f599b200640 1 -- 172.21.15.37:0/2429157434 shutdown_connections 2024-03-28T18:10:37.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.095+0000 7f599b200640 1 -- 172.21.15.37:0/2429157434 wait complete. 2024-03-28T18:10:37.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.097+0000 7f599b200640 1 Processor -- start 2024-03-28T18:10:37.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.097+0000 7f599b200640 1 -- start start 2024-03-28T18:10:37.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.097+0000 7f599b200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f599c0ffdf0 0x7f599c19de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:37.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.097+0000 7f599b200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f599c102760 0x7f599c19e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:37.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.097+0000 7f599b200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f599c105110 0x7f599c1a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:37.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.097+0000 7f599b200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599c113820 con 0x7f599c105110 2024-03-28T18:10:37.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.097+0000 7f599b200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599c1136a0 con 0x7f599c0ffdf0 2024-03-28T18:10:37.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.097+0000 7f599b200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599c1139a0 con 0x7f599c102760 2024-03-28T18:10:37.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.098+0000 7f5999e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f599c0ffdf0 0x7f599c19de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:37.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.098+0000 7f5993e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f599c102760 0x7f599c19e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:37.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.098+0000 7f5999e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f599c0ffdf0 0x7f599c19de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:37668/0 (socket says 172.21.15.37:37668) 2024-03-28T18:10:37.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.098+0000 7f5999e00640 1 -- 172.21.15.37:0/3200491542 learned_addr learned my addr 172.21.15.37:0/3200491542 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:37.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.098+0000 7f599a800640 1 --2- 172.21.15.37:0/3200491542 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f599c105110 0x7f599c1a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:37.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.098+0000 7f5993e00640 1 -- 172.21.15.37:0/3200491542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f599c0ffdf0 msgr2=0x7f599c19de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:37.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.098+0000 7f5993e00640 1 --2- 172.21.15.37:0/3200491542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f599c0ffdf0 0x7f599c19de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:37.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.098+0000 7f5993e00640 1 -- 172.21.15.37:0/3200491542 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f599c105110 msgr2=0x7f599c1a53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:37.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.098+0000 7f5993e00640 1 --2- 172.21.15.37:0/3200491542 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f599c105110 0x7f599c1a53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:37.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.098+0000 7f5993e00640 1 -- 172.21.15.37:0/3200491542 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f599c1a5a20 con 0x7f599c102760 2024-03-28T18:10:37.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.098+0000 7f599a800640 1 --2- 172.21.15.37:0/3200491542 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f599c105110 0x7f599c1a53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:37.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.098+0000 7f5993e00640 1 --2- 172.21.15.37:0/3200491542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f599c102760 0x7f599c19e340 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5984036040 tx=0x7f598402c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:37.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.099+0000 7f5992a00640 1 -- 172.21.15.37:0/3200491542 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984043020 con 0x7f599c102760 2024-03-28T18:10:37.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.099+0000 7f5992a00640 1 -- 172.21.15.37:0/3200491542 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59840054e0 con 0x7f599c102760 2024-03-28T18:10:37.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.099+0000 7f5992a00640 1 -- 172.21.15.37:0/3200491542 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984003ca0 con 0x7f599c102760 2024-03-28T18:10:37.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.099+0000 7f599b200640 1 -- 172.21.15.37:0/3200491542 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f599c1a5cb0 con 0x7f599c102760 2024-03-28T18:10:37.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.099+0000 7f599b200640 1 -- 172.21.15.37:0/3200491542 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f599c1a61d0 con 0x7f599c102760 2024-03-28T18:10:37.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.100+0000 7f599b200640 1 -- 172.21.15.37:0/3200491542 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f599c0fd500 con 0x7f599c102760 2024-03-28T18:10:37.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.102+0000 7f5992a00640 1 -- 172.21.15.37:0/3200491542 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5984046020 con 0x7f599c102760 2024-03-28T18:10:37.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.103+0000 7f5992a00640 1 --2- 172.21.15.37:0/3200491542 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f59800a2800 0x7f59800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:37.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.103+0000 7f5992a00640 1 -- 172.21.15.37:0/3200491542 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f59840f0b80 con 0x7f599c102760 2024-03-28T18:10:37.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.103+0000 7f5999e00640 1 --2- 172.21.15.37:0/3200491542 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f59800a2800 0x7f59800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:37.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.104+0000 7f5999e00640 1 --2- 172.21.15.37:0/3200491542 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f59800a2800 0x7f59800a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f598c009c90 tx=0x7f598c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:37.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.111+0000 7f5992a00640 1 -- 172.21.15.37:0/3200491542 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f598404d020 con 0x7f599c102760 2024-03-28T18:10:37.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.317+0000 7f599b200640 1 -- 172.21.15.37:0/3200491542 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f599c02d070 con 0x7f599c102760 2024-03-28T18:10:37.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.318+0000 7f5992a00640 1 -- 172.21.15.37:0/3200491542 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f59840bdcf0 con 0x7f599c102760 2024-03-28T18:10:37.318 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 64 2024-03-28T18:10:37.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.321+0000 7f599b200640 1 -- 172.21.15.37:0/3200491542 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f59800a2800 msgr2=0x7f59800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:37.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.321+0000 7f599b200640 1 --2- 172.21.15.37:0/3200491542 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f59800a2800 0x7f59800a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f598c009c90 tx=0x7f598c005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:37.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.321+0000 7f599b200640 1 -- 172.21.15.37:0/3200491542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f599c102760 msgr2=0x7f599c19e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:37.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.321+0000 7f599b200640 1 --2- 172.21.15.37:0/3200491542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f599c102760 0x7f599c19e340 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5984036040 tx=0x7f598402c6c0 comp rx=0 tx=0).stop 2024-03-28T18:10:37.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.321+0000 7f599b200640 1 -- 172.21.15.37:0/3200491542 shutdown_connections 2024-03-28T18:10:37.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.321+0000 7f599b200640 1 --2- 172.21.15.37:0/3200491542 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f59800a2800 0x7f59800a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:37.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.321+0000 7f599b200640 1 --2- 172.21.15.37:0/3200491542 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f599c105110 0x7f599c1a53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:37.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.321+0000 7f599b200640 1 --2- 172.21.15.37:0/3200491542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f599c102760 0x7f599c19e340 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:37.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.321+0000 7f599b200640 1 --2- 172.21.15.37:0/3200491542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f599c0ffdf0 0x7f599c19de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:37.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.321+0000 7f599b200640 1 -- 172.21.15.37:0/3200491542 >> 172.21.15.37:0/3200491542 conn(0x7f599c0fba10 msgr2=0x7f599c100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:37.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.321+0000 7f599b200640 1 -- 172.21.15.37:0/3200491542 shutdown_connections 2024-03-28T18:10:37.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.321+0000 7f599b200640 1 -- 172.21.15.37:0/3200491542 wait complete. 2024-03-28T18:10:37.339 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:10:37.406 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:37 smithi037 ceph-mon[31460]: pgmap v388: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:10:37.406 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:37 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2367001077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:37.406 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:37 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3577911923' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:37.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:37 smithi059 ceph-mon[36190]: pgmap v388: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:10:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:37 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2367001077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:37 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3577911923' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:37.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:37 smithi170 ceph-mon[36280]: pgmap v388: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:10:37.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:37 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2367001077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:37.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:37 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3577911923' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:37.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.540+0000 7f09bea00640 1 -- 172.21.15.37:0/1339003155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09b81026f0 msgr2=0x7f09b8102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:37.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.540+0000 7f09bea00640 1 --2- 172.21.15.37:0/1339003155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09b81026f0 0x7f09b8102b70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009900 tx=0x7f09a402bc50 comp rx=0 tx=0).stop 2024-03-28T18:10:37.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.541+0000 7f09bea00640 1 -- 172.21.15.37:0/1339003155 shutdown_connections 2024-03-28T18:10:37.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.541+0000 7f09bea00640 1 --2- 172.21.15.37:0/1339003155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09b81026f0 0x7f09b8102b70 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:37.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.541+0000 7f09bea00640 1 --2- 172.21.15.37:0/1339003155 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09b81030b0 0x7f09b8109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:37.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.541+0000 7f09bea00640 1 --2- 172.21.15.37:0/1339003155 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09b8101500 0x7f09b8101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:37.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.541+0000 7f09bea00640 1 -- 172.21.15.37:0/1339003155 >> 172.21.15.37:0/1339003155 conn(0x7f09b80fca70 msgr2=0x7f09b80feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:37.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.541+0000 7f09bea00640 1 -- 172.21.15.37:0/1339003155 shutdown_connections 2024-03-28T18:10:37.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.541+0000 7f09bea00640 1 -- 172.21.15.37:0/1339003155 wait complete. 2024-03-28T18:10:37.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.542+0000 7f09bea00640 1 Processor -- start 2024-03-28T18:10:37.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.542+0000 7f09bea00640 1 -- start start 2024-03-28T18:10:37.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.542+0000 7f09bea00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09b8101500 0x7f09b8199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:37.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.542+0000 7f09bea00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09b81026f0 0x7f09b8199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:37.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.542+0000 7f09bea00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09b81030b0 0x7f09b81a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:37.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.542+0000 7f09bea00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b810d1e0 con 0x7f09b8101500 2024-03-28T18:10:37.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.542+0000 7f09bea00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b810d060 con 0x7f09b81030b0 2024-03-28T18:10:37.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.542+0000 7f09bea00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b810d360 con 0x7f09b81026f0 2024-03-28T18:10:37.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.542+0000 7f09bd600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09b8101500 0x7f09b8199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:37.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.542+0000 7f09bcc00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09b81026f0 0x7f09b8199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:37.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.542+0000 7f09bcc00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09b81026f0 0x7f09b8199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55128/0 (socket says 172.21.15.37:55128) 2024-03-28T18:10:37.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.542+0000 7f09bcc00640 1 -- 172.21.15.37:0/1713450998 learned_addr learned my addr 172.21.15.37:0/1713450998 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:37.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.542+0000 7f09be000640 1 --2- 172.21.15.37:0/1713450998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09b81030b0 0x7f09b81a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:37.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.543+0000 7f09bcc00640 1 -- 172.21.15.37:0/1713450998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09b81030b0 msgr2=0x7f09b81a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:37.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.543+0000 7f09bcc00640 1 --2- 172.21.15.37:0/1713450998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09b81030b0 0x7f09b81a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:37.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.543+0000 7f09bcc00640 1 -- 172.21.15.37:0/1713450998 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09b8101500 msgr2=0x7f09b8199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:37.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.543+0000 7f09bcc00640 1 --2- 172.21.15.37:0/1713450998 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09b8101500 0x7f09b8199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:37.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.543+0000 7f09bcc00640 1 -- 172.21.15.37:0/1713450998 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b81a1640 con 0x7f09b81026f0 2024-03-28T18:10:37.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.543+0000 7f09bcc00640 1 --2- 172.21.15.37:0/1713450998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09b81026f0 0x7f09b8199f60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f09a402c260 tx=0x7f09a402c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:37.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.543+0000 7f09b6000640 1 -- 172.21.15.37:0/1713450998 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a4041070 con 0x7f09b81026f0 2024-03-28T18:10:37.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.544+0000 7f09b6000640 1 -- 172.21.15.37:0/1713450998 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09a4004450 con 0x7f09b81026f0 2024-03-28T18:10:37.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.544+0000 7f09bea00640 1 -- 172.21.15.37:0/1713450998 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b81a18d0 con 0x7f09b81026f0 2024-03-28T18:10:37.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.544+0000 7f09b6000640 1 -- 172.21.15.37:0/1713450998 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a4034440 con 0x7f09b81026f0 2024-03-28T18:10:37.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.544+0000 7f09bea00640 1 -- 172.21.15.37:0/1713450998 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b81a1e70 con 0x7f09b81026f0 2024-03-28T18:10:37.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.545+0000 7f09bea00640 1 -- 172.21.15.37:0/1713450998 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b81a2040 con 0x7f09b81026f0 2024-03-28T18:10:37.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.548+0000 7f09b6000640 1 -- 172.21.15.37:0/1713450998 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f09a4044070 con 0x7f09b81026f0 2024-03-28T18:10:37.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.550+0000 7f09b6000640 1 --2- 172.21.15.37:0/1713450998 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f09a00a2880 0x7f09a00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:37.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.550+0000 7f09b6000640 1 -- 172.21.15.37:0/1713450998 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f09a40f0470 con 0x7f09b81026f0 2024-03-28T18:10:37.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.550+0000 7f09bd600640 1 --2- 172.21.15.37:0/1713450998 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f09a00a2880 0x7f09a00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:37.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.551+0000 7f09bd600640 1 --2- 172.21.15.37:0/1713450998 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f09a00a2880 0x7f09a00a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f09b8102550 tx=0x7f09ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:37.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.554+0000 7f09b6000640 1 -- 172.21.15.37:0/1713450998 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f09a40bced0 con 0x7f09b81026f0 2024-03-28T18:10:37.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.902+0000 7f09bea00640 1 -- 172.21.15.37:0/1713450998 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09b81a2210 con 0x7f09b81026f0 2024-03-28T18:10:37.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.903+0000 7f09b6000640 1 -- 172.21.15.37:0/1713450998 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11383 (secure 0 0 0) 0x7f09a40c0f70 con 0x7f09b81026f0 2024-03-28T18:10:37.904 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:37.904 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1926270371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6838","nonce":1926270371},{"type":"v1","addr":"172.21.15.37:6839","nonce":1926270371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:08:19.968894+0000","modified":"2024-03-28T18:08:25.415653+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24580,"mds_1":24569,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6837/4034607523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":4034607523},{"type":"v1","addr":"172.21.15.59:6837","nonce":4034607523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6839/1575518279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":1575518279},{"type":"v1","addr":"172.21.15.170:6839","nonce":1575518279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6839/2091629078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":2091629078},{"type":"v1","addr":"172.21.15.59:6839","nonce":2091629078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24580":{"gid":24580,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.37:6841/1923263622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6840","nonce":1923263622},{"type":"v1","addr":"172.21.15.37:6841","nonce":1923263622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24580,"qdb_cluster":[24550,24569,24580]},"id":1}]} 2024-03-28T18:10:37.904 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 22 2024-03-28T18:10:37.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.906+0000 7f09bea00640 1 -- 172.21.15.37:0/1713450998 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f09a00a2880 msgr2=0x7f09a00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:37.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.906+0000 7f09bea00640 1 --2- 172.21.15.37:0/1713450998 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f09a00a2880 0x7f09a00a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f09b8102550 tx=0x7f09ac005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:37.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.906+0000 7f09bea00640 1 -- 172.21.15.37:0/1713450998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09b81026f0 msgr2=0x7f09b8199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:37.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.906+0000 7f09bea00640 1 --2- 172.21.15.37:0/1713450998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09b81026f0 0x7f09b8199f60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f09a402c260 tx=0x7f09a402c670 comp rx=0 tx=0).stop 2024-03-28T18:10:37.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.907+0000 7f09bea00640 1 -- 172.21.15.37:0/1713450998 shutdown_connections 2024-03-28T18:10:37.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.907+0000 7f09bea00640 1 --2- 172.21.15.37:0/1713450998 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f09a00a2880 0x7f09a00a4d40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:37.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.907+0000 7f09bea00640 1 --2- 172.21.15.37:0/1713450998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09b81030b0 0x7f09b81a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:37.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.907+0000 7f09bea00640 1 --2- 172.21.15.37:0/1713450998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09b81026f0 0x7f09b8199f60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:37.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.907+0000 7f09bea00640 1 --2- 172.21.15.37:0/1713450998 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09b8101500 0x7f09b8199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:37.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.907+0000 7f09bea00640 1 -- 172.21.15.37:0/1713450998 >> 172.21.15.37:0/1713450998 conn(0x7f09b80fca70 msgr2=0x7f09b80738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:37.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.907+0000 7f09bea00640 1 -- 172.21.15.37:0/1713450998 shutdown_connections 2024-03-28T18:10:37.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:37.907+0000 7f09bea00640 1 -- 172.21.15.37:0/1713450998 wait complete. 2024-03-28T18:10:37.929 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:10:38.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.116+0000 7ff06e600640 1 -- 172.21.15.37:0/294571043 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff068105080 msgr2=0x7ff068105500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:38.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.116+0000 7ff06e600640 1 --2- 172.21.15.37:0/294571043 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff068105080 0x7ff068105500 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff060009be0 tx=0x7ff06002bef0 comp rx=0 tx=0).stop 2024-03-28T18:10:38.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.116+0000 7ff06e600640 1 -- 172.21.15.37:0/294571043 shutdown_connections 2024-03-28T18:10:38.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.116+0000 7ff06e600640 1 --2- 172.21.15.37:0/294571043 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff068105080 0x7ff068105500 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:38.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.116+0000 7ff06e600640 1 --2- 172.21.15.37:0/294571043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0681026d0 0x7ff068104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:38.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.116+0000 7ff06e600640 1 --2- 172.21.15.37:0/294571043 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0680ffd60 0x7ff068102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:38.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.116+0000 7ff06e600640 1 -- 172.21.15.37:0/294571043 >> 172.21.15.37:0/294571043 conn(0x7ff0680fb980 msgr2=0x7ff0680fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:38.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.116+0000 7ff06e600640 1 -- 172.21.15.37:0/294571043 shutdown_connections 2024-03-28T18:10:38.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.116+0000 7ff06e600640 1 -- 172.21.15.37:0/294571043 wait complete. 2024-03-28T18:10:38.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.117+0000 7ff06e600640 1 Processor -- start 2024-03-28T18:10:38.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.117+0000 7ff06e600640 1 -- start start 2024-03-28T18:10:38.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.118+0000 7ff06e600640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0680ffd60 0x7ff06819dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:38.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.118+0000 7ff06e600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0681026d0 0x7ff06819e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:38.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.118+0000 7ff06e600640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff068105080 0x7ff0681a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:38.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.118+0000 7ff06e600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff068113790 con 0x7ff0681026d0 2024-03-28T18:10:38.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.118+0000 7ff06e600640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff068113610 con 0x7ff068105080 2024-03-28T18:10:38.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.118+0000 7ff06e600640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff068113910 con 0x7ff0680ffd60 2024-03-28T18:10:38.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.118+0000 7ff06d200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0680ffd60 0x7ff06819dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:38.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.118+0000 7ff06d200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0680ffd60 0x7ff06819dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55144/0 (socket says 172.21.15.37:55144) 2024-03-28T18:10:38.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.118+0000 7ff06dc00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff068105080 0x7ff0681a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:38.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.118+0000 7ff06d200640 1 -- 172.21.15.37:0/2790201379 learned_addr learned my addr 172.21.15.37:0/2790201379 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:38.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.118+0000 7ff067e00640 1 --2- 172.21.15.37:0/2790201379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0681026d0 0x7ff06819e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:38.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.118+0000 7ff06d200640 1 -- 172.21.15.37:0/2790201379 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff068105080 msgr2=0x7ff0681a5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:38.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.118+0000 7ff06d200640 1 --2- 172.21.15.37:0/2790201379 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff068105080 0x7ff0681a5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:38.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.118+0000 7ff06d200640 1 -- 172.21.15.37:0/2790201379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0681026d0 msgr2=0x7ff06819e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:38.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.118+0000 7ff06d200640 1 --2- 172.21.15.37:0/2790201379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0681026d0 0x7ff06819e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:38.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.118+0000 7ff06d200640 1 -- 172.21.15.37:0/2790201379 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0681a5970 con 0x7ff0680ffd60 2024-03-28T18:10:38.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.119+0000 7ff06d200640 1 --2- 172.21.15.37:0/2790201379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0680ffd60 0x7ff06819dd50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff05c00e9c0 tx=0x7ff05c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:38.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.119+0000 7ff065600640 1 -- 172.21.15.37:0/2790201379 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c00cde0 con 0x7ff0680ffd60 2024-03-28T18:10:38.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.119+0000 7ff065600640 1 -- 172.21.15.37:0/2790201379 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff05c0035f0 con 0x7ff0680ffd60 2024-03-28T18:10:38.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.119+0000 7ff065600640 1 -- 172.21.15.37:0/2790201379 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c010690 con 0x7ff0680ffd60 2024-03-28T18:10:38.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.119+0000 7ff06e600640 1 -- 172.21.15.37:0/2790201379 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0681a5c60 con 0x7ff0680ffd60 2024-03-28T18:10:38.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.119+0000 7ff06e600640 1 -- 172.21.15.37:0/2790201379 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0681a61b0 con 0x7ff0680ffd60 2024-03-28T18:10:38.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.120+0000 7ff06e600640 1 -- 172.21.15.37:0/2790201379 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0680fd470 con 0x7ff0680ffd60 2024-03-28T18:10:38.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.122+0000 7ff065600640 1 -- 172.21.15.37:0/2790201379 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff05c0037a0 con 0x7ff0680ffd60 2024-03-28T18:10:38.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.123+0000 7ff065600640 1 --2- 172.21.15.37:0/2790201379 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff0500a2800 0x7ff0500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:38.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.124+0000 7ff065600640 1 -- 172.21.15.37:0/2790201379 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff05c0cee30 con 0x7ff0680ffd60 2024-03-28T18:10:38.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.124+0000 7ff067e00640 1 --2- 172.21.15.37:0/2790201379 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff0500a2800 0x7ff0500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:38.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.125+0000 7ff067e00640 1 --2- 172.21.15.37:0/2790201379 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff0500a2800 0x7ff0500a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff06819f260 tx=0x7ff054005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:38.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.127+0000 7ff065600640 1 -- 172.21.15.37:0/2790201379 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff05c09bee0 con 0x7ff0680ffd60 2024-03-28T18:10:38.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.349+0000 7ff06e600640 1 -- 172.21.15.37:0/2790201379 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0680622d0 con 0x7ff0680ffd60 2024-03-28T18:10:38.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.351+0000 7ff065600640 1 -- 172.21.15.37:0/2790201379 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22575 (secure 0 0 0) 0x7ff05c09b7e0 con 0x7ff0680ffd60 2024-03-28T18:10:38.351 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:38.352 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":87,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","created":"2024-03-28T17:59:05.949847+0000","modified":"2024-03-28T18:09:15.709901+0000","last_up_change":"2024-03-28T18:07:21.021030+0000","last_in_change":"2024-03-28T18:07:01.416192+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:03:00.517967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:08:18.365827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:08:19.446282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db60618d-56f7-4130-8a90-84cbc480965b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6803","nonce":3562602035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6805","nonce":3562602035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6809","nonce":3562602035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6807","nonce":3562602035}]},"public_addr":"172.21.15.37:6803/3562602035","cluster_addr":"172.21.15.37:6805/3562602035","heartbeat_back_addr":"172.21.15.37:6809/3562602035","heartbeat_front_addr":"172.21.15.37:6807/3562602035","state":["exists","up"]},{"osd":1,"uuid":"7a4b7ad6-905e-4258-9574-693d4277dfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6801","nonce":3326537211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6803","nonce":3326537211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6807","nonce":3326537211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6805","nonce":3326537211}]},"public_addr":"172.21.15.59:6801/3326537211","cluster_addr":"172.21.15.59:6803/3326537211","heartbeat_back_addr":"172.21.15.59:6807/3326537211","heartbeat_front_addr":"172.21.15.59:6805/3326537211","state":["exists","up"]},{"osd":2,"uuid":"8b76ec1c-46bf-4336-89b5-12d82906e8b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6801","nonce":1269398427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6803","nonce":1269398427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6807","nonce":1269398427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6805","nonce":1269398427}]},"public_addr":"172.21.15.170:6801/1269398427","cluster_addr":"172.21.15.170:6803/1269398427","heartbeat_back_addr":"172.21.15.170:6807/1269398427","heartbeat_front_addr":"172.21.15.170:6805/1269398427","state":["exists","up"]},{"osd":3,"uuid":"66eca1da-2fbd-44e1-8bbf-5635a2c70ad5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6811","nonce":317805240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6813","nonce":317805240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6817","nonce":317805240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6815","nonce":317805240}]},"public_addr":"172.21.15.37:6811/317805240","cluster_addr":"172.21.15.37:6813/317805240","heartbeat_back_addr":"172.21.15.37:6817/317805240","heartbeat_front_addr":"172.21.15.37:6815/317805240","state":["exists","up"]},{"osd":4,"uuid":"4c877335-e42d-4d8e-9f48-a1d0a89ae438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6809","nonce":2410632623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6811","nonce":2410632623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6815","nonce":2410632623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6813","nonce":2410632623}]},"public_addr":"172.21.15.59:6809/2410632623","cluster_addr":"172.21.15.59:6811/2410632623","heartbeat_back_addr":"172.21.15.59:6815/2410632623","heartbeat_front_addr":"172.21.15.59:6813/2410632623","state":["exists","up"]},{"osd":5,"uuid":"b884842a-86e2-455b-9570-ff163b3bc0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6809","nonce":1218943899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6811","nonce":1218943899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6815","nonce":1218943899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6813","nonce":1218943899}]},"public_addr":"172.21.15.170:6809/1218943899","cluster_addr":"172.21.15.170:6811/1218943899","heartbeat_back_addr":"172.21.15.170:6815/1218943899","heartbeat_front_addr":"172.21.15.170:6813/1218943899","state":["exists","up"]},{"osd":6,"uuid":"249a1200-77f4-44ad-981f-1d352f732b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6819","nonce":1899652046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6821","nonce":1899652046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6825","nonce":1899652046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6823","nonce":1899652046}]},"public_addr":"172.21.15.37:6819/1899652046","cluster_addr":"172.21.15.37:6821/1899652046","heartbeat_back_addr":"172.21.15.37:6825/1899652046","heartbeat_front_addr":"172.21.15.37:6823/1899652046","state":["exists","up"]},{"osd":7,"uuid":"c73338e6-f505-4517-a84b-f35e04750a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6817","nonce":1343675690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6819","nonce":1343675690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6823","nonce":1343675690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6821","nonce":1343675690}]},"public_addr":"172.21.15.59:6817/1343675690","cluster_addr":"172.21.15.59:6819/1343675690","heartbeat_back_addr":"172.21.15.59:6823/1343675690","heartbeat_front_addr":"172.21.15.59:6821/1343675690","state":["exists","up"]},{"osd":8,"uuid":"149b8297-ff2d-4a88-9b1e-0def021cc516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6817","nonce":3235794556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6819","nonce":3235794556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6823","nonce":3235794556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6821","nonce":3235794556}]},"public_addr":"172.21.15.170:6817/3235794556","cluster_addr":"172.21.15.170:6819/3235794556","heartbeat_back_addr":"172.21.15.170:6823/3235794556","heartbeat_front_addr":"172.21.15.170:6821/3235794556","state":["exists","up"]},{"osd":9,"uuid":"55c27cc2-c771-44b7-90fe-8f0feca44162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6827","nonce":2307153008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6829","nonce":2307153008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6833","nonce":2307153008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6831","nonce":2307153008}]},"public_addr":"172.21.15.37:6827/2307153008","cluster_addr":"172.21.15.37:6829/2307153008","heartbeat_back_addr":"172.21.15.37:6833/2307153008","heartbeat_front_addr":"172.21.15.37:6831/2307153008","state":["exists","up"]},{"osd":10,"uuid":"c0595135-ab81-49d3-8c23-19f960c03e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6825","nonce":2113830275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6827","nonce":2113830275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6831","nonce":2113830275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6829","nonce":2113830275}]},"public_addr":"172.21.15.59:6825/2113830275","cluster_addr":"172.21.15.59:6827/2113830275","heartbeat_back_addr":"172.21.15.59:6831/2113830275","heartbeat_front_addr":"172.21.15.59:6829/2113830275","state":["exists","up"]},{"osd":11,"uuid":"5871e177-c17b-4637-a797-4fa5125920dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6825","nonce":3082439360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6827","nonce":3082439360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6831","nonce":3082439360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6829","nonce":3082439360}]},"public_addr":"172.21.15.170:6825/3082439360","cluster_addr":"172.21.15.170:6827/3082439360","heartbeat_back_addr":"172.21.15.170:6831/3082439360","heartbeat_front_addr":"172.21.15.170:6829/3082439360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:07.690545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:33.183396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:57.400263+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:30.597211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:57.259044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:22.627823+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:57.585863+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:25.000476+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:50.876688+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:25.859761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:53.563146+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:07:19.304115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3023379428":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1443710231":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6800/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6801/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6801/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6800/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1007625490":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/2895698584":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/497313034":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/3293943475":"2024-03-29T17:59:39.162038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:10:38.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.354+0000 7ff06e600640 1 -- 172.21.15.37:0/2790201379 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff0500a2800 msgr2=0x7ff0500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:38.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.354+0000 7ff06e600640 1 --2- 172.21.15.37:0/2790201379 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff0500a2800 0x7ff0500a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff06819f260 tx=0x7ff054005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:38.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.354+0000 7ff06e600640 1 -- 172.21.15.37:0/2790201379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0680ffd60 msgr2=0x7ff06819dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:38.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.354+0000 7ff06e600640 1 --2- 172.21.15.37:0/2790201379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0680ffd60 0x7ff06819dd50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff05c00e9c0 tx=0x7ff05c00ee80 comp rx=0 tx=0).stop 2024-03-28T18:10:38.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.354+0000 7ff06e600640 1 -- 172.21.15.37:0/2790201379 shutdown_connections 2024-03-28T18:10:38.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.354+0000 7ff06e600640 1 --2- 172.21.15.37:0/2790201379 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff0500a2800 0x7ff0500a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:38.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.354+0000 7ff06e600640 1 --2- 172.21.15.37:0/2790201379 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff068105080 0x7ff0681a5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:38.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.354+0000 7ff06e600640 1 --2- 172.21.15.37:0/2790201379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0681026d0 0x7ff06819e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:38.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.354+0000 7ff06e600640 1 --2- 172.21.15.37:0/2790201379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0680ffd60 0x7ff06819dd50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:38.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.354+0000 7ff06e600640 1 -- 172.21.15.37:0/2790201379 >> 172.21.15.37:0/2790201379 conn(0x7ff0680fb980 msgr2=0x7ff068100a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:38.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.354+0000 7ff06e600640 1 -- 172.21.15.37:0/2790201379 shutdown_connections 2024-03-28T18:10:38.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.354+0000 7ff06e600640 1 -- 172.21.15.37:0/2790201379 wait complete. 2024-03-28T18:10:38.369 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:38 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3200491542' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:38.369 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:38 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1713450998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:38.372 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:03:00.517967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:08:18.365827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:08:19.446282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:10:38.373 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:10:38.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:38 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3200491542' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:38.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:38 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1713450998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:38.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:38 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3200491542' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:38.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:38 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1713450998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:38.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.573+0000 7fe775c00640 1 -- 172.21.15.37:0/2577861080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7701037c0 msgr2=0x7fe770103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:38.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.573+0000 7fe775c00640 1 --2- 172.21.15.37:0/2577861080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7701037c0 0x7fe770103be0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe760009900 tx=0x7fe76002bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:38.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.573+0000 7fe775c00640 1 -- 172.21.15.37:0/2577861080 shutdown_connections 2024-03-28T18:10:38.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.573+0000 7fe775c00640 1 --2- 172.21.15.37:0/2577861080 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe770105370 0x7fe77010bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:38.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.573+0000 7fe775c00640 1 --2- 172.21.15.37:0/2577861080 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7701049b0 0x7fe770104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:38.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.573+0000 7fe775c00640 1 --2- 172.21.15.37:0/2577861080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7701037c0 0x7fe770103be0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:38.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.573+0000 7fe775c00640 1 -- 172.21.15.37:0/2577861080 >> 172.21.15.37:0/2577861080 conn(0x7fe7700fed50 msgr2=0x7fe770101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:38.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.573+0000 7fe775c00640 1 -- 172.21.15.37:0/2577861080 shutdown_connections 2024-03-28T18:10:38.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.573+0000 7fe775c00640 1 -- 172.21.15.37:0/2577861080 wait complete. 2024-03-28T18:10:38.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.575+0000 7fe775c00640 1 Processor -- start 2024-03-28T18:10:38.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.575+0000 7fe775c00640 1 -- start start 2024-03-28T18:10:38.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.576+0000 7fe775c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7701037c0 0x7fe770199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:38.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.576+0000 7fe775c00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7701049b0 0x7fe770199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:38.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.576+0000 7fe775c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe770105370 0x7fe7701a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:38.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.576+0000 7fe775c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77010f490 con 0x7fe7701037c0 2024-03-28T18:10:38.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.576+0000 7fe775c00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77010f310 con 0x7fe770105370 2024-03-28T18:10:38.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.576+0000 7fe775c00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77010f610 con 0x7fe7701049b0 2024-03-28T18:10:38.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.576+0000 7fe76fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7701037c0 0x7fe770199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:38.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.576+0000 7fe76fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7701037c0 0x7fe770199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49090/0 (socket says 172.21.15.37:49090) 2024-03-28T18:10:38.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.576+0000 7fe76fe00640 1 -- 172.21.15.37:0/1322372722 learned_addr learned my addr 172.21.15.37:0/1322372722 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:38.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.576+0000 7fe76f400640 1 --2- 172.21.15.37:0/1322372722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7701049b0 0x7fe770199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:38.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.576+0000 7fe775200640 1 --2- 172.21.15.37:0/1322372722 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe770105370 0x7fe7701a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:38.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.577+0000 7fe76f400640 1 -- 172.21.15.37:0/1322372722 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe770105370 msgr2=0x7fe7701a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:38.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.577+0000 7fe76f400640 1 --2- 172.21.15.37:0/1322372722 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe770105370 0x7fe7701a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:38.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.577+0000 7fe76f400640 1 -- 172.21.15.37:0/1322372722 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7701037c0 msgr2=0x7fe770199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:38.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.577+0000 7fe76f400640 1 --2- 172.21.15.37:0/1322372722 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7701037c0 0x7fe770199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:38.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.577+0000 7fe76f400640 1 -- 172.21.15.37:0/1322372722 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7701a16a0 con 0x7fe7701049b0 2024-03-28T18:10:38.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.577+0000 7fe76f400640 1 --2- 172.21.15.37:0/1322372722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7701049b0 0x7fe770199fc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe75c00e9f0 tx=0x7fe75c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:38.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.577+0000 7fe76cc00640 1 -- 172.21.15.37:0/1322372722 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c00ccc0 con 0x7fe7701049b0 2024-03-28T18:10:38.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.577+0000 7fe76cc00640 1 -- 172.21.15.37:0/1322372722 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe75c010910 con 0x7fe7701049b0 2024-03-28T18:10:38.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.577+0000 7fe775c00640 1 -- 172.21.15.37:0/1322372722 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7701a1990 con 0x7fe7701049b0 2024-03-28T18:10:38.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.577+0000 7fe76cc00640 1 -- 172.21.15.37:0/1322372722 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c018960 con 0x7fe7701049b0 2024-03-28T18:10:38.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.577+0000 7fe775c00640 1 -- 172.21.15.37:0/1322372722 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7701a1ee0 con 0x7fe7701049b0 2024-03-28T18:10:38.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.578+0000 7fe775c00640 1 -- 172.21.15.37:0/1322372722 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe77010aa80 con 0x7fe7701049b0 2024-03-28T18:10:38.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.581+0000 7fe76cc00640 1 -- 172.21.15.37:0/1322372722 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe75c010ac0 con 0x7fe7701049b0 2024-03-28T18:10:38.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.582+0000 7fe76cc00640 1 --2- 172.21.15.37:0/1322372722 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe7540a2800 0x7fe7540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:38.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.582+0000 7fe76cc00640 1 -- 172.21.15.37:0/1322372722 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe75c0cee60 con 0x7fe7701049b0 2024-03-28T18:10:38.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.582+0000 7fe76fe00640 1 --2- 172.21.15.37:0/1322372722 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe7540a2800 0x7fe7540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:38.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.583+0000 7fe76fe00640 1 --2- 172.21.15.37:0/1322372722 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe7540a2800 0x7fe7540a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe760006010 tx=0x7fe76002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:38.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.586+0000 7fe76cc00640 1 -- 172.21.15.37:0/1322372722 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe75c09b860 con 0x7fe7701049b0 2024-03-28T18:10:38.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.780+0000 7fe775c00640 1 -- 172.21.15.37:0/1322372722 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe7700622a0 con 0x7fe7701049b0 2024-03-28T18:10:38.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.781+0000 7fe76cc00640 1 -- 172.21.15.37:0/1322372722 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe75c09f900 con 0x7fe7701049b0 2024-03-28T18:10:38.781 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 1 2024-03-28T18:10:38.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.783+0000 7fe775c00640 1 -- 172.21.15.37:0/1322372722 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe7540a2800 msgr2=0x7fe7540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:38.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.783+0000 7fe775c00640 1 --2- 172.21.15.37:0/1322372722 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe7540a2800 0x7fe7540a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe760006010 tx=0x7fe76002df90 comp rx=0 tx=0).stop 2024-03-28T18:10:38.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.783+0000 7fe775c00640 1 -- 172.21.15.37:0/1322372722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7701049b0 msgr2=0x7fe770199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:38.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.783+0000 7fe775c00640 1 --2- 172.21.15.37:0/1322372722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7701049b0 0x7fe770199fc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe75c00e9f0 tx=0x7fe75c00eeb0 comp rx=0 tx=0).stop 2024-03-28T18:10:38.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.783+0000 7fe775c00640 1 -- 172.21.15.37:0/1322372722 shutdown_connections 2024-03-28T18:10:38.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.783+0000 7fe775c00640 1 --2- 172.21.15.37:0/1322372722 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe7540a2800 0x7fe7540a4cc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:38.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.783+0000 7fe775c00640 1 --2- 172.21.15.37:0/1322372722 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe770105370 0x7fe7701a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:38.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.783+0000 7fe775c00640 1 --2- 172.21.15.37:0/1322372722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7701049b0 0x7fe770199fc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:38.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.783+0000 7fe775c00640 1 --2- 172.21.15.37:0/1322372722 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7701037c0 0x7fe770199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:38.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.783+0000 7fe775c00640 1 -- 172.21.15.37:0/1322372722 >> 172.21.15.37:0/1322372722 conn(0x7fe7700fed50 msgr2=0x7fe770107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:38.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.783+0000 7fe775c00640 1 -- 172.21.15.37:0/1322372722 shutdown_connections 2024-03-28T18:10:38.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:38.783+0000 7fe775c00640 1 -- 172.21.15.37:0/1322372722 wait complete. 2024-03-28T18:10:38.801 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:10:39.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.007+0000 7f2e69e00640 1 -- 172.21.15.37:0/4048528700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e64105490 msgr2=0x7f2e6410bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:39.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.007+0000 7f2e69e00640 1 --2- 172.21.15.37:0/4048528700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e64105490 0x7f2e6410bd20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c009ca0 tx=0x7f2e5c02c100 comp rx=0 tx=0).stop 2024-03-28T18:10:39.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.007+0000 7f2e69e00640 1 -- 172.21.15.37:0/4048528700 shutdown_connections 2024-03-28T18:10:39.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.007+0000 7f2e69e00640 1 --2- 172.21.15.37:0/4048528700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e64105490 0x7f2e6410bd20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.007+0000 7f2e69e00640 1 --2- 172.21.15.37:0/4048528700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e64104ad0 0x7f2e64104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.007+0000 7f2e69e00640 1 --2- 172.21.15.37:0/4048528700 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e641038e0 0x7f2e64103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.007+0000 7f2e69e00640 1 -- 172.21.15.37:0/4048528700 >> 172.21.15.37:0/4048528700 conn(0x7f2e640fee50 msgr2=0x7f2e64101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:39.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.007+0000 7f2e69e00640 1 -- 172.21.15.37:0/4048528700 shutdown_connections 2024-03-28T18:10:39.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.007+0000 7f2e69e00640 1 -- 172.21.15.37:0/4048528700 wait complete. 2024-03-28T18:10:39.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.008+0000 7f2e69e00640 1 Processor -- start 2024-03-28T18:10:39.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.008+0000 7f2e69e00640 1 -- start start 2024-03-28T18:10:39.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.009+0000 7f2e69e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e641038e0 0x7f2e64199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:39.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.009+0000 7f2e69e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e64104ad0 0x7f2e6419a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:39.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.009+0000 7f2e69e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e64105490 0x7f2e641a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:39.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.009+0000 7f2e69e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6410f540 con 0x7f2e641038e0 2024-03-28T18:10:39.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.009+0000 7f2e69e00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6410f3c0 con 0x7f2e64104ad0 2024-03-28T18:10:39.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.009+0000 7f2e69e00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6410f6c0 con 0x7f2e64105490 2024-03-28T18:10:39.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.009+0000 7f2e62000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e64104ad0 0x7f2e6419a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:39.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.009+0000 7f2e62000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e64104ad0 0x7f2e6419a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:37760/0 (socket says 172.21.15.37:37760) 2024-03-28T18:10:39.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.009+0000 7f2e62000640 1 -- 172.21.15.37:0/1100634885 learned_addr learned my addr 172.21.15.37:0/1100634885 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:39.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.009+0000 7f2e63400640 1 --2- 172.21.15.37:0/1100634885 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e64105490 0x7f2e641a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:39.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.009+0000 7f2e62000640 1 -- 172.21.15.37:0/1100634885 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e64105490 msgr2=0x7f2e641a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:39.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.009+0000 7f2e62000640 1 --2- 172.21.15.37:0/1100634885 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e64105490 0x7f2e641a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.009+0000 7f2e62000640 1 -- 172.21.15.37:0/1100634885 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e641038e0 msgr2=0x7f2e64199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:39.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.009+0000 7f2e62a00640 1 --2- 172.21.15.37:0/1100634885 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e641038e0 0x7f2e64199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:39.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.009+0000 7f2e62000640 1 --2- 172.21.15.37:0/1100634885 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e641038e0 0x7f2e64199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.009+0000 7f2e62000640 1 -- 172.21.15.37:0/1100634885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e641a1770 con 0x7f2e64104ad0 2024-03-28T18:10:39.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.010+0000 7f2e62000640 1 --2- 172.21.15.37:0/1100634885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e64104ad0 0x7f2e6419a090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2e5000e9f0 tx=0x7f2e5000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:39.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.010+0000 7f2e68a00640 1 -- 172.21.15.37:0/1100634885 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e5000cd80 con 0x7f2e64104ad0 2024-03-28T18:10:39.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.010+0000 7f2e68a00640 1 -- 172.21.15.37:0/1100634885 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e50010910 con 0x7f2e64104ad0 2024-03-28T18:10:39.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.010+0000 7f2e69e00640 1 -- 172.21.15.37:0/1100634885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e641a1a60 con 0x7f2e64104ad0 2024-03-28T18:10:39.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.010+0000 7f2e68a00640 1 -- 172.21.15.37:0/1100634885 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e50018990 con 0x7f2e64104ad0 2024-03-28T18:10:39.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.010+0000 7f2e69e00640 1 -- 172.21.15.37:0/1100634885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e641a1fb0 con 0x7f2e64104ad0 2024-03-28T18:10:39.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.011+0000 7f2e69e00640 1 -- 172.21.15.37:0/1100634885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e6410aba0 con 0x7f2e64104ad0 2024-03-28T18:10:39.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.014+0000 7f2e68a00640 1 -- 172.21.15.37:0/1100634885 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2e50010ac0 con 0x7f2e64104ad0 2024-03-28T18:10:39.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.015+0000 7f2e68a00640 1 --2- 172.21.15.37:0/1100634885 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f2e480a2800 0x7f2e480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:39.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.015+0000 7f2e68a00640 1 -- 172.21.15.37:0/1100634885 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f2e500ceeb0 con 0x7f2e64104ad0 2024-03-28T18:10:39.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.015+0000 7f2e62a00640 1 --2- 172.21.15.37:0/1100634885 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f2e480a2800 0x7f2e480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:39.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.016+0000 7f2e62a00640 1 --2- 172.21.15.37:0/1100634885 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f2e480a2800 0x7f2e480a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2e64104930 tx=0x7f2e54005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:39.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.019+0000 7f2e68a00640 1 -- 172.21.15.37:0/1100634885 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2e5009bf00 con 0x7f2e64104ad0 2024-03-28T18:10:39.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.214+0000 7f2e69e00640 1 -- 172.21.15.37:0/1100634885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2e640622a0 con 0x7f2e64104ad0 2024-03-28T18:10:39.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.214+0000 7f2e68a00640 1 -- 172.21.15.37:0/1100634885 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2e5009b800 con 0x7f2e64104ad0 2024-03-28T18:10:39.215 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 32 2024-03-28T18:10:39.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.218+0000 7f2e69e00640 1 -- 172.21.15.37:0/1100634885 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f2e480a2800 msgr2=0x7f2e480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:39.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.218+0000 7f2e69e00640 1 --2- 172.21.15.37:0/1100634885 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f2e480a2800 0x7f2e480a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2e64104930 tx=0x7f2e54005c60 comp rx=0 tx=0).stop 2024-03-28T18:10:39.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.218+0000 7f2e69e00640 1 -- 172.21.15.37:0/1100634885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e64104ad0 msgr2=0x7f2e6419a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:39.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.218+0000 7f2e69e00640 1 --2- 172.21.15.37:0/1100634885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e64104ad0 0x7f2e6419a090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2e5000e9f0 tx=0x7f2e5000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.218+0000 7f2e69e00640 1 -- 172.21.15.37:0/1100634885 shutdown_connections 2024-03-28T18:10:39.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.218+0000 7f2e69e00640 1 --2- 172.21.15.37:0/1100634885 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f2e480a2800 0x7f2e480a4cc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.218+0000 7f2e69e00640 1 --2- 172.21.15.37:0/1100634885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e64104ad0 0x7f2e6419a090 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.218+0000 7f2e69e00640 1 --2- 172.21.15.37:0/1100634885 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e64105490 0x7f2e641a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.218+0000 7f2e69e00640 1 --2- 172.21.15.37:0/1100634885 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e641038e0 0x7f2e64199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.218+0000 7f2e69e00640 1 -- 172.21.15.37:0/1100634885 >> 172.21.15.37:0/1100634885 conn(0x7f2e640fee50 msgr2=0x7f2e64107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:39.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.218+0000 7f2e69e00640 1 -- 172.21.15.37:0/1100634885 shutdown_connections 2024-03-28T18:10:39.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.218+0000 7f2e69e00640 1 -- 172.21.15.37:0/1100634885 wait complete. 2024-03-28T18:10:39.242 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:10:39.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.456+0000 7f1216c00640 1 -- 172.21.15.37:0/3880654505 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12101015a0 msgr2=0x7f12101019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:39.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.456+0000 7f1216c00640 1 --2- 172.21.15.37:0/3880654505 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12101015a0 0x7f12101019c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1204009900 tx=0x7f120402bc40 comp rx=0 tx=0).stop 2024-03-28T18:10:39.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.456+0000 7f1216c00640 1 -- 172.21.15.37:0/3880654505 shutdown_connections 2024-03-28T18:10:39.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.456+0000 7f1216c00640 1 --2- 172.21.15.37:0/3880654505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1210102790 0x7f1210102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.456+0000 7f1216c00640 1 --2- 172.21.15.37:0/3880654505 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1210103150 0x7f12101099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.456+0000 7f1216c00640 1 --2- 172.21.15.37:0/3880654505 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12101015a0 0x7f12101019c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.456+0000 7f1216c00640 1 -- 172.21.15.37:0/3880654505 >> 172.21.15.37:0/3880654505 conn(0x7f12100fcb10 msgr2=0x7f12100fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:39.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.457+0000 7f1216c00640 1 -- 172.21.15.37:0/3880654505 shutdown_connections 2024-03-28T18:10:39.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.457+0000 7f1216c00640 1 -- 172.21.15.37:0/3880654505 wait complete. 2024-03-28T18:10:39.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.457+0000 7f1216c00640 1 Processor -- start 2024-03-28T18:10:39.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.458+0000 7f1216c00640 1 -- start start 2024-03-28T18:10:39.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.458+0000 7f1216c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12101015a0 0x7f1210199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:39.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.458+0000 7f1216c00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1210102790 0x7f1210199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:39.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.458+0000 7f1216c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1210103150 0x7f12101a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:39.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.458+0000 7f1216c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121010d240 con 0x7f1210103150 2024-03-28T18:10:39.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.458+0000 7f1216c00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121010d0c0 con 0x7f12101015a0 2024-03-28T18:10:39.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.458+0000 7f1216c00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121010d3c0 con 0x7f1210102790 2024-03-28T18:10:39.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.458+0000 7f1215800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12101015a0 0x7f1210199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:39.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.458+0000 7f1214e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1210102790 0x7f1210199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:39.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.458+0000 7f1215800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12101015a0 0x7f1210199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:37782/0 (socket says 172.21.15.37:37782) 2024-03-28T18:10:39.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.458+0000 7f1215800640 1 -- 172.21.15.37:0/3292714362 learned_addr learned my addr 172.21.15.37:0/3292714362 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:39.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.458+0000 7f1216200640 1 --2- 172.21.15.37:0/3292714362 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1210103150 0x7f12101a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:39.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.458+0000 7f1215800640 1 -- 172.21.15.37:0/3292714362 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1210102790 msgr2=0x7f1210199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:39.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.458+0000 7f1215800640 1 --2- 172.21.15.37:0/3292714362 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1210102790 0x7f1210199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.458+0000 7f1215800640 1 -- 172.21.15.37:0/3292714362 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1210103150 msgr2=0x7f12101a1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:39.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.458+0000 7f1215800640 1 --2- 172.21.15.37:0/3292714362 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1210103150 0x7f12101a1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.459+0000 7f1215800640 1 -- 172.21.15.37:0/3292714362 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12101a16b0 con 0x7f12101015a0 2024-03-28T18:10:39.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.459+0000 7f1215800640 1 --2- 172.21.15.37:0/3292714362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12101015a0 0x7f1210199a90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f120402efd0 tx=0x7f1204005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:39.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.459+0000 7f120e000640 1 -- 172.21.15.37:0/3292714362 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1204041070 con 0x7f12101015a0 2024-03-28T18:10:39.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.459+0000 7f120e000640 1 -- 172.21.15.37:0/3292714362 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12040044d0 con 0x7f12101015a0 2024-03-28T18:10:39.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.459+0000 7f1216c00640 1 -- 172.21.15.37:0/3292714362 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12101a1940 con 0x7f12101015a0 2024-03-28T18:10:39.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.459+0000 7f120e000640 1 -- 172.21.15.37:0/3292714362 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1204034440 con 0x7f12101015a0 2024-03-28T18:10:39.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.459+0000 7f1216c00640 1 -- 172.21.15.37:0/3292714362 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12101a1e00 con 0x7f12101015a0 2024-03-28T18:10:39.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.460+0000 7f1216c00640 1 -- 172.21.15.37:0/3292714362 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12101a2030 con 0x7f12101015a0 2024-03-28T18:10:39.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.462+0000 7f120e000640 1 -- 172.21.15.37:0/3292714362 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1204044070 con 0x7f12101015a0 2024-03-28T18:10:39.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.463+0000 7f120e000640 1 --2- 172.21.15.37:0/3292714362 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f11f80a2800 0x7f11f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:39.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.463+0000 7f120e000640 1 -- 172.21.15.37:0/3292714362 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f12040f0640 con 0x7f12101015a0 2024-03-28T18:10:39.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.463+0000 7f1214e00640 1 --2- 172.21.15.37:0/3292714362 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f11f80a2800 0x7f11f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:39.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.464+0000 7f1214e00640 1 --2- 172.21.15.37:0/3292714362 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f11f80a2800 0x7f11f80a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f121019afa0 tx=0x7f11fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:39.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.467+0000 7f120e000640 1 -- 172.21.15.37:0/3292714362 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f12040bd040 con 0x7f12101015a0 2024-03-28T18:10:39.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:39 smithi059 ceph-mon[36190]: pgmap v389: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:39.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:39 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2790201379' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:39.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:39 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1322372722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:39.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:39 smithi170 ceph-mon[36280]: pgmap v389: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:39.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:39 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2790201379' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:39.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:39 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1322372722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:39.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:39 smithi037 ceph-mon[31460]: pgmap v389: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:39.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:39 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2790201379' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:39.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:39 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1322372722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:39.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.667+0000 7f1216c00640 1 -- 172.21.15.37:0/3292714362 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f12101a2200 con 0x7f12101015a0 2024-03-28T18:10:39.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.667+0000 7f120e000640 1 -- 172.21.15.37:0/3292714362 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f12040c10e0 con 0x7f12101015a0 2024-03-28T18:10:39.667 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 64 2024-03-28T18:10:39.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.670+0000 7f1216c00640 1 -- 172.21.15.37:0/3292714362 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f11f80a2800 msgr2=0x7f11f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:39.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.670+0000 7f1216c00640 1 --2- 172.21.15.37:0/3292714362 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f11f80a2800 0x7f11f80a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f121019afa0 tx=0x7f11fc005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:39.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.670+0000 7f1216c00640 1 -- 172.21.15.37:0/3292714362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12101015a0 msgr2=0x7f1210199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:39.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.670+0000 7f1216c00640 1 --2- 172.21.15.37:0/3292714362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12101015a0 0x7f1210199a90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f120402efd0 tx=0x7f1204005d50 comp rx=0 tx=0).stop 2024-03-28T18:10:39.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.671+0000 7f1216c00640 1 -- 172.21.15.37:0/3292714362 shutdown_connections 2024-03-28T18:10:39.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.671+0000 7f1216c00640 1 --2- 172.21.15.37:0/3292714362 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f11f80a2800 0x7f11f80a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.671+0000 7f1216c00640 1 --2- 172.21.15.37:0/3292714362 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1210103150 0x7f12101a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.671+0000 7f1216c00640 1 --2- 172.21.15.37:0/3292714362 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1210102790 0x7f1210199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.671+0000 7f1216c00640 1 --2- 172.21.15.37:0/3292714362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12101015a0 0x7f1210199a90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.671+0000 7f1216c00640 1 -- 172.21.15.37:0/3292714362 >> 172.21.15.37:0/3292714362 conn(0x7f12100fcb10 msgr2=0x7f12100738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:39.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.671+0000 7f1216c00640 1 -- 172.21.15.37:0/3292714362 shutdown_connections 2024-03-28T18:10:39.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.671+0000 7f1216c00640 1 -- 172.21.15.37:0/3292714362 wait complete. 2024-03-28T18:10:39.691 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:10:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.909+0000 7f7168000640 1 -- 172.21.15.37:0/3435527897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71601028a0 msgr2=0x7f7160104d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.909+0000 7f7168000640 1 --2- 172.21.15.37:0/3435527897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71601028a0 0x7f7160104d10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f714c009900 tx=0x7f714c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.909+0000 7f7168000640 1 -- 172.21.15.37:0/3435527897 shutdown_connections 2024-03-28T18:10:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.909+0000 7f7168000640 1 --2- 172.21.15.37:0/3435527897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71601028a0 0x7f7160104d10 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.909+0000 7f7168000640 1 --2- 172.21.15.37:0/3435527897 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7160105250 0x7f71601056d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.909+0000 7f7168000640 1 --2- 172.21.15.37:0/3435527897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f71600fff30 0x7f7160102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.909+0000 7f7168000640 1 -- 172.21.15.37:0/3435527897 >> 172.21.15.37:0/3435527897 conn(0x7f71600fbb50 msgr2=0x7f71600fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.909+0000 7f7168000640 1 -- 172.21.15.37:0/3435527897 shutdown_connections 2024-03-28T18:10:39.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.909+0000 7f7168000640 1 -- 172.21.15.37:0/3435527897 wait complete. 2024-03-28T18:10:39.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.910+0000 7f7168000640 1 Processor -- start 2024-03-28T18:10:39.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.910+0000 7f7168000640 1 -- start start 2024-03-28T18:10:39.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.911+0000 7f7168000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f71600fff30 0x7f716019df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.911+0000 7f7168000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71601028a0 0x7f716019e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.911+0000 7f7168000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7160105250 0x7f71601a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.911+0000 7f7168000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7160113960 con 0x7f71600fff30 2024-03-28T18:10:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.911+0000 7f7168000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71601137e0 con 0x7f71601028a0 2024-03-28T18:10:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.911+0000 7f7168000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7160113ae0 con 0x7f7160105250 2024-03-28T18:10:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.911+0000 7f7165600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f71600fff30 0x7f716019df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.911+0000 7f7164c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71601028a0 0x7f716019e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.911+0000 7f7165600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f71600fff30 0x7f716019df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49130/0 (socket says 172.21.15.37:49130) 2024-03-28T18:10:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.911+0000 7f7165600640 1 -- 172.21.15.37:0/2697011757 learned_addr learned my addr 172.21.15.37:0/2697011757 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.911+0000 7f7166000640 1 --2- 172.21.15.37:0/2697011757 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7160105250 0x7f71601a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.912+0000 7f7164c00640 1 -- 172.21.15.37:0/2697011757 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7160105250 msgr2=0x7f71601a5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.912+0000 7f7164c00640 1 --2- 172.21.15.37:0/2697011757 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7160105250 0x7f71601a5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.912+0000 7f7164c00640 1 -- 172.21.15.37:0/2697011757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f71600fff30 msgr2=0x7f716019df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.912+0000 7f7164c00640 1 --2- 172.21.15.37:0/2697011757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f71600fff30 0x7f716019df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.912+0000 7f7164c00640 1 -- 172.21.15.37:0/2697011757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71601a5bb0 con 0x7f71601028a0 2024-03-28T18:10:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.912+0000 7f7166000640 1 --2- 172.21.15.37:0/2697011757 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7160105250 0x7f71601a5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.912+0000 7f7165600640 1 --2- 172.21.15.37:0/2697011757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f71600fff30 0x7f716019df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.912+0000 7f7164c00640 1 --2- 172.21.15.37:0/2697011757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71601028a0 0x7f716019e4d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f714c036040 tx=0x7f714c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.912+0000 7f715e000640 1 -- 172.21.15.37:0/2697011757 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c043020 con 0x7f71601028a0 2024-03-28T18:10:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.912+0000 7f715e000640 1 -- 172.21.15.37:0/2697011757 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f714c0054e0 con 0x7f71601028a0 2024-03-28T18:10:39.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.912+0000 7f715e000640 1 -- 172.21.15.37:0/2697011757 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c003c30 con 0x7f71601028a0 2024-03-28T18:10:39.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.912+0000 7f7168000640 1 -- 172.21.15.37:0/2697011757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71601a5e40 con 0x7f71601028a0 2024-03-28T18:10:39.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.913+0000 7f7168000640 1 -- 172.21.15.37:0/2697011757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71601a62b0 con 0x7f71601028a0 2024-03-28T18:10:39.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.914+0000 7f7168000640 1 -- 172.21.15.37:0/2697011757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71600fd640 con 0x7f71601028a0 2024-03-28T18:10:39.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.915+0000 7f715e000640 1 -- 172.21.15.37:0/2697011757 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f714c046020 con 0x7f71601028a0 2024-03-28T18:10:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.917+0000 7f715e000640 1 --2- 172.21.15.37:0/2697011757 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f71480a2800 0x7f71480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:39.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.917+0000 7f715e000640 1 -- 172.21.15.37:0/2697011757 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f714c0f0ae0 con 0x7f71601028a0 2024-03-28T18:10:39.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.917+0000 7f7165600640 1 --2- 172.21.15.37:0/2697011757 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f71480a2800 0x7f71480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:39.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.918+0000 7f7165600640 1 --2- 172.21.15.37:0/2697011757 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f71480a2800 0x7f71480a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7154011550 tx=0x7f7154005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:39.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:39.921+0000 7f715e000640 1 -- 172.21.15.37:0/2697011757 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f714c04d020 con 0x7f71601028a0 2024-03-28T18:10:40.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.227+0000 7f7168000640 1 -- 172.21.15.37:0/2697011757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71600622a0 con 0x7f71601028a0 2024-03-28T18:10:40.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.228+0000 7f715e000640 1 -- 172.21.15.37:0/2697011757 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11383 (secure 0 0 0) 0x7f714c0bdb30 con 0x7f71601028a0 2024-03-28T18:10:40.228 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:40.228 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1926270371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6838","nonce":1926270371},{"type":"v1","addr":"172.21.15.37:6839","nonce":1926270371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:08:19.968894+0000","modified":"2024-03-28T18:08:25.415653+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24580,"mds_1":24569,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6837/4034607523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":4034607523},{"type":"v1","addr":"172.21.15.59:6837","nonce":4034607523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6839/1575518279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":1575518279},{"type":"v1","addr":"172.21.15.170:6839","nonce":1575518279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6839/2091629078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":2091629078},{"type":"v1","addr":"172.21.15.59:6839","nonce":2091629078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24580":{"gid":24580,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.37:6841/1923263622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6840","nonce":1923263622},{"type":"v1","addr":"172.21.15.37:6841","nonce":1923263622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24580,"qdb_cluster":[24550,24569,24580]},"id":1}]} 2024-03-28T18:10:40.228 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 22 2024-03-28T18:10:40.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.230+0000 7f7168000640 1 -- 172.21.15.37:0/2697011757 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f71480a2800 msgr2=0x7f71480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:40.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.230+0000 7f7168000640 1 --2- 172.21.15.37:0/2697011757 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f71480a2800 0x7f71480a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7154011550 tx=0x7f7154005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:40.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.230+0000 7f7168000640 1 -- 172.21.15.37:0/2697011757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71601028a0 msgr2=0x7f716019e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:40.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.230+0000 7f7168000640 1 --2- 172.21.15.37:0/2697011757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71601028a0 0x7f716019e4d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f714c036040 tx=0x7f714c02c650 comp rx=0 tx=0).stop 2024-03-28T18:10:40.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.230+0000 7f7168000640 1 -- 172.21.15.37:0/2697011757 shutdown_connections 2024-03-28T18:10:40.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.230+0000 7f7168000640 1 --2- 172.21.15.37:0/2697011757 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f71480a2800 0x7f71480a4cc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.230+0000 7f7168000640 1 --2- 172.21.15.37:0/2697011757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71601028a0 0x7f716019e4d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.230+0000 7f7168000640 1 --2- 172.21.15.37:0/2697011757 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7160105250 0x7f71601a5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.230+0000 7f7168000640 1 --2- 172.21.15.37:0/2697011757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f71600fff30 0x7f716019df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.230+0000 7f7168000640 1 -- 172.21.15.37:0/2697011757 >> 172.21.15.37:0/2697011757 conn(0x7f71600fbb50 msgr2=0x7f71601008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:40.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.230+0000 7f7168000640 1 -- 172.21.15.37:0/2697011757 shutdown_connections 2024-03-28T18:10:40.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.230+0000 7f7168000640 1 -- 172.21.15.37:0/2697011757 wait complete. 2024-03-28T18:10:40.248 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:10:40.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.447+0000 7f0135000640 1 -- 172.21.15.37:0/3049554919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01301048e0 msgr2=0x7f0130104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:40.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.447+0000 7f0135000640 1 --2- 172.21.15.37:0/3049554919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01301048e0 0x7f0130104d60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f011c009860 tx=0x7f011c02bb70 comp rx=0 tx=0).stop 2024-03-28T18:10:40.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.447+0000 7f0135000640 1 -- 172.21.15.37:0/3049554919 shutdown_connections 2024-03-28T18:10:40.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.447+0000 7f0135000640 1 --2- 172.21.15.37:0/3049554919 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f01301052a0 0x7f013010bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.447+0000 7f0135000640 1 --2- 172.21.15.37:0/3049554919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01301048e0 0x7f0130104d60 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.447+0000 7f0135000640 1 --2- 172.21.15.37:0/3049554919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01301036f0 0x7f0130103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.447+0000 7f0135000640 1 -- 172.21.15.37:0/3049554919 >> 172.21.15.37:0/3049554919 conn(0x7f01300fec80 msgr2=0x7f01301010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:40.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.447+0000 7f0135000640 1 -- 172.21.15.37:0/3049554919 shutdown_connections 2024-03-28T18:10:40.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.447+0000 7f0135000640 1 -- 172.21.15.37:0/3049554919 wait complete. 2024-03-28T18:10:40.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.448+0000 7f0135000640 1 Processor -- start 2024-03-28T18:10:40.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.448+0000 7f0135000640 1 -- start start 2024-03-28T18:10:40.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.449+0000 7f0135000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01301036f0 0x7f0130199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.449+0000 7f0135000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f01301048e0 0x7f0130199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.449+0000 7f0135000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01301052a0 0x7f01301a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.449+0000 7f0135000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013010f370 con 0x7f01301048e0 2024-03-28T18:10:40.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.449+0000 7f0135000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013010f1f0 con 0x7f01301052a0 2024-03-28T18:10:40.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.449+0000 7f0135000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013010f4f0 con 0x7f01301036f0 2024-03-28T18:10:40.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.449+0000 7f012f400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01301036f0 0x7f0130199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.449+0000 7f012f400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01301036f0 0x7f0130199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55240/0 (socket says 172.21.15.37:55240) 2024-03-28T18:10:40.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.449+0000 7f012f400640 1 -- 172.21.15.37:0/3308870547 learned_addr learned my addr 172.21.15.37:0/3308870547 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:40.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.449+0000 7f012fe00640 1 --2- 172.21.15.37:0/3308870547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01301052a0 0x7f01301a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.449+0000 7f012ea00640 1 --2- 172.21.15.37:0/3308870547 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f01301048e0 0x7f0130199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.449+0000 7f012f400640 1 -- 172.21.15.37:0/3308870547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01301052a0 msgr2=0x7f01301a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:40.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.449+0000 7f012f400640 1 --2- 172.21.15.37:0/3308870547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01301052a0 0x7f01301a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.449+0000 7f012f400640 1 -- 172.21.15.37:0/3308870547 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f01301048e0 msgr2=0x7f0130199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:40.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.449+0000 7f012f400640 1 --2- 172.21.15.37:0/3308870547 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f01301048e0 0x7f0130199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.449+0000 7f012f400640 1 -- 172.21.15.37:0/3308870547 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01301a1660 con 0x7f01301036f0 2024-03-28T18:10:40.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.449+0000 7f012ea00640 1 --2- 172.21.15.37:0/3308870547 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f01301048e0 0x7f0130199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:40.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.450+0000 7f012f400640 1 --2- 172.21.15.37:0/3308870547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01301036f0 0x7f0130199a40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f012000e9f0 tx=0x7f012000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:40.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.450+0000 7f0127e00640 1 -- 172.21.15.37:0/3308870547 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012000ccc0 con 0x7f01301036f0 2024-03-28T18:10:40.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.450+0000 7f0127e00640 1 -- 172.21.15.37:0/3308870547 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01200035f0 con 0x7f01301036f0 2024-03-28T18:10:40.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.450+0000 7f0135000640 1 -- 172.21.15.37:0/3308870547 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01301a1950 con 0x7f01301036f0 2024-03-28T18:10:40.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.450+0000 7f0127e00640 1 -- 172.21.15.37:0/3308870547 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012000ce60 con 0x7f01301036f0 2024-03-28T18:10:40.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.450+0000 7f0135000640 1 -- 172.21.15.37:0/3308870547 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01301a1e70 con 0x7f01301036f0 2024-03-28T18:10:40.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.451+0000 7f0135000640 1 -- 172.21.15.37:0/3308870547 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f013010aa30 con 0x7f01301036f0 2024-03-28T18:10:40.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.453+0000 7f0127e00640 1 -- 172.21.15.37:0/3308870547 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0120004030 con 0x7f01301036f0 2024-03-28T18:10:40.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.454+0000 7f0127e00640 1 --2- 172.21.15.37:0/3308870547 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f01140a2880 0x7f01140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.454+0000 7f0127e00640 1 -- 172.21.15.37:0/3308870547 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f01200ced80 con 0x7f01301036f0 2024-03-28T18:10:40.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.454+0000 7f012ea00640 1 --2- 172.21.15.37:0/3308870547 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f01140a2880 0x7f01140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.455+0000 7f012ea00640 1 --2- 172.21.15.37:0/3308870547 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f01140a2880 0x7f01140a4d40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f011c009830 tx=0x7f011c005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:40.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.461+0000 7f0127e00640 1 -- 172.21.15.37:0/3308870547 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f012009bdd0 con 0x7f01301036f0 2024-03-28T18:10:40.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:40 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1100634885' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:40.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:40 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3292714362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:40.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:40 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1100634885' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:40.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:40 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3292714362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:40.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:40 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1100634885' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:40.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:40 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3292714362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:40.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.666+0000 7f0135000640 1 -- 172.21.15.37:0/3308870547 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01300622a0 con 0x7f01301036f0 2024-03-28T18:10:40.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.668+0000 7f0127e00640 1 -- 172.21.15.37:0/3308870547 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22575 (secure 0 0 0) 0x7f01200860e0 con 0x7f01301036f0 2024-03-28T18:10:40.669 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:40.669 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":87,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","created":"2024-03-28T17:59:05.949847+0000","modified":"2024-03-28T18:09:15.709901+0000","last_up_change":"2024-03-28T18:07:21.021030+0000","last_in_change":"2024-03-28T18:07:01.416192+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:03:00.517967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:08:18.365827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:08:19.446282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db60618d-56f7-4130-8a90-84cbc480965b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6803","nonce":3562602035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6805","nonce":3562602035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6809","nonce":3562602035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6807","nonce":3562602035}]},"public_addr":"172.21.15.37:6803/3562602035","cluster_addr":"172.21.15.37:6805/3562602035","heartbeat_back_addr":"172.21.15.37:6809/3562602035","heartbeat_front_addr":"172.21.15.37:6807/3562602035","state":["exists","up"]},{"osd":1,"uuid":"7a4b7ad6-905e-4258-9574-693d4277dfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6801","nonce":3326537211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6803","nonce":3326537211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6807","nonce":3326537211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6805","nonce":3326537211}]},"public_addr":"172.21.15.59:6801/3326537211","cluster_addr":"172.21.15.59:6803/3326537211","heartbeat_back_addr":"172.21.15.59:6807/3326537211","heartbeat_front_addr":"172.21.15.59:6805/3326537211","state":["exists","up"]},{"osd":2,"uuid":"8b76ec1c-46bf-4336-89b5-12d82906e8b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6801","nonce":1269398427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6803","nonce":1269398427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6807","nonce":1269398427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6805","nonce":1269398427}]},"public_addr":"172.21.15.170:6801/1269398427","cluster_addr":"172.21.15.170:6803/1269398427","heartbeat_back_addr":"172.21.15.170:6807/1269398427","heartbeat_front_addr":"172.21.15.170:6805/1269398427","state":["exists","up"]},{"osd":3,"uuid":"66eca1da-2fbd-44e1-8bbf-5635a2c70ad5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6811","nonce":317805240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6813","nonce":317805240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6817","nonce":317805240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6815","nonce":317805240}]},"public_addr":"172.21.15.37:6811/317805240","cluster_addr":"172.21.15.37:6813/317805240","heartbeat_back_addr":"172.21.15.37:6817/317805240","heartbeat_front_addr":"172.21.15.37:6815/317805240","state":["exists","up"]},{"osd":4,"uuid":"4c877335-e42d-4d8e-9f48-a1d0a89ae438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6809","nonce":2410632623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6811","nonce":2410632623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6815","nonce":2410632623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6813","nonce":2410632623}]},"public_addr":"172.21.15.59:6809/2410632623","cluster_addr":"172.21.15.59:6811/2410632623","heartbeat_back_addr":"172.21.15.59:6815/2410632623","heartbeat_front_addr":"172.21.15.59:6813/2410632623","state":["exists","up"]},{"osd":5,"uuid":"b884842a-86e2-455b-9570-ff163b3bc0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6809","nonce":1218943899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6811","nonce":1218943899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6815","nonce":1218943899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6813","nonce":1218943899}]},"public_addr":"172.21.15.170:6809/1218943899","cluster_addr":"172.21.15.170:6811/1218943899","heartbeat_back_addr":"172.21.15.170:6815/1218943899","heartbeat_front_addr":"172.21.15.170:6813/1218943899","state":["exists","up"]},{"osd":6,"uuid":"249a1200-77f4-44ad-981f-1d352f732b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6819","nonce":1899652046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6821","nonce":1899652046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6825","nonce":1899652046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6823","nonce":1899652046}]},"public_addr":"172.21.15.37:6819/1899652046","cluster_addr":"172.21.15.37:6821/1899652046","heartbeat_back_addr":"172.21.15.37:6825/1899652046","heartbeat_front_addr":"172.21.15.37:6823/1899652046","state":["exists","up"]},{"osd":7,"uuid":"c73338e6-f505-4517-a84b-f35e04750a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6817","nonce":1343675690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6819","nonce":1343675690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6823","nonce":1343675690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6821","nonce":1343675690}]},"public_addr":"172.21.15.59:6817/1343675690","cluster_addr":"172.21.15.59:6819/1343675690","heartbeat_back_addr":"172.21.15.59:6823/1343675690","heartbeat_front_addr":"172.21.15.59:6821/1343675690","state":["exists","up"]},{"osd":8,"uuid":"149b8297-ff2d-4a88-9b1e-0def021cc516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6817","nonce":3235794556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6819","nonce":3235794556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6823","nonce":3235794556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6821","nonce":3235794556}]},"public_addr":"172.21.15.170:6817/3235794556","cluster_addr":"172.21.15.170:6819/3235794556","heartbeat_back_addr":"172.21.15.170:6823/3235794556","heartbeat_front_addr":"172.21.15.170:6821/3235794556","state":["exists","up"]},{"osd":9,"uuid":"55c27cc2-c771-44b7-90fe-8f0feca44162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6827","nonce":2307153008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6829","nonce":2307153008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6833","nonce":2307153008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6831","nonce":2307153008}]},"public_addr":"172.21.15.37:6827/2307153008","cluster_addr":"172.21.15.37:6829/2307153008","heartbeat_back_addr":"172.21.15.37:6833/2307153008","heartbeat_front_addr":"172.21.15.37:6831/2307153008","state":["exists","up"]},{"osd":10,"uuid":"c0595135-ab81-49d3-8c23-19f960c03e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6825","nonce":2113830275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6827","nonce":2113830275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6831","nonce":2113830275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6829","nonce":2113830275}]},"public_addr":"172.21.15.59:6825/2113830275","cluster_addr":"172.21.15.59:6827/2113830275","heartbeat_back_addr":"172.21.15.59:6831/2113830275","heartbeat_front_addr":"172.21.15.59:6829/2113830275","state":["exists","up"]},{"osd":11,"uuid":"5871e177-c17b-4637-a797-4fa5125920dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6825","nonce":3082439360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6827","nonce":3082439360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6831","nonce":3082439360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6829","nonce":3082439360}]},"public_addr":"172.21.15.170:6825/3082439360","cluster_addr":"172.21.15.170:6827/3082439360","heartbeat_back_addr":"172.21.15.170:6831/3082439360","heartbeat_front_addr":"172.21.15.170:6829/3082439360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:07.690545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:33.183396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:57.400263+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:30.597211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:57.259044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:22.627823+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:57.585863+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:25.000476+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:50.876688+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:25.859761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:53.563146+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:07:19.304115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3023379428":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1443710231":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6800/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6801/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6801/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6800/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1007625490":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/2895698584":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/497313034":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/3293943475":"2024-03-29T17:59:39.162038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:10:40.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.671+0000 7f0135000640 1 -- 172.21.15.37:0/3308870547 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f01140a2880 msgr2=0x7f01140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:40.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.671+0000 7f0135000640 1 --2- 172.21.15.37:0/3308870547 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f01140a2880 0x7f01140a4d40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f011c009830 tx=0x7f011c005d20 comp rx=0 tx=0).stop 2024-03-28T18:10:40.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.671+0000 7f0135000640 1 -- 172.21.15.37:0/3308870547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01301036f0 msgr2=0x7f0130199a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:40.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.671+0000 7f0135000640 1 --2- 172.21.15.37:0/3308870547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01301036f0 0x7f0130199a40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f012000e9f0 tx=0x7f012000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.671+0000 7f0135000640 1 -- 172.21.15.37:0/3308870547 shutdown_connections 2024-03-28T18:10:40.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.671+0000 7f0135000640 1 --2- 172.21.15.37:0/3308870547 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f01140a2880 0x7f01140a4d40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.671+0000 7f0135000640 1 --2- 172.21.15.37:0/3308870547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01301052a0 0x7f01301a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.671+0000 7f0135000640 1 --2- 172.21.15.37:0/3308870547 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f01301048e0 0x7f0130199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.671+0000 7f0135000640 1 --2- 172.21.15.37:0/3308870547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01301036f0 0x7f0130199a40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.671+0000 7f0135000640 1 -- 172.21.15.37:0/3308870547 >> 172.21.15.37:0/3308870547 conn(0x7f01300fec80 msgr2=0x7f0130107c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:40.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.671+0000 7f0135000640 1 -- 172.21.15.37:0/3308870547 shutdown_connections 2024-03-28T18:10:40.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.671+0000 7f0135000640 1 -- 172.21.15.37:0/3308870547 wait complete. 2024-03-28T18:10:40.689 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T18:08:19.968894+0000', 'modified': '2024-03-28T18:08:25.415653+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24580, 'mds_1': 24569, 'mds_2': 24550}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24512': {'gid': 24512, 'name': 'h', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.59:6837/4034607523', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 4034607523}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 4034607523}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24517': {'gid': 24517, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.170:6835/4133178144', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6834', 'nonce': 4133178144}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 4133178144}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24529': {'gid': 24529, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.170:6837/1308157923', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6836', 'nonce': 1308157923}, {'type': 'v1', 'addr': '172.21.15.170:6837', 'nonce': 1308157923}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24550': {'gid': 24550, 'name': 'i', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.170:6839/1575518279', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6838', 'nonce': 1575518279}, {'type': 'v1', 'addr': '172.21.15.170:6839', 'nonce': 1575518279}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24569': {'gid': 24569, 'name': 'b', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.59:6839/2091629078', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6838', 'nonce': 2091629078}, {'type': 'v1', 'addr': '172.21.15.59:6839', 'nonce': 2091629078}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24580': {'gid': 24580, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.37:6841/1923263622', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.37:6840', 'nonce': 1923263622}, {'type': 'v1', 'addr': '172.21.15.37:6841', 'nonce': 1923263622}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24580, 'qdb_cluster': [24550, 24569, 24580]} 2024-03-28T18:10:40.689 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-28T18:10:40.689 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-28T18:10:40.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.886+0000 7f9873e00640 1 -- 172.21.15.37:0/3685822880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9874102000 msgr2=0x7f98741043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:40.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.886+0000 7f9873e00640 1 --2- 172.21.15.37:0/3685822880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9874102000 0x7f98741043f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9864009ca0 tx=0x7f986402c100 comp rx=0 tx=0).stop 2024-03-28T18:10:40.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.886+0000 7f9873e00640 1 -- 172.21.15.37:0/3685822880 shutdown_connections 2024-03-28T18:10:40.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.886+0000 7f9873e00640 1 --2- 172.21.15.37:0/3685822880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9874104930 0x7f9874106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.886+0000 7f9873e00640 1 --2- 172.21.15.37:0/3685822880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9874107260 0x7f98741096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.886+0000 7f9873e00640 1 --2- 172.21.15.37:0/3685822880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9874102000 0x7f98741043f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.886+0000 7f9873e00640 1 -- 172.21.15.37:0/3685822880 >> 172.21.15.37:0/3685822880 conn(0x7f98740fba10 msgr2=0x7f98740fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:40.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.886+0000 7f9873e00640 1 -- 172.21.15.37:0/3685822880 shutdown_connections 2024-03-28T18:10:40.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.886+0000 7f9873e00640 1 -- 172.21.15.37:0/3685822880 wait complete. 2024-03-28T18:10:40.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.887+0000 7f9873e00640 1 Processor -- start 2024-03-28T18:10:40.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.888+0000 7f9873e00640 1 -- start start 2024-03-28T18:10:40.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.888+0000 7f9873e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9874102000 0x7f98741978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.888+0000 7f9873e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9874104930 0x7f9874197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.888+0000 7f9873e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9874107260 0x7f987419ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.888+0000 7f9873e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987410d210 con 0x7f9874104930 2024-03-28T18:10:40.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.888+0000 7f9873e00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987410d090 con 0x7f9874107260 2024-03-28T18:10:40.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.888+0000 7f9873e00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987410d390 con 0x7f9874102000 2024-03-28T18:10:40.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.888+0000 7f9872a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9874102000 0x7f98741978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.888+0000 7f9872a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9874102000 0x7f98741978c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55256/0 (socket says 172.21.15.37:55256) 2024-03-28T18:10:40.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.888+0000 7f9872a00640 1 -- 172.21.15.37:0/1040599167 learned_addr learned my addr 172.21.15.37:0/1040599167 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:40.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.888+0000 7f9873400640 1 --2- 172.21.15.37:0/1040599167 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9874107260 0x7f987419ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.888+0000 7f9872000640 1 --2- 172.21.15.37:0/1040599167 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9874104930 0x7f9874197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.889+0000 7f9872a00640 1 -- 172.21.15.37:0/1040599167 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9874107260 msgr2=0x7f987419ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:40.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.889+0000 7f9872a00640 1 --2- 172.21.15.37:0/1040599167 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9874107260 0x7f987419ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.889+0000 7f9872a00640 1 -- 172.21.15.37:0/1040599167 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9874104930 msgr2=0x7f9874197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:40.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.889+0000 7f9872a00640 1 --2- 172.21.15.37:0/1040599167 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9874104930 0x7f9874197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.889+0000 7f9872a00640 1 -- 172.21.15.37:0/1040599167 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987419f4e0 con 0x7f9874102000 2024-03-28T18:10:40.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.889+0000 7f9872000640 1 --2- 172.21.15.37:0/1040599167 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9874104930 0x7f9874197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:40.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.889+0000 7f9872a00640 1 --2- 172.21.15.37:0/1040599167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9874102000 0x7f98741978c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f98640325f0 tx=0x7f9864035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:40.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.889+0000 7f986b400640 1 -- 172.21.15.37:0/1040599167 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9864039040 con 0x7f9874102000 2024-03-28T18:10:40.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.889+0000 7f986b400640 1 -- 172.21.15.37:0/1040599167 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9864044070 con 0x7f9874102000 2024-03-28T18:10:40.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.889+0000 7f9873e00640 1 -- 172.21.15.37:0/1040599167 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f987419f770 con 0x7f9874102000 2024-03-28T18:10:40.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.889+0000 7f986b400640 1 -- 172.21.15.37:0/1040599167 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9864004460 con 0x7f9874102000 2024-03-28T18:10:40.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.889+0000 7f9873e00640 1 -- 172.21.15.37:0/1040599167 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f987419fc80 con 0x7f9874102000 2024-03-28T18:10:40.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.892+0000 7f986b400640 1 -- 172.21.15.37:0/1040599167 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9864004600 con 0x7f9874102000 2024-03-28T18:10:40.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.892+0000 7f987c5a3740 1 Processor -- start 2024-03-28T18:10:40.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.893+0000 7f987c5a3740 1 -- start start 2024-03-28T18:10:40.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.893+0000 7f987c5a3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2190 0x561f022c25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.893+0000 7f987c5a3740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c2b80 0x561f022c7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.893+0000 7f987c5a3740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c7cf0 0x561f022ca180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.893+0000 7f987c5a3740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f02222c90 con 0x561f022c2b80 2024-03-28T18:10:40.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.893+0000 7f987c5a3740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f021cb980 con 0x561f022c7cf0 2024-03-28T18:10:40.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.893+0000 7f987c5a3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f0221ed30 con 0x561f022c2190 2024-03-28T18:10:40.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.893+0000 7f9873400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2190 0x561f022c25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.893+0000 7f9872000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c7cf0 0x561f022ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.893+0000 7f9873400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2190 0x561f022c25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55266/0 (socket says 172.21.15.37:55266) 2024-03-28T18:10:40.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.893+0000 7f9872000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c7cf0 0x561f022ca180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:37858/0 (socket says 172.21.15.37:37858) 2024-03-28T18:10:40.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.893+0000 7f9873400640 1 -- 172.21.15.37:0/3671435322 learned_addr learned my addr 172.21.15.37:0/3671435322 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:40.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.893+0000 7f9872a00640 1 --2- 172.21.15.37:0/3671435322 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c2b80 0x561f022c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.894+0000 7f9873400640 1 -- 172.21.15.37:0/3671435322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c7cf0 msgr2=0x561f022ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:40.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.894+0000 7f9873400640 1 --2- 172.21.15.37:0/3671435322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c7cf0 0x561f022ca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.894+0000 7f9873400640 1 -- 172.21.15.37:0/3671435322 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c2b80 msgr2=0x561f022c7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:40.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.894+0000 7f9873400640 1 --2- 172.21.15.37:0/3671435322 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c2b80 0x561f022c7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.894+0000 7f9873400640 1 -- 172.21.15.37:0/3671435322 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f021e3510 con 0x561f022c2190 2024-03-28T18:10:40.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.894+0000 7f9872a00640 1 --2- 172.21.15.37:0/3671435322 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c2b80 0x561f022c7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:40.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.894+0000 7f9873400640 1 --2- 172.21.15.37:0/3671435322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2190 0x561f022c25b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f986c009c90 tx=0x7f986c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7312a5fe4e56f559 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:40.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.894+0000 7f986b400640 1 --2- 172.21.15.37:0/1040599167 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f98580a2880 0x7f98580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.894+0000 7f9863e00640 1 -- 172.21.15.37:0/3671435322 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c004230 con 0x561f022c2190 2024-03-28T18:10:40.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.894+0000 7f9863e00640 1 -- 172.21.15.37:0/3671435322 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f986c009e40 con 0x561f022c2190 2024-03-28T18:10:40.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.894+0000 7f9863e00640 1 -- 172.21.15.37:0/3671435322 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c0197e0 con 0x561f022c2190 2024-03-28T18:10:40.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.894+0000 7f986b400640 1 -- 172.21.15.37:0/1040599167 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f98640f0fe0 con 0x7f9874102000 2024-03-28T18:10:40.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.894+0000 7f9872a00640 1 --2- 172.21.15.37:0/1040599167 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f98580a2880 0x7f98580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.895+0000 7f987c5a3740 1 -- 172.21.15.37:0/3671435322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2190 msgr2=0x561f022c25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:40.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.895+0000 7f987c5a3740 1 --2- 172.21.15.37:0/3671435322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2190 0x561f022c25b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f986c009c90 tx=0x7f986c011810 comp rx=0 tx=0).stop 2024-03-28T18:10:40.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.895+0000 7f9872a00640 1 --2- 172.21.15.37:0/1040599167 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f98580a2880 0x7f98580a4d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f98640eee60 tx=0x7f986402c100 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:40.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.895+0000 7f987c5a3740 1 -- 172.21.15.37:0/3671435322 shutdown_connections 2024-03-28T18:10:40.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.895+0000 7f987c5a3740 1 --2- 172.21.15.37:0/3671435322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c7cf0 0x561f022ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.895+0000 7f987c5a3740 1 --2- 172.21.15.37:0/3671435322 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c2b80 0x561f022c7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.895+0000 7f987c5a3740 1 --2- 172.21.15.37:0/3671435322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2190 0x561f022c25b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.895+0000 7f987c5a3740 1 -- 172.21.15.37:0/3671435322 >> 172.21.15.37:0/3671435322 conn(0x561f022b92f0 msgr2=0x561f022b96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:40.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.895+0000 7f987c5a3740 1 -- 172.21.15.37:0/3671435322 shutdown_connections 2024-03-28T18:10:40.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.895+0000 7f987c5a3740 1 -- 172.21.15.37:0/3671435322 wait complete. 2024-03-28T18:10:40.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.896+0000 7f987c5a3740 1 -- start start 2024-03-28T18:10:40.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.896+0000 7f987c5a3740 1 Processor -- start 2024-03-28T18:10:40.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.896+0000 7f987c5a3740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c2190 0x561f022cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.896+0000 7f987c5a3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2b80 0x561f022ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.896+0000 7f987c5a3740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c7cf0 0x561f022d3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.896+0000 7f987c5a3740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f022caca0 con 0x561f022c7cf0 2024-03-28T18:10:40.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.896+0000 7f987c5a3740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f021dedd0 con 0x561f022c2190 2024-03-28T18:10:40.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.896+0000 7f987c5a3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f022cae20 con 0x561f022c2b80 2024-03-28T18:10:40.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.896+0000 7f987c5a3740 10 client.0 _ll_register_callbacks cb 0x561f01f36ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:10:40.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.897+0000 7f9873400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c2190 0x561f022cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.897+0000 7f9872000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2b80 0x561f022ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.897+0000 7f9872000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2b80 0x561f022ccc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55268/0 (socket says 172.21.15.37:55268) 2024-03-28T18:10:40.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.897+0000 7f9872000640 1 -- 172.21.15.37:0/2395008847 learned_addr learned my addr 172.21.15.37:0/2395008847 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:40.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.897+0000 7f9873400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c2190 0x561f022cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:37872/0 (socket says 172.21.15.37:37872) 2024-03-28T18:10:40.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.897+0000 7f9873400640 1 --2- 172.21.15.37:0/2395008847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c7cf0 0x561f022d3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.897+0000 7f9872000640 1 -- 172.21.15.37:0/2395008847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c2190 msgr2=0x561f022cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:40.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.897+0000 7f9872000640 1 --2- 172.21.15.37:0/2395008847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c2190 0x561f022cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.897+0000 7f9872000640 1 -- 172.21.15.37:0/2395008847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c7cf0 msgr2=0x561f022d3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:40.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.897+0000 7f9872000640 1 --2- 172.21.15.37:0/2395008847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c7cf0 0x561f022d3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.897+0000 7f9872000640 1 -- 172.21.15.37:0/2395008847 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561f01f372d0 con 0x561f022c2b80 2024-03-28T18:10:40.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.897+0000 7f9873400640 1 --2- 172.21.15.37:0/2395008847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c2190 0x561f022cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:40.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.897+0000 7f9872000640 1 --2- 172.21.15.37:0/2395008847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2b80 0x561f022ccc70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f985c012700 tx=0x7f985c012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57ea9bf5f7a157e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:40.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.897+0000 7f9860c00640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:10:40.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.897+0000 7f9860c00640 1 -- 172.21.15.37:0/2395008847 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c004320 con 0x561f022c2b80 2024-03-28T18:10:40.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.898+0000 7f9860c00640 1 -- 172.21.15.37:0/2395008847 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f985c011940 con 0x561f022c2b80 2024-03-28T18:10:40.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.898+0000 7f987c5a3740 1 -- 172.21.15.37:0/2395008847 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561f021c4520 con 0x561f022c2b80 2024-03-28T18:10:40.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.898+0000 7f9860c00640 1 -- 172.21.15.37:0/2395008847 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f985c0044c0 con 0x561f022c2b80 2024-03-28T18:10:40.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.898+0000 7f987c5a3740 10 client.25165 fetch_fsmap learned FSMap version 22 2024-03-28T18:10:40.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.898+0000 7f987c5a3740 1 -- 172.21.15.37:0/2395008847 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x561f020016b0 con 0x561f022c2b80 2024-03-28T18:10:40.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.898+0000 7f9860c00640 1 -- 172.21.15.37:0/2395008847 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x561f021c4520 con 0x561f022c2b80 2024-03-28T18:10:40.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.898+0000 7f9860c00640 1 -- 172.21.15.37:0/2395008847 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f985c00f440 con 0x561f022c2b80 2024-03-28T18:10:40.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.898+0000 7f987c5a3740 10 client.25165 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T18:10:40.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.898+0000 7f987c5a3740 10 client.25165 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T18:10:40.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.898+0000 7f987c5a3740 20 client.25165 populate_metadata read hostname 'smithi037' 2024-03-28T18:10:40.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.898+0000 7f987c5a3740 1 --2- 172.21.15.37:0/2395008847 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x561f022d8f90 0x561f022d8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.898+0000 7f987c5a3740 4 client.25165 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:10:40.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.898+0000 7f987c5a3740 1 -- 172.21.15.37:0/2395008847 --> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561f022301d0 con 0x561f022d8f90 2024-03-28T18:10:40.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.899+0000 7f9873400640 1 --2- 172.21.15.37:0/2395008847 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x561f022d8f90 0x561f022d8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.899+0000 7f9873400640 1 --2- 172.21.15.37:0/2395008847 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x561f022d8f90 0x561f022d8050 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f4017f604ca0ffd2 server_cookie=39de6b290b21b382 in_seq=0 out_seq=0 2024-03-28T18:10:40.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.899+0000 7f9860c00640 10 client.25165 ms_handle_connect on v2:172.21.15.170:6838/1575518279 2024-03-28T18:10:40.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.900+0000 7f9860c00640 1 -- 172.21.15.37:0/2395008847 <== mds.2 v2:172.21.15.170:6838/1575518279 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561f022301d0 con 0x561f022d8f90 2024-03-28T18:10:40.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.900+0000 7f9860c00640 10 client.25165 handle_command_reply: tid=0 2024-03-28T18:10:40.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.900+0000 7f987c5a3740 1 client.25165 shutdown 2024-03-28T18:10:40.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.900+0000 7f987c5a3740 1 -- 172.21.15.37:0/2395008847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2b80 msgr2=0x561f022ccc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:40.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.900+0000 7f987c5a3740 1 --2- 172.21.15.37:0/2395008847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2b80 0x561f022ccc70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f985c012700 tx=0x7f985c012bc0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.900+0000 7f987c5a3740 1 -- 172.21.15.37:0/2395008847 shutdown_connections 2024-03-28T18:10:40.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.900+0000 7f987c5a3740 1 --2- 172.21.15.37:0/2395008847 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x561f022d8f90 0x561f022d8050 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.900+0000 7f987c5a3740 1 --2- 172.21.15.37:0/2395008847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c7cf0 0x561f022d3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.900+0000 7f987c5a3740 1 --2- 172.21.15.37:0/2395008847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2b80 0x561f022ccc70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.900+0000 7f987c5a3740 1 --2- 172.21.15.37:0/2395008847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c2190 0x561f022cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.900+0000 7f9860c00640 0 client.25165 ms_handle_reset on v2:172.21.15.170:6838/1575518279 2024-03-28T18:10:40.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.900+0000 7f987c5a3740 1 -- 172.21.15.37:0/2395008847 >> 172.21.15.37:0/2395008847 conn(0x561f022b92f0 msgr2=0x561f022b84f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:40.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.900+0000 7f987c5a3740 1 -- 172.21.15.37:0/2395008847 shutdown_connections 2024-03-28T18:10:40.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.900+0000 7f987c5a3740 1 -- 172.21.15.37:0/2395008847 wait complete. 2024-03-28T18:10:40.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.900+0000 7f987c5a3740 20 client.25165 trim_cache size 0 max 16384 2024-03-28T18:10:40.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.918+0000 7f987c5a3740 1 Processor -- start 2024-03-28T18:10:40.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.918+0000 7f987c5a3740 1 -- start start 2024-03-28T18:10:40.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.918+0000 7f987c5a3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2190 0x561f022cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.918+0000 7f987c5a3740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c2b80 0x561f022b96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.919+0000 7f987c5a3740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c7cf0 0x561f022b7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.919+0000 7f987c5a3740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f022d87b0 con 0x561f022c2b80 2024-03-28T18:10:40.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.919+0000 7f987c5a3740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f022d8e10 con 0x561f022c7cf0 2024-03-28T18:10:40.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.919+0000 7f987c5a3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f022d9470 con 0x561f022c2190 2024-03-28T18:10:40.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.919+0000 7f9873400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2190 0x561f022cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.919+0000 7f9873400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2190 0x561f022cb1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55270/0 (socket says 172.21.15.37:55270) 2024-03-28T18:10:40.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.919+0000 7f9873400640 1 -- 172.21.15.37:0/350706011 learned_addr learned my addr 172.21.15.37:0/350706011 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:40.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.919+0000 7f9873400640 1 --2- 172.21.15.37:0/350706011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c7cf0 0x561f022b7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.919+0000 7f9872000640 1 --2- 172.21.15.37:0/350706011 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c2b80 0x561f022b96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.919+0000 7f9873400640 1 -- 172.21.15.37:0/350706011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c7cf0 msgr2=0x561f022b7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:40.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.919+0000 7f9873400640 1 --2- 172.21.15.37:0/350706011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c7cf0 0x561f022b7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.919+0000 7f9873400640 1 -- 172.21.15.37:0/350706011 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c2b80 msgr2=0x561f022b96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:40.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.919+0000 7f9873400640 1 --2- 172.21.15.37:0/350706011 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c2b80 0x561f022b96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.919+0000 7f9873400640 1 -- 172.21.15.37:0/350706011 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f022cd720 con 0x561f022c2190 2024-03-28T18:10:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.919+0000 7f9873400640 1 --2- 172.21.15.37:0/350706011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2190 0x561f022cb1c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f986c009c30 tx=0x7f986c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5adfda7851823425 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.920+0000 7f9863400640 1 -- 172.21.15.37:0/350706011 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c02c0b0 con 0x561f022c2190 2024-03-28T18:10:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.920+0000 7f9863400640 1 -- 172.21.15.37:0/350706011 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f986c02c250 con 0x561f022c2190 2024-03-28T18:10:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.920+0000 7f9863400640 1 -- 172.21.15.37:0/350706011 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c0038e0 con 0x561f022c2190 2024-03-28T18:10:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.920+0000 7f987c5a3740 1 -- 172.21.15.37:0/350706011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2190 msgr2=0x561f022cb1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.920+0000 7f987c5a3740 1 --2- 172.21.15.37:0/350706011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2190 0x561f022cb1c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f986c009c30 tx=0x7f986c005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.920+0000 7f987c5a3740 1 -- 172.21.15.37:0/350706011 shutdown_connections 2024-03-28T18:10:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.920+0000 7f987c5a3740 1 --2- 172.21.15.37:0/350706011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c7cf0 0x561f022b7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.920+0000 7f987c5a3740 1 --2- 172.21.15.37:0/350706011 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c2b80 0x561f022b96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.920+0000 7f987c5a3740 1 --2- 172.21.15.37:0/350706011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c2190 0x561f022cb1c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.920+0000 7f987c5a3740 1 -- 172.21.15.37:0/350706011 >> 172.21.15.37:0/350706011 conn(0x561f022b92f0 msgr2=0x561f022c3620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.920+0000 7f987c5a3740 1 -- 172.21.15.37:0/350706011 shutdown_connections 2024-03-28T18:10:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.920+0000 7f987c5a3740 1 -- 172.21.15.37:0/350706011 wait complete. 2024-03-28T18:10:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.921+0000 7f987c5a3740 1 -- start start 2024-03-28T18:10:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.921+0000 7f987c5a3740 1 Processor -- start 2024-03-28T18:10:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.921+0000 7f987c5a3740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c2190 0x561f022cf560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.921+0000 7f987c5a3740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c2b80 0x561f022cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.921+0000 7f987c5a3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c7cf0 0x561f022d2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.921+0000 7f987c5a3740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f022da130 con 0x561f022c2b80 2024-03-28T18:10:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.921+0000 7f987c5a3740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f022d9ad0 con 0x561f022c2190 2024-03-28T18:10:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.921+0000 7f987c5a3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f022da790 con 0x561f022c7cf0 2024-03-28T18:10:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.921+0000 7f987c5a3740 10 client.0 _ll_register_callbacks cb 0x7f985c009860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:10:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.922+0000 7f9873400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c2190 0x561f022cf560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.922+0000 7f9872000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c2b80 0x561f022cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.922+0000 7f9873400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c2190 0x561f022cf560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:37900/0 (socket says 172.21.15.37:37900) 2024-03-28T18:10:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.922+0000 7f9873400640 1 -- 172.21.15.37:0/221856623 learned_addr learned my addr 172.21.15.37:0/221856623 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.922+0000 7f9873400640 1 --2- 172.21.15.37:0/221856623 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c7cf0 0x561f022d2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.922+0000 7f9873400640 1 -- 172.21.15.37:0/221856623 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c7cf0 msgr2=0x561f022d2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.922+0000 7f9873400640 1 --2- 172.21.15.37:0/221856623 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c7cf0 0x561f022d2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.922+0000 7f9873400640 1 -- 172.21.15.37:0/221856623 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c2b80 msgr2=0x561f022cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.922+0000 7f9873400640 1 --2- 172.21.15.37:0/221856623 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c2b80 0x561f022cfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.922+0000 7f9873400640 1 -- 172.21.15.37:0/221856623 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561f022d30b0 con 0x561f022c2190 2024-03-28T18:10:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.922+0000 7f9873400640 1 --2- 172.21.15.37:0/221856623 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c2190 0x561f022cf560 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f986c02c220 tx=0x7f986c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c306ee5cec0d396 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.922+0000 7f9860c00640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:10:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.922+0000 7f9860c00640 1 -- 172.21.15.37:0/221856623 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c005530 con 0x561f022c2190 2024-03-28T18:10:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.923+0000 7f9860c00640 1 -- 172.21.15.37:0/221856623 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f986c02c850 con 0x561f022c2190 2024-03-28T18:10:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.923+0000 7f987c5a3740 1 -- 172.21.15.37:0/221856623 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f986c02c250 con 0x561f022c2190 2024-03-28T18:10:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.923+0000 7f9860c00640 1 -- 172.21.15.37:0/221856623 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f986c02ca50 con 0x561f022c2190 2024-03-28T18:10:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.923+0000 7f987c5a3740 10 client.25145 fetch_fsmap learned FSMap version 22 2024-03-28T18:10:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.923+0000 7f987c5a3740 1 -- 172.21.15.37:0/221856623 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x561f023349d0 con 0x561f022c2190 2024-03-28T18:10:40.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.923+0000 7f9860c00640 1 -- 172.21.15.37:0/221856623 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f986c02c250 con 0x561f022c2190 2024-03-28T18:10:40.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.923+0000 7f9860c00640 1 -- 172.21.15.37:0/221856623 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f986c018070 con 0x561f022c2190 2024-03-28T18:10:40.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.923+0000 7f987c5a3740 10 client.25145 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T18:10:40.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.923+0000 7f987c5a3740 10 client.25145 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T18:10:40.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.923+0000 7f987c5a3740 20 client.25145 populate_metadata read hostname 'smithi037' 2024-03-28T18:10:40.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.923+0000 7f987c5a3740 1 --2- 172.21.15.37:0/221856623 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x561f02338880 0x561f02334ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:40.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.923+0000 7f987c5a3740 4 client.25145 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:10:40.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.923+0000 7f987c5a3740 1 -- 172.21.15.37:0/221856623 --> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561f0222a520 con 0x561f02338880 2024-03-28T18:10:40.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.924+0000 7f9872000640 1 --2- 172.21.15.37:0/221856623 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x561f02338880 0x561f02334ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:40.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.924+0000 7f9872000640 1 --2- 172.21.15.37:0/221856623 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x561f02338880 0x561f02334ba0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2d2a028c0d731218 server_cookie=8448e9178c5cba87 in_seq=0 out_seq=0 2024-03-28T18:10:40.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.924+0000 7f9860c00640 10 client.25145 ms_handle_connect on v2:172.21.15.170:6838/1575518279 2024-03-28T18:10:40.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.924+0000 7f9860c00640 1 -- 172.21.15.37:0/221856623 <== mds.2 v2:172.21.15.170:6838/1575518279 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x561f0222a520 con 0x561f02338880 2024-03-28T18:10:40.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.924+0000 7f9860c00640 10 client.25145 handle_command_reply: tid=0 2024-03-28T18:10:40.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.924+0000 7f987c5a3740 1 client.25145 shutdown 2024-03-28T18:10:40.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.924+0000 7f987c5a3740 1 -- 172.21.15.37:0/221856623 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c2190 msgr2=0x561f022cf560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:40.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.924+0000 7f987c5a3740 1 --2- 172.21.15.37:0/221856623 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c2190 0x561f022cf560 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f986c02c220 tx=0x7f986c011810 comp rx=0 tx=0).stop 2024-03-28T18:10:40.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.925+0000 7f987c5a3740 1 -- 172.21.15.37:0/221856623 shutdown_connections 2024-03-28T18:10:40.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.925+0000 7f987c5a3740 1 --2- 172.21.15.37:0/221856623 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x561f02338880 0x561f02334ba0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.925+0000 7f987c5a3740 1 --2- 172.21.15.37:0/221856623 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f022c7cf0 0x561f022d2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.925+0000 7f987c5a3740 1 --2- 172.21.15.37:0/221856623 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x561f022c2b80 0x561f022cfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.925+0000 7f987c5a3740 1 --2- 172.21.15.37:0/221856623 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561f022c2190 0x561f022cf560 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.925+0000 7f987c5a3740 1 -- 172.21.15.37:0/221856623 >> 172.21.15.37:0/221856623 conn(0x561f022b92f0 msgr2=0x561f022cb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:40.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.925+0000 7f9860c00640 0 client.25145 ms_handle_reset on v2:172.21.15.170:6838/1575518279 2024-03-28T18:10:40.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.925+0000 7f987c5a3740 1 -- 172.21.15.37:0/221856623 shutdown_connections 2024-03-28T18:10:40.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.925+0000 7f987c5a3740 1 -- 172.21.15.37:0/221856623 wait complete. 2024-03-28T18:10:40.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.925+0000 7f987c5a3740 20 client.25145 trim_cache size 0 max 16384 2024-03-28T18:10:40.929 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:10:40.930 INFO:teuthology.orchestra.run.smithi037.stdout: "cluster_fsid": "b5ad257c-ed2c-11ee-95d0-87774f69a715", 2024-03-28T18:10:40.930 INFO:teuthology.orchestra.run.smithi037.stdout: "whoami": 2, 2024-03-28T18:10:40.930 INFO:teuthology.orchestra.run.smithi037.stdout: "id": 24550, 2024-03-28T18:10:40.930 INFO:teuthology.orchestra.run.smithi037.stdout: "want_state": "up:active", 2024-03-28T18:10:40.930 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "up:active", 2024-03-28T18:10:40.930 INFO:teuthology.orchestra.run.smithi037.stdout: "fs_name": "cephfs", 2024-03-28T18:10:40.930 INFO:teuthology.orchestra.run.smithi037.stdout: "rank_uptime": 136.51821519500001, 2024-03-28T18:10:40.930 INFO:teuthology.orchestra.run.smithi037.stdout: "mdsmap_epoch": 22, 2024-03-28T18:10:40.930 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap_epoch": 87, 2024-03-28T18:10:40.930 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap_epoch_barrier": 85, 2024-03-28T18:10:40.930 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 158.607843532 2024-03-28T18:10:40.930 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:10:40.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.925+0000 7f9873e00640 1 -- 172.21.15.37:0/1040599167 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f98580a2880 msgr2=0x7f98580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:40.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.925+0000 7f9873e00640 1 --2- 172.21.15.37:0/1040599167 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f98580a2880 0x7f98580a4d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f98640eee60 tx=0x7f986402c100 comp rx=0 tx=0).stop 2024-03-28T18:10:40.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.925+0000 7f9873e00640 1 -- 172.21.15.37:0/1040599167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9874102000 msgr2=0x7f98741978c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:40.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.925+0000 7f9873e00640 1 --2- 172.21.15.37:0/1040599167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9874102000 0x7f98741978c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f98640325f0 tx=0x7f9864035d10 comp rx=0 tx=0).stop 2024-03-28T18:10:40.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.926+0000 7f9873e00640 1 -- 172.21.15.37:0/1040599167 shutdown_connections 2024-03-28T18:10:40.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.926+0000 7f9873e00640 1 --2- 172.21.15.37:0/1040599167 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f98580a2880 0x7f98580a4d40 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.926+0000 7f9873e00640 1 --2- 172.21.15.37:0/1040599167 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9874107260 0x7f987419ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.926+0000 7f9873e00640 1 --2- 172.21.15.37:0/1040599167 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9874104930 0x7f9874197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.926+0000 7f9873e00640 1 --2- 172.21.15.37:0/1040599167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9874102000 0x7f98741978c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:40.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.926+0000 7f9873e00640 1 -- 172.21.15.37:0/1040599167 >> 172.21.15.37:0/1040599167 conn(0x7f98740fba10 msgr2=0x7f9874102940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:40.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.926+0000 7f9873e00640 1 -- 172.21.15.37:0/1040599167 shutdown_connections 2024-03-28T18:10:40.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:40.926+0000 7f9873e00640 1 -- 172.21.15.37:0/1040599167 wait complete. 2024-03-28T18:10:40.944 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T18:10:41.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.152+0000 7f8558a00640 1 -- 172.21.15.37:0/723871372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8554104930 msgr2=0x7f8554106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:41.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.152+0000 7f8558a00640 1 --2- 172.21.15.37:0/723871372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8554104930 0x7f8554106d20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8540009900 tx=0x7f854002bc50 comp rx=0 tx=0).stop 2024-03-28T18:10:41.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.152+0000 7f8558a00640 1 -- 172.21.15.37:0/723871372 shutdown_connections 2024-03-28T18:10:41.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.152+0000 7f8558a00640 1 --2- 172.21.15.37:0/723871372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8554104930 0x7f8554106d20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.152+0000 7f8558a00640 1 --2- 172.21.15.37:0/723871372 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8554107260 0x7f85541096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.152+0000 7f8558a00640 1 --2- 172.21.15.37:0/723871372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8554102000 0x7f85541043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.152+0000 7f8558a00640 1 -- 172.21.15.37:0/723871372 >> 172.21.15.37:0/723871372 conn(0x7f85540fba10 msgr2=0x7f85540fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:41.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.152+0000 7f8558a00640 1 -- 172.21.15.37:0/723871372 shutdown_connections 2024-03-28T18:10:41.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.152+0000 7f8558a00640 1 -- 172.21.15.37:0/723871372 wait complete. 2024-03-28T18:10:41.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.154+0000 7f8558a00640 1 Processor -- start 2024-03-28T18:10:41.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.154+0000 7f8558a00640 1 -- start start 2024-03-28T18:10:41.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.155+0000 7f8558a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8554102000 0x7f8554101970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.155+0000 7f8558a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8554104930 0x7f85540fffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.155+0000 7f8558a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8554107260 0x7f85541004f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.155+0000 7f8558a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855410d260 con 0x7f8554107260 2024-03-28T18:10:41.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.155+0000 7f8558a00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855410d0e0 con 0x7f8554102000 2024-03-28T18:10:41.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.155+0000 7f8558a00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855410d3e0 con 0x7f8554104930 2024-03-28T18:10:41.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.155+0000 7f8553400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8554102000 0x7f8554101970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.155+0000 7f8553400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8554102000 0x7f8554101970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:37932/0 (socket says 172.21.15.37:37932) 2024-03-28T18:10:41.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.155+0000 7f8553400640 1 -- 172.21.15.37:0/3772291491 learned_addr learned my addr 172.21.15.37:0/3772291491 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:41.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.155+0000 7f8552a00640 1 --2- 172.21.15.37:0/3772291491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8554104930 0x7f85540fffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.155+0000 7f8553e00640 1 --2- 172.21.15.37:0/3772291491 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8554107260 0x7f85541004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.155+0000 7f8553400640 1 -- 172.21.15.37:0/3772291491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8554104930 msgr2=0x7f85540fffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:41.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.155+0000 7f8553400640 1 --2- 172.21.15.37:0/3772291491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8554104930 0x7f85540fffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.155+0000 7f8553400640 1 -- 172.21.15.37:0/3772291491 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8554107260 msgr2=0x7f85541004f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:41.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.155+0000 7f8553400640 1 --2- 172.21.15.37:0/3772291491 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8554107260 0x7f85541004f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.155+0000 7f8553400640 1 -- 172.21.15.37:0/3772291491 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8554100d80 con 0x7f8554102000 2024-03-28T18:10:41.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.155+0000 7f8553e00640 1 --2- 172.21.15.37:0/3772291491 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8554107260 0x7f85541004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:10:41.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.156+0000 7f8553400640 1 --2- 172.21.15.37:0/3772291491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8554102000 0x7f8554101970 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f854400eb90 tx=0x7f854400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:41.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.156+0000 7f854be00640 1 -- 172.21.15.37:0/3772291491 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8544004d20 con 0x7f8554102000 2024-03-28T18:10:41.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.156+0000 7f854be00640 1 -- 172.21.15.37:0/3772291491 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f854400ee90 con 0x7f8554102000 2024-03-28T18:10:41.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.156+0000 7f854be00640 1 -- 172.21.15.37:0/3772291491 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8544010830 con 0x7f8554102000 2024-03-28T18:10:41.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.156+0000 7f8558a00640 1 -- 172.21.15.37:0/3772291491 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85541a19d0 con 0x7f8554102000 2024-03-28T18:10:41.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.156+0000 7f8558a00640 1 -- 172.21.15.37:0/3772291491 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85541a1ef0 con 0x7f8554102000 2024-03-28T18:10:41.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.159+0000 7f855c9fd740 1 Processor -- start 2024-03-28T18:10:41.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.159+0000 7f854be00640 1 -- 172.21.15.37:0/3772291491 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8544005020 con 0x7f8554102000 2024-03-28T18:10:41.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.159+0000 7f855c9fd740 1 -- start start 2024-03-28T18:10:41.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.160+0000 7f855c9fd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38bbd80 0x5645b38bc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.160+0000 7f855c9fd740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38bc770 0x5645b38c1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.160+0000 7f855c9fd740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38c18e0 0x5645b38c3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.160+0000 7f855c9fd740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645b3820c90 con 0x5645b38bc770 2024-03-28T18:10:41.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.160+0000 7f855c9fd740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645b37c9980 con 0x5645b38c18e0 2024-03-28T18:10:41.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.160+0000 7f855c9fd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645b381cd30 con 0x5645b38bbd80 2024-03-28T18:10:41.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.160+0000 7f8553e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38bbd80 0x5645b38bc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.160+0000 7f8552a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38c18e0 0x5645b38c3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.160+0000 7f8553e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38bbd80 0x5645b38bc1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55316/0 (socket says 172.21.15.37:55316) 2024-03-28T18:10:41.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.160+0000 7f8553e00640 1 -- 172.21.15.37:0/162401948 learned_addr learned my addr 172.21.15.37:0/162401948 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:41.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.160+0000 7f8553400640 1 --2- 172.21.15.37:0/162401948 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38bc770 0x5645b38c1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.160+0000 7f8553e00640 1 -- 172.21.15.37:0/162401948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38c18e0 msgr2=0x5645b38c3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:41.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.160+0000 7f8553e00640 1 --2- 172.21.15.37:0/162401948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38c18e0 0x5645b38c3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.160+0000 7f8553e00640 1 -- 172.21.15.37:0/162401948 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38bc770 msgr2=0x5645b38c1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:41.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.160+0000 7f8553e00640 1 --2- 172.21.15.37:0/162401948 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38bc770 0x5645b38c1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.160+0000 7f8553e00640 1 -- 172.21.15.37:0/162401948 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5645b37e1510 con 0x5645b38bbd80 2024-03-28T18:10:41.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.160+0000 7f8553400640 1 --2- 172.21.15.37:0/162401948 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38bc770 0x5645b38c1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:10:41.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.161+0000 7f8553e00640 1 --2- 172.21.15.37:0/162401948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38bbd80 0x5645b38bc1a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f854c009bd0 tx=0x7f854c00e6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=276164393870cf2b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:41.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.161+0000 7f853fe00640 1 -- 172.21.15.37:0/162401948 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c016070 con 0x5645b38bbd80 2024-03-28T18:10:41.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.161+0000 7f853fe00640 1 -- 172.21.15.37:0/162401948 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f854c011950 con 0x5645b38bbd80 2024-03-28T18:10:41.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.161+0000 7f853fe00640 1 -- 172.21.15.37:0/162401948 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c010910 con 0x5645b38bbd80 2024-03-28T18:10:41.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.161+0000 7f854be00640 1 --2- 172.21.15.37:0/3772291491 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f85380a2880 0x7f85380a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.161+0000 7f854be00640 1 -- 172.21.15.37:0/3772291491 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f85440cf140 con 0x7f8554102000 2024-03-28T18:10:41.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.161+0000 7f8553400640 1 --2- 172.21.15.37:0/3772291491 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f85380a2880 0x7f85380a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.161+0000 7f855c9fd740 1 -- 172.21.15.37:0/162401948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38bbd80 msgr2=0x5645b38bc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:41.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.161+0000 7f855c9fd740 1 --2- 172.21.15.37:0/162401948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38bbd80 0x5645b38bc1a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f854c009bd0 tx=0x7f854c00e6c0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.162+0000 7f855c9fd740 1 -- 172.21.15.37:0/162401948 shutdown_connections 2024-03-28T18:10:41.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.162+0000 7f855c9fd740 1 --2- 172.21.15.37:0/162401948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38c18e0 0x5645b38c3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.162+0000 7f855c9fd740 1 --2- 172.21.15.37:0/162401948 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38bc770 0x5645b38c1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.162+0000 7f855c9fd740 1 --2- 172.21.15.37:0/162401948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38bbd80 0x5645b38bc1a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.162+0000 7f855c9fd740 1 -- 172.21.15.37:0/162401948 >> 172.21.15.37:0/162401948 conn(0x5645b38b72f0 msgr2=0x5645b38b76f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:41.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.162+0000 7f855c9fd740 1 -- 172.21.15.37:0/162401948 shutdown_connections 2024-03-28T18:10:41.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.162+0000 7f855c9fd740 1 -- 172.21.15.37:0/162401948 wait complete. 2024-03-28T18:10:41.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.162+0000 7f8553400640 1 --2- 172.21.15.37:0/3772291491 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f85380a2880 0x7f85380a4d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f85440a3d00 tx=0x7f854400cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:41.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.162+0000 7f855c9fd740 1 -- start start 2024-03-28T18:10:41.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.163+0000 7f855c9fd740 1 Processor -- start 2024-03-28T18:10:41.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.163+0000 7f855c9fd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38bbd80 0x5645b38c6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.163+0000 7f855c9fd740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38bc770 0x5645b38c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.163+0000 7f855c9fd740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38c18e0 0x5645b38cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.163+0000 7f855c9fd740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645b38c4890 con 0x5645b38c18e0 2024-03-28T18:10:41.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.163+0000 7f855c9fd740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645b37dcdd0 con 0x5645b38bc770 2024-03-28T18:10:41.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.163+0000 7f855c9fd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645b38c4a10 con 0x5645b38bbd80 2024-03-28T18:10:41.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.163+0000 7f855c9fd740 10 client.0 _ll_register_callbacks cb 0x5645b3534ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:10:41.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.163+0000 7f8552a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38bc770 0x5645b38c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.163+0000 7f8552a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38bc770 0x5645b38c67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:37946/0 (socket says 172.21.15.37:37946) 2024-03-28T18:10:41.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.163+0000 7f8552a00640 1 -- 172.21.15.37:0/3099233665 learned_addr learned my addr 172.21.15.37:0/3099233665 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:41.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.163+0000 7f8553e00640 1 --2- 172.21.15.37:0/3099233665 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38bbd80 0x5645b38c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.163+0000 7f8553e00640 1 --2- 172.21.15.37:0/3099233665 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38c18e0 0x5645b38cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.164+0000 7f8552a00640 1 -- 172.21.15.37:0/3099233665 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38bbd80 msgr2=0x5645b38c6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:41.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.164+0000 7f8552a00640 1 --2- 172.21.15.37:0/3099233665 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38bbd80 0x5645b38c6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.164+0000 7f8552a00640 1 -- 172.21.15.37:0/3099233665 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38c18e0 msgr2=0x5645b38cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:41.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.164+0000 7f8552a00640 1 --2- 172.21.15.37:0/3099233665 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38c18e0 0x5645b38cd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.164+0000 7f8552a00640 1 -- 172.21.15.37:0/3099233665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5645b35352d0 con 0x5645b38bc770 2024-03-28T18:10:41.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.164+0000 7f8553e00640 1 --2- 172.21.15.37:0/3099233665 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38c18e0 0x5645b38cd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:41.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.164+0000 7f8552a00640 1 --2- 172.21.15.37:0/3099233665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38bc770 0x5645b38c67b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8540005ab0 tx=0x7f8540004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc77d7eb4e24bb16 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:41.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.164+0000 7f853cc00640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:10:41.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.164+0000 7f853cc00640 1 -- 172.21.15.37:0/3099233665 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8540005d50 con 0x5645b38bc770 2024-03-28T18:10:41.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.164+0000 7f853cc00640 1 -- 172.21.15.37:0/3099233665 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f85400048c0 con 0x5645b38bc770 2024-03-28T18:10:41.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.164+0000 7f855c9fd740 1 -- 172.21.15.37:0/3099233665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f854c011950 con 0x5645b38bc770 2024-03-28T18:10:41.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.165+0000 7f853cc00640 1 -- 172.21.15.37:0/3099233665 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8540004ac0 con 0x5645b38bc770 2024-03-28T18:10:41.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.165+0000 7f855c9fd740 10 client.25157 fetch_fsmap learned FSMap version 22 2024-03-28T18:10:41.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.165+0000 7f855c9fd740 1 -- 172.21.15.37:0/3099233665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5645b35ff6b0 con 0x5645b38bc770 2024-03-28T18:10:41.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.165+0000 7f853cc00640 1 -- 172.21.15.37:0/3099233665 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f854c011950 con 0x5645b38bc770 2024-03-28T18:10:41.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.165+0000 7f853cc00640 1 -- 172.21.15.37:0/3099233665 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8540046070 con 0x5645b38bc770 2024-03-28T18:10:41.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.165+0000 7f855c9fd740 10 client.25157 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T18:10:41.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.165+0000 7f855c9fd740 10 client.25157 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T18:10:41.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.165+0000 7f855c9fd740 20 client.25157 populate_metadata read hostname 'smithi037' 2024-03-28T18:10:41.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.165+0000 7f855c9fd740 1 --2- 172.21.15.37:0/3099233665 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x5645b38d29a0 0x5645b38d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.165+0000 7f855c9fd740 4 client.25157 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:10:41.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.165+0000 7f855c9fd740 1 -- 172.21.15.37:0/3099233665 --> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5645b37c2520 con 0x5645b38d29a0 2024-03-28T18:10:41.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.166+0000 7f8553e00640 1 --2- 172.21.15.37:0/3099233665 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x5645b38d29a0 0x5645b38d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.166+0000 7f8553e00640 1 --2- 172.21.15.37:0/3099233665 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x5645b38d29a0 0x5645b38d1a40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2d6b1b1b6fd59a56 server_cookie=fdc2861f64e0ba09 in_seq=0 out_seq=0 2024-03-28T18:10:41.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.166+0000 7f853cc00640 10 client.25157 ms_handle_connect on v2:172.21.15.59:6838/2091629078 2024-03-28T18:10:41.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.167+0000 7f853cc00640 1 -- 172.21.15.37:0/3099233665 <== mds.1 v2:172.21.15.59:6838/2091629078 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5645b37c2520 con 0x5645b38d29a0 2024-03-28T18:10:41.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.167+0000 7f853cc00640 10 client.25157 handle_command_reply: tid=0 2024-03-28T18:10:41.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.167+0000 7f855c9fd740 1 client.25157 shutdown 2024-03-28T18:10:41.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.167+0000 7f855c9fd740 1 -- 172.21.15.37:0/3099233665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38bc770 msgr2=0x5645b38c67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:41.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.167+0000 7f855c9fd740 1 --2- 172.21.15.37:0/3099233665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38bc770 0x5645b38c67b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8540005ab0 tx=0x7f8540004270 comp rx=0 tx=0).stop 2024-03-28T18:10:41.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.167+0000 7f855c9fd740 1 -- 172.21.15.37:0/3099233665 shutdown_connections 2024-03-28T18:10:41.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.167+0000 7f855c9fd740 1 --2- 172.21.15.37:0/3099233665 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x5645b38d29a0 0x5645b38d1a40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.167+0000 7f855c9fd740 1 --2- 172.21.15.37:0/3099233665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38bc770 0x5645b38c67b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.167+0000 7f855c9fd740 1 --2- 172.21.15.37:0/3099233665 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38c18e0 0x5645b38cd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.167+0000 7f853cc00640 0 client.25157 ms_handle_reset on v2:172.21.15.59:6838/2091629078 2024-03-28T18:10:41.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.167+0000 7f855c9fd740 1 --2- 172.21.15.37:0/3099233665 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38bbd80 0x5645b38c6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.167+0000 7f855c9fd740 1 -- 172.21.15.37:0/3099233665 >> 172.21.15.37:0/3099233665 conn(0x5645b38b72f0 msgr2=0x5645b38b64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:41.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.167+0000 7f855c9fd740 1 -- 172.21.15.37:0/3099233665 shutdown_connections 2024-03-28T18:10:41.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.167+0000 7f855c9fd740 1 -- 172.21.15.37:0/3099233665 wait complete. 2024-03-28T18:10:41.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.167+0000 7f855c9fd740 20 client.25157 trim_cache size 0 max 16384 2024-03-28T18:10:41.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.187+0000 7f855c9fd740 1 Processor -- start 2024-03-28T18:10:41.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.187+0000 7f855c9fd740 1 -- start start 2024-03-28T18:10:41.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.188+0000 7f855c9fd740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38bbd80 0x5645b38d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.188+0000 7f855c9fd740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38bc770 0x5645b38b76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.188+0000 7f855c9fd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38c18e0 0x5645b38b5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.188+0000 7f855c9fd740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645b38d21c0 con 0x5645b38bbd80 2024-03-28T18:10:41.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.188+0000 7f855c9fd740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645b38d2820 con 0x5645b38bc770 2024-03-28T18:10:41.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.188+0000 7f855c9fd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645b38d2e80 con 0x5645b38c18e0 2024-03-28T18:10:41.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.188+0000 7f8552a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38bc770 0x5645b38b76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.188+0000 7f8553e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38bbd80 0x5645b38d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.188+0000 7f8552a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38bc770 0x5645b38b76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:37950/0 (socket says 172.21.15.37:37950) 2024-03-28T18:10:41.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.188+0000 7f8552a00640 1 -- 172.21.15.37:0/905259198 learned_addr learned my addr 172.21.15.37:0/905259198 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:41.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.188+0000 7f8553e00640 1 --2- 172.21.15.37:0/905259198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38c18e0 0x5645b38b5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.188+0000 7f8552a00640 1 -- 172.21.15.37:0/905259198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38c18e0 msgr2=0x5645b38b5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:41.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.188+0000 7f8552a00640 1 --2- 172.21.15.37:0/905259198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38c18e0 0x5645b38b5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.188+0000 7f8552a00640 1 -- 172.21.15.37:0/905259198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38bbd80 msgr2=0x5645b38d1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:41.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.188+0000 7f8552a00640 1 --2- 172.21.15.37:0/905259198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38bbd80 0x5645b38d1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.188+0000 7f8552a00640 1 -- 172.21.15.37:0/905259198 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5645b38bd000 con 0x5645b38bc770 2024-03-28T18:10:41.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.188+0000 7f8553e00640 1 --2- 172.21.15.37:0/905259198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38c18e0 0x5645b38b5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:41.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.189+0000 7f8552a00640 1 --2- 172.21.15.37:0/905259198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38bc770 0x5645b38b76d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f854002c190 tx=0x7f85400098b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93aeed50b4036f72 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:41.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.190+0000 7f853f400640 1 -- 172.21.15.37:0/905259198 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8540057070 con 0x5645b38bc770 2024-03-28T18:10:41.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.190+0000 7f853f400640 1 -- 172.21.15.37:0/905259198 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f854004e030 con 0x5645b38bc770 2024-03-28T18:10:41.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.190+0000 7f853f400640 1 -- 172.21.15.37:0/905259198 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8540000e60 con 0x5645b38bc770 2024-03-28T18:10:41.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.191+0000 7f855c9fd740 1 -- 172.21.15.37:0/905259198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38bc770 msgr2=0x5645b38b76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:41.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.191+0000 7f855c9fd740 1 --2- 172.21.15.37:0/905259198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38bc770 0x5645b38b76d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f854002c190 tx=0x7f85400098b0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.191+0000 7f855c9fd740 1 -- 172.21.15.37:0/905259198 shutdown_connections 2024-03-28T18:10:41.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.191+0000 7f855c9fd740 1 --2- 172.21.15.37:0/905259198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38bc770 0x5645b38b76d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.191+0000 7f855c9fd740 1 --2- 172.21.15.37:0/905259198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38c18e0 0x5645b38b5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.191+0000 7f855c9fd740 1 --2- 172.21.15.37:0/905259198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38bbd80 0x5645b38d1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.191+0000 7f855c9fd740 1 -- 172.21.15.37:0/905259198 >> 172.21.15.37:0/905259198 conn(0x5645b38b72f0 msgr2=0x5645b38c4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:41.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.191+0000 7f855c9fd740 1 -- 172.21.15.37:0/905259198 shutdown_connections 2024-03-28T18:10:41.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.191+0000 7f855c9fd740 1 -- 172.21.15.37:0/905259198 wait complete. 2024-03-28T18:10:41.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.191+0000 7f855c9fd740 1 -- start start 2024-03-28T18:10:41.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.192+0000 7f855c9fd740 1 Processor -- start 2024-03-28T18:10:41.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.192+0000 7f855c9fd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38bbd80 0x5645b38cca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.192+0000 7f855c9fd740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38bc770 0x5645b38ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.192+0000 7f855c9fd740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38c18e0 0x5645b38c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.192+0000 7f855c9fd740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645b38d3b40 con 0x5645b38bc770 2024-03-28T18:10:41.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.192+0000 7f855c9fd740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645b38d34e0 con 0x5645b38c18e0 2024-03-28T18:10:41.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.192+0000 7f855c9fd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645b38d41a0 con 0x5645b38bbd80 2024-03-28T18:10:41.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.192+0000 7f855c9fd740 10 client.0 _ll_register_callbacks cb 0x7f8540009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:10:41.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.192+0000 7f8553e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38bbd80 0x5645b38cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.192+0000 7f8553e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38bbd80 0x5645b38cca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55346/0 (socket says 172.21.15.37:55346) 2024-03-28T18:10:41.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.192+0000 7f8553e00640 1 -- 172.21.15.37:0/967343201 learned_addr learned my addr 172.21.15.37:0/967343201 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:41.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.192+0000 7f8553e00640 1 --2- 172.21.15.37:0/967343201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38c18e0 0x5645b38c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.193+0000 7f8552a00640 1 --2- 172.21.15.37:0/967343201 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38bc770 0x5645b38ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.193+0000 7f8553e00640 1 -- 172.21.15.37:0/967343201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38c18e0 msgr2=0x5645b38c99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:41.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.193+0000 7f8553e00640 1 --2- 172.21.15.37:0/967343201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38c18e0 0x5645b38c99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.193+0000 7f8553e00640 1 -- 172.21.15.37:0/967343201 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38bc770 msgr2=0x5645b38ccf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:41.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.193+0000 7f8553e00640 1 --2- 172.21.15.37:0/967343201 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38bc770 0x5645b38ccf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.193+0000 7f8553e00640 1 -- 172.21.15.37:0/967343201 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5645b38ca1f0 con 0x5645b38bbd80 2024-03-28T18:10:41.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.193+0000 7f8553e00640 1 --2- 172.21.15.37:0/967343201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38bbd80 0x5645b38cca10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f854c00e700 tx=0x7f854c009f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc5749df5c0c2b9c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:41.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.193+0000 7f853cc00640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:10:41.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.193+0000 7f853cc00640 1 -- 172.21.15.37:0/967343201 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c016040 con 0x5645b38bbd80 2024-03-28T18:10:41.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.193+0000 7f853cc00640 1 -- 172.21.15.37:0/967343201 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f854c02b4b0 con 0x5645b38bbd80 2024-03-28T18:10:41.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.193+0000 7f855c9fd740 1 -- 172.21.15.37:0/967343201 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f854004e030 con 0x5645b38bbd80 2024-03-28T18:10:41.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.193+0000 7f853cc00640 1 -- 172.21.15.37:0/967343201 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f854c03e050 con 0x5645b38bbd80 2024-03-28T18:10:41.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.194+0000 7f855c9fd740 10 client.25189 fetch_fsmap learned FSMap version 22 2024-03-28T18:10:41.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.194+0000 7f855c9fd740 1 -- 172.21.15.37:0/967343201 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5645b3937000 con 0x5645b38bbd80 2024-03-28T18:10:41.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.194+0000 7f853cc00640 1 -- 172.21.15.37:0/967343201 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f854004e030 con 0x5645b38bbd80 2024-03-28T18:10:41.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.194+0000 7f853cc00640 1 -- 172.21.15.37:0/967343201 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f854c00f040 con 0x5645b38bbd80 2024-03-28T18:10:41.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.194+0000 7f855c9fd740 10 client.25189 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T18:10:41.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.194+0000 7f855c9fd740 10 client.25189 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T18:10:41.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.194+0000 7f855c9fd740 20 client.25189 populate_metadata read hostname 'smithi037' 2024-03-28T18:10:41.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.194+0000 7f855c9fd740 1 --2- 172.21.15.37:0/967343201 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x5645b393b500 0x5645b3937340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.194+0000 7f855c9fd740 4 client.25189 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:10:41.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.194+0000 7f855c9fd740 1 -- 172.21.15.37:0/967343201 --> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5645b382e1d0 con 0x5645b393b500 2024-03-28T18:10:41.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.194+0000 7f8552a00640 1 --2- 172.21.15.37:0/967343201 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x5645b393b500 0x5645b3937340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.195+0000 7f8552a00640 1 --2- 172.21.15.37:0/967343201 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x5645b393b500 0x5645b3937340 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1bfbc481ec58842 server_cookie=103598c11ca93d11 in_seq=0 out_seq=0 2024-03-28T18:10:41.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.195+0000 7f853cc00640 10 client.25189 ms_handle_connect on v2:172.21.15.59:6838/2091629078 2024-03-28T18:10:41.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.195+0000 7f853cc00640 1 -- 172.21.15.37:0/967343201 <== mds.1 v2:172.21.15.59:6838/2091629078 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5645b382e1d0 con 0x5645b393b500 2024-03-28T18:10:41.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.195+0000 7f853cc00640 10 client.25189 handle_command_reply: tid=0 2024-03-28T18:10:41.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.195+0000 7f855c9fd740 1 client.25189 shutdown 2024-03-28T18:10:41.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.195+0000 7f855c9fd740 1 -- 172.21.15.37:0/967343201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38bbd80 msgr2=0x5645b38cca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:41.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.195+0000 7f855c9fd740 1 --2- 172.21.15.37:0/967343201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38bbd80 0x5645b38cca10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f854c00e700 tx=0x7f854c009f60 comp rx=0 tx=0).stop 2024-03-28T18:10:41.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.195+0000 7f855c9fd740 1 -- 172.21.15.37:0/967343201 shutdown_connections 2024-03-28T18:10:41.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.195+0000 7f855c9fd740 1 --2- 172.21.15.37:0/967343201 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x5645b393b500 0x5645b3937340 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.195+0000 7f855c9fd740 1 --2- 172.21.15.37:0/967343201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645b38c18e0 0x5645b38c99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.195+0000 7f855c9fd740 1 --2- 172.21.15.37:0/967343201 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5645b38bc770 0x5645b38ccf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.195+0000 7f855c9fd740 1 --2- 172.21.15.37:0/967343201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5645b38bbd80 0x5645b38cca10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.195+0000 7f855c9fd740 1 -- 172.21.15.37:0/967343201 >> 172.21.15.37:0/967343201 conn(0x5645b38b72f0 msgr2=0x5645b38d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:41.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.195+0000 7f853cc00640 0 client.25189 ms_handle_reset on v2:172.21.15.59:6838/2091629078 2024-03-28T18:10:41.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.195+0000 7f855c9fd740 1 -- 172.21.15.37:0/967343201 shutdown_connections 2024-03-28T18:10:41.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.196+0000 7f855c9fd740 1 -- 172.21.15.37:0/967343201 wait complete. 2024-03-28T18:10:41.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.196+0000 7f855c9fd740 20 client.25189 trim_cache size 0 max 16384 2024-03-28T18:10:41.200 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:10:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: "cluster_fsid": "b5ad257c-ed2c-11ee-95d0-87774f69a715", 2024-03-28T18:10:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: "whoami": 1, 2024-03-28T18:10:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: "id": 24569, 2024-03-28T18:10:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: "want_state": "up:active", 2024-03-28T18:10:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "up:active", 2024-03-28T18:10:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: "fs_name": "cephfs", 2024-03-28T18:10:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: "rank_uptime": 137.81619733900001, 2024-03-28T18:10:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: "mdsmap_epoch": 22, 2024-03-28T18:10:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap_epoch": 87, 2024-03-28T18:10:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap_epoch_barrier": 85, 2024-03-28T18:10:41.200 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 147.55448227799999 2024-03-28T18:10:41.200 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:10:41.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.196+0000 7f8558a00640 1 -- 172.21.15.37:0/3772291491 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f85380a2880 msgr2=0x7f85380a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:41.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.196+0000 7f8558a00640 1 --2- 172.21.15.37:0/3772291491 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f85380a2880 0x7f85380a4d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f85440a3d00 tx=0x7f854400cf90 comp rx=0 tx=0).stop 2024-03-28T18:10:41.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.196+0000 7f8558a00640 1 -- 172.21.15.37:0/3772291491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8554102000 msgr2=0x7f8554101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:41.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.196+0000 7f8558a00640 1 --2- 172.21.15.37:0/3772291491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8554102000 0x7f8554101970 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f854400eb90 tx=0x7f854400c6a0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.196+0000 7f8558a00640 1 -- 172.21.15.37:0/3772291491 shutdown_connections 2024-03-28T18:10:41.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.196+0000 7f8558a00640 1 --2- 172.21.15.37:0/3772291491 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f85380a2880 0x7f85380a4d40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.196+0000 7f8558a00640 1 --2- 172.21.15.37:0/3772291491 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8554107260 0x7f85541004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.196+0000 7f8558a00640 1 --2- 172.21.15.37:0/3772291491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8554104930 0x7f85540fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.196+0000 7f8558a00640 1 --2- 172.21.15.37:0/3772291491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8554102000 0x7f8554101970 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.196+0000 7f8558a00640 1 -- 172.21.15.37:0/3772291491 >> 172.21.15.37:0/3772291491 conn(0x7f85540fba10 msgr2=0x7f8554102940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:41.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.196+0000 7f8558a00640 1 -- 172.21.15.37:0/3772291491 shutdown_connections 2024-03-28T18:10:41.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.197+0000 7f8558a00640 1 -- 172.21.15.37:0/3772291491 wait complete. 2024-03-28T18:10:41.210 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:41 smithi037 ceph-mon[31460]: pgmap v390: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:10:41.210 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:41 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2697011757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:41.210 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:41 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3308870547' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:41.214 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-28T18:10:41.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.413+0000 7ffaf3200640 1 -- 172.21.15.37:0/923524955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffaf4107260 msgr2=0x7ffaf41096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:41.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.413+0000 7ffaf3200640 1 --2- 172.21.15.37:0/923524955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffaf4107260 0x7ffaf41096b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ffaec009be0 tx=0x7ffaec02bef0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.413+0000 7ffaf3200640 1 -- 172.21.15.37:0/923524955 shutdown_connections 2024-03-28T18:10:41.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.413+0000 7ffaf3200640 1 --2- 172.21.15.37:0/923524955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffaf4107260 0x7ffaf41096b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.413+0000 7ffaf3200640 1 --2- 172.21.15.37:0/923524955 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffaf4104930 0x7ffaf4106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.413+0000 7ffaf3200640 1 --2- 172.21.15.37:0/923524955 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaf4102000 0x7ffaf41043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.413+0000 7ffaf3200640 1 -- 172.21.15.37:0/923524955 >> 172.21.15.37:0/923524955 conn(0x7ffaf40fba10 msgr2=0x7ffaf40fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:41.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.413+0000 7ffaf3200640 1 -- 172.21.15.37:0/923524955 shutdown_connections 2024-03-28T18:10:41.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.413+0000 7ffaf3200640 1 -- 172.21.15.37:0/923524955 wait complete. 2024-03-28T18:10:41.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.414+0000 7ffaf3200640 1 Processor -- start 2024-03-28T18:10:41.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.414+0000 7ffaf3200640 1 -- start start 2024-03-28T18:10:41.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.415+0000 7ffaf3200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffaf4102000 0x7ffaf4101970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.415+0000 7ffaf3200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffaf4104930 0x7ffaf40fffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.415+0000 7ffaf3200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaf4107260 0x7ffaf41004f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.415+0000 7ffaf3200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf410d260 con 0x7ffaf4102000 2024-03-28T18:10:41.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.415+0000 7ffaf3200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf410d0e0 con 0x7ffaf4107260 2024-03-28T18:10:41.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.415+0000 7ffaf3200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf410d3e0 con 0x7ffaf4104930 2024-03-28T18:10:41.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.415+0000 7ffaebe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffaf4104930 0x7ffaf40fffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.415+0000 7ffaf1e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffaf4102000 0x7ffaf4101970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.415+0000 7ffaebe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffaf4104930 0x7ffaf40fffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55376/0 (socket says 172.21.15.37:55376) 2024-03-28T18:10:41.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.415+0000 7ffaf1e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffaf4102000 0x7ffaf4101970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49280/0 (socket says 172.21.15.37:49280) 2024-03-28T18:10:41.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.415+0000 7ffaebe00640 1 -- 172.21.15.37:0/1908450050 learned_addr learned my addr 172.21.15.37:0/1908450050 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:41.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.415+0000 7ffaf2800640 1 --2- 172.21.15.37:0/1908450050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaf4107260 0x7ffaf41004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.415+0000 7ffaebe00640 1 -- 172.21.15.37:0/1908450050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaf4107260 msgr2=0x7ffaf41004f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:41.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.415+0000 7ffaebe00640 1 --2- 172.21.15.37:0/1908450050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaf4107260 0x7ffaf41004f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.415+0000 7ffaebe00640 1 -- 172.21.15.37:0/1908450050 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffaf4102000 msgr2=0x7ffaf4101970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:41.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.415+0000 7ffaebe00640 1 --2- 172.21.15.37:0/1908450050 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffaf4102000 0x7ffaf4101970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.415+0000 7ffaebe00640 1 -- 172.21.15.37:0/1908450050 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf4100d80 con 0x7ffaf4104930 2024-03-28T18:10:41.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.416+0000 7ffaebe00640 1 --2- 172.21.15.37:0/1908450050 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffaf4104930 0x7ffaf40fffb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ffadc00c970 tx=0x7ffadc00ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:41.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.416+0000 7ffaeaa00640 1 -- 172.21.15.37:0/1908450050 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc014070 con 0x7ffaf4104930 2024-03-28T18:10:41.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.416+0000 7ffaeaa00640 1 -- 172.21.15.37:0/1908450050 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffadc004440 con 0x7ffaf4104930 2024-03-28T18:10:41.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.416+0000 7ffaeaa00640 1 -- 172.21.15.37:0/1908450050 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc00f450 con 0x7ffaf4104930 2024-03-28T18:10:41.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.416+0000 7ffaf3200640 1 -- 172.21.15.37:0/1908450050 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf41a1880 con 0x7ffaf4104930 2024-03-28T18:10:41.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.416+0000 7ffaf3200640 1 -- 172.21.15.37:0/1908450050 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf41a1d80 con 0x7ffaf4104930 2024-03-28T18:10:41.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.419+0000 7ffaeaa00640 1 -- 172.21.15.37:0/1908450050 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ffadc017070 con 0x7ffaf4104930 2024-03-28T18:10:41.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.419+0000 7ffafbe64740 1 Processor -- start 2024-03-28T18:10:41.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.420+0000 7ffafbe64740 1 -- start start 2024-03-28T18:10:41.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.420+0000 7ffafbe64740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232a61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.420+0000 7ffafbe64740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232a6770 0x556d232ab270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.420+0000 7ffafbe64740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232ab8e0 0x556d232add70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.420+0000 7ffafbe64740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d2320ac90 con 0x556d232a6770 2024-03-28T18:10:41.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.420+0000 7ffafbe64740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d231b3980 con 0x556d232a5d80 2024-03-28T18:10:41.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.420+0000 7ffafbe64740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d23206d30 con 0x556d232ab8e0 2024-03-28T18:10:41.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.420+0000 7ffaf2800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232a61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.420+0000 7ffaf2800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232a61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:37984/0 (socket says 172.21.15.37:37984) 2024-03-28T18:10:41.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.420+0000 7ffaf2800640 1 -- 172.21.15.37:0/1919308080 learned_addr learned my addr 172.21.15.37:0/1919308080 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:41.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.420+0000 7ffaebe00640 1 --2- 172.21.15.37:0/1919308080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232ab8e0 0x556d232add70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.420+0000 7ffaf1e00640 1 --2- 172.21.15.37:0/1919308080 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232a6770 0x556d232ab270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.421+0000 7ffaf2800640 1 -- 172.21.15.37:0/1919308080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232ab8e0 msgr2=0x556d232add70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:41.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.421+0000 7ffaf2800640 1 --2- 172.21.15.37:0/1919308080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232ab8e0 0x556d232add70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.421+0000 7ffaf2800640 1 -- 172.21.15.37:0/1919308080 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232a6770 msgr2=0x556d232ab270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:41.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.421+0000 7ffaf2800640 1 --2- 172.21.15.37:0/1919308080 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232a6770 0x556d232ab270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.421+0000 7ffaf2800640 1 -- 172.21.15.37:0/1919308080 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d231cb510 con 0x556d232a5d80 2024-03-28T18:10:41.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.421+0000 7ffaf1e00640 1 --2- 172.21.15.37:0/1919308080 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232a6770 0x556d232ab270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:41.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.421+0000 7ffaeaa00640 1 --2- 172.21.15.37:0/1908450050 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ffad80a2880 0x7ffad80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.421+0000 7ffaf2800640 1 --2- 172.21.15.37:0/1919308080 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232a61a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffaec02c500 tx=0x7ffaec033b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a89a613abc63c48b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:41.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.421+0000 7ffaeaa00640 1 -- 172.21.15.37:0/1908450050 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ffadc0ce840 con 0x7ffaf4104930 2024-03-28T18:10:41.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.421+0000 7ffaf1e00640 1 --2- 172.21.15.37:0/1908450050 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ffad80a2880 0x7ffad80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.421+0000 7ffae3400640 1 -- 172.21.15.37:0/1919308080 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec043030 con 0x556d232a5d80 2024-03-28T18:10:41.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.421+0000 7ffae3400640 1 -- 172.21.15.37:0/1919308080 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffaec033cb0 con 0x556d232a5d80 2024-03-28T18:10:41.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.421+0000 7ffae3400640 1 -- 172.21.15.37:0/1919308080 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec02ca50 con 0x556d232a5d80 2024-03-28T18:10:41.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.422+0000 7ffafbe64740 1 -- 172.21.15.37:0/1919308080 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 msgr2=0x556d232a61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:41.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.422+0000 7ffafbe64740 1 --2- 172.21.15.37:0/1919308080 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232a61a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffaec02c500 tx=0x7ffaec033b40 comp rx=0 tx=0).stop 2024-03-28T18:10:41.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.422+0000 7ffaf1e00640 1 --2- 172.21.15.37:0/1908450050 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ffad80a2880 0x7ffad80a4d40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ffae40096c0 tx=0x7ffae4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:41.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.422+0000 7ffafbe64740 1 -- 172.21.15.37:0/1919308080 shutdown_connections 2024-03-28T18:10:41.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.422+0000 7ffafbe64740 1 --2- 172.21.15.37:0/1919308080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232ab8e0 0x556d232add70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.422+0000 7ffafbe64740 1 --2- 172.21.15.37:0/1919308080 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232a6770 0x556d232ab270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.422+0000 7ffafbe64740 1 --2- 172.21.15.37:0/1919308080 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232a61a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.422+0000 7ffafbe64740 1 -- 172.21.15.37:0/1919308080 >> 172.21.15.37:0/1919308080 conn(0x556d232a12f0 msgr2=0x556d232a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:41.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.422+0000 7ffafbe64740 1 -- 172.21.15.37:0/1919308080 shutdown_connections 2024-03-28T18:10:41.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.422+0000 7ffafbe64740 1 -- 172.21.15.37:0/1919308080 wait complete. 2024-03-28T18:10:41.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.423+0000 7ffafbe64740 1 -- start start 2024-03-28T18:10:41.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.423+0000 7ffafbe64740 1 Processor -- start 2024-03-28T18:10:41.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.423+0000 7ffafbe64740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232b0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.423+0000 7ffafbe64740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232a6770 0x556d232b07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.423+0000 7ffafbe64740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232ab8e0 0x556d232b7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.423+0000 7ffafbe64740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d232ae830 con 0x556d232a6770 2024-03-28T18:10:41.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.423+0000 7ffafbe64740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d231c6dd0 con 0x556d232a5d80 2024-03-28T18:10:41.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.423+0000 7ffafbe64740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d232ae9b0 con 0x556d232ab8e0 2024-03-28T18:10:41.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.423+0000 7ffafbe64740 10 client.0 _ll_register_callbacks cb 0x556d22f1eac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:10:41.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.423+0000 7ffaf1e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.424+0000 7ffaf1e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232b0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:38000/0 (socket says 172.21.15.37:38000) 2024-03-28T18:10:41.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.424+0000 7ffaf1e00640 1 -- 172.21.15.37:0/4108382417 learned_addr learned my addr 172.21.15.37:0/4108382417 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:41.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.424+0000 7ffaf2800640 1 --2- 172.21.15.37:0/4108382417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232ab8e0 0x556d232b7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.424+0000 7ffaebe00640 1 --2- 172.21.15.37:0/4108382417 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232a6770 0x556d232b07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.424+0000 7ffaf2800640 1 -- 172.21.15.37:0/4108382417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 msgr2=0x556d232b0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:41.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.424+0000 7ffaf2800640 1 --2- 172.21.15.37:0/4108382417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232b0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.424+0000 7ffaf2800640 1 -- 172.21.15.37:0/4108382417 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232a6770 msgr2=0x556d232b07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:41.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.424+0000 7ffaf2800640 1 --2- 172.21.15.37:0/4108382417 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232a6770 0x556d232b07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.424+0000 7ffaf2800640 1 -- 172.21.15.37:0/4108382417 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556d22f1f2d0 con 0x556d232ab8e0 2024-03-28T18:10:41.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.424+0000 7ffaf2800640 1 --2- 172.21.15.37:0/4108382417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232ab8e0 0x556d232b7810 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ffaec02c430 tx=0x7ffaec033e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24f599345c648e3d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:41.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.424+0000 7ffad7e00640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:10:41.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.424+0000 7ffad7e00640 1 -- 172.21.15.37:0/4108382417 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec04c070 con 0x556d232ab8e0 2024-03-28T18:10:41.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.424+0000 7ffad7e00640 1 -- 172.21.15.37:0/4108382417 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ffaec033780 con 0x556d232ab8e0 2024-03-28T18:10:41.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.425+0000 7ffafbe64740 1 -- 172.21.15.37:0/4108382417 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffaec033cb0 con 0x556d232ab8e0 2024-03-28T18:10:41.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.425+0000 7ffad7e00640 1 -- 172.21.15.37:0/4108382417 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffaec057050 con 0x556d232ab8e0 2024-03-28T18:10:41.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.425+0000 7ffafbe64740 10 client.25201 fetch_fsmap learned FSMap version 22 2024-03-28T18:10:41.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.425+0000 7ffafbe64740 1 -- 172.21.15.37:0/4108382417 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x556d22fe96b0 con 0x556d232ab8e0 2024-03-28T18:10:41.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.425+0000 7ffad7e00640 1 -- 172.21.15.37:0/4108382417 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffaec033cb0 con 0x556d232ab8e0 2024-03-28T18:10:41.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.425+0000 7ffad7e00640 1 -- 172.21.15.37:0/4108382417 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ffaec004680 con 0x556d232ab8e0 2024-03-28T18:10:41.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.425+0000 7ffafbe64740 10 client.25201 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T18:10:41.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.425+0000 7ffafbe64740 10 client.25201 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T18:10:41.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.425+0000 7ffafbe64740 20 client.25201 populate_metadata read hostname 'smithi037' 2024-03-28T18:10:41.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.425+0000 7ffafbe64740 1 --2- 172.21.15.37:0/4108382417 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556d232bc420 0x556d232c08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.425+0000 7ffafbe64740 4 client.25201 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:10:41.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.425+0000 7ffafbe64740 1 -- 172.21.15.37:0/4108382417 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556d231ac520 con 0x556d232bc420 2024-03-28T18:10:41.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.426+0000 7ffaf1e00640 1 --2- 172.21.15.37:0/4108382417 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556d232bc420 0x556d232c08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.426+0000 7ffaf1e00640 1 --2- 172.21.15.37:0/4108382417 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556d232bc420 0x556d232c08a0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57409b7f48409174 server_cookie=e26010fb9879558a in_seq=0 out_seq=0 2024-03-28T18:10:41.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.426+0000 7ffad7e00640 10 client.25201 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:10:41.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.427+0000 7ffad7e00640 1 -- 172.21.15.37:0/4108382417 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556d231ac520 con 0x556d232bc420 2024-03-28T18:10:41.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.427+0000 7ffad7e00640 10 client.25201 handle_command_reply: tid=0 2024-03-28T18:10:41.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.427+0000 7ffafbe64740 1 client.25201 shutdown 2024-03-28T18:10:41.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.427+0000 7ffafbe64740 1 -- 172.21.15.37:0/4108382417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232ab8e0 msgr2=0x556d232b7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:41.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.427+0000 7ffafbe64740 1 --2- 172.21.15.37:0/4108382417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232ab8e0 0x556d232b7810 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ffaec02c430 tx=0x7ffaec033e60 comp rx=0 tx=0).stop 2024-03-28T18:10:41.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.427+0000 7ffafbe64740 1 -- 172.21.15.37:0/4108382417 shutdown_connections 2024-03-28T18:10:41.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.427+0000 7ffafbe64740 1 --2- 172.21.15.37:0/4108382417 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556d232bc420 0x556d232c08a0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.427+0000 7ffafbe64740 1 --2- 172.21.15.37:0/4108382417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232ab8e0 0x556d232b7810 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.427+0000 7ffafbe64740 1 --2- 172.21.15.37:0/4108382417 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232a6770 0x556d232b07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.427+0000 7ffafbe64740 1 --2- 172.21.15.37:0/4108382417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232b0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.427+0000 7ffafbe64740 1 -- 172.21.15.37:0/4108382417 >> 172.21.15.37:0/4108382417 conn(0x556d232a12f0 msgr2=0x556d232a04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:41.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.427+0000 7ffad7e00640 0 client.25201 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:10:41.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.428+0000 7ffafbe64740 1 -- 172.21.15.37:0/4108382417 shutdown_connections 2024-03-28T18:10:41.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.428+0000 7ffafbe64740 1 -- 172.21.15.37:0/4108382417 wait complete. 2024-03-28T18:10:41.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.428+0000 7ffafbe64740 20 client.25201 trim_cache size 0 max 16384 2024-03-28T18:10:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.446+0000 7ffafbe64740 1 Processor -- start 2024-03-28T18:10:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.446+0000 7ffafbe64740 1 -- start start 2024-03-28T18:10:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.446+0000 7ffafbe64740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232adcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.446+0000 7ffafbe64740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232a6770 0x556d232b7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.446+0000 7ffafbe64740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232ab8e0 0x556d232b0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.447+0000 7ffafbe64740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d232bbc40 con 0x556d232ab8e0 2024-03-28T18:10:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.447+0000 7ffafbe64740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d232bc2a0 con 0x556d232a5d80 2024-03-28T18:10:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.447+0000 7ffafbe64740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d232bc900 con 0x556d232a6770 2024-03-28T18:10:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.447+0000 7ffaf1e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232adcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.447+0000 7ffaebe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232a6770 0x556d232b7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.447+0000 7ffaf1e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232adcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:38006/0 (socket says 172.21.15.37:38006) 2024-03-28T18:10:41.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.447+0000 7ffaf2800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232ab8e0 0x556d232b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.447+0000 7ffaf1e00640 1 -- 172.21.15.37:0/3978968908 learned_addr learned my addr 172.21.15.37:0/3978968908 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:41.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.447+0000 7ffaebe00640 1 -- 172.21.15.37:0/3978968908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 msgr2=0x556d232adcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:41.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.447+0000 7ffaebe00640 1 --2- 172.21.15.37:0/3978968908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232adcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.447+0000 7ffaebe00640 1 -- 172.21.15.37:0/3978968908 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232ab8e0 msgr2=0x556d232b0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:41.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.447+0000 7ffaebe00640 1 --2- 172.21.15.37:0/3978968908 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232ab8e0 0x556d232b0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.447+0000 7ffaebe00640 1 -- 172.21.15.37:0/3978968908 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d232a2060 con 0x556d232a6770 2024-03-28T18:10:41.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.447+0000 7ffaf1e00640 1 --2- 172.21.15.37:0/3978968908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232adcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:41.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.447+0000 7ffaf2800640 1 --2- 172.21.15.37:0/3978968908 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232ab8e0 0x556d232b0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:41.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.447+0000 7ffaebe00640 1 --2- 172.21.15.37:0/3978968908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232a6770 0x556d232b7730 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ffadc007590 tx=0x7ffadc007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ef0c80927f710a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:41.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.448+0000 7ffae2a00640 1 -- 172.21.15.37:0/3978968908 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc09b190 con 0x556d232a6770 2024-03-28T18:10:41.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.448+0000 7ffae2a00640 1 -- 172.21.15.37:0/3978968908 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffadc09f2e0 con 0x556d232a6770 2024-03-28T18:10:41.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.448+0000 7ffae2a00640 1 -- 172.21.15.37:0/3978968908 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc097800 con 0x556d232a6770 2024-03-28T18:10:41.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.448+0000 7ffafbe64740 1 -- 172.21.15.37:0/3978968908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232a6770 msgr2=0x556d232b7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:41.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.448+0000 7ffafbe64740 1 --2- 172.21.15.37:0/3978968908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232a6770 0x556d232b7730 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ffadc007590 tx=0x7ffadc007f60 comp rx=0 tx=0).stop 2024-03-28T18:10:41.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.448+0000 7ffafbe64740 1 -- 172.21.15.37:0/3978968908 shutdown_connections 2024-03-28T18:10:41.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.448+0000 7ffafbe64740 1 --2- 172.21.15.37:0/3978968908 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232ab8e0 0x556d232b0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.448+0000 7ffafbe64740 1 --2- 172.21.15.37:0/3978968908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232a6770 0x556d232b7730 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.448+0000 7ffafbe64740 1 --2- 172.21.15.37:0/3978968908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232adcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.448+0000 7ffafbe64740 1 -- 172.21.15.37:0/3978968908 >> 172.21.15.37:0/3978968908 conn(0x556d232a12f0 msgr2=0x556d232aed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:41.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.448+0000 7ffafbe64740 1 -- 172.21.15.37:0/3978968908 shutdown_connections 2024-03-28T18:10:41.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.449+0000 7ffafbe64740 1 -- 172.21.15.37:0/3978968908 wait complete. 2024-03-28T18:10:41.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.449+0000 7ffafbe64740 1 -- start start 2024-03-28T18:10:41.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.449+0000 7ffafbe64740 1 Processor -- start 2024-03-28T18:10:41.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.449+0000 7ffafbe64740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232b32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.449+0000 7ffafbe64740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232a6770 0x556d232b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.449+0000 7ffafbe64740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232ab8e0 0x556d232b6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.449+0000 7ffafbe64740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d232bd5c0 con 0x556d232a6770 2024-03-28T18:10:41.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.449+0000 7ffafbe64740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d232bcf60 con 0x556d232a5d80 2024-03-28T18:10:41.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.449+0000 7ffafbe64740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d232bdc20 con 0x556d232ab8e0 2024-03-28T18:10:41.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.449+0000 7ffafbe64740 10 client.0 _ll_register_callbacks cb 0x7ffadc0d4400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:10:41.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.450+0000 7ffaf1e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232b32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.450+0000 7ffaebe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232a6770 0x556d232b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.450+0000 7ffaf2800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232ab8e0 0x556d232b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.450+0000 7ffaf1e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232b32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:38020/0 (socket says 172.21.15.37:38020) 2024-03-28T18:10:41.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.450+0000 7ffaf1e00640 1 -- 172.21.15.37:0/857446607 learned_addr learned my addr 172.21.15.37:0/857446607 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:41.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.450+0000 7ffaf1e00640 1 -- 172.21.15.37:0/857446607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232ab8e0 msgr2=0x556d232b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:41.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.450+0000 7ffaf1e00640 1 --2- 172.21.15.37:0/857446607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232ab8e0 0x556d232b6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.450+0000 7ffaf1e00640 1 -- 172.21.15.37:0/857446607 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232a6770 msgr2=0x556d232b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:41.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.450+0000 7ffaf1e00640 1 --2- 172.21.15.37:0/857446607 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232a6770 0x556d232b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.450+0000 7ffaf1e00640 1 -- 172.21.15.37:0/857446607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556d232b6e00 con 0x556d232a5d80 2024-03-28T18:10:41.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.450+0000 7ffaf2800640 1 --2- 172.21.15.37:0/857446607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232ab8e0 0x556d232b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:41.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.450+0000 7ffaf1e00640 1 --2- 172.21.15.37:0/857446607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232b32b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x556d232b7d50 tx=0x7ffae40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caa726b81d24b98c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:41.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.450+0000 7ffad7e00640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:10:41.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.451+0000 7ffad7e00640 1 -- 172.21.15.37:0/857446607 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae402d220 con 0x556d232a5d80 2024-03-28T18:10:41.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.451+0000 7ffad7e00640 1 -- 172.21.15.37:0/857446607 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ffae402da70 con 0x556d232a5d80 2024-03-28T18:10:41.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.451+0000 7ffafbe64740 1 -- 172.21.15.37:0/857446607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffadc09f2e0 con 0x556d232a5d80 2024-03-28T18:10:41.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.451+0000 7ffad7e00640 1 -- 172.21.15.37:0/857446607 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffae402dc70 con 0x556d232a5d80 2024-03-28T18:10:41.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.451+0000 7ffafbe64740 10 client.25181 fetch_fsmap learned FSMap version 22 2024-03-28T18:10:41.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.451+0000 7ffafbe64740 1 -- 172.21.15.37:0/857446607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x556d23317e70 con 0x556d232a5d80 2024-03-28T18:10:41.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.451+0000 7ffad7e00640 1 -- 172.21.15.37:0/857446607 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffadc09f2e0 con 0x556d232a5d80 2024-03-28T18:10:41.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.451+0000 7ffad7e00640 1 -- 172.21.15.37:0/857446607 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ffae40047a0 con 0x556d232a5d80 2024-03-28T18:10:41.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.452+0000 7ffafbe64740 10 client.25181 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T18:10:41.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.452+0000 7ffafbe64740 10 client.25181 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T18:10:41.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.452+0000 7ffafbe64740 20 client.25181 populate_metadata read hostname 'smithi037' 2024-03-28T18:10:41.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.452+0000 7ffafbe64740 1 --2- 172.21.15.37:0/857446607 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556d2331c5c0 0x556d23318160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:41.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.452+0000 7ffafbe64740 4 client.25181 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:10:41.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.452+0000 7ffafbe64740 1 -- 172.21.15.37:0/857446607 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556d232181d0 con 0x556d2331c5c0 2024-03-28T18:10:41.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.452+0000 7ffaebe00640 1 --2- 172.21.15.37:0/857446607 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556d2331c5c0 0x556d23318160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:41.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.452+0000 7ffaebe00640 1 --2- 172.21.15.37:0/857446607 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556d2331c5c0 0x556d23318160 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45771fbb89ad6c8d server_cookie=219ef9a311b337e7 in_seq=0 out_seq=0 2024-03-28T18:10:41.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.452+0000 7ffad7e00640 10 client.25181 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:10:41.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.453+0000 7ffad7e00640 1 -- 172.21.15.37:0/857446607 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x556d232181d0 con 0x556d2331c5c0 2024-03-28T18:10:41.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.453+0000 7ffad7e00640 10 client.25181 handle_command_reply: tid=0 2024-03-28T18:10:41.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.453+0000 7ffafbe64740 1 client.25181 shutdown 2024-03-28T18:10:41.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.453+0000 7ffafbe64740 1 -- 172.21.15.37:0/857446607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 msgr2=0x556d232b32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:41.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.453+0000 7ffafbe64740 1 --2- 172.21.15.37:0/857446607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232b32b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x556d232b7d50 tx=0x7ffae40046c0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.453+0000 7ffafbe64740 1 -- 172.21.15.37:0/857446607 shutdown_connections 2024-03-28T18:10:41.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.453+0000 7ffafbe64740 1 --2- 172.21.15.37:0/857446607 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556d2331c5c0 0x556d23318160 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.453+0000 7ffafbe64740 1 --2- 172.21.15.37:0/857446607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d232ab8e0 0x556d232b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.453+0000 7ffafbe64740 1 --2- 172.21.15.37:0/857446607 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556d232a6770 0x556d232b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.453+0000 7ffad7e00640 0 client.25181 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:10:41.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.453+0000 7ffafbe64740 1 --2- 172.21.15.37:0/857446607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d232a5d80 0x556d232b32b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.453+0000 7ffafbe64740 1 -- 172.21.15.37:0/857446607 >> 172.21.15.37:0/857446607 conn(0x556d232a12f0 msgr2=0x556d232b0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:41.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.453+0000 7ffafbe64740 1 -- 172.21.15.37:0/857446607 shutdown_connections 2024-03-28T18:10:41.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.453+0000 7ffafbe64740 1 -- 172.21.15.37:0/857446607 wait complete. 2024-03-28T18:10:41.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.453+0000 7ffafbe64740 20 client.25181 trim_cache size 0 max 16384 2024-03-28T18:10:41.458 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:10:41.458 INFO:teuthology.orchestra.run.smithi037.stdout: "cluster_fsid": "b5ad257c-ed2c-11ee-95d0-87774f69a715", 2024-03-28T18:10:41.458 INFO:teuthology.orchestra.run.smithi037.stdout: "whoami": 0, 2024-03-28T18:10:41.458 INFO:teuthology.orchestra.run.smithi037.stdout: "id": 24580, 2024-03-28T18:10:41.458 INFO:teuthology.orchestra.run.smithi037.stdout: "want_state": "up:active", 2024-03-28T18:10:41.458 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "up:active", 2024-03-28T18:10:41.458 INFO:teuthology.orchestra.run.smithi037.stdout: "fs_name": "cephfs", 2024-03-28T18:10:41.458 INFO:teuthology.orchestra.run.smithi037.stdout: "rank_uptime": 140.940741944, 2024-03-28T18:10:41.458 INFO:teuthology.orchestra.run.smithi037.stdout: "mdsmap_epoch": 22, 2024-03-28T18:10:41.459 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap_epoch": 87, 2024-03-28T18:10:41.459 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap_epoch_barrier": 85, 2024-03-28T18:10:41.459 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 150.87590278900001 2024-03-28T18:10:41.459 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:10:41.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.454+0000 7ffaf3200640 1 -- 172.21.15.37:0/1908450050 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ffad80a2880 msgr2=0x7ffad80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:41.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.454+0000 7ffaf3200640 1 --2- 172.21.15.37:0/1908450050 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ffad80a2880 0x7ffad80a4d40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ffae40096c0 tx=0x7ffae4009260 comp rx=0 tx=0).stop 2024-03-28T18:10:41.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.454+0000 7ffaf3200640 1 -- 172.21.15.37:0/1908450050 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffaf4104930 msgr2=0x7ffaf40fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:41.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.454+0000 7ffaf3200640 1 --2- 172.21.15.37:0/1908450050 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffaf4104930 0x7ffaf40fffb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ffadc00c970 tx=0x7ffadc00ce30 comp rx=0 tx=0).stop 2024-03-28T18:10:41.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.454+0000 7ffaf3200640 1 -- 172.21.15.37:0/1908450050 shutdown_connections 2024-03-28T18:10:41.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.454+0000 7ffaf3200640 1 --2- 172.21.15.37:0/1908450050 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ffad80a2880 0x7ffad80a4d40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.454+0000 7ffaf3200640 1 --2- 172.21.15.37:0/1908450050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaf4107260 0x7ffaf41004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.454+0000 7ffaf3200640 1 --2- 172.21.15.37:0/1908450050 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffaf4104930 0x7ffaf40fffb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.454+0000 7ffaf3200640 1 --2- 172.21.15.37:0/1908450050 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffaf4102000 0x7ffaf4101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:41.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.454+0000 7ffaf3200640 1 -- 172.21.15.37:0/1908450050 >> 172.21.15.37:0/1908450050 conn(0x7ffaf40fba10 msgr2=0x7ffaf4102920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:41.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.454+0000 7ffaf3200640 1 -- 172.21.15.37:0/1908450050 shutdown_connections 2024-03-28T18:10:41.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:41.454+0000 7ffaf3200640 1 -- 172.21.15.37:0/1908450050 wait complete. 2024-03-28T18:10:41.474 INFO:teuthology.run_tasks:Running task kclient... 2024-03-28T18:10:41.499 INFO:tasks.kclient:Mounting kernel clients... 2024-03-28T18:10:41.499 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-28T18:10:41.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:10:41.500 DEBUG:teuthology.orchestra.run.smithi037:> ip netns list 2024-03-28T18:10:41.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:41 smithi059 ceph-mon[36190]: pgmap v390: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:10:41.537 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:41 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2697011757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:41.537 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:41 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3308870547' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:41.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:41 smithi170 ceph-mon[36280]: pgmap v390: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:10:41.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:41 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2697011757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:41.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:41 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3308870547' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:41.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:10:41.552 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip link delete ceph-brx 2024-03-28T18:10:41.648 INFO:teuthology.orchestra.run.smithi037.stderr:Cannot find device "ceph-brx" 2024-03-28T18:10:41.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:10:41.654 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-28T18:10:41.654 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T18:10:41.654 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T18:10:41.654 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5ad257c-ed2c-11ee-95d0-87774f69a715 -- ceph fs ls 2024-03-28T18:10:42.123 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:10:43.529 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:43 smithi037 ceph-mon[31460]: pgmap v391: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:10:43.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:43 smithi059 ceph-mon[36190]: pgmap v391: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:10:43.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:43 smithi170 ceph-mon[36280]: pgmap v391: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:10:44.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.389+0000 7f455b000700 1 -- 172.21.15.37:0/1754926619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4554104fb0 msgr2=0x7f4554109000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:44.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.389+0000 7f455b000700 1 --2- 172.21.15.37:0/1754926619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4554104fb0 0x7f4554109000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f453c0099c0 tx=0x7f453c009cd0 comp rx=0 tx=0).stop 2024-03-28T18:10:44.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.390+0000 7f455b000700 1 -- 172.21.15.37:0/1754926619 shutdown_connections 2024-03-28T18:10:44.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.390+0000 7f455b000700 1 --2- 172.21.15.37:0/1754926619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45541096e0 0x7f45541112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:44.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.390+0000 7f455b000700 1 --2- 172.21.15.37:0/1754926619 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4554104600 0x7f45541049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:44.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.390+0000 7f455b000700 1 --2- 172.21.15.37:0/1754926619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4554104fb0 0x7f4554109000 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:44.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.390+0000 7f455b000700 1 -- 172.21.15.37:0/1754926619 >> 172.21.15.37:0/1754926619 conn(0x7f45540ffe90 msgr2=0x7f45541022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:44.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.390+0000 7f455b000700 1 -- 172.21.15.37:0/1754926619 shutdown_connections 2024-03-28T18:10:44.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.390+0000 7f455b000700 1 -- 172.21.15.37:0/1754926619 wait complete. 2024-03-28T18:10:44.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.391+0000 7f455b000700 1 Processor -- start 2024-03-28T18:10:44.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.391+0000 7f455b000700 1 -- start start 2024-03-28T18:10:44.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.391+0000 7f455b000700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4554104600 0x7f45541a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:44.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.392+0000 7f455b000700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4554104fb0 0x7f45541a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:44.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.392+0000 7f455b000700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45541096e0 0x7f455419a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:44.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.392+0000 7f455b000700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4554114710 con 0x7f4554104600 2024-03-28T18:10:44.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.392+0000 7f455b000700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4554114590 con 0x7f4554104fb0 2024-03-28T18:10:44.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.392+0000 7f455b000700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4554114890 con 0x7f45541096e0 2024-03-28T18:10:44.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.392+0000 7f4553e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4554104600 0x7f45541a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:44.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.392+0000 7f4553e00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4554104600 0x7f45541a07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:48154/0 (socket says 172.21.15.37:48154) 2024-03-28T18:10:44.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.392+0000 7f4553e00700 1 -- 172.21.15.37:0/1288723646 learned_addr learned my addr 172.21.15.37:0/1288723646 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:44.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.392+0000 7f4559000700 1 --2- 172.21.15.37:0/1288723646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45541096e0 0x7f455419a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:44.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.392+0000 7f454be00700 1 --2- 172.21.15.37:0/1288723646 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4554104fb0 0x7f45541a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:44.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.392+0000 7f4559000700 1 -- 172.21.15.37:0/1288723646 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4554104fb0 msgr2=0x7f45541a0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:44.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.392+0000 7f4559000700 1 --2- 172.21.15.37:0/1288723646 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4554104fb0 0x7f45541a0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:44.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.392+0000 7f4559000700 1 -- 172.21.15.37:0/1288723646 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4554104600 msgr2=0x7f45541a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:44.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.392+0000 7f4559000700 1 --2- 172.21.15.37:0/1288723646 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4554104600 0x7f45541a07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:44.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.392+0000 7f4559000700 1 -- 172.21.15.37:0/1288723646 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455419b110 con 0x7f45541096e0 2024-03-28T18:10:44.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.393+0000 7f4559000700 1 --2- 172.21.15.37:0/1288723646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45541096e0 0x7f455419a9b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f454c00efc0 tx=0x7f454c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:44.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.393+0000 7f4551600700 1 -- 172.21.15.37:0/1288723646 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454c0092e0 con 0x7f45541096e0 2024-03-28T18:10:44.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.393+0000 7f4551600700 1 -- 172.21.15.37:0/1288723646 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f454c004bb0 con 0x7f45541096e0 2024-03-28T18:10:44.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.393+0000 7f455b000700 1 -- 172.21.15.37:0/1288723646 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f455419b3a0 con 0x7f45541096e0 2024-03-28T18:10:44.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.393+0000 7f4551600700 1 -- 172.21.15.37:0/1288723646 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454c004280 con 0x7f45541096e0 2024-03-28T18:10:44.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.393+0000 7f455b000700 1 -- 172.21.15.37:0/1288723646 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45541a76c0 con 0x7f45541096e0 2024-03-28T18:10:44.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.394+0000 7f455b000700 1 -- 172.21.15.37:0/1288723646 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4554068e50 con 0x7f45541096e0 2024-03-28T18:10:44.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.397+0000 7f4551600700 1 -- 172.21.15.37:0/1288723646 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f454c01b030 con 0x7f45541096e0 2024-03-28T18:10:44.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.398+0000 7f4551600700 1 --2- 172.21.15.37:0/1288723646 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f45380a2910 0x7f45380a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:44.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.398+0000 7f4551600700 1 -- 172.21.15.37:0/1288723646 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f454c0ce500 con 0x7f45541096e0 2024-03-28T18:10:44.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.398+0000 7f4553e00700 1 --2- 172.21.15.37:0/1288723646 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f45380a2910 0x7f45380a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:44.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.399+0000 7f4553e00700 1 --2- 172.21.15.37:0/1288723646 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f45380a2910 0x7f45380a4dd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f455419baf0 tx=0x7f4544009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:44.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.402+0000 7f4551600700 1 -- 172.21.15.37:0/1288723646 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f454c022020 con 0x7f45541096e0 2024-03-28T18:10:44.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.700+0000 7f455b000700 1 -- 172.21.15.37:0/1288723646 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f455402da30 con 0x7f45541096e0 2024-03-28T18:10:44.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.700+0000 7f4551600700 1 -- 172.21.15.37:0/1288723646 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v22) v1 ==== 53+0+73 (secure 0 0 0) 0x7f454c010040 con 0x7f45541096e0 2024-03-28T18:10:44.701 INFO:teuthology.orchestra.run.smithi037.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-28T18:10:44.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.703+0000 7f455b000700 1 -- 172.21.15.37:0/1288723646 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f45380a2910 msgr2=0x7f45380a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:44.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.703+0000 7f455b000700 1 --2- 172.21.15.37:0/1288723646 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f45380a2910 0x7f45380a4dd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f455419baf0 tx=0x7f4544009260 comp rx=0 tx=0).stop 2024-03-28T18:10:44.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.703+0000 7f455b000700 1 -- 172.21.15.37:0/1288723646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45541096e0 msgr2=0x7f455419a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:44.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.703+0000 7f455b000700 1 --2- 172.21.15.37:0/1288723646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45541096e0 0x7f455419a9b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f454c00efc0 tx=0x7f454c00c3a0 comp rx=0 tx=0).stop 2024-03-28T18:10:44.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.703+0000 7f455b000700 1 -- 172.21.15.37:0/1288723646 shutdown_connections 2024-03-28T18:10:44.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.703+0000 7f455b000700 1 --2- 172.21.15.37:0/1288723646 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f45380a2910 0x7f45380a4dd0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:44.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.703+0000 7f455b000700 1 --2- 172.21.15.37:0/1288723646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45541096e0 0x7f455419a9b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:44.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.703+0000 7f455b000700 1 --2- 172.21.15.37:0/1288723646 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4554104600 0x7f45541a07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:44.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.703+0000 7f455b000700 1 --2- 172.21.15.37:0/1288723646 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4554104fb0 0x7f45541a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:44.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.703+0000 7f455b000700 1 -- 172.21.15.37:0/1288723646 >> 172.21.15.37:0/1288723646 conn(0x7f45540ffe90 msgr2=0x7f4554102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:44.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.703+0000 7f455b000700 1 -- 172.21.15.37:0/1288723646 shutdown_connections 2024-03-28T18:10:44.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:44.703+0000 7f455b000700 1 -- 172.21.15.37:0/1288723646 wait complete. 2024-03-28T18:10:45.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:45 smithi037 ceph-mon[31460]: pgmap v392: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:45.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:45 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1288723646' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T18:10:45.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:45 smithi059 ceph-mon[36190]: pgmap v392: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:45.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:45 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1288723646' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T18:10:45.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:45 smithi170 ceph-mon[36280]: pgmap v392: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:45.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:45 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1288723646' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T18:10:45.542 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-28T18:10:45.542 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T18:10:45.543 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi037.front.sepia.ceph.com 2024-03-28T18:10:45.543 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-28T18:10:45.543 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T18:10:45.543 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-28T18:10:45.543 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T18:10:45.543 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T18:10:45.543 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-28T18:10:45.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:10:45.543 DEBUG:teuthology.orchestra.run.smithi037:> ip addr 2024-03-28T18:10:45.604 INFO:teuthology.orchestra.run.smithi037.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T18:10:45.604 INFO:teuthology.orchestra.run.smithi037.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T18:10:45.604 INFO:teuthology.orchestra.run.smithi037.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T18:10:45.605 INFO:teuthology.orchestra.run.smithi037.stdout: valid_lft forever preferred_lft forever 2024-03-28T18:10:45.605 INFO:teuthology.orchestra.run.smithi037.stdout: inet6 ::1/128 scope host 2024-03-28T18:10:45.605 INFO:teuthology.orchestra.run.smithi037.stdout: valid_lft forever preferred_lft forever 2024-03-28T18:10:45.605 INFO:teuthology.orchestra.run.smithi037.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T18:10:45.605 INFO:teuthology.orchestra.run.smithi037.stdout: link/ether 0c:c4:7a:6b:fd:18 brd ff:ff:ff:ff:ff:ff 2024-03-28T18:10:45.605 INFO:teuthology.orchestra.run.smithi037.stdout: altname enp7s0f0 2024-03-28T18:10:45.605 INFO:teuthology.orchestra.run.smithi037.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T18:10:45.605 INFO:teuthology.orchestra.run.smithi037.stdout: link/ether 0c:c4:7a:6b:fd:19 brd ff:ff:ff:ff:ff:ff 2024-03-28T18:10:45.605 INFO:teuthology.orchestra.run.smithi037.stdout: altname enp7s0f1 2024-03-28T18:10:45.605 INFO:teuthology.orchestra.run.smithi037.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T18:10:45.605 INFO:teuthology.orchestra.run.smithi037.stdout: link/ether 0c:c4:7a:88:77:94 brd ff:ff:ff:ff:ff:ff 2024-03-28T18:10:45.605 INFO:teuthology.orchestra.run.smithi037.stdout: altname enp1s0f0 2024-03-28T18:10:45.605 INFO:teuthology.orchestra.run.smithi037.stdout: inet 172.21.15.37/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-28T18:10:45.605 INFO:teuthology.orchestra.run.smithi037.stdout: valid_lft 41926sec preferred_lft 41926sec 2024-03-28T18:10:45.606 INFO:teuthology.orchestra.run.smithi037.stdout: inet6 fe80::ec4:7aff:fe88:7794/64 scope link noprefixroute 2024-03-28T18:10:45.606 INFO:teuthology.orchestra.run.smithi037.stdout: valid_lft forever preferred_lft forever 2024-03-28T18:10:45.606 INFO:teuthology.orchestra.run.smithi037.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T18:10:45.606 INFO:teuthology.orchestra.run.smithi037.stdout: link/ether 0c:c4:7a:88:77:95 brd ff:ff:ff:ff:ff:ff 2024-03-28T18:10:45.606 INFO:teuthology.orchestra.run.smithi037.stdout: altname enp1s0f1 2024-03-28T18:10:45.607 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T18:10:45.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:10:45.607 DEBUG:teuthology.orchestra.run.smithi037:> (cd / && exec bash -c ' 2024-03-28T18:10:45.608 DEBUG:teuthology.orchestra.run.smithi037:> set -e 2024-03-28T18:10:45.608 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip link add name ceph-brx type bridge 2024-03-28T18:10:45.608 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip addr flush dev ceph-brx 2024-03-28T18:10:45.608 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip link set ceph-brx up 2024-03-28T18:10:45.608 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T18:10:45.608 DEBUG:teuthology.orchestra.run.smithi037:> ') 2024-03-28T18:10:46.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:10:46.063 DEBUG:teuthology.orchestra.run.smithi037:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T18:10:46.146 INFO:teuthology.orchestra.run.smithi037.stdout:1 2024-03-28T18:10:46.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:10:46.149 DEBUG:teuthology.orchestra.run.smithi037:> route 2024-03-28T18:10:46.203 INFO:teuthology.orchestra.run.smithi037.stdout:Kernel IP routing table 2024-03-28T18:10:46.203 INFO:teuthology.orchestra.run.smithi037.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T18:10:46.203 INFO:teuthology.orchestra.run.smithi037.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-28T18:10:46.204 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-28T18:10:46.204 INFO:teuthology.orchestra.run.smithi037.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T18:10:46.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:10:46.205 DEBUG:teuthology.orchestra.run.smithi037:> (cd / && exec bash -c ' 2024-03-28T18:10:46.205 DEBUG:teuthology.orchestra.run.smithi037:> set -e 2024-03-28T18:10:46.205 DEBUG:teuthology.orchestra.run.smithi037:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-28T18:10:46.205 DEBUG:teuthology.orchestra.run.smithi037:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-28T18:10:46.205 DEBUG:teuthology.orchestra.run.smithi037:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-28T18:10:46.205 DEBUG:teuthology.orchestra.run.smithi037:> ') 2024-03-28T18:10:46.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:10:46.974 DEBUG:teuthology.orchestra.run.smithi037:> ip netns list 2024-03-28T18:10:47.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:10:47.027 DEBUG:teuthology.orchestra.run.smithi037:> ip netns list-id 2024-03-28T18:10:47.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:10:47.073 DEBUG:teuthology.orchestra.run.smithi037:> (cd / && exec bash -c ' 2024-03-28T18:10:47.073 DEBUG:teuthology.orchestra.run.smithi037:> set -e 2024-03-28T18:10:47.073 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T18:10:47.073 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-28T18:10:47.073 DEBUG:teuthology.orchestra.run.smithi037:> ') 2024-03-28T18:10:47.244 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:47 smithi037 ceph-mon[31460]: pgmap v393: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:10:47.257 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-28T18:10:47.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:10:47.257 DEBUG:teuthology.orchestra.run.smithi037:> (cd / && exec bash -c ' 2024-03-28T18:10:47.257 DEBUG:teuthology.orchestra.run.smithi037:> set -e 2024-03-28T18:10:47.257 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-28T18:10:47.257 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T18:10:47.258 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-28T18:10:47.258 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-28T18:10:47.258 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-28T18:10:47.258 DEBUG:teuthology.orchestra.run.smithi037:> ') 2024-03-28T18:10:47.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:47 smithi059 ceph-mon[36190]: pgmap v393: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:10:47.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:47 smithi170 ceph-mon[36280]: pgmap v393: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:10:47.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:10:47.554 DEBUG:teuthology.orchestra.run.smithi037:> (cd / && exec bash -c ' 2024-03-28T18:10:47.554 DEBUG:teuthology.orchestra.run.smithi037:> set -e 2024-03-28T18:10:47.554 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip link set brx.0 up 2024-03-28T18:10:47.555 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T18:10:47.555 DEBUG:teuthology.orchestra.run.smithi037:> ') 2024-03-28T18:10:47.725 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T18:10:47.725 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-28T18:10:47.775 INFO:teuthology.orchestra.run.smithi037.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-28T18:10:47.776 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T18:10:47.776 DEBUG:teuthology.orchestra.run.smithi037:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-28T18:10:47.818 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-28T18:10:47.818 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-28T18:10:47.818 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/6fbe70b0-f754-450d-9fbd-670541c15017 2024-03-28T18:10:47.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:10:47.819 DEBUG:teuthology.orchestra.run.smithi037:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/6fbe70b0-f754-450d-9fbd-670541c15017 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-03-28T18:10:47.995 INFO:teuthology.orchestra.run.smithi037.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-03-28T18:10:47.995 INFO:teuthology.orchestra.run.smithi037.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-03-28T18:10:48.626 INFO:teuthology.orchestra.run.smithi037.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-03-28T18:10:48.626 INFO:teuthology.orchestra.run.smithi037.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-03-28T18:10:48.626 INFO:teuthology.orchestra.run.smithi037.stdout:mount.ceph: resolved to: "172.21.15.37:6789,172.21.15.59:6789,172.21.15.170:6789" 2024-03-28T18:10:48.626 INFO:teuthology.orchestra.run.smithi037.stdout:mount.ceph: trying mount with new device syntax: 0@b5ad257c-ed2c-11ee-95d0-87774f69a715.cephfs=/volumes/qa/sv_1/6fbe70b0-f754-450d-9fbd-670541c15017 2024-03-28T18:10:48.626 INFO:teuthology.orchestra.run.smithi037.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync,name=0,key=0,mon_addr=172.21.15.37:6789/172.21.15.59:6789/172.21.15.170:6789" will pass to kernel 2024-03-28T18:10:48.634 INFO:teuthology.orchestra.run.smithi037.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-28T18:10:48.635 INFO:teuthology.orchestra.run.smithi037.stdout: You just mounted a file system that supports labels which does not 2024-03-28T18:10:48.635 INFO:teuthology.orchestra.run.smithi037.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-28T18:10:48.635 INFO:teuthology.orchestra.run.smithi037.stdout: applications will generate AVC messages and not be allowed access to 2024-03-28T18:10:48.635 INFO:teuthology.orchestra.run.smithi037.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-28T18:10:48.643 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-28T18:10:48.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:10:48.643 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-28T18:10:48.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:10:48.746 DEBUG:teuthology.orchestra.run.smithi037:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T18:10:48.801 INFO:teuthology.orchestra.run.smithi037.stdout:client25216 2024-03-28T18:10:48.801 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-28T18:10:48.801 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T18:10:48.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:10:48.852 DEBUG:teuthology.orchestra.run.smithi037:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T18:10:48.947 INFO:teuthology.orchestra.run.smithi037.stdout:client25216 2024-03-28T18:10:48.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:10:48.947 DEBUG:teuthology.orchestra.run.smithi037:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T18:10:48.999 INFO:teuthology.orchestra.run.smithi037.stdout:client25216 2024-03-28T18:10:48.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:10:48.999 DEBUG:teuthology.orchestra.run.smithi037:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715.client25216/status') 2024-03-28T18:10:49.074 INFO:teuthology.orchestra.run.smithi037.stdout:instance: client.25216 (3)172.21.15.37:0/167536383 2024-03-28T18:10:49.074 INFO:teuthology.orchestra.run.smithi037.stdout:blocklisted: false 2024-03-28T18:10:49.074 INFO:teuthology.orchestra.run.smithi037.stderr:0+1 records in 2024-03-28T18:10:49.074 INFO:teuthology.orchestra.run.smithi037.stderr:0+1 records out 2024-03-28T18:10:49.075 INFO:teuthology.orchestra.run.smithi037.stderr:70 bytes copied, 6.702e-05 s, 1.0 MB/s 2024-03-28T18:10:49.080 INFO:teuthology.run_tasks:Running task exec... 2024-03-28T18:10:49.090 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T18:10:49.090 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi037.front.sepia.ceph.com 2024-03-28T18:10:49.090 DEBUG:teuthology.orchestra.run.smithi037:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-28T18:10:49.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.296+0000 7ff5e5200640 1 -- 172.21.15.37:0/3659364494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5e0103900 msgr2=0x7ff5e0103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:49.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.296+0000 7ff5e5200640 1 --2- 172.21.15.37:0/3659364494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5e0103900 0x7ff5e0103d20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0009860 tx=0x7ff5d002bb70 comp rx=0 tx=0).stop 2024-03-28T18:10:49.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.296+0000 7ff5e5200640 1 -- 172.21.15.37:0/3659364494 shutdown_connections 2024-03-28T18:10:49.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.296+0000 7ff5e5200640 1 --2- 172.21.15.37:0/3659364494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5e01054b0 0x7ff5e010bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:49.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.296+0000 7ff5e5200640 1 --2- 172.21.15.37:0/3659364494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5e0104af0 0x7ff5e0104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:49.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.296+0000 7ff5e5200640 1 --2- 172.21.15.37:0/3659364494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5e0103900 0x7ff5e0103d20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:49.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.296+0000 7ff5e5200640 1 -- 172.21.15.37:0/3659364494 >> 172.21.15.37:0/3659364494 conn(0x7ff5e00fee90 msgr2=0x7ff5e01012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:49.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.297+0000 7ff5e5200640 1 -- 172.21.15.37:0/3659364494 shutdown_connections 2024-03-28T18:10:49.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.297+0000 7ff5e5200640 1 -- 172.21.15.37:0/3659364494 wait complete. 2024-03-28T18:10:49.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.298+0000 7ff5e5200640 1 Processor -- start 2024-03-28T18:10:49.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.298+0000 7ff5e5200640 1 -- start start 2024-03-28T18:10:49.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.298+0000 7ff5e5200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5e0103900 0x7ff5e0199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:49.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.298+0000 7ff5e5200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5e0104af0 0x7ff5e019a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:49.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.298+0000 7ff5e5200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5e01054b0 0x7ff5e01a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:49.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.298+0000 7ff5e5200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e010f550 con 0x7ff5e0103900 2024-03-28T18:10:49.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.298+0000 7ff5e5200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e010f3d0 con 0x7ff5e0104af0 2024-03-28T18:10:49.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.298+0000 7ff5e5200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e010f6d0 con 0x7ff5e01054b0 2024-03-28T18:10:49.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.299+0000 7ff5d7e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5e0104af0 0x7ff5e019a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:49.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.299+0000 7ff5d7e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5e0104af0 0x7ff5e019a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:54126/0 (socket says 172.21.15.37:54126) 2024-03-28T18:10:49.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.299+0000 7ff5d7e00640 1 -- 172.21.15.37:0/3067142632 learned_addr learned my addr 172.21.15.37:0/3067142632 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:49.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.299+0000 7ff5de800640 1 --2- 172.21.15.37:0/3067142632 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5e01054b0 0x7ff5e01a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:49.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.299+0000 7ff5dde00640 1 --2- 172.21.15.37:0/3067142632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5e0103900 0x7ff5e0199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:49.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.299+0000 7ff5d7e00640 1 -- 172.21.15.37:0/3067142632 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5e01054b0 msgr2=0x7ff5e01a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:49.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.299+0000 7ff5d7e00640 1 --2- 172.21.15.37:0/3067142632 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5e01054b0 0x7ff5e01a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:49.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.299+0000 7ff5d7e00640 1 -- 172.21.15.37:0/3067142632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5e0103900 msgr2=0x7ff5e0199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:49.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.299+0000 7ff5d7e00640 1 --2- 172.21.15.37:0/3067142632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5e0103900 0x7ff5e0199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:49.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.299+0000 7ff5d7e00640 1 -- 172.21.15.37:0/3067142632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e01a1750 con 0x7ff5e0104af0 2024-03-28T18:10:49.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.299+0000 7ff5d7e00640 1 --2- 172.21.15.37:0/3067142632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5e0104af0 0x7ff5e019a070 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff5c800e9f0 tx=0x7ff5c800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:49.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.300+0000 7ff5d6a00640 1 -- 172.21.15.37:0/3067142632 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c800cd80 con 0x7ff5e0104af0 2024-03-28T18:10:49.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.300+0000 7ff5d6a00640 1 -- 172.21.15.37:0/3067142632 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff5c8010910 con 0x7ff5e0104af0 2024-03-28T18:10:49.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.300+0000 7ff5d6a00640 1 -- 172.21.15.37:0/3067142632 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c8018990 con 0x7ff5e0104af0 2024-03-28T18:10:49.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.300+0000 7ff5e5200640 1 -- 172.21.15.37:0/3067142632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e01a1a40 con 0x7ff5e0104af0 2024-03-28T18:10:49.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.300+0000 7ff5e5200640 1 -- 172.21.15.37:0/3067142632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e01a1fe0 con 0x7ff5e0104af0 2024-03-28T18:10:49.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.301+0000 7ff5e5200640 1 -- 172.21.15.37:0/3067142632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e010abc0 con 0x7ff5e0104af0 2024-03-28T18:10:49.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.302+0000 7ff5d6a00640 1 -- 172.21.15.37:0/3067142632 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff5c8010ac0 con 0x7ff5e0104af0 2024-03-28T18:10:49.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.304+0000 7ff5d6a00640 1 --2- 172.21.15.37:0/3067142632 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff5c40a2920 0x7ff5c40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:49.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.304+0000 7ff5d6a00640 1 -- 172.21.15.37:0/3067142632 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff5c80ced60 con 0x7ff5e0104af0 2024-03-28T18:10:49.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.304+0000 7ff5dde00640 1 --2- 172.21.15.37:0/3067142632 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff5c40a2920 0x7ff5c40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:49.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.305+0000 7ff5dde00640 1 --2- 172.21.15.37:0/3067142632 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff5c40a2920 0x7ff5c40a4de0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00349e0 tx=0x7ff5d0034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:49.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.308+0000 7ff5d6a00640 1 -- 172.21.15.37:0/3067142632 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff5c809b760 con 0x7ff5e0104af0 2024-03-28T18:10:49.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:49 smithi037 ceph-mon[31460]: pgmap v394: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:49.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:49 smithi059 ceph-mon[36190]: pgmap v394: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:49.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:49 smithi170 ceph-mon[36280]: pgmap v394: 97 pgs: 97 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:49.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:49.563+0000 7ff5e5200640 1 -- 172.21.15.37:0/3067142632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) v1 -- 0x7ff5e00622a0 con 0x7ff5e0104af0 2024-03-28T18:10:50.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.217+0000 7ff5d6a00640 1 -- 172.21.15.37:0/3067142632 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v23) v1 ==== 117+0+0 (secure 0 0 0) 0x7ff5c80bb020 con 0x7ff5e0104af0 2024-03-28T18:10:50.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.220+0000 7ff5e5200640 1 -- 172.21.15.37:0/3067142632 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff5c40a2920 msgr2=0x7ff5c40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:50.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.220+0000 7ff5e5200640 1 --2- 172.21.15.37:0/3067142632 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff5c40a2920 0x7ff5c40a4de0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00349e0 tx=0x7ff5d0034ab0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.220+0000 7ff5e5200640 1 -- 172.21.15.37:0/3067142632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5e0104af0 msgr2=0x7ff5e019a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:50.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.220+0000 7ff5e5200640 1 --2- 172.21.15.37:0/3067142632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5e0104af0 0x7ff5e019a070 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff5c800e9f0 tx=0x7ff5c800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.220+0000 7ff5e5200640 1 -- 172.21.15.37:0/3067142632 shutdown_connections 2024-03-28T18:10:50.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.220+0000 7ff5e5200640 1 --2- 172.21.15.37:0/3067142632 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff5c40a2920 0x7ff5c40a4de0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.221+0000 7ff5e5200640 1 --2- 172.21.15.37:0/3067142632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5e0104af0 0x7ff5e019a070 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.221+0000 7ff5e5200640 1 --2- 172.21.15.37:0/3067142632 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5e01054b0 0x7ff5e01a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.221+0000 7ff5e5200640 1 --2- 172.21.15.37:0/3067142632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5e0103900 0x7ff5e0199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.221+0000 7ff5e5200640 1 -- 172.21.15.37:0/3067142632 >> 172.21.15.37:0/3067142632 conn(0x7ff5e00fee90 msgr2=0x7ff5e0107d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:50.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.221+0000 7ff5e5200640 1 -- 172.21.15.37:0/3067142632 shutdown_connections 2024-03-28T18:10:50.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.221+0000 7ff5e5200640 1 -- 172.21.15.37:0/3067142632 wait complete. 2024-03-28T18:10:50.243 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-28T18:10:50.254 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-28T18:10:50.296 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:10:50.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.489+0000 7f8423000640 1 -- 172.21.15.37:0/2257573101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f841c1016d0 msgr2=0x7f841c101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:50.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.489+0000 7f8423000640 1 --2- 172.21.15.37:0/2257573101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f841c1016d0 0x7f841c101af0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f840c009900 tx=0x7f840c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:50.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.489+0000 7f8423000640 1 -- 172.21.15.37:0/2257573101 shutdown_connections 2024-03-28T18:10:50.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.489+0000 7f8423000640 1 --2- 172.21.15.37:0/2257573101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f841c103280 0x7f841c109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.489+0000 7f8423000640 1 --2- 172.21.15.37:0/2257573101 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f841c1028c0 0x7f841c102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.489+0000 7f8423000640 1 --2- 172.21.15.37:0/2257573101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f841c1016d0 0x7f841c101af0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.489+0000 7f8423000640 1 -- 172.21.15.37:0/2257573101 >> 172.21.15.37:0/2257573101 conn(0x7f841c0fcc40 msgr2=0x7f841c0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:50.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.489+0000 7f8423000640 1 -- 172.21.15.37:0/2257573101 shutdown_connections 2024-03-28T18:10:50.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.489+0000 7f8423000640 1 -- 172.21.15.37:0/2257573101 wait complete. 2024-03-28T18:10:50.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.490+0000 7f8423000640 1 Processor -- start 2024-03-28T18:10:50.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.490+0000 7f8423000640 1 -- start start 2024-03-28T18:10:50.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.490+0000 7f8423000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f841c1016d0 0x7f841c199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:50.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.490+0000 7f8423000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f841c1028c0 0x7f841c19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:50.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.491+0000 7f8423000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f841c103280 0x7f841c1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:50.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.491+0000 7f8423000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c10d3a0 con 0x7f841c103280 2024-03-28T18:10:50.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.491+0000 7f8423000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c10d220 con 0x7f841c1028c0 2024-03-28T18:10:50.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.491+0000 7f8423000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c10d520 con 0x7f841c1016d0 2024-03-28T18:10:50.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.491+0000 7f841be00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f841c1016d0 0x7f841c199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:50.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.491+0000 7f841b400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f841c1028c0 0x7f841c19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:50.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.491+0000 7f841be00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f841c1016d0 0x7f841c199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:59540/0 (socket says 172.21.15.37:59540) 2024-03-28T18:10:50.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.491+0000 7f841be00640 1 -- 172.21.15.37:0/55545654 learned_addr learned my addr 172.21.15.37:0/55545654 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:50.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.491+0000 7f841b400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f841c1028c0 0x7f841c19a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:54146/0 (socket says 172.21.15.37:54146) 2024-03-28T18:10:50.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.491+0000 7f8421000640 1 --2- 172.21.15.37:0/55545654 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f841c103280 0x7f841c1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:50.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.491+0000 7f841b400640 1 -- 172.21.15.37:0/55545654 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f841c1016d0 msgr2=0x7f841c199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:50.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.491+0000 7f841b400640 1 --2- 172.21.15.37:0/55545654 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f841c1016d0 0x7f841c199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.491+0000 7f841b400640 1 -- 172.21.15.37:0/55545654 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f841c103280 msgr2=0x7f841c1a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:50.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.491+0000 7f841b400640 1 --2- 172.21.15.37:0/55545654 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f841c103280 0x7f841c1a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.491+0000 7f841b400640 1 -- 172.21.15.37:0/55545654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f841c1a1830 con 0x7f841c1028c0 2024-03-28T18:10:50.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.491+0000 7f8421000640 1 --2- 172.21.15.37:0/55545654 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f841c103280 0x7f841c1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:50.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.491+0000 7f841be00640 1 --2- 172.21.15.37:0/55545654 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f841c1016d0 0x7f841c199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:50.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.492+0000 7f841b400640 1 --2- 172.21.15.37:0/55545654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f841c1028c0 0x7f841c19a160 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8408005fd0 tx=0x7f840800dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:50.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.492+0000 7f8418c00640 1 -- 172.21.15.37:0/55545654 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8408009c30 con 0x7f841c1028c0 2024-03-28T18:10:50.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.492+0000 7f8423000640 1 -- 172.21.15.37:0/55545654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f841c1a1b20 con 0x7f841c1028c0 2024-03-28T18:10:50.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.492+0000 7f8418c00640 1 -- 172.21.15.37:0/55545654 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8408009dd0 con 0x7f841c1028c0 2024-03-28T18:10:50.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.492+0000 7f8418c00640 1 -- 172.21.15.37:0/55545654 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8408010460 con 0x7f841c1028c0 2024-03-28T18:10:50.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.492+0000 7f8423000640 1 -- 172.21.15.37:0/55545654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f841c1a20c0 con 0x7f841c1028c0 2024-03-28T18:10:50.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.493+0000 7f8423000640 1 -- 172.21.15.37:0/55545654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f841c1a2290 con 0x7f841c1028c0 2024-03-28T18:10:50.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.496+0000 7f8418c00640 1 -- 172.21.15.37:0/55545654 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8408003f50 con 0x7f841c1028c0 2024-03-28T18:10:50.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.497+0000 7f8418c00640 1 --2- 172.21.15.37:0/55545654 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f84000a2800 0x7f84000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:50.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.497+0000 7f8418c00640 1 -- 172.21.15.37:0/55545654 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f84080ceb50 con 0x7f841c1028c0 2024-03-28T18:10:50.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.497+0000 7f841be00640 1 --2- 172.21.15.37:0/55545654 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f84000a2800 0x7f84000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:50.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.498+0000 7f841be00640 1 --2- 172.21.15.37:0/55545654 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f84000a2800 0x7f84000a4cc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f841c19b0f0 tx=0x7f840c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:50.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.506+0000 7f8418c00640 1 -- 172.21.15.37:0/55545654 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8408017020 con 0x7f841c1028c0 2024-03-28T18:10:50.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:50 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3067142632' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-28T18:10:50.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:50 smithi059 ceph-mon[36190]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-28T18:10:50.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:50 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:10:50.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:50 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3067142632' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-28T18:10:50.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:50 smithi170 ceph-mon[36280]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-28T18:10:50.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:50 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:10:50.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:50 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3067142632' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-28T18:10:50.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:50 smithi037 ceph-mon[31460]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-28T18:10:50.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:50 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:10:50.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.740+0000 7f8423000640 1 -- 172.21.15.37:0/55545654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f841c1a2580 con 0x7f841c1028c0 2024-03-28T18:10:50.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.742+0000 7f8418c00640 1 -- 172.21.15.37:0/55545654 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22575 (secure 0 0 0) 0x7f840809bba0 con 0x7f841c1028c0 2024-03-28T18:10:50.742 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:50.742 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":87,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","created":"2024-03-28T17:59:05.949847+0000","modified":"2024-03-28T18:09:15.709901+0000","last_up_change":"2024-03-28T18:07:21.021030+0000","last_in_change":"2024-03-28T18:07:01.416192+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:03:00.517967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:08:18.365827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:08:19.446282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db60618d-56f7-4130-8a90-84cbc480965b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6803","nonce":3562602035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6805","nonce":3562602035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6809","nonce":3562602035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6807","nonce":3562602035}]},"public_addr":"172.21.15.37:6803/3562602035","cluster_addr":"172.21.15.37:6805/3562602035","heartbeat_back_addr":"172.21.15.37:6809/3562602035","heartbeat_front_addr":"172.21.15.37:6807/3562602035","state":["exists","up"]},{"osd":1,"uuid":"7a4b7ad6-905e-4258-9574-693d4277dfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6801","nonce":3326537211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6803","nonce":3326537211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6807","nonce":3326537211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6805","nonce":3326537211}]},"public_addr":"172.21.15.59:6801/3326537211","cluster_addr":"172.21.15.59:6803/3326537211","heartbeat_back_addr":"172.21.15.59:6807/3326537211","heartbeat_front_addr":"172.21.15.59:6805/3326537211","state":["exists","up"]},{"osd":2,"uuid":"8b76ec1c-46bf-4336-89b5-12d82906e8b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6801","nonce":1269398427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6803","nonce":1269398427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6807","nonce":1269398427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6805","nonce":1269398427}]},"public_addr":"172.21.15.170:6801/1269398427","cluster_addr":"172.21.15.170:6803/1269398427","heartbeat_back_addr":"172.21.15.170:6807/1269398427","heartbeat_front_addr":"172.21.15.170:6805/1269398427","state":["exists","up"]},{"osd":3,"uuid":"66eca1da-2fbd-44e1-8bbf-5635a2c70ad5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6811","nonce":317805240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6813","nonce":317805240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6817","nonce":317805240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6815","nonce":317805240}]},"public_addr":"172.21.15.37:6811/317805240","cluster_addr":"172.21.15.37:6813/317805240","heartbeat_back_addr":"172.21.15.37:6817/317805240","heartbeat_front_addr":"172.21.15.37:6815/317805240","state":["exists","up"]},{"osd":4,"uuid":"4c877335-e42d-4d8e-9f48-a1d0a89ae438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6809","nonce":2410632623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6811","nonce":2410632623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6815","nonce":2410632623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6813","nonce":2410632623}]},"public_addr":"172.21.15.59:6809/2410632623","cluster_addr":"172.21.15.59:6811/2410632623","heartbeat_back_addr":"172.21.15.59:6815/2410632623","heartbeat_front_addr":"172.21.15.59:6813/2410632623","state":["exists","up"]},{"osd":5,"uuid":"b884842a-86e2-455b-9570-ff163b3bc0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6809","nonce":1218943899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6811","nonce":1218943899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6815","nonce":1218943899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6813","nonce":1218943899}]},"public_addr":"172.21.15.170:6809/1218943899","cluster_addr":"172.21.15.170:6811/1218943899","heartbeat_back_addr":"172.21.15.170:6815/1218943899","heartbeat_front_addr":"172.21.15.170:6813/1218943899","state":["exists","up"]},{"osd":6,"uuid":"249a1200-77f4-44ad-981f-1d352f732b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6819","nonce":1899652046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6821","nonce":1899652046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6825","nonce":1899652046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6823","nonce":1899652046}]},"public_addr":"172.21.15.37:6819/1899652046","cluster_addr":"172.21.15.37:6821/1899652046","heartbeat_back_addr":"172.21.15.37:6825/1899652046","heartbeat_front_addr":"172.21.15.37:6823/1899652046","state":["exists","up"]},{"osd":7,"uuid":"c73338e6-f505-4517-a84b-f35e04750a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6817","nonce":1343675690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6819","nonce":1343675690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6823","nonce":1343675690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6821","nonce":1343675690}]},"public_addr":"172.21.15.59:6817/1343675690","cluster_addr":"172.21.15.59:6819/1343675690","heartbeat_back_addr":"172.21.15.59:6823/1343675690","heartbeat_front_addr":"172.21.15.59:6821/1343675690","state":["exists","up"]},{"osd":8,"uuid":"149b8297-ff2d-4a88-9b1e-0def021cc516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6817","nonce":3235794556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6819","nonce":3235794556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6823","nonce":3235794556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6821","nonce":3235794556}]},"public_addr":"172.21.15.170:6817/3235794556","cluster_addr":"172.21.15.170:6819/3235794556","heartbeat_back_addr":"172.21.15.170:6823/3235794556","heartbeat_front_addr":"172.21.15.170:6821/3235794556","state":["exists","up"]},{"osd":9,"uuid":"55c27cc2-c771-44b7-90fe-8f0feca44162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6827","nonce":2307153008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6829","nonce":2307153008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6833","nonce":2307153008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6831","nonce":2307153008}]},"public_addr":"172.21.15.37:6827/2307153008","cluster_addr":"172.21.15.37:6829/2307153008","heartbeat_back_addr":"172.21.15.37:6833/2307153008","heartbeat_front_addr":"172.21.15.37:6831/2307153008","state":["exists","up"]},{"osd":10,"uuid":"c0595135-ab81-49d3-8c23-19f960c03e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6825","nonce":2113830275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6827","nonce":2113830275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6831","nonce":2113830275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6829","nonce":2113830275}]},"public_addr":"172.21.15.59:6825/2113830275","cluster_addr":"172.21.15.59:6827/2113830275","heartbeat_back_addr":"172.21.15.59:6831/2113830275","heartbeat_front_addr":"172.21.15.59:6829/2113830275","state":["exists","up"]},{"osd":11,"uuid":"5871e177-c17b-4637-a797-4fa5125920dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6825","nonce":3082439360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6827","nonce":3082439360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6831","nonce":3082439360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6829","nonce":3082439360}]},"public_addr":"172.21.15.170:6825/3082439360","cluster_addr":"172.21.15.170:6827/3082439360","heartbeat_back_addr":"172.21.15.170:6831/3082439360","heartbeat_front_addr":"172.21.15.170:6829/3082439360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:07.690545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:33.183396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:57.400263+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:30.597211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:57.259044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:22.627823+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:57.585863+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:25.000476+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:50.876688+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:25.859761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:53.563146+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:07:19.304115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3023379428":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1443710231":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6800/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6801/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6801/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6800/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1007625490":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/2895698584":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/497313034":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/3293943475":"2024-03-29T17:59:39.162038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:10:50.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.745+0000 7f8423000640 1 -- 172.21.15.37:0/55545654 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f84000a2800 msgr2=0x7f84000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:50.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.745+0000 7f8423000640 1 --2- 172.21.15.37:0/55545654 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f84000a2800 0x7f84000a4cc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f841c19b0f0 tx=0x7f840c02df90 comp rx=0 tx=0).stop 2024-03-28T18:10:50.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.745+0000 7f8423000640 1 -- 172.21.15.37:0/55545654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f841c1028c0 msgr2=0x7f841c19a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:50.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.745+0000 7f8423000640 1 --2- 172.21.15.37:0/55545654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f841c1028c0 0x7f841c19a160 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8408005fd0 tx=0x7f840800dde0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.745+0000 7f8423000640 1 -- 172.21.15.37:0/55545654 shutdown_connections 2024-03-28T18:10:50.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.745+0000 7f8423000640 1 --2- 172.21.15.37:0/55545654 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f84000a2800 0x7f84000a4cc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.745+0000 7f8423000640 1 --2- 172.21.15.37:0/55545654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f841c1028c0 0x7f841c19a160 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.745+0000 7f8423000640 1 --2- 172.21.15.37:0/55545654 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f841c103280 0x7f841c1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.745+0000 7f8423000640 1 --2- 172.21.15.37:0/55545654 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f841c1016d0 0x7f841c199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.745+0000 7f8423000640 1 -- 172.21.15.37:0/55545654 >> 172.21.15.37:0/55545654 conn(0x7f841c0fcc40 msgr2=0x7f841c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:50.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.745+0000 7f8423000640 1 -- 172.21.15.37:0/55545654 shutdown_connections 2024-03-28T18:10:50.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.746+0000 7f8423000640 1 -- 172.21.15.37:0/55545654 wait complete. 2024-03-28T18:10:50.766 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:03:00.517967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:08:18.365827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:08:19.446282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:10:50.767 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:10:50.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.984+0000 7f8d8ca00640 1 -- 172.21.15.37:0/3854872515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d88104ab0 msgr2=0x7f8d88104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:50.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.984+0000 7f8d8ca00640 1 --2- 172.21.15.37:0/3854872515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d88104ab0 0x7f8d88104f30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8d74009900 tx=0x7f8d7402bc50 comp rx=0 tx=0).stop 2024-03-28T18:10:50.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.984+0000 7f8d8ca00640 1 -- 172.21.15.37:0/3854872515 shutdown_connections 2024-03-28T18:10:50.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.984+0000 7f8d8ca00640 1 --2- 172.21.15.37:0/3854872515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d88105470 0x7f8d8810bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.984+0000 7f8d8ca00640 1 --2- 172.21.15.37:0/3854872515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d88104ab0 0x7f8d88104f30 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.984+0000 7f8d8ca00640 1 --2- 172.21.15.37:0/3854872515 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d881038c0 0x7f8d88103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.984+0000 7f8d8ca00640 1 -- 172.21.15.37:0/3854872515 >> 172.21.15.37:0/3854872515 conn(0x7f8d880fee50 msgr2=0x7f8d88101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:50.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.984+0000 7f8d8ca00640 1 -- 172.21.15.37:0/3854872515 shutdown_connections 2024-03-28T18:10:50.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.984+0000 7f8d8ca00640 1 -- 172.21.15.37:0/3854872515 wait complete. 2024-03-28T18:10:50.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.985+0000 7f8d8ca00640 1 Processor -- start 2024-03-28T18:10:50.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.986+0000 7f8d8ca00640 1 -- start start 2024-03-28T18:10:50.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.986+0000 7f8d8ca00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d881038c0 0x7f8d88199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:50.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.986+0000 7f8d8ca00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d88104ab0 0x7f8d8819a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:50.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.986+0000 7f8d8ca00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d88105470 0x7f8d881a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:50.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.986+0000 7f8d8ca00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8810f570 con 0x7f8d88104ab0 2024-03-28T18:10:50.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.986+0000 7f8d8ca00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8810f3f0 con 0x7f8d88105470 2024-03-28T18:10:50.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.986+0000 7f8d8ca00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8810f6f0 con 0x7f8d881038c0 2024-03-28T18:10:50.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.986+0000 7f8d85e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d881038c0 0x7f8d88199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:50.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.986+0000 7f8d85e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d881038c0 0x7f8d88199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:59560/0 (socket says 172.21.15.37:59560) 2024-03-28T18:10:50.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.986+0000 7f8d85e00640 1 -- 172.21.15.37:0/561742877 learned_addr learned my addr 172.21.15.37:0/561742877 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:50.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.986+0000 7f8d86800640 1 --2- 172.21.15.37:0/561742877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d88105470 0x7f8d881a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:50.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.987+0000 7f8d85400640 1 --2- 172.21.15.37:0/561742877 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d88104ab0 0x7f8d8819a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:50.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.987+0000 7f8d85e00640 1 -- 172.21.15.37:0/561742877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d88105470 msgr2=0x7f8d881a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:50.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.987+0000 7f8d85e00640 1 --2- 172.21.15.37:0/561742877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d88105470 0x7f8d881a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.987+0000 7f8d85e00640 1 -- 172.21.15.37:0/561742877 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d88104ab0 msgr2=0x7f8d8819a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:50.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.987+0000 7f8d85e00640 1 --2- 172.21.15.37:0/561742877 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d88104ab0 0x7f8d8819a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:50.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.987+0000 7f8d85e00640 1 -- 172.21.15.37:0/561742877 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d881a17e0 con 0x7f8d881038c0 2024-03-28T18:10:50.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.987+0000 7f8d85400640 1 --2- 172.21.15.37:0/561742877 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d88104ab0 0x7f8d8819a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:10:50.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.987+0000 7f8d85e00640 1 --2- 172.21.15.37:0/561742877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d881038c0 0x7f8d88199bc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8d7800e9c0 tx=0x7f8d7800edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:50.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.987+0000 7f8d7ea00640 1 -- 172.21.15.37:0/561742877 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7800ccc0 con 0x7f8d881038c0 2024-03-28T18:10:50.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.987+0000 7f8d7ea00640 1 -- 172.21.15.37:0/561742877 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d780035f0 con 0x7f8d881038c0 2024-03-28T18:10:50.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.987+0000 7f8d7ea00640 1 -- 172.21.15.37:0/561742877 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7800ce60 con 0x7f8d881038c0 2024-03-28T18:10:50.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.987+0000 7f8d8ca00640 1 -- 172.21.15.37:0/561742877 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d881a1ad0 con 0x7f8d881038c0 2024-03-28T18:10:50.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.988+0000 7f8d8ca00640 1 -- 172.21.15.37:0/561742877 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d881a1fd0 con 0x7f8d881038c0 2024-03-28T18:10:50.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.989+0000 7f8d8ca00640 1 -- 172.21.15.37:0/561742877 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d8810abe0 con 0x7f8d881038c0 2024-03-28T18:10:50.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.992+0000 7f8d7ea00640 1 -- 172.21.15.37:0/561742877 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8d78004020 con 0x7f8d881038c0 2024-03-28T18:10:50.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.993+0000 7f8d7ea00640 1 --2- 172.21.15.37:0/561742877 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8d6c0a2880 0x7f8d6c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:50.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.994+0000 7f8d7ea00640 1 -- 172.21.15.37:0/561742877 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f8d780cee70 con 0x7f8d881038c0 2024-03-28T18:10:50.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.994+0000 7f8d85400640 1 --2- 172.21.15.37:0/561742877 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8d6c0a2880 0x7f8d6c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:50.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.994+0000 7f8d85400640 1 --2- 172.21.15.37:0/561742877 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8d6c0a2880 0x7f8d6c0a4d40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8d740096c0 tx=0x7f8d74005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:50.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:50.997+0000 7f8d7ea00640 1 -- 172.21.15.37:0/561742877 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8d7809bec0 con 0x7f8d881038c0 2024-03-28T18:10:51.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.212+0000 7f8d8ca00640 1 -- 172.21.15.37:0/561742877 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8d88062340 con 0x7f8d881038c0 2024-03-28T18:10:51.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.212+0000 7f8d7ea00640 1 -- 172.21.15.37:0/561742877 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8d7809b7c0 con 0x7f8d881038c0 2024-03-28T18:10:51.212 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 1 2024-03-28T18:10:51.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.215+0000 7f8d8ca00640 1 -- 172.21.15.37:0/561742877 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8d6c0a2880 msgr2=0x7f8d6c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:51.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.215+0000 7f8d8ca00640 1 --2- 172.21.15.37:0/561742877 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8d6c0a2880 0x7f8d6c0a4d40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8d740096c0 tx=0x7f8d74005fd0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.215+0000 7f8d8ca00640 1 -- 172.21.15.37:0/561742877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d881038c0 msgr2=0x7f8d88199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:51.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.215+0000 7f8d8ca00640 1 --2- 172.21.15.37:0/561742877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d881038c0 0x7f8d88199bc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8d7800e9c0 tx=0x7f8d7800edc0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.216+0000 7f8d8ca00640 1 -- 172.21.15.37:0/561742877 shutdown_connections 2024-03-28T18:10:51.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.216+0000 7f8d8ca00640 1 --2- 172.21.15.37:0/561742877 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8d6c0a2880 0x7f8d6c0a4d40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.216+0000 7f8d8ca00640 1 --2- 172.21.15.37:0/561742877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d88105470 0x7f8d881a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.216+0000 7f8d8ca00640 1 --2- 172.21.15.37:0/561742877 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d88104ab0 0x7f8d8819a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.216+0000 7f8d8ca00640 1 --2- 172.21.15.37:0/561742877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d881038c0 0x7f8d88199bc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.216+0000 7f8d8ca00640 1 -- 172.21.15.37:0/561742877 >> 172.21.15.37:0/561742877 conn(0x7f8d880fee50 msgr2=0x7f8d88107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:51.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.216+0000 7f8d8ca00640 1 -- 172.21.15.37:0/561742877 shutdown_connections 2024-03-28T18:10:51.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.216+0000 7f8d8ca00640 1 -- 172.21.15.37:0/561742877 wait complete. 2024-03-28T18:10:51.233 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:10:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:51 smithi170 ceph-mon[36280]: pgmap v395: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T18:10:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:51 smithi170 ceph-mon[36280]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-28T18:10:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:51 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:10:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:51 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:51 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/55545654' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:51 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:51 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:51 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:51 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/561742877' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:51.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.428+0000 7f33a5a00640 1 -- 172.21.15.37:0/4151662585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33a0074100 msgr2=0x7f33a0074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:51.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.428+0000 7f33a5a00640 1 --2- 172.21.15.37:0/4151662585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33a0074100 0x7f33a0074580 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f338c009900 tx=0x7f338c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:51.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.428+0000 7f33a5a00640 1 -- 172.21.15.37:0/4151662585 shutdown_connections 2024-03-28T18:10:51.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.428+0000 7f33a5a00640 1 --2- 172.21.15.37:0/4151662585 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33a010db70 0x7f33a010ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.428+0000 7f33a5a00640 1 --2- 172.21.15.37:0/4151662585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33a0074100 0x7f33a0074580 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.428+0000 7f33a5a00640 1 --2- 172.21.15.37:0/4151662585 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33a00756e0 0x7f33a0073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.428+0000 7f33a5a00640 1 -- 172.21.15.37:0/4151662585 >> 172.21.15.37:0/4151662585 conn(0x7f33a00fcb10 msgr2=0x7f33a00fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:51.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.428+0000 7f33a5a00640 1 -- 172.21.15.37:0/4151662585 shutdown_connections 2024-03-28T18:10:51.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.428+0000 7f33a5a00640 1 -- 172.21.15.37:0/4151662585 wait complete. 2024-03-28T18:10:51.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.429+0000 7f33a5a00640 1 Processor -- start 2024-03-28T18:10:51.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.429+0000 7f33a5a00640 1 -- start start 2024-03-28T18:10:51.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.430+0000 7f33a5a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33a0074100 0x7f33a019de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:51.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.430+0000 7f33a5a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33a00756e0 0x7f33a019e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:51.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.430+0000 7f33a5a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33a010db70 0x7f33a01a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:51.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.430+0000 7f33a5a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a0113840 con 0x7f33a010db70 2024-03-28T18:10:51.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.430+0000 7f33a5a00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a01136c0 con 0x7f33a0074100 2024-03-28T18:10:51.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.430+0000 7f33a5a00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a01139c0 con 0x7f33a00756e0 2024-03-28T18:10:51.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.430+0000 7f339fe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33a0074100 0x7f33a019de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:51.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.430+0000 7f339fe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33a0074100 0x7f33a019de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:54182/0 (socket says 172.21.15.37:54182) 2024-03-28T18:10:51.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.430+0000 7f339fe00640 1 -- 172.21.15.37:0/1396632297 learned_addr learned my addr 172.21.15.37:0/1396632297 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:51.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.430+0000 7f339f400640 1 --2- 172.21.15.37:0/1396632297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33a00756e0 0x7f33a019e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:51.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.430+0000 7f33a5000640 1 --2- 172.21.15.37:0/1396632297 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33a010db70 0x7f33a01a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:51.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.430+0000 7f339fe00640 1 -- 172.21.15.37:0/1396632297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33a00756e0 msgr2=0x7f33a019e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:51.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.430+0000 7f339fe00640 1 --2- 172.21.15.37:0/1396632297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33a00756e0 0x7f33a019e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.430+0000 7f339fe00640 1 -- 172.21.15.37:0/1396632297 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33a010db70 msgr2=0x7f33a01a5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:51.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.430+0000 7f339fe00640 1 --2- 172.21.15.37:0/1396632297 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33a010db70 0x7f33a01a5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.431+0000 7f339fe00640 1 -- 172.21.15.37:0/1396632297 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a01a5a80 con 0x7f33a0074100 2024-03-28T18:10:51.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.431+0000 7f339fe00640 1 --2- 172.21.15.37:0/1396632297 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33a0074100 0x7f33a019de60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f339000ea60 tx=0x7f339000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:51.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.431+0000 7f339cc00640 1 -- 172.21.15.37:0/1396632297 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339000cde0 con 0x7f33a0074100 2024-03-28T18:10:51.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.431+0000 7f339cc00640 1 -- 172.21.15.37:0/1396632297 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33900035f0 con 0x7f33a0074100 2024-03-28T18:10:51.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.431+0000 7f339cc00640 1 -- 172.21.15.37:0/1396632297 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3390010670 con 0x7f33a0074100 2024-03-28T18:10:51.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.431+0000 7f33a5a00640 1 -- 172.21.15.37:0/1396632297 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a01a5d70 con 0x7f33a0074100 2024-03-28T18:10:51.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.431+0000 7f33a5a00640 1 -- 172.21.15.37:0/1396632297 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a01a6180 con 0x7f33a0074100 2024-03-28T18:10:51.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.432+0000 7f33a5a00640 1 -- 172.21.15.37:0/1396632297 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33a0103d40 con 0x7f33a0074100 2024-03-28T18:10:51.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.436+0000 7f339cc00640 1 -- 172.21.15.37:0/1396632297 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f33900037a0 con 0x7f33a0074100 2024-03-28T18:10:51.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.437+0000 7f339cc00640 1 --2- 172.21.15.37:0/1396632297 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f33840a2880 0x7f33840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:51.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.437+0000 7f339cc00640 1 -- 172.21.15.37:0/1396632297 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f33900ceda0 con 0x7f33a0074100 2024-03-28T18:10:51.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.437+0000 7f339f400640 1 --2- 172.21.15.37:0/1396632297 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f33840a2880 0x7f33840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:51.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.438+0000 7f339f400640 1 --2- 172.21.15.37:0/1396632297 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f33840a2880 0x7f33840a4d40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f338c02c240 tx=0x7f338c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:51.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.441+0000 7f339cc00640 1 -- 172.21.15.37:0/1396632297 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f339009bdf0 con 0x7f33a0074100 2024-03-28T18:10:51.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:51 smithi059 ceph-mon[36190]: pgmap v395: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T18:10:51.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:51 smithi059 ceph-mon[36190]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-28T18:10:51.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:51 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:10:51.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:51 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:51.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:51 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/55545654' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:51.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:51 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:51.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:51 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:51.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:51 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:51.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:51 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/561742877' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:51.628 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:51 smithi037 ceph-mon[31460]: pgmap v395: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T18:10:51.628 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:51 smithi037 ceph-mon[31460]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-28T18:10:51.628 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:51 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:10:51.629 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:51 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:51.629 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:51 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/55545654' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:51.629 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:51 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:51.629 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:51 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:51.629 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:51 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:51.629 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:51 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/561742877' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:51.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.650+0000 7f33a5a00640 1 -- 172.21.15.37:0/1396632297 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f33a00622d0 con 0x7f33a0074100 2024-03-28T18:10:51.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.651+0000 7f339cc00640 1 -- 172.21.15.37:0/1396632297 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f339009b6f0 con 0x7f33a0074100 2024-03-28T18:10:51.651 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 32 2024-03-28T18:10:51.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.653+0000 7f33a5a00640 1 -- 172.21.15.37:0/1396632297 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f33840a2880 msgr2=0x7f33840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:51.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.653+0000 7f33a5a00640 1 --2- 172.21.15.37:0/1396632297 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f33840a2880 0x7f33840a4d40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f338c02c240 tx=0x7f338c000bc0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.653+0000 7f33a5a00640 1 -- 172.21.15.37:0/1396632297 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33a0074100 msgr2=0x7f33a019de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:51.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.653+0000 7f33a5a00640 1 --2- 172.21.15.37:0/1396632297 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33a0074100 0x7f33a019de60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f339000ea60 tx=0x7f339000ef20 comp rx=0 tx=0).stop 2024-03-28T18:10:51.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.653+0000 7f33a5a00640 1 -- 172.21.15.37:0/1396632297 shutdown_connections 2024-03-28T18:10:51.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.653+0000 7f33a5a00640 1 --2- 172.21.15.37:0/1396632297 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f33840a2880 0x7f33840a4d40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.653+0000 7f33a5a00640 1 --2- 172.21.15.37:0/1396632297 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33a010db70 0x7f33a01a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.653+0000 7f33a5a00640 1 --2- 172.21.15.37:0/1396632297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33a00756e0 0x7f33a019e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.653+0000 7f33a5a00640 1 --2- 172.21.15.37:0/1396632297 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33a0074100 0x7f33a019de60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.653+0000 7f33a5a00640 1 -- 172.21.15.37:0/1396632297 >> 172.21.15.37:0/1396632297 conn(0x7f33a00fcb10 msgr2=0x7f33a0100f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:51.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.653+0000 7f33a5a00640 1 -- 172.21.15.37:0/1396632297 shutdown_connections 2024-03-28T18:10:51.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.654+0000 7f33a5a00640 1 -- 172.21.15.37:0/1396632297 wait complete. 2024-03-28T18:10:51.672 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:10:51.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.899+0000 7f1a55600640 1 -- 172.21.15.37:0/3430854264 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a501038e0 msgr2=0x7f1a50103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:51.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.899+0000 7f1a55600640 1 --2- 172.21.15.37:0/3430854264 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a501038e0 0x7f1a50103d00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1a40009900 tx=0x7f1a4002bc40 comp rx=0 tx=0).stop 2024-03-28T18:10:51.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.899+0000 7f1a55600640 1 -- 172.21.15.37:0/3430854264 shutdown_connections 2024-03-28T18:10:51.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.899+0000 7f1a55600640 1 --2- 172.21.15.37:0/3430854264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a50105490 0x7f1a5010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.899+0000 7f1a55600640 1 --2- 172.21.15.37:0/3430854264 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a50104ad0 0x7f1a50104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.899+0000 7f1a55600640 1 --2- 172.21.15.37:0/3430854264 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a501038e0 0x7f1a50103d00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.899+0000 7f1a55600640 1 -- 172.21.15.37:0/3430854264 >> 172.21.15.37:0/3430854264 conn(0x7f1a500fee50 msgr2=0x7f1a50101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:51.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.900+0000 7f1a55600640 1 -- 172.21.15.37:0/3430854264 shutdown_connections 2024-03-28T18:10:51.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.900+0000 7f1a55600640 1 -- 172.21.15.37:0/3430854264 wait complete. 2024-03-28T18:10:51.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.901+0000 7f1a55600640 1 Processor -- start 2024-03-28T18:10:51.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.901+0000 7f1a55600640 1 -- start start 2024-03-28T18:10:51.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.902+0000 7f1a55600640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a501038e0 0x7f1a50199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:51.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.902+0000 7f1a55600640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a50104ad0 0x7f1a5019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:51.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.902+0000 7f1a55600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a50105490 0x7f1a501a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:51.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.902+0000 7f1a55600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5010f5b0 con 0x7f1a50105490 2024-03-28T18:10:51.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.902+0000 7f1a55600640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5010f430 con 0x7f1a501038e0 2024-03-28T18:10:51.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.902+0000 7f1a55600640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5010f730 con 0x7f1a50104ad0 2024-03-28T18:10:51.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.902+0000 7f1a4ea00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a501038e0 0x7f1a50199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:51.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.902+0000 7f1a4ea00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a501038e0 0x7f1a50199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:54208/0 (socket says 172.21.15.37:54208) 2024-03-28T18:10:51.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.902+0000 7f1a4ea00640 1 -- 172.21.15.37:0/1093243152 learned_addr learned my addr 172.21.15.37:0/1093243152 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:51.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.902+0000 7f1a4e000640 1 --2- 172.21.15.37:0/1093243152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a50104ad0 0x7f1a5019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:51.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.902+0000 7f1a4f400640 1 --2- 172.21.15.37:0/1093243152 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a50105490 0x7f1a501a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:51.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.902+0000 7f1a4ea00640 1 -- 172.21.15.37:0/1093243152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a50104ad0 msgr2=0x7f1a5019a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:51.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.902+0000 7f1a4ea00640 1 --2- 172.21.15.37:0/1093243152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a50104ad0 0x7f1a5019a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.902+0000 7f1a4ea00640 1 -- 172.21.15.37:0/1093243152 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a50105490 msgr2=0x7f1a501a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:51.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.902+0000 7f1a4ea00640 1 --2- 172.21.15.37:0/1093243152 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a50105490 0x7f1a501a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:51.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.902+0000 7f1a4ea00640 1 -- 172.21.15.37:0/1093243152 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a501a1810 con 0x7f1a501038e0 2024-03-28T18:10:51.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.902+0000 7f1a4e000640 1 --2- 172.21.15.37:0/1093243152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a50104ad0 0x7f1a5019a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:51.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.903+0000 7f1a4ea00640 1 --2- 172.21.15.37:0/1093243152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a501038e0 0x7f1a50199bf0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1a4002efd0 tx=0x7f1a4002c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:51.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.903+0000 7f1a47400640 1 -- 172.21.15.37:0/1093243152 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a40041070 con 0x7f1a501038e0 2024-03-28T18:10:51.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.903+0000 7f1a47400640 1 -- 172.21.15.37:0/1093243152 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a40004450 con 0x7f1a501038e0 2024-03-28T18:10:51.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.903+0000 7f1a47400640 1 -- 172.21.15.37:0/1093243152 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a40003e50 con 0x7f1a501038e0 2024-03-28T18:10:51.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.903+0000 7f1a55600640 1 -- 172.21.15.37:0/1093243152 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a501a1aa0 con 0x7f1a501038e0 2024-03-28T18:10:51.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.903+0000 7f1a55600640 1 -- 172.21.15.37:0/1093243152 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a501a1f10 con 0x7f1a501038e0 2024-03-28T18:10:51.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.905+0000 7f1a55600640 1 -- 172.21.15.37:0/1093243152 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5010aba0 con 0x7f1a501038e0 2024-03-28T18:10:51.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.906+0000 7f1a47400640 1 -- 172.21.15.37:0/1093243152 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1a40044070 con 0x7f1a501038e0 2024-03-28T18:10:51.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.908+0000 7f1a47400640 1 --2- 172.21.15.37:0/1093243152 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f1a340a2a20 0x7f1a340a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:51.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.909+0000 7f1a47400640 1 -- 172.21.15.37:0/1093243152 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f1a400f06a0 con 0x7f1a501038e0 2024-03-28T18:10:51.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.909+0000 7f1a4e000640 1 --2- 172.21.15.37:0/1093243152 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f1a340a2a20 0x7f1a340a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:51.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.909+0000 7f1a4e000640 1 --2- 172.21.15.37:0/1093243152 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f1a340a2a20 0x7f1a340a4ee0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1a5019b100 tx=0x7f1a3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:51.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:51.917+0000 7f1a47400640 1 -- 172.21.15.37:0/1093243152 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1a400bd6f0 con 0x7f1a501038e0 2024-03-28T18:10:52.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.166+0000 7f1a55600640 1 -- 172.21.15.37:0/1093243152 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1a500622a0 con 0x7f1a501038e0 2024-03-28T18:10:52.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.167+0000 7f1a47400640 1 -- 172.21.15.37:0/1093243152 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1a400bcff0 con 0x7f1a501038e0 2024-03-28T18:10:52.167 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 64 2024-03-28T18:10:52.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.170+0000 7f1a55600640 1 -- 172.21.15.37:0/1093243152 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f1a340a2a20 msgr2=0x7f1a340a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:52.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.170+0000 7f1a55600640 1 --2- 172.21.15.37:0/1093243152 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f1a340a2a20 0x7f1a340a4ee0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1a5019b100 tx=0x7f1a3c005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:52.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.170+0000 7f1a55600640 1 -- 172.21.15.37:0/1093243152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a501038e0 msgr2=0x7f1a50199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:52.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.170+0000 7f1a55600640 1 --2- 172.21.15.37:0/1093243152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a501038e0 0x7f1a50199bf0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1a4002efd0 tx=0x7f1a4002c660 comp rx=0 tx=0).stop 2024-03-28T18:10:52.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.170+0000 7f1a55600640 1 -- 172.21.15.37:0/1093243152 shutdown_connections 2024-03-28T18:10:52.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.170+0000 7f1a55600640 1 --2- 172.21.15.37:0/1093243152 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f1a340a2a20 0x7f1a340a4ee0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:52.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.170+0000 7f1a55600640 1 --2- 172.21.15.37:0/1093243152 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a50105490 0x7f1a501a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:52.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.170+0000 7f1a55600640 1 --2- 172.21.15.37:0/1093243152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a50104ad0 0x7f1a5019a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:52.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.170+0000 7f1a55600640 1 --2- 172.21.15.37:0/1093243152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a501038e0 0x7f1a50199bf0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:52.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.170+0000 7f1a55600640 1 -- 172.21.15.37:0/1093243152 >> 172.21.15.37:0/1093243152 conn(0x7f1a500fee50 msgr2=0x7f1a50107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:52.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.170+0000 7f1a55600640 1 -- 172.21.15.37:0/1093243152 shutdown_connections 2024-03-28T18:10:52.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.171+0000 7f1a55600640 1 -- 172.21.15.37:0/1093243152 wait complete. 2024-03-28T18:10:52.192 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:10:52.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.385+0000 7f3d67200640 1 -- 172.21.15.37:0/1024378817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d68101500 msgr2=0x7f3d68101920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:52.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.385+0000 7f3d67200640 1 --2- 172.21.15.37:0/1024378817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d68101500 0x7f3d68101920 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3d58009860 tx=0x7f3d5802bb70 comp rx=0 tx=0).stop 2024-03-28T18:10:52.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.385+0000 7f3d67200640 1 -- 172.21.15.37:0/1024378817 shutdown_connections 2024-03-28T18:10:52.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.385+0000 7f3d67200640 1 --2- 172.21.15.37:0/1024378817 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d681030b0 0x7f3d68109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:52.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.385+0000 7f3d67200640 1 --2- 172.21.15.37:0/1024378817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d681026f0 0x7f3d68102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:52.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.385+0000 7f3d67200640 1 --2- 172.21.15.37:0/1024378817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d68101500 0x7f3d68101920 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:52.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.385+0000 7f3d67200640 1 -- 172.21.15.37:0/1024378817 >> 172.21.15.37:0/1024378817 conn(0x7f3d680fca70 msgr2=0x7f3d680feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:52.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.385+0000 7f3d67200640 1 -- 172.21.15.37:0/1024378817 shutdown_connections 2024-03-28T18:10:52.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.385+0000 7f3d67200640 1 -- 172.21.15.37:0/1024378817 wait complete. 2024-03-28T18:10:52.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.386+0000 7f3d67200640 1 Processor -- start 2024-03-28T18:10:52.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.386+0000 7f3d67200640 1 -- start start 2024-03-28T18:10:52.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.386+0000 7f3d67200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d68101500 0x7f3d68199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:52.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.386+0000 7f3d67200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d681026f0 0x7f3d68199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:52.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.386+0000 7f3d67200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d681030b0 0x7f3d681a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:52.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.386+0000 7f3d67200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6810d160 con 0x7f3d68101500 2024-03-28T18:10:52.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.386+0000 7f3d67200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6810cfe0 con 0x7f3d681026f0 2024-03-28T18:10:52.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.386+0000 7f3d67200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6810d2e0 con 0x7f3d681030b0 2024-03-28T18:10:52.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.386+0000 7f3d65400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d681026f0 0x7f3d68199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:52.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.387+0000 7f3d66800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d681030b0 0x7f3d681a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:52.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.387+0000 7f3d65400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d681026f0 0x7f3d68199fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:54222/0 (socket says 172.21.15.37:54222) 2024-03-28T18:10:52.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.387+0000 7f3d65400640 1 -- 172.21.15.37:0/3345438805 learned_addr learned my addr 172.21.15.37:0/3345438805 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:52.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.387+0000 7f3d65e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d68101500 0x7f3d68199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:52.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.387+0000 7f3d66800640 1 -- 172.21.15.37:0/3345438805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d681026f0 msgr2=0x7f3d68199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:52.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.387+0000 7f3d66800640 1 --2- 172.21.15.37:0/3345438805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d681026f0 0x7f3d68199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:52.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.387+0000 7f3d66800640 1 -- 172.21.15.37:0/3345438805 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d68101500 msgr2=0x7f3d68199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:52.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.387+0000 7f3d66800640 1 --2- 172.21.15.37:0/3345438805 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d68101500 0x7f3d68199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:52.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.387+0000 7f3d66800640 1 -- 172.21.15.37:0/3345438805 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d681a1690 con 0x7f3d681030b0 2024-03-28T18:10:52.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.387+0000 7f3d66800640 1 --2- 172.21.15.37:0/3345438805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d681030b0 0x7f3d681a1020 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3d60009fd0 tx=0x7f3d6000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:52.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.387+0000 7f3d5ea00640 1 -- 172.21.15.37:0/3345438805 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6000ce20 con 0x7f3d681030b0 2024-03-28T18:10:52.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.388+0000 7f3d5ea00640 1 -- 172.21.15.37:0/3345438805 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d60005020 con 0x7f3d681030b0 2024-03-28T18:10:52.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.388+0000 7f3d67200640 1 -- 172.21.15.37:0/3345438805 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d681a1980 con 0x7f3d681030b0 2024-03-28T18:10:52.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.388+0000 7f3d5ea00640 1 -- 172.21.15.37:0/3345438805 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d60010720 con 0x7f3d681030b0 2024-03-28T18:10:52.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.388+0000 7f3d67200640 1 -- 172.21.15.37:0/3345438805 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d681a1e80 con 0x7f3d681030b0 2024-03-28T18:10:52.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.389+0000 7f3d67200640 1 -- 172.21.15.37:0/3345438805 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d681a20b0 con 0x7f3d681030b0 2024-03-28T18:10:52.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.392+0000 7f3d5ea00640 1 -- 172.21.15.37:0/3345438805 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3d60005540 con 0x7f3d681030b0 2024-03-28T18:10:52.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.393+0000 7f3d5ea00640 1 --2- 172.21.15.37:0/3345438805 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f3d4c0a2800 0x7f3d4c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:52.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.393+0000 7f3d5ea00640 1 -- 172.21.15.37:0/3345438805 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3d600cf190 con 0x7f3d681030b0 2024-03-28T18:10:52.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.393+0000 7f3d65e00640 1 --2- 172.21.15.37:0/3345438805 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f3d4c0a2800 0x7f3d4c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:52.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.394+0000 7f3d65e00640 1 --2- 172.21.15.37:0/3345438805 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f3d4c0a2800 0x7f3d4c0a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3d580349e0 tx=0x7f3d58034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:52.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.400+0000 7f3d5ea00640 1 -- 172.21.15.37:0/3345438805 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3d6009c1e0 con 0x7f3d681030b0 2024-03-28T18:10:52.472 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:52 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1396632297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:52.472 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:52 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:52.472 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:52 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:52.472 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:52 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1093243152' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:52.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:52 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1396632297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:52.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:52 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:52.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:52 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:52.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:52 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1093243152' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:52.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:52 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1396632297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:52.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:52 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:52.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:52 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:52.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:52 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1093243152' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:52.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.673+0000 7f3d67200640 1 -- 172.21.15.37:0/3345438805 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d681a2320 con 0x7f3d681030b0 2024-03-28T18:10:52.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.675+0000 7f3d5ea00640 1 -- 172.21.15.37:0/3345438805 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22575 (secure 0 0 0) 0x7f3d6009bae0 con 0x7f3d681030b0 2024-03-28T18:10:52.676 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:52.676 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":87,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","created":"2024-03-28T17:59:05.949847+0000","modified":"2024-03-28T18:09:15.709901+0000","last_up_change":"2024-03-28T18:07:21.021030+0000","last_in_change":"2024-03-28T18:07:01.416192+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:03:00.517967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:08:18.365827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:08:19.446282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db60618d-56f7-4130-8a90-84cbc480965b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6803","nonce":3562602035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6805","nonce":3562602035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6809","nonce":3562602035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6807","nonce":3562602035}]},"public_addr":"172.21.15.37:6803/3562602035","cluster_addr":"172.21.15.37:6805/3562602035","heartbeat_back_addr":"172.21.15.37:6809/3562602035","heartbeat_front_addr":"172.21.15.37:6807/3562602035","state":["exists","up"]},{"osd":1,"uuid":"7a4b7ad6-905e-4258-9574-693d4277dfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6801","nonce":3326537211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6803","nonce":3326537211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6807","nonce":3326537211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6805","nonce":3326537211}]},"public_addr":"172.21.15.59:6801/3326537211","cluster_addr":"172.21.15.59:6803/3326537211","heartbeat_back_addr":"172.21.15.59:6807/3326537211","heartbeat_front_addr":"172.21.15.59:6805/3326537211","state":["exists","up"]},{"osd":2,"uuid":"8b76ec1c-46bf-4336-89b5-12d82906e8b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6801","nonce":1269398427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6803","nonce":1269398427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6807","nonce":1269398427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6805","nonce":1269398427}]},"public_addr":"172.21.15.170:6801/1269398427","cluster_addr":"172.21.15.170:6803/1269398427","heartbeat_back_addr":"172.21.15.170:6807/1269398427","heartbeat_front_addr":"172.21.15.170:6805/1269398427","state":["exists","up"]},{"osd":3,"uuid":"66eca1da-2fbd-44e1-8bbf-5635a2c70ad5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6811","nonce":317805240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6813","nonce":317805240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6817","nonce":317805240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6815","nonce":317805240}]},"public_addr":"172.21.15.37:6811/317805240","cluster_addr":"172.21.15.37:6813/317805240","heartbeat_back_addr":"172.21.15.37:6817/317805240","heartbeat_front_addr":"172.21.15.37:6815/317805240","state":["exists","up"]},{"osd":4,"uuid":"4c877335-e42d-4d8e-9f48-a1d0a89ae438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6809","nonce":2410632623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6811","nonce":2410632623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6815","nonce":2410632623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6813","nonce":2410632623}]},"public_addr":"172.21.15.59:6809/2410632623","cluster_addr":"172.21.15.59:6811/2410632623","heartbeat_back_addr":"172.21.15.59:6815/2410632623","heartbeat_front_addr":"172.21.15.59:6813/2410632623","state":["exists","up"]},{"osd":5,"uuid":"b884842a-86e2-455b-9570-ff163b3bc0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6809","nonce":1218943899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6811","nonce":1218943899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6815","nonce":1218943899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6813","nonce":1218943899}]},"public_addr":"172.21.15.170:6809/1218943899","cluster_addr":"172.21.15.170:6811/1218943899","heartbeat_back_addr":"172.21.15.170:6815/1218943899","heartbeat_front_addr":"172.21.15.170:6813/1218943899","state":["exists","up"]},{"osd":6,"uuid":"249a1200-77f4-44ad-981f-1d352f732b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6819","nonce":1899652046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6821","nonce":1899652046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6825","nonce":1899652046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6823","nonce":1899652046}]},"public_addr":"172.21.15.37:6819/1899652046","cluster_addr":"172.21.15.37:6821/1899652046","heartbeat_back_addr":"172.21.15.37:6825/1899652046","heartbeat_front_addr":"172.21.15.37:6823/1899652046","state":["exists","up"]},{"osd":7,"uuid":"c73338e6-f505-4517-a84b-f35e04750a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6817","nonce":1343675690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6819","nonce":1343675690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6823","nonce":1343675690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6821","nonce":1343675690}]},"public_addr":"172.21.15.59:6817/1343675690","cluster_addr":"172.21.15.59:6819/1343675690","heartbeat_back_addr":"172.21.15.59:6823/1343675690","heartbeat_front_addr":"172.21.15.59:6821/1343675690","state":["exists","up"]},{"osd":8,"uuid":"149b8297-ff2d-4a88-9b1e-0def021cc516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6817","nonce":3235794556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6819","nonce":3235794556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6823","nonce":3235794556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6821","nonce":3235794556}]},"public_addr":"172.21.15.170:6817/3235794556","cluster_addr":"172.21.15.170:6819/3235794556","heartbeat_back_addr":"172.21.15.170:6823/3235794556","heartbeat_front_addr":"172.21.15.170:6821/3235794556","state":["exists","up"]},{"osd":9,"uuid":"55c27cc2-c771-44b7-90fe-8f0feca44162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6827","nonce":2307153008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6829","nonce":2307153008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6833","nonce":2307153008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6831","nonce":2307153008}]},"public_addr":"172.21.15.37:6827/2307153008","cluster_addr":"172.21.15.37:6829/2307153008","heartbeat_back_addr":"172.21.15.37:6833/2307153008","heartbeat_front_addr":"172.21.15.37:6831/2307153008","state":["exists","up"]},{"osd":10,"uuid":"c0595135-ab81-49d3-8c23-19f960c03e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6825","nonce":2113830275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6827","nonce":2113830275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6831","nonce":2113830275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6829","nonce":2113830275}]},"public_addr":"172.21.15.59:6825/2113830275","cluster_addr":"172.21.15.59:6827/2113830275","heartbeat_back_addr":"172.21.15.59:6831/2113830275","heartbeat_front_addr":"172.21.15.59:6829/2113830275","state":["exists","up"]},{"osd":11,"uuid":"5871e177-c17b-4637-a797-4fa5125920dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6825","nonce":3082439360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6827","nonce":3082439360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6831","nonce":3082439360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6829","nonce":3082439360}]},"public_addr":"172.21.15.170:6825/3082439360","cluster_addr":"172.21.15.170:6827/3082439360","heartbeat_back_addr":"172.21.15.170:6831/3082439360","heartbeat_front_addr":"172.21.15.170:6829/3082439360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:07.690545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:33.183396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:57.400263+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:30.597211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:57.259044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:22.627823+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:57.585863+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:25.000476+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:50.876688+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:25.859761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:53.563146+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:07:19.304115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3023379428":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1443710231":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6800/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6801/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6801/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6800/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1007625490":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/2895698584":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/497313034":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/3293943475":"2024-03-29T17:59:39.162038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:10:52.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.678+0000 7f3d67200640 1 -- 172.21.15.37:0/3345438805 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f3d4c0a2800 msgr2=0x7f3d4c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:52.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.678+0000 7f3d67200640 1 --2- 172.21.15.37:0/3345438805 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f3d4c0a2800 0x7f3d4c0a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3d580349e0 tx=0x7f3d58034ab0 comp rx=0 tx=0).stop 2024-03-28T18:10:52.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.678+0000 7f3d67200640 1 -- 172.21.15.37:0/3345438805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d681030b0 msgr2=0x7f3d681a1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:52.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.678+0000 7f3d67200640 1 --2- 172.21.15.37:0/3345438805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d681030b0 0x7f3d681a1020 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3d60009fd0 tx=0x7f3d6000ef10 comp rx=0 tx=0).stop 2024-03-28T18:10:52.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.678+0000 7f3d67200640 1 -- 172.21.15.37:0/3345438805 shutdown_connections 2024-03-28T18:10:52.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.678+0000 7f3d67200640 1 --2- 172.21.15.37:0/3345438805 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f3d4c0a2800 0x7f3d4c0a4cc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:52.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.678+0000 7f3d67200640 1 --2- 172.21.15.37:0/3345438805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d681026f0 0x7f3d68199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:52.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.678+0000 7f3d67200640 1 --2- 172.21.15.37:0/3345438805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d681030b0 0x7f3d681a1020 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:52.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.678+0000 7f3d67200640 1 --2- 172.21.15.37:0/3345438805 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d68101500 0x7f3d68199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:52.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.678+0000 7f3d67200640 1 -- 172.21.15.37:0/3345438805 >> 172.21.15.37:0/3345438805 conn(0x7f3d680fca70 msgr2=0x7f3d680738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:52.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.678+0000 7f3d67200640 1 -- 172.21.15.37:0/3345438805 shutdown_connections 2024-03-28T18:10:52.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.678+0000 7f3d67200640 1 -- 172.21.15.37:0/3345438805 wait complete. 2024-03-28T18:10:52.698 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:03:00.517967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:08:18.365827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:08:19.446282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:10:52.699 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:10:52.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.901+0000 7f2853000640 1 -- 172.21.15.37:0/4060804379 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f284c102770 msgr2=0x7f284c104be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:52.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.901+0000 7f2853000640 1 --2- 172.21.15.37:0/4060804379 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f284c102770 0x7f284c104be0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2838009900 tx=0x7f283802bc40 comp rx=0 tx=0).stop 2024-03-28T18:10:52.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.901+0000 7f2853000640 1 -- 172.21.15.37:0/4060804379 shutdown_connections 2024-03-28T18:10:52.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.901+0000 7f2853000640 1 --2- 172.21.15.37:0/4060804379 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f284c102770 0x7f284c104be0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:52.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.901+0000 7f2853000640 1 --2- 172.21.15.37:0/4060804379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f284c105120 0x7f284c1055a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:52.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.901+0000 7f2853000640 1 --2- 172.21.15.37:0/4060804379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f284c0ffe00 0x7f284c102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:52.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.901+0000 7f2853000640 1 -- 172.21.15.37:0/4060804379 >> 172.21.15.37:0/4060804379 conn(0x7f284c0fba20 msgr2=0x7f284c0fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:52.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.901+0000 7f2853000640 1 -- 172.21.15.37:0/4060804379 shutdown_connections 2024-03-28T18:10:52.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.901+0000 7f2853000640 1 -- 172.21.15.37:0/4060804379 wait complete. 2024-03-28T18:10:52.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.902+0000 7f2853000640 1 Processor -- start 2024-03-28T18:10:52.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.903+0000 7f2853000640 1 -- start start 2024-03-28T18:10:52.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.903+0000 7f2853000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f284c0ffe00 0x7f284c19de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:52.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.903+0000 7f2853000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f284c102770 0x7f284c19e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:52.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.903+0000 7f2853000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f284c105120 0x7f284c1a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:52.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.903+0000 7f2853000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284c113850 con 0x7f284c0ffe00 2024-03-28T18:10:52.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.903+0000 7f2853000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284c1136d0 con 0x7f284c105120 2024-03-28T18:10:52.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.903+0000 7f2853000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284c1139d0 con 0x7f284c102770 2024-03-28T18:10:52.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.903+0000 7f2851c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f284c0ffe00 0x7f284c19de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:52.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.903+0000 7f2851c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f284c0ffe00 0x7f284c19de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:48294/0 (socket says 172.21.15.37:48294) 2024-03-28T18:10:52.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.903+0000 7f2851c00640 1 -- 172.21.15.37:0/4076956604 learned_addr learned my addr 172.21.15.37:0/4076956604 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:52.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.903+0000 7f2851200640 1 --2- 172.21.15.37:0/4076956604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f284c102770 0x7f284c19e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:52.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.903+0000 7f2852600640 1 --2- 172.21.15.37:0/4076956604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f284c105120 0x7f284c1a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:52.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.904+0000 7f2851200640 1 -- 172.21.15.37:0/4076956604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f284c105120 msgr2=0x7f284c1a5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:52.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.904+0000 7f2851200640 1 --2- 172.21.15.37:0/4076956604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f284c105120 0x7f284c1a5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:52.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.904+0000 7f2851200640 1 -- 172.21.15.37:0/4076956604 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f284c0ffe00 msgr2=0x7f284c19de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:52.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.904+0000 7f2851200640 1 --2- 172.21.15.37:0/4076956604 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f284c0ffe00 0x7f284c19de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:52.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.904+0000 7f2851200640 1 -- 172.21.15.37:0/4076956604 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284c1a5a90 con 0x7f284c102770 2024-03-28T18:10:52.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.904+0000 7f2851c00640 1 --2- 172.21.15.37:0/4076956604 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f284c0ffe00 0x7f284c19de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:52.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.904+0000 7f2851200640 1 --2- 172.21.15.37:0/4076956604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f284c102770 0x7f284c19e3b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2838036040 tx=0x7f2838005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:52.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.904+0000 7f284a000640 1 -- 172.21.15.37:0/4076956604 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2838041070 con 0x7f284c102770 2024-03-28T18:10:52.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.904+0000 7f284a000640 1 -- 172.21.15.37:0/4076956604 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28380044d0 con 0x7f284c102770 2024-03-28T18:10:52.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.905+0000 7f2853000640 1 -- 172.21.15.37:0/4076956604 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f284c1a5d20 con 0x7f284c102770 2024-03-28T18:10:52.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.905+0000 7f284a000640 1 -- 172.21.15.37:0/4076956604 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2838034440 con 0x7f284c102770 2024-03-28T18:10:52.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.905+0000 7f2853000640 1 -- 172.21.15.37:0/4076956604 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f284c1a61d0 con 0x7f284c102770 2024-03-28T18:10:52.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.906+0000 7f2853000640 1 -- 172.21.15.37:0/4076956604 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f284c0fd510 con 0x7f284c102770 2024-03-28T18:10:52.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.907+0000 7f284a000640 1 -- 172.21.15.37:0/4076956604 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2838044070 con 0x7f284c102770 2024-03-28T18:10:52.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.909+0000 7f284a000640 1 --2- 172.21.15.37:0/4076956604 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f28340a2880 0x7f28340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:52.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.909+0000 7f284a000640 1 -- 172.21.15.37:0/4076956604 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f28380f04d0 con 0x7f284c102770 2024-03-28T18:10:52.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.909+0000 7f2851c00640 1 --2- 172.21.15.37:0/4076956604 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f28340a2880 0x7f28340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:52.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.910+0000 7f2851c00640 1 --2- 172.21.15.37:0/4076956604 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f28340a2880 0x7f28340a4d40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2840011500 tx=0x7f2840005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:52.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:52.917+0000 7f284a000640 1 -- 172.21.15.37:0/4076956604 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f28380bced0 con 0x7f284c102770 2024-03-28T18:10:53.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.132+0000 7f2853000640 1 -- 172.21.15.37:0/4076956604 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f284c02d070 con 0x7f284c102770 2024-03-28T18:10:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.133+0000 7f284a000640 1 -- 172.21.15.37:0/4076956604 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f28380c0f70 con 0x7f284c102770 2024-03-28T18:10:53.133 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 1 2024-03-28T18:10:53.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.136+0000 7f2853000640 1 -- 172.21.15.37:0/4076956604 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f28340a2880 msgr2=0x7f28340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:53.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.136+0000 7f2853000640 1 --2- 172.21.15.37:0/4076956604 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f28340a2880 0x7f28340a4d40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2840011500 tx=0x7f2840005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:53.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.136+0000 7f2853000640 1 -- 172.21.15.37:0/4076956604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f284c102770 msgr2=0x7f284c19e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:53.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.136+0000 7f2853000640 1 --2- 172.21.15.37:0/4076956604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f284c102770 0x7f284c19e3b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2838036040 tx=0x7f2838005d50 comp rx=0 tx=0).stop 2024-03-28T18:10:53.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.136+0000 7f2853000640 1 -- 172.21.15.37:0/4076956604 shutdown_connections 2024-03-28T18:10:53.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.136+0000 7f2853000640 1 --2- 172.21.15.37:0/4076956604 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f28340a2880 0x7f28340a4d40 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:53.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.136+0000 7f2853000640 1 --2- 172.21.15.37:0/4076956604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f284c105120 0x7f284c1a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:53.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.136+0000 7f2853000640 1 --2- 172.21.15.37:0/4076956604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f284c102770 0x7f284c19e3b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:53.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.136+0000 7f2853000640 1 --2- 172.21.15.37:0/4076956604 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f284c0ffe00 0x7f284c19de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:53.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.136+0000 7f2853000640 1 -- 172.21.15.37:0/4076956604 >> 172.21.15.37:0/4076956604 conn(0x7f284c0fba20 msgr2=0x7f284c100740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:53.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.136+0000 7f2853000640 1 -- 172.21.15.37:0/4076956604 shutdown_connections 2024-03-28T18:10:53.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.136+0000 7f2853000640 1 -- 172.21.15.37:0/4076956604 wait complete. 2024-03-28T18:10:53.156 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:10:53.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.354+0000 7ff9dd000640 1 -- 172.21.15.37:0/4123408582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9d8103300 msgr2=0x7ff9d8109b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:53.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.354+0000 7ff9dd000640 1 --2- 172.21.15.37:0/4123408582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9d8103300 0x7ff9d8109b90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009cd0 tx=0x7ff9d002c0f0 comp rx=0 tx=0).stop 2024-03-28T18:10:53.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.354+0000 7ff9dd000640 1 -- 172.21.15.37:0/4123408582 shutdown_connections 2024-03-28T18:10:53.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.354+0000 7ff9dd000640 1 --2- 172.21.15.37:0/4123408582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9d8102940 0x7ff9d8102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:53.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.354+0000 7ff9dd000640 1 --2- 172.21.15.37:0/4123408582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9d8103300 0x7ff9d8109b90 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:53.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.354+0000 7ff9dd000640 1 --2- 172.21.15.37:0/4123408582 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9d80ff970 0x7ff9d80ffd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:53.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.354+0000 7ff9dd000640 1 -- 172.21.15.37:0/4123408582 >> 172.21.15.37:0/4123408582 conn(0x7ff9d80fb2c0 msgr2=0x7ff9d80fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:53.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.354+0000 7ff9dd000640 1 -- 172.21.15.37:0/4123408582 shutdown_connections 2024-03-28T18:10:53.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.354+0000 7ff9dd000640 1 -- 172.21.15.37:0/4123408582 wait complete. 2024-03-28T18:10:53.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.355+0000 7ff9dd000640 1 Processor -- start 2024-03-28T18:10:53.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.356+0000 7ff9dd000640 1 -- start start 2024-03-28T18:10:53.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.356+0000 7ff9dd000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9d80ff970 0x7ff9d81022b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:53.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.356+0000 7ff9dd000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9d8102940 0x7ff9d81008f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:53.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.356+0000 7ff9dd000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9d8103300 0x7ff9d8100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:53.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.356+0000 7ff9dd000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d810d350 con 0x7ff9d80ff970 2024-03-28T18:10:53.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.356+0000 7ff9dd000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d810d1d0 con 0x7ff9d8103300 2024-03-28T18:10:53.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.356+0000 7ff9dd000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d810d4d0 con 0x7ff9d8102940 2024-03-28T18:10:53.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.356+0000 7ff9d5400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9d8102940 0x7ff9d81008f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:53.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.356+0000 7ff9d6800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9d8103300 0x7ff9d8100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:53.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.356+0000 7ff9d5400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9d8102940 0x7ff9d81008f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:59652/0 (socket says 172.21.15.37:59652) 2024-03-28T18:10:53.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.356+0000 7ff9d5400640 1 -- 172.21.15.37:0/203097408 learned_addr learned my addr 172.21.15.37:0/203097408 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:53.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.357+0000 7ff9d5e00640 1 --2- 172.21.15.37:0/203097408 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9d80ff970 0x7ff9d81022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:53.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.357+0000 7ff9d5400640 1 -- 172.21.15.37:0/203097408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9d8103300 msgr2=0x7ff9d8100e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:53.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.357+0000 7ff9d5400640 1 --2- 172.21.15.37:0/203097408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9d8103300 0x7ff9d8100e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:53.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.357+0000 7ff9d5400640 1 -- 172.21.15.37:0/203097408 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9d80ff970 msgr2=0x7ff9d81022b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:53.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.357+0000 7ff9d5400640 1 --2- 172.21.15.37:0/203097408 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9d80ff970 0x7ff9d81022b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:53.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.357+0000 7ff9d5400640 1 -- 172.21.15.37:0/203097408 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d81016c0 con 0x7ff9d8102940 2024-03-28T18:10:53.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.357+0000 7ff9d5e00640 1 --2- 172.21.15.37:0/203097408 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9d80ff970 0x7ff9d81022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:53.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.357+0000 7ff9d6800640 1 --2- 172.21.15.37:0/203097408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9d8103300 0x7ff9d8100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:53.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.357+0000 7ff9d5400640 1 --2- 172.21.15.37:0/203097408 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9d8102940 0x7ff9d81008f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400e9f0 tx=0x7ff9c400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:53.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.357+0000 7ff9cea00640 1 -- 172.21.15.37:0/203097408 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c400cd80 con 0x7ff9d8102940 2024-03-28T18:10:53.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.358+0000 7ff9dd000640 1 -- 172.21.15.37:0/203097408 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d81a1ac0 con 0x7ff9d8102940 2024-03-28T18:10:53.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.358+0000 7ff9cea00640 1 -- 172.21.15.37:0/203097408 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9c4005500 con 0x7ff9d8102940 2024-03-28T18:10:53.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.358+0000 7ff9cea00640 1 -- 172.21.15.37:0/203097408 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4010690 con 0x7ff9d8102940 2024-03-28T18:10:53.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.358+0000 7ff9dd000640 1 -- 172.21.15.37:0/203097408 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d81a2010 con 0x7ff9d8102940 2024-03-28T18:10:53.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.359+0000 7ff9dd000640 1 -- 172.21.15.37:0/203097408 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d8108a70 con 0x7ff9d8102940 2024-03-28T18:10:53.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.362+0000 7ff9cea00640 1 -- 172.21.15.37:0/203097408 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff9c40046c0 con 0x7ff9d8102940 2024-03-28T18:10:53.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.366+0000 7ff9cea00640 1 --2- 172.21.15.37:0/203097408 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff9bc0a2800 0x7ff9bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:53.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.366+0000 7ff9cea00640 1 -- 172.21.15.37:0/203097408 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff9c40cefe0 con 0x7ff9d8102940 2024-03-28T18:10:53.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.366+0000 7ff9d5e00640 1 --2- 172.21.15.37:0/203097408 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff9bc0a2800 0x7ff9bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:53.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.367+0000 7ff9d5e00640 1 --2- 172.21.15.37:0/203097408 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff9bc0a2800 0x7ff9bc0a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8069b10 tx=0x7ff9c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:53.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.372+0000 7ff9cea00640 1 -- 172.21.15.37:0/203097408 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff9c409c030 con 0x7ff9d8102940 2024-03-28T18:10:53.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.601+0000 7ff9dd000640 1 -- 172.21.15.37:0/203097408 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff9d80622a0 con 0x7ff9d8102940 2024-03-28T18:10:53.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.602+0000 7ff9cea00640 1 -- 172.21.15.37:0/203097408 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff9c409b930 con 0x7ff9d8102940 2024-03-28T18:10:53.602 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 32 2024-03-28T18:10:53.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.605+0000 7ff9dd000640 1 -- 172.21.15.37:0/203097408 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff9bc0a2800 msgr2=0x7ff9bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:53.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.605+0000 7ff9dd000640 1 --2- 172.21.15.37:0/203097408 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff9bc0a2800 0x7ff9bc0a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8069b10 tx=0x7ff9c8005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:53.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.605+0000 7ff9dd000640 1 -- 172.21.15.37:0/203097408 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9d8102940 msgr2=0x7ff9d81008f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:53.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.605+0000 7ff9dd000640 1 --2- 172.21.15.37:0/203097408 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9d8102940 0x7ff9d81008f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400e9f0 tx=0x7ff9c400edf0 comp rx=0 tx=0).stop 2024-03-28T18:10:53.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.605+0000 7ff9dd000640 1 -- 172.21.15.37:0/203097408 shutdown_connections 2024-03-28T18:10:53.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.605+0000 7ff9dd000640 1 --2- 172.21.15.37:0/203097408 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff9bc0a2800 0x7ff9bc0a4cc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:53.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.605+0000 7ff9dd000640 1 --2- 172.21.15.37:0/203097408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9d8103300 0x7ff9d8100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:53.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.605+0000 7ff9dd000640 1 --2- 172.21.15.37:0/203097408 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9d8102940 0x7ff9d81008f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:53.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.605+0000 7ff9dd000640 1 --2- 172.21.15.37:0/203097408 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9d80ff970 0x7ff9d81022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:53.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.605+0000 7ff9dd000640 1 -- 172.21.15.37:0/203097408 >> 172.21.15.37:0/203097408 conn(0x7ff9d80fb2c0 msgr2=0x7ff9d8105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:53.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.606+0000 7ff9dd000640 1 -- 172.21.15.37:0/203097408 shutdown_connections 2024-03-28T18:10:53.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.606+0000 7ff9dd000640 1 -- 172.21.15.37:0/203097408 wait complete. 2024-03-28T18:10:53.625 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:10:53.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.835+0000 7faf2b200640 1 -- 172.21.15.37:0/3585384726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf2c1049b0 msgr2=0x7faf2c106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:53.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.835+0000 7faf2b200640 1 --2- 172.21.15.37:0/3585384726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf2c1049b0 0x7faf2c106da0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faf18009900 tx=0x7faf1802bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:53.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.835+0000 7faf2b200640 1 -- 172.21.15.37:0/3585384726 shutdown_connections 2024-03-28T18:10:53.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.835+0000 7faf2b200640 1 --2- 172.21.15.37:0/3585384726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf2c1049b0 0x7faf2c106da0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:53.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.835+0000 7faf2b200640 1 --2- 172.21.15.37:0/3585384726 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf2c1072e0 0x7faf2c1096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:53.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.835+0000 7faf2b200640 1 --2- 172.21.15.37:0/3585384726 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faf2c102080 0x7faf2c104470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:53.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.835+0000 7faf2b200640 1 -- 172.21.15.37:0/3585384726 >> 172.21.15.37:0/3585384726 conn(0x7faf2c0fba10 msgr2=0x7faf2c0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:53.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.835+0000 7faf2b200640 1 -- 172.21.15.37:0/3585384726 shutdown_connections 2024-03-28T18:10:53.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.835+0000 7faf2b200640 1 -- 172.21.15.37:0/3585384726 wait complete. 2024-03-28T18:10:53.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.836+0000 7faf2b200640 1 Processor -- start 2024-03-28T18:10:53.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.836+0000 7faf2b200640 1 -- start start 2024-03-28T18:10:53.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.836+0000 7faf2b200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf2c102080 0x7faf2c199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:53.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.837+0000 7faf2b200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faf2c1049b0 0x7faf2c19a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:53.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.837+0000 7faf2b200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf2c1072e0 0x7faf2c1a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:53.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.837+0000 7faf2b200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c10d1d0 con 0x7faf2c1049b0 2024-03-28T18:10:53.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.837+0000 7faf2b200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c10d050 con 0x7faf2c1072e0 2024-03-28T18:10:53.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.837+0000 7faf2b200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c10d350 con 0x7faf2c102080 2024-03-28T18:10:53.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.837+0000 7faf29e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf2c102080 0x7faf2c199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:53.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.837+0000 7faf29400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faf2c1049b0 0x7faf2c19a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:53.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.837+0000 7faf2a800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf2c1072e0 0x7faf2c1a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:53.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.837+0000 7faf29e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf2c102080 0x7faf2c199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55752/0 (socket says 172.21.15.37:55752) 2024-03-28T18:10:53.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.837+0000 7faf29e00640 1 -- 172.21.15.37:0/4150024134 learned_addr learned my addr 172.21.15.37:0/4150024134 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:53.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.837+0000 7faf29e00640 1 -- 172.21.15.37:0/4150024134 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf2c1072e0 msgr2=0x7faf2c1a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:53.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.837+0000 7faf29e00640 1 --2- 172.21.15.37:0/4150024134 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf2c1072e0 0x7faf2c1a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:53.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.837+0000 7faf29e00640 1 -- 172.21.15.37:0/4150024134 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faf2c1049b0 msgr2=0x7faf2c19a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:53.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.837+0000 7faf29e00640 1 --2- 172.21.15.37:0/4150024134 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faf2c1049b0 0x7faf2c19a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:53.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.837+0000 7faf29e00640 1 -- 172.21.15.37:0/4150024134 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2c1a16e0 con 0x7faf2c102080 2024-03-28T18:10:53.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.837+0000 7faf2a800640 1 --2- 172.21.15.37:0/4150024134 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf2c1072e0 0x7faf2c1a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:53.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.838+0000 7faf29e00640 1 --2- 172.21.15.37:0/4150024134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf2c102080 0x7faf2c199ac0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faf1c00ea60 tx=0x7faf1c00ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:53.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.838+0000 7faf22a00640 1 -- 172.21.15.37:0/4150024134 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1c00cd20 con 0x7faf2c102080 2024-03-28T18:10:53.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.838+0000 7faf22a00640 1 -- 172.21.15.37:0/4150024134 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf1c0035f0 con 0x7faf2c102080 2024-03-28T18:10:53.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.838+0000 7faf2b200640 1 -- 172.21.15.37:0/4150024134 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf2c1a19d0 con 0x7faf2c102080 2024-03-28T18:10:53.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.838+0000 7faf22a00640 1 -- 172.21.15.37:0/4150024134 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1c010690 con 0x7faf2c102080 2024-03-28T18:10:53.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.838+0000 7faf2b200640 1 -- 172.21.15.37:0/4150024134 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf2c1a1f40 con 0x7faf2c102080 2024-03-28T18:10:53.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.839+0000 7faf2b200640 1 -- 172.21.15.37:0/4150024134 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf2c0683b0 con 0x7faf2c102080 2024-03-28T18:10:53.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.841+0000 7faf22a00640 1 -- 172.21.15.37:0/4150024134 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7faf1c0037a0 con 0x7faf2c102080 2024-03-28T18:10:53.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.842+0000 7faf22a00640 1 --2- 172.21.15.37:0/4150024134 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7faf100a2880 0x7faf100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:53.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.843+0000 7faf22a00640 1 -- 172.21.15.37:0/4150024134 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7faf1c0ced30 con 0x7faf2c102080 2024-03-28T18:10:53.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.843+0000 7faf29400640 1 --2- 172.21.15.37:0/4150024134 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7faf100a2880 0x7faf100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:53.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.843+0000 7faf29400640 1 --2- 172.21.15.37:0/4150024134 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7faf100a2880 0x7faf100a4d40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7faf18000c00 tx=0x7faf18005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:53.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:53.846+0000 7faf22a00640 1 -- 172.21.15.37:0/4150024134 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faf1c09bd80 con 0x7faf2c102080 2024-03-28T18:10:53.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:53 smithi037 ceph-mon[31460]: pgmap v396: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T18:10:53.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:53.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:53 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:53.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:53 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/3345438805' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:53.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:53 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/4076956604' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:54.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:53 smithi059 ceph-mon[36190]: pgmap v396: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T18:10:54.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:54.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:53 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:54.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:53 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/3345438805' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:54.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:53 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/4076956604' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:54.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:53 smithi170 ceph-mon[36280]: pgmap v396: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T18:10:54.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:54.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:53 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:54.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:53 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/3345438805' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:54.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:53 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/4076956604' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:54.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.071+0000 7faf2b200640 1 -- 172.21.15.37:0/4150024134 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7faf2c0622a0 con 0x7faf2c102080 2024-03-28T18:10:54.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.071+0000 7faf22a00640 1 -- 172.21.15.37:0/4150024134 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7faf1c09b680 con 0x7faf2c102080 2024-03-28T18:10:54.071 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 64 2024-03-28T18:10:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.074+0000 7faf2b200640 1 -- 172.21.15.37:0/4150024134 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7faf100a2880 msgr2=0x7faf100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.074+0000 7faf2b200640 1 --2- 172.21.15.37:0/4150024134 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7faf100a2880 0x7faf100a4d40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7faf18000c00 tx=0x7faf18005fd0 comp rx=0 tx=0).stop 2024-03-28T18:10:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.074+0000 7faf2b200640 1 -- 172.21.15.37:0/4150024134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf2c102080 msgr2=0x7faf2c199ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.074+0000 7faf2b200640 1 --2- 172.21.15.37:0/4150024134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf2c102080 0x7faf2c199ac0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faf1c00ea60 tx=0x7faf1c00ee60 comp rx=0 tx=0).stop 2024-03-28T18:10:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.074+0000 7faf2b200640 1 -- 172.21.15.37:0/4150024134 shutdown_connections 2024-03-28T18:10:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.074+0000 7faf2b200640 1 --2- 172.21.15.37:0/4150024134 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7faf100a2880 0x7faf100a4d40 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.074+0000 7faf2b200640 1 --2- 172.21.15.37:0/4150024134 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf2c1072e0 0x7faf2c1a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.074+0000 7faf2b200640 1 --2- 172.21.15.37:0/4150024134 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faf2c1049b0 0x7faf2c19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.074+0000 7faf2b200640 1 --2- 172.21.15.37:0/4150024134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf2c102080 0x7faf2c199ac0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.074+0000 7faf2b200640 1 -- 172.21.15.37:0/4150024134 >> 172.21.15.37:0/4150024134 conn(0x7faf2c0fba10 msgr2=0x7faf2c101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.074+0000 7faf2b200640 1 -- 172.21.15.37:0/4150024134 shutdown_connections 2024-03-28T18:10:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.074+0000 7faf2b200640 1 -- 172.21.15.37:0/4150024134 wait complete. 2024-03-28T18:10:54.095 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-28T18:10:54.095 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:10:54.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.322+0000 7f302d800640 1 -- 172.21.15.37:0/708806720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30281030b0 msgr2=0x7f3028109940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:54.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.322+0000 7f302d800640 1 --2- 172.21.15.37:0/708806720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30281030b0 0x7f3028109940 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3020009ca0 tx=0x7f302002c0f0 comp rx=0 tx=0).stop 2024-03-28T18:10:54.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.323+0000 7f302d800640 1 -- 172.21.15.37:0/708806720 shutdown_connections 2024-03-28T18:10:54.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.323+0000 7f302d800640 1 --2- 172.21.15.37:0/708806720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30281030b0 0x7f3028109940 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:54.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.323+0000 7f302d800640 1 --2- 172.21.15.37:0/708806720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30281026f0 0x7f3028102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:54.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.323+0000 7f302d800640 1 --2- 172.21.15.37:0/708806720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3028101500 0x7f3028101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:54.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.323+0000 7f302d800640 1 -- 172.21.15.37:0/708806720 >> 172.21.15.37:0/708806720 conn(0x7f30280fca70 msgr2=0x7f30280feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:54.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.323+0000 7f302d800640 1 -- 172.21.15.37:0/708806720 shutdown_connections 2024-03-28T18:10:54.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.323+0000 7f302d800640 1 -- 172.21.15.37:0/708806720 wait complete. 2024-03-28T18:10:54.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.324+0000 7f302d800640 1 Processor -- start 2024-03-28T18:10:54.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.324+0000 7f302d800640 1 -- start start 2024-03-28T18:10:54.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.324+0000 7f302d800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3028101500 0x7f3028199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:54.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.324+0000 7f302d800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30281026f0 0x7f3028199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:54.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.324+0000 7f302d800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30281030b0 0x7f30281a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:54.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.324+0000 7f302d800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302810d160 con 0x7f30281026f0 2024-03-28T18:10:54.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.324+0000 7f302d800640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302810cfe0 con 0x7f30281030b0 2024-03-28T18:10:54.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.324+0000 7f302d800640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302810d2e0 con 0x7f3028101500 2024-03-28T18:10:54.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.324+0000 7f3027e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3028101500 0x7f3028199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:54.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.324+0000 7f3027e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3028101500 0x7f3028199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55770/0 (socket says 172.21.15.37:55770) 2024-03-28T18:10:54.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.324+0000 7f3027e00640 1 -- 172.21.15.37:0/1038418910 learned_addr learned my addr 172.21.15.37:0/1038418910 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:54.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.325+0000 7f3027400640 1 --2- 172.21.15.37:0/1038418910 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30281026f0 0x7f3028199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:54.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.325+0000 7f302ce00640 1 --2- 172.21.15.37:0/1038418910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30281030b0 0x7f30281a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:54.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.325+0000 7f3027e00640 1 -- 172.21.15.37:0/1038418910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30281030b0 msgr2=0x7f30281a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:54.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.325+0000 7f3027e00640 1 --2- 172.21.15.37:0/1038418910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30281030b0 0x7f30281a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:54.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.325+0000 7f3027e00640 1 -- 172.21.15.37:0/1038418910 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30281026f0 msgr2=0x7f3028199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:54.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.325+0000 7f3027e00640 1 --2- 172.21.15.37:0/1038418910 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30281026f0 0x7f3028199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:54.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.325+0000 7f3027e00640 1 -- 172.21.15.37:0/1038418910 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30281a1630 con 0x7f3028101500 2024-03-28T18:10:54.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.325+0000 7f3027e00640 1 --2- 172.21.15.37:0/1038418910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3028101500 0x7f3028199a10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f301800d950 tx=0x7f301800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:54.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.326+0000 7f3024c00640 1 -- 172.21.15.37:0/1038418910 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3018017020 con 0x7f3028101500 2024-03-28T18:10:54.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.326+0000 7f3024c00640 1 -- 172.21.15.37:0/1038418910 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3018009c30 con 0x7f3028101500 2024-03-28T18:10:54.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.326+0000 7f3024c00640 1 -- 172.21.15.37:0/1038418910 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301800f720 con 0x7f3028101500 2024-03-28T18:10:54.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.326+0000 7f302d800640 1 -- 172.21.15.37:0/1038418910 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30281a1920 con 0x7f3028101500 2024-03-28T18:10:54.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.326+0000 7f302d800640 1 -- 172.21.15.37:0/1038418910 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30281a1e20 con 0x7f3028101500 2024-03-28T18:10:54.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.327+0000 7f302d800640 1 -- 172.21.15.37:0/1038418910 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30280683b0 con 0x7f3028101500 2024-03-28T18:10:54.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.329+0000 7f3024c00640 1 -- 172.21.15.37:0/1038418910 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f301801a020 con 0x7f3028101500 2024-03-28T18:10:54.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.330+0000 7f3024c00640 1 --2- 172.21.15.37:0/1038418910 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f300c0a2880 0x7f300c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:54.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.330+0000 7f3024c00640 1 -- 172.21.15.37:0/1038418910 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f30180ce860 con 0x7f3028101500 2024-03-28T18:10:54.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.330+0000 7f3027400640 1 --2- 172.21.15.37:0/1038418910 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f300c0a2880 0x7f300c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:54.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.331+0000 7f3027400640 1 --2- 172.21.15.37:0/1038418910 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f300c0a2880 0x7f300c0a4d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f302819af20 tx=0x7f3014005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:54.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.338+0000 7f3024c00640 1 -- 172.21.15.37:0/1038418910 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3018021020 con 0x7f3028101500 2024-03-28T18:10:54.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.614+0000 7f302d800640 1 -- 172.21.15.37:0/1038418910 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30281a2100 con 0x7f3028101500 2024-03-28T18:10:54.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.615+0000 7f3024c00640 1 -- 172.21.15.37:0/1038418910 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11383 (secure 0 0 0) 0x7f3018009de0 con 0x7f3028101500 2024-03-28T18:10:54.615 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:54.615 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1926270371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6838","nonce":1926270371},{"type":"v1","addr":"172.21.15.37:6839","nonce":1926270371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:08:19.968894+0000","modified":"2024-03-28T18:10:49.565429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24580,"mds_1":24569,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6837/4034607523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":4034607523},{"type":"v1","addr":"172.21.15.59:6837","nonce":4034607523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6839/1575518279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":1575518279},{"type":"v1","addr":"172.21.15.170:6839","nonce":1575518279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6839/2091629078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":2091629078},{"type":"v1","addr":"172.21.15.59:6839","nonce":2091629078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24580":{"gid":24580,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.37:6841/1923263622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6840","nonce":1923263622},{"type":"v1","addr":"172.21.15.37:6841","nonce":1923263622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24580,"qdb_cluster":[24580,24550,24569]},"id":1}]} 2024-03-28T18:10:54.615 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 23 2024-03-28T18:10:54.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.617+0000 7f302d800640 1 -- 172.21.15.37:0/1038418910 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f300c0a2880 msgr2=0x7f300c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:54.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.617+0000 7f302d800640 1 --2- 172.21.15.37:0/1038418910 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f300c0a2880 0x7f300c0a4d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f302819af20 tx=0x7f3014005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:54.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.617+0000 7f302d800640 1 -- 172.21.15.37:0/1038418910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3028101500 msgr2=0x7f3028199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:54.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.617+0000 7f302d800640 1 --2- 172.21.15.37:0/1038418910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3028101500 0x7f3028199a10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f301800d950 tx=0x7f301800de10 comp rx=0 tx=0).stop 2024-03-28T18:10:54.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.618+0000 7f302d800640 1 -- 172.21.15.37:0/1038418910 shutdown_connections 2024-03-28T18:10:54.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.618+0000 7f302d800640 1 --2- 172.21.15.37:0/1038418910 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f300c0a2880 0x7f300c0a4d40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:54.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.618+0000 7f302d800640 1 --2- 172.21.15.37:0/1038418910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30281030b0 0x7f30281a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:54.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.618+0000 7f302d800640 1 --2- 172.21.15.37:0/1038418910 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30281026f0 0x7f3028199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:54.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.618+0000 7f302d800640 1 --2- 172.21.15.37:0/1038418910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3028101500 0x7f3028199a10 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:54.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.618+0000 7f302d800640 1 -- 172.21.15.37:0/1038418910 >> 172.21.15.37:0/1038418910 conn(0x7f30280fca70 msgr2=0x7f30280738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:54.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.618+0000 7f302d800640 1 -- 172.21.15.37:0/1038418910 shutdown_connections 2024-03-28T18:10:54.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.618+0000 7f302d800640 1 -- 172.21.15.37:0/1038418910 wait complete. 2024-03-28T18:10:54.635 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-28T18:10:54.635 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-28T18:10:54.635 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:10:54.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.814+0000 7fe4ce200640 1 -- 172.21.15.37:0/695207851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4c81016c0 msgr2=0x7fe4c8101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:54.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.814+0000 7fe4ce200640 1 --2- 172.21.15.37:0/695207851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4c81016c0 0x7fe4c8101ae0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8009900 tx=0x7fe4b802bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:54.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.814+0000 7fe4ce200640 1 -- 172.21.15.37:0/695207851 shutdown_connections 2024-03-28T18:10:54.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.814+0000 7fe4ce200640 1 --2- 172.21.15.37:0/695207851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4c8103270 0x7fe4c8109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:54.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.814+0000 7fe4ce200640 1 --2- 172.21.15.37:0/695207851 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4c81028b0 0x7fe4c8102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:54.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.814+0000 7fe4ce200640 1 --2- 172.21.15.37:0/695207851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4c81016c0 0x7fe4c8101ae0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:54.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.814+0000 7fe4ce200640 1 -- 172.21.15.37:0/695207851 >> 172.21.15.37:0/695207851 conn(0x7fe4c80fcc50 msgr2=0x7fe4c80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:54.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.814+0000 7fe4ce200640 1 -- 172.21.15.37:0/695207851 shutdown_connections 2024-03-28T18:10:54.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.814+0000 7fe4ce200640 1 -- 172.21.15.37:0/695207851 wait complete. 2024-03-28T18:10:54.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.815+0000 7fe4ce200640 1 Processor -- start 2024-03-28T18:10:54.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.815+0000 7fe4ce200640 1 -- start start 2024-03-28T18:10:54.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.815+0000 7fe4ce200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4c81016c0 0x7fe4c8199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:54.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.815+0000 7fe4ce200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4c81028b0 0x7fe4c819a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:54.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.815+0000 7fe4ce200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4c8103270 0x7fe4c81a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:54.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.815+0000 7fe4ce200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c810d360 con 0x7fe4c8103270 2024-03-28T18:10:54.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.815+0000 7fe4ce200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c810d1e0 con 0x7fe4c81016c0 2024-03-28T18:10:54.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.815+0000 7fe4ce200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c810d4e0 con 0x7fe4c81028b0 2024-03-28T18:10:54.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.816+0000 7fe4c7400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4c81016c0 0x7fe4c8199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:54.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.816+0000 7fe4c7400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4c81016c0 0x7fe4c8199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:36902/0 (socket says 172.21.15.37:36902) 2024-03-28T18:10:54.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.816+0000 7fe4c7400640 1 -- 172.21.15.37:0/879643298 learned_addr learned my addr 172.21.15.37:0/879643298 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:54.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.816+0000 7fe4c6a00640 1 --2- 172.21.15.37:0/879643298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4c81028b0 0x7fe4c819a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:54.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.816+0000 7fe4c7e00640 1 --2- 172.21.15.37:0/879643298 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4c8103270 0x7fe4c81a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:54.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.816+0000 7fe4c7400640 1 -- 172.21.15.37:0/879643298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4c81028b0 msgr2=0x7fe4c819a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:54.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.816+0000 7fe4c7400640 1 --2- 172.21.15.37:0/879643298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4c81028b0 0x7fe4c819a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:54.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.816+0000 7fe4c7400640 1 -- 172.21.15.37:0/879643298 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4c8103270 msgr2=0x7fe4c81a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:54.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.816+0000 7fe4c7400640 1 --2- 172.21.15.37:0/879643298 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4c8103270 0x7fe4c81a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:54.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.816+0000 7fe4c7400640 1 -- 172.21.15.37:0/879643298 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c81a1840 con 0x7fe4c81016c0 2024-03-28T18:10:54.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.816+0000 7fe4c7e00640 1 --2- 172.21.15.37:0/879643298 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4c8103270 0x7fe4c81a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:54.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.816+0000 7fe4c7400640 1 --2- 172.21.15.37:0/879643298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4c81016c0 0x7fe4c8199c30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8000c00 tx=0x7fe4b802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:54.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.817+0000 7fe4bfe00640 1 -- 172.21.15.37:0/879643298 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b8043020 con 0x7fe4c81016c0 2024-03-28T18:10:54.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.817+0000 7fe4ce200640 1 -- 172.21.15.37:0/879643298 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c81a1ad0 con 0x7fe4c81016c0 2024-03-28T18:10:54.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.817+0000 7fe4bfe00640 1 -- 172.21.15.37:0/879643298 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4b80054e0 con 0x7fe4c81016c0 2024-03-28T18:10:54.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.817+0000 7fe4bfe00640 1 -- 172.21.15.37:0/879643298 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b8003c40 con 0x7fe4c81016c0 2024-03-28T18:10:54.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.817+0000 7fe4ce200640 1 -- 172.21.15.37:0/879643298 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c81a2020 con 0x7fe4c81016c0 2024-03-28T18:10:54.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.818+0000 7fe4ce200640 1 -- 172.21.15.37:0/879643298 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c80688e0 con 0x7fe4c81016c0 2024-03-28T18:10:54.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.819+0000 7fe4bfe00640 1 -- 172.21.15.37:0/879643298 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe4b8046020 con 0x7fe4c81016c0 2024-03-28T18:10:54.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.821+0000 7fe4bfe00640 1 --2- 172.21.15.37:0/879643298 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe4ac0a2800 0x7fe4ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:54.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.821+0000 7fe4bfe00640 1 -- 172.21.15.37:0/879643298 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe4b80f0b90 con 0x7fe4c81016c0 2024-03-28T18:10:54.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.821+0000 7fe4c6a00640 1 --2- 172.21.15.37:0/879643298 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe4ac0a2800 0x7fe4ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:54.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.822+0000 7fe4c6a00640 1 --2- 172.21.15.37:0/879643298 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe4ac0a2800 0x7fe4ac0a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4009bd0 tx=0x7fe4b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:54.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:54.825+0000 7fe4bfe00640 1 -- 172.21.15.37:0/879643298 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe4b804d020 con 0x7fe4c81016c0 2024-03-28T18:10:54.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:54 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/203097408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:54.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:54 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:54.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:54 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:54.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:54 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/4150024134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:55.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.027+0000 7fe4ce200640 1 -- 172.21.15.37:0/879643298 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe4c8074ff0 con 0x7fe4ac0a2800 2024-03-28T18:10:55.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:54 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/203097408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:55.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:54 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:55.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:54 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:55.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:54 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/4150024134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:55.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:54 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/203097408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:55.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:54 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:55.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:54 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:55.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:54 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/4150024134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:55.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.036+0000 7fe4bfe00640 1 -- 172.21.15.37:0/879643298 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259036 (secure 0 0 0) 0x7fe4c8074ff0 con 0x7fe4ac0a2800 2024-03-28T18:10:55.040 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:55.042 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-03-28T18:10:55.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.041+0000 7fe4ce200640 1 -- 172.21.15.37:0/879643298 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe4ac0a2800 msgr2=0x7fe4ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:55.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.041+0000 7fe4ce200640 1 --2- 172.21.15.37:0/879643298 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe4ac0a2800 0x7fe4ac0a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4009bd0 tx=0x7fe4b4005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:55.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.041+0000 7fe4ce200640 1 -- 172.21.15.37:0/879643298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4c81016c0 msgr2=0x7fe4c8199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:55.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.041+0000 7fe4ce200640 1 --2- 172.21.15.37:0/879643298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4c81016c0 0x7fe4c8199c30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8000c00 tx=0x7fe4b802c650 comp rx=0 tx=0).stop 2024-03-28T18:10:55.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.041+0000 7fe4ce200640 1 -- 172.21.15.37:0/879643298 shutdown_connections 2024-03-28T18:10:55.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.041+0000 7fe4ce200640 1 --2- 172.21.15.37:0/879643298 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe4ac0a2800 0x7fe4ac0a4cc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:55.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.041+0000 7fe4ce200640 1 --2- 172.21.15.37:0/879643298 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4c8103270 0x7fe4c81a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:55.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.041+0000 7fe4ce200640 1 --2- 172.21.15.37:0/879643298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4c81028b0 0x7fe4c819a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:55.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.041+0000 7fe4ce200640 1 --2- 172.21.15.37:0/879643298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4c81016c0 0x7fe4c8199c30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:55.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.041+0000 7fe4ce200640 1 -- 172.21.15.37:0/879643298 >> 172.21.15.37:0/879643298 conn(0x7fe4c80fcc50 msgr2=0x7fe4c80738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:55.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.041+0000 7fe4ce200640 1 -- 172.21.15.37:0/879643298 shutdown_connections 2024-03-28T18:10:55.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.041+0000 7fe4ce200640 1 -- 172.21.15.37:0/879643298 wait complete. 2024-03-28T18:10:55.058 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-03-28T18:10:54.014176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657501,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":1037,"num_read_kb":641,"num_write":357,"num_write_kb":3061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1181430,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":788,"ondisk_log_size":788,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377364,"kb_used_data":11572,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564460,"statfs":{"total":1151940427776,"available":1151554007040,"internally_reserved":0,"allocated":11849728,"data_stored":7444197,"data_compressed":113751,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":612,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":32,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004862"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.005929+0000","last_change":"2024-03-28T18:08:20.025576+0000","last_active":"2024-03-28T18:09:44.005929+0000","last_peered":"2024-03-28T18:09:44.005929+0000","last_clean":"2024-03-28T18:09:44.005929+0000","last_became_active":"2024-03-28T18:08:20.025177+0000","last_became_peered":"2024-03-28T18:08:20.025177+0000","last_unstale":"2024-03-28T18:09:44.005929+0000","last_undegraded":"2024-03-28T18:09:44.005929+0000","last_fullsized":"2024-03-28T18:09:44.005929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:14.521170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.018719+0000","last_change":"2024-03-28T18:08:20.027418+0000","last_active":"2024-03-28T18:09:44.018719+0000","last_peered":"2024-03-28T18:09:44.018719+0000","last_clean":"2024-03-28T18:09:44.018719+0000","last_became_active":"2024-03-28T18:08:20.026731+0000","last_became_peered":"2024-03-28T18:08:20.026731+0000","last_unstale":"2024-03-28T18:09:44.018719+0000","last_undegraded":"2024-03-28T18:09:44.018719+0000","last_fullsized":"2024-03-28T18:09:44.018719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:52:58.328738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.001347+0000","last_change":"2024-03-28T18:08:20.029821+0000","last_active":"2024-03-28T18:09:44.001347+0000","last_peered":"2024-03-28T18:09:44.001347+0000","last_clean":"2024-03-28T18:09:44.001347+0000","last_became_active":"2024-03-28T18:08:20.028245+0000","last_became_peered":"2024-03-28T18:08:20.028245+0000","last_unstale":"2024-03-28T18:09:44.001347+0000","last_undegraded":"2024-03-28T18:09:44.001347+0000","last_fullsized":"2024-03-28T18:09:44.001347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:44.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.966606+0000","last_change":"2024-03-28T18:08:20.015682+0000","last_active":"2024-03-28T18:09:43.966606+0000","last_peered":"2024-03-28T18:09:43.966606+0000","last_clean":"2024-03-28T18:09:43.966606+0000","last_became_active":"2024-03-28T18:08:20.015303+0000","last_became_peered":"2024-03-28T18:08:20.015303+0000","last_unstale":"2024-03-28T18:09:43.966606+0000","last_undegraded":"2024-03-28T18:09:43.966606+0000","last_fullsized":"2024-03-28T18:09:43.966606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:57.456052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.998456+0000","last_change":"2024-03-28T18:08:20.020970+0000","last_active":"2024-03-28T18:09:43.998456+0000","last_peered":"2024-03-28T18:09:43.998456+0000","last_clean":"2024-03-28T18:09:43.998456+0000","last_became_active":"2024-03-28T18:08:20.020646+0000","last_became_peered":"2024-03-28T18:08:20.020646+0000","last_unstale":"2024-03-28T18:09:43.998456+0000","last_undegraded":"2024-03-28T18:09:43.998456+0000","last_fullsized":"2024-03-28T18:09:43.998456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:57:20.422546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.970526+0000","last_change":"2024-03-28T18:08:20.022395+0000","last_active":"2024-03-28T18:09:43.970526+0000","last_peered":"2024-03-28T18:09:43.970526+0000","last_clean":"2024-03-28T18:09:43.970526+0000","last_became_active":"2024-03-28T18:08:20.022022+0000","last_became_peered":"2024-03-28T18:08:20.022022+0000","last_unstale":"2024-03-28T18:09:43.970526+0000","last_undegraded":"2024-03-28T18:09:43.970526+0000","last_fullsized":"2024-03-28T18:09:43.970526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:32.590751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.972192+0000","last_change":"2024-03-28T18:08:20.019990+0000","last_active":"2024-03-28T18:09:43.972192+0000","last_peered":"2024-03-28T18:09:43.972192+0000","last_clean":"2024-03-28T18:09:43.972192+0000","last_became_active":"2024-03-28T18:08:20.019618+0000","last_became_peered":"2024-03-28T18:08:20.019618+0000","last_unstale":"2024-03-28T18:09:43.972192+0000","last_undegraded":"2024-03-28T18:09:43.972192+0000","last_fullsized":"2024-03-28T18:09:43.972192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:48.442579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.968407+0000","last_change":"2024-03-28T18:09:15.889503+0000","last_active":"2024-03-28T18:09:43.968407+0000","last_peered":"2024-03-28T18:09:43.968407+0000","last_clean":"2024-03-28T18:09:43.968407+0000","last_became_active":"2024-03-28T18:09:15.888272+0000","last_became_peered":"2024-03-28T18:09:15.888272+0000","last_unstale":"2024-03-28T18:09:43.968407+0000","last_undegraded":"2024-03-28T18:09:43.968407+0000","last_fullsized":"2024-03-28T18:09:43.968407+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:49.222646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.980289+0000","last_change":"2024-03-28T18:08:20.009732+0000","last_active":"2024-03-28T18:09:43.980289+0000","last_peered":"2024-03-28T18:09:43.980289+0000","last_clean":"2024-03-28T18:09:43.980289+0000","last_became_active":"2024-03-28T18:08:20.009288+0000","last_became_peered":"2024-03-28T18:08:20.009288+0000","last_unstale":"2024-03-28T18:09:43.980289+0000","last_undegraded":"2024-03-28T18:09:43.980289+0000","last_fullsized":"2024-03-28T18:09:43.980289+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:20:54.917964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.964990+0000","last_change":"2024-03-28T18:08:20.009687+0000","last_active":"2024-03-28T18:09:43.964990+0000","last_peered":"2024-03-28T18:09:43.964990+0000","last_clean":"2024-03-28T18:09:43.964990+0000","last_became_active":"2024-03-28T18:08:20.008364+0000","last_became_peered":"2024-03-28T18:08:20.008364+0000","last_unstale":"2024-03-28T18:09:43.964990+0000","last_undegraded":"2024-03-28T18:09:43.964990+0000","last_fullsized":"2024-03-28T18:09:43.964990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:08.419202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.978300+0000","last_change":"2024-03-28T18:08:20.011508+0000","last_active":"2024-03-28T18:09:43.978300+0000","last_peered":"2024-03-28T18:09:43.978300+0000","last_clean":"2024-03-28T18:09:43.978300+0000","last_became_active":"2024-03-28T18:08:20.011117+0000","last_became_peered":"2024-03-28T18:08:20.011117+0000","last_unstale":"2024-03-28T18:09:43.978300+0000","last_undegraded":"2024-03-28T18:09:43.978300+0000","last_fullsized":"2024-03-28T18:09:43.978300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:06:19.278931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.966450+0000","last_change":"2024-03-28T18:08:20.011403+0000","last_active":"2024-03-28T18:09:43.966450+0000","last_peered":"2024-03-28T18:09:43.966450+0000","last_clean":"2024-03-28T18:09:43.966450+0000","last_became_active":"2024-03-28T18:08:20.011025+0000","last_became_peered":"2024-03-28T18:08:20.011025+0000","last_unstale":"2024-03-28T18:09:43.966450+0000","last_undegraded":"2024-03-28T18:09:43.966450+0000","last_fullsized":"2024-03-28T18:09:43.966450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:13.599432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.005479+0000","last_change":"2024-03-28T18:08:20.001664+0000","last_active":"2024-03-28T18:09:44.005479+0000","last_peered":"2024-03-28T18:09:44.005479+0000","last_clean":"2024-03-28T18:09:44.005479+0000","last_became_active":"2024-03-28T18:08:20.000910+0000","last_became_peered":"2024-03-28T18:08:20.000910+0000","last_unstale":"2024-03-28T18:09:44.005479+0000","last_undegraded":"2024-03-28T18:09:44.005479+0000","last_fullsized":"2024-03-28T18:09:44.005479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:00:23.796302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.008215+0000","last_change":"2024-03-28T18:08:20.027174+0000","last_active":"2024-03-28T18:09:44.008215+0000","last_peered":"2024-03-28T18:09:44.008215+0000","last_clean":"2024-03-28T18:09:44.008215+0000","last_became_active":"2024-03-28T18:08:20.026113+0000","last_became_peered":"2024-03-28T18:08:20.026113+0000","last_unstale":"2024-03-28T18:09:44.008215+0000","last_undegraded":"2024-03-28T18:09:44.008215+0000","last_fullsized":"2024-03-28T18:09:44.008215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:55:00.061696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.010537+0000","last_change":"2024-03-28T18:08:20.020060+0000","last_active":"2024-03-28T18:09:44.010537+0000","last_peered":"2024-03-28T18:09:44.010537+0000","last_clean":"2024-03-28T18:09:44.010537+0000","last_became_active":"2024-03-28T18:08:20.019691+0000","last_became_peered":"2024-03-28T18:08:20.019691+0000","last_unstale":"2024-03-28T18:09:44.010537+0000","last_undegraded":"2024-03-28T18:09:44.010537+0000","last_fullsized":"2024-03-28T18:09:44.010537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:27.889737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.003319+0000","last_change":"2024-03-28T18:08:20.024239+0000","last_active":"2024-03-28T18:09:44.003319+0000","last_peered":"2024-03-28T18:09:44.003319+0000","last_clean":"2024-03-28T18:09:44.003319+0000","last_became_active":"2024-03-28T18:08:20.023874+0000","last_became_peered":"2024-03-28T18:08:20.023874+0000","last_unstale":"2024-03-28T18:09:44.003319+0000","last_undegraded":"2024-03-28T18:09:44.003319+0000","last_fullsized":"2024-03-28T18:09:44.003319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:05:32.903195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.008671+0000","last_change":"2024-03-28T18:08:20.033322+0000","last_active":"2024-03-28T18:09:44.008671+0000","last_peered":"2024-03-28T18:09:44.008671+0000","last_clean":"2024-03-28T18:09:44.008671+0000","last_became_active":"2024-03-28T18:08:20.033048+0000","last_became_peered":"2024-03-28T18:08:20.033048+0000","last_unstale":"2024-03-28T18:09:44.008671+0000","last_undegraded":"2024-03-28T18:09:44.008671+0000","last_fullsized":"2024-03-28T18:09:44.008671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:18:59.234141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.986420+0000","last_change":"2024-03-28T18:08:20.022093+0000","last_active":"2024-03-28T18:09:43.986420+0000","last_peered":"2024-03-28T18:09:43.986420+0000","last_clean":"2024-03-28T18:09:43.986420+0000","last_became_active":"2024-03-28T18:08:20.021411+0000","last_became_peered":"2024-03-28T18:08:20.021411+0000","last_unstale":"2024-03-28T18:09:43.986420+0000","last_undegraded":"2024-03-28T18:09:43.986420+0000","last_fullsized":"2024-03-28T18:09:43.986420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:09:40.366628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.002040+0000","last_change":"2024-03-28T18:08:20.028893+0000","last_active":"2024-03-28T18:09:44.002040+0000","last_peered":"2024-03-28T18:09:44.002040+0000","last_clean":"2024-03-28T18:09:44.002040+0000","last_became_active":"2024-03-28T18:08:20.028634+0000","last_became_peered":"2024-03-28T18:08:20.028634+0000","last_unstale":"2024-03-28T18:09:44.002040+0000","last_undegraded":"2024-03-28T18:09:44.002040+0000","last_fullsized":"2024-03-28T18:09:44.002040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:32:27.454406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":46,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.734956+0000","last_change":"2024-03-28T18:09:15.734404+0000","last_active":"2024-03-28T18:09:15.734956+0000","last_peered":"2024-03-28T18:09:15.734956+0000","last_clean":"2024-03-28T18:09:15.734956+0000","last_became_active":"2024-03-28T18:09:15.723870+0000","last_became_peered":"2024-03-28T18:09:15.723870+0000","last_unstale":"2024-03-28T18:09:15.734956+0000","last_undegraded":"2024-03-28T18:09:15.734956+0000","last_fullsized":"2024-03-28T18:09:15.734956+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:32.736966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.004329+0000","last_change":"2024-03-28T18:08:20.027855+0000","last_active":"2024-03-28T18:09:44.004329+0000","last_peered":"2024-03-28T18:09:44.004329+0000","last_clean":"2024-03-28T18:09:44.004329+0000","last_became_active":"2024-03-28T18:08:20.027565+0000","last_became_peered":"2024-03-28T18:08:20.027565+0000","last_unstale":"2024-03-28T18:09:44.004329+0000","last_undegraded":"2024-03-28T18:09:44.004329+0000","last_fullsized":"2024-03-28T18:09:44.004329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:36:37.596573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.725789+0000","last_change":"2024-03-28T18:08:19.499055+0000","last_active":"2024-03-28T18:09:15.725789+0000","last_peered":"2024-03-28T18:09:15.725789+0000","last_clean":"2024-03-28T18:09:15.725789+0000","last_became_active":"2024-03-28T18:08:19.497964+0000","last_became_peered":"2024-03-28T18:08:19.497964+0000","last_unstale":"2024-03-28T18:09:15.725789+0000","last_undegraded":"2024-03-28T18:09:15.725789+0000","last_fullsized":"2024-03-28T18:09:15.725789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:38:45.620406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.012271+0000","last_change":"2024-03-28T18:08:20.028967+0000","last_active":"2024-03-28T18:09:44.012271+0000","last_peered":"2024-03-28T18:09:44.012271+0000","last_clean":"2024-03-28T18:09:44.012271+0000","last_became_active":"2024-03-28T18:08:20.028607+0000","last_became_peered":"2024-03-28T18:08:20.028607+0000","last_unstale":"2024-03-28T18:09:44.012271+0000","last_undegraded":"2024-03-28T18:09:44.012271+0000","last_fullsized":"2024-03-28T18:09:44.012271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:21:23.994077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.893027+0000","last_change":"2024-03-28T18:09:15.892649+0000","last_active":"2024-03-28T18:09:15.893027+0000","last_peered":"2024-03-28T18:09:15.893027+0000","last_clean":"2024-03-28T18:09:15.893027+0000","last_became_active":"2024-03-28T18:09:15.888520+0000","last_became_peered":"2024-03-28T18:09:15.888520+0000","last_unstale":"2024-03-28T18:09:15.893027+0000","last_undegraded":"2024-03-28T18:09:15.893027+0000","last_fullsized":"2024-03-28T18:09:15.893027+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:34.085738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.013841+0000","last_change":"2024-03-28T18:08:20.009878+0000","last_active":"2024-03-28T18:09:44.013841+0000","last_peered":"2024-03-28T18:09:44.013841+0000","last_clean":"2024-03-28T18:09:44.013841+0000","last_became_active":"2024-03-28T18:08:20.009534+0000","last_became_peered":"2024-03-28T18:08:20.009534+0000","last_unstale":"2024-03-28T18:09:44.013841+0000","last_undegraded":"2024-03-28T18:09:44.013841+0000","last_fullsized":"2024-03-28T18:09:44.013841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:58.975854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716112+0000","last_change":"2024-03-28T18:08:19.472613+0000","last_active":"2024-03-28T18:09:15.716112+0000","last_peered":"2024-03-28T18:09:15.716112+0000","last_clean":"2024-03-28T18:09:15.716112+0000","last_became_active":"2024-03-28T18:08:19.472186+0000","last_became_peered":"2024-03-28T18:08:19.472186+0000","last_unstale":"2024-03-28T18:09:15.716112+0000","last_undegraded":"2024-03-28T18:09:15.716112+0000","last_fullsized":"2024-03-28T18:09:15.716112+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:50.807609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.974502+0000","last_change":"2024-03-28T18:08:20.008242+0000","last_active":"2024-03-28T18:09:43.974502+0000","last_peered":"2024-03-28T18:09:43.974502+0000","last_clean":"2024-03-28T18:09:43.974502+0000","last_became_active":"2024-03-28T18:08:20.007942+0000","last_became_peered":"2024-03-28T18:08:20.007942+0000","last_unstale":"2024-03-28T18:09:43.974502+0000","last_undegraded":"2024-03-28T18:09:43.974502+0000","last_fullsized":"2024-03-28T18:09:43.974502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:23.288978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'8","reported_seq":179,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:50.579007+0000","last_change":"2024-03-28T18:08:19.480869+0000","last_active":"2024-03-28T18:10:50.579007+0000","last_peered":"2024-03-28T18:10:50.579007+0000","last_clean":"2024-03-28T18:10:50.579007+0000","last_became_active":"2024-03-28T18:08:19.480441+0000","last_became_peered":"2024-03-28T18:08:19.480441+0000","last_unstale":"2024-03-28T18:10:50.579007+0000","last_undegraded":"2024-03-28T18:10:50.579007+0000","last_fullsized":"2024-03-28T18:10:50.579007+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:26.640605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1460,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":151,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.960050+0000","last_change":"2024-03-28T18:08:20.021530+0000","last_active":"2024-03-28T18:09:43.960050+0000","last_peered":"2024-03-28T18:09:43.960050+0000","last_clean":"2024-03-28T18:09:43.960050+0000","last_became_active":"2024-03-28T18:08:20.021179+0000","last_became_peered":"2024-03-28T18:08:20.021179+0000","last_unstale":"2024-03-28T18:09:43.960050+0000","last_undegraded":"2024-03-28T18:09:43.960050+0000","last_fullsized":"2024-03-28T18:09:43.960050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:43:07.300300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887313+0000","last_change":"2024-03-28T18:08:19.494597+0000","last_active":"2024-03-28T18:09:15.887313+0000","last_peered":"2024-03-28T18:09:15.887313+0000","last_clean":"2024-03-28T18:09:15.887313+0000","last_became_active":"2024-03-28T18:08:19.494178+0000","last_became_peered":"2024-03-28T18:08:19.494178+0000","last_unstale":"2024-03-28T18:09:15.887313+0000","last_undegraded":"2024-03-28T18:09:15.887313+0000","last_fullsized":"2024-03-28T18:09:15.887313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:08:10.257768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.991904+0000","last_change":"2024-03-28T18:08:20.021062+0000","last_active":"2024-03-28T18:09:43.991904+0000","last_peered":"2024-03-28T18:09:43.991904+0000","last_clean":"2024-03-28T18:09:43.991904+0000","last_became_active":"2024-03-28T18:08:20.020758+0000","last_became_peered":"2024-03-28T18:08:20.020758+0000","last_unstale":"2024-03-28T18:09:43.991904+0000","last_undegraded":"2024-03-28T18:09:43.991904+0000","last_fullsized":"2024-03-28T18:09:43.991904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:47:39.656513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.889759+0000","last_change":"2024-03-28T18:08:19.480189+0000","last_active":"2024-03-28T18:09:15.889759+0000","last_peered":"2024-03-28T18:09:15.889759+0000","last_clean":"2024-03-28T18:09:15.889759+0000","last_became_active":"2024-03-28T18:08:19.479812+0000","last_became_peered":"2024-03-28T18:08:19.479812+0000","last_unstale":"2024-03-28T18:09:15.889759+0000","last_undegraded":"2024-03-28T18:09:15.889759+0000","last_fullsized":"2024-03-28T18:09:15.889759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:46.408565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.007167+0000","last_change":"2024-03-28T18:08:20.023635+0000","last_active":"2024-03-28T18:09:44.007167+0000","last_peered":"2024-03-28T18:09:44.007167+0000","last_clean":"2024-03-28T18:09:44.007167+0000","last_became_active":"2024-03-28T18:08:20.023199+0000","last_became_peered":"2024-03-28T18:08:20.023199+0000","last_unstale":"2024-03-28T18:09:44.007167+0000","last_undegraded":"2024-03-28T18:09:44.007167+0000","last_fullsized":"2024-03-28T18:09:44.007167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:12.477899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.886913+0000","last_change":"2024-03-28T18:08:19.480858+0000","last_active":"2024-03-28T18:09:15.886913+0000","last_peered":"2024-03-28T18:09:15.886913+0000","last_clean":"2024-03-28T18:09:15.886913+0000","last_became_active":"2024-03-28T18:08:19.480418+0000","last_became_peered":"2024-03-28T18:08:19.480418+0000","last_unstale":"2024-03-28T18:09:15.886913+0000","last_undegraded":"2024-03-28T18:09:15.886913+0000","last_fullsized":"2024-03-28T18:09:15.886913+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:59:57.788915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.019886+0000","last_change":"2024-03-28T18:08:20.031719+0000","last_active":"2024-03-28T18:09:44.019886+0000","last_peered":"2024-03-28T18:09:44.019886+0000","last_clean":"2024-03-28T18:09:44.019886+0000","last_became_active":"2024-03-28T18:08:20.029482+0000","last_became_peered":"2024-03-28T18:08:20.029482+0000","last_unstale":"2024-03-28T18:09:44.019886+0000","last_undegraded":"2024-03-28T18:09:44.019886+0000","last_fullsized":"2024-03-28T18:09:44.019886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:13.226596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'5","reported_seq":170,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:48.619711+0000","last_change":"2024-03-28T18:08:19.483346+0000","last_active":"2024-03-28T18:10:48.619711+0000","last_peered":"2024-03-28T18:10:48.619711+0000","last_clean":"2024-03-28T18:10:48.619711+0000","last_became_active":"2024-03-28T18:08:19.483016+0000","last_became_peered":"2024-03-28T18:08:19.483016+0000","last_unstale":"2024-03-28T18:10:48.619711+0000","last_undegraded":"2024-03-28T18:10:48.619711+0000","last_fullsized":"2024-03-28T18:10:48.619711+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:34:33.008477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":145,"num_read_kb":145,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.984960+0000","last_change":"2024-03-28T18:08:20.020478+0000","last_active":"2024-03-28T18:09:43.984960+0000","last_peered":"2024-03-28T18:09:43.984960+0000","last_clean":"2024-03-28T18:09:43.984960+0000","last_became_active":"2024-03-28T18:08:20.020054+0000","last_became_peered":"2024-03-28T18:08:20.020054+0000","last_unstale":"2024-03-28T18:09:43.984960+0000","last_undegraded":"2024-03-28T18:09:43.984960+0000","last_fullsized":"2024-03-28T18:09:43.984960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:13:43.655549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716206+0000","last_change":"2024-03-28T18:08:19.479386+0000","last_active":"2024-03-28T18:09:15.716206+0000","last_peered":"2024-03-28T18:09:15.716206+0000","last_clean":"2024-03-28T18:09:15.716206+0000","last_became_active":"2024-03-28T18:08:19.478985+0000","last_became_peered":"2024-03-28T18:08:19.478985+0000","last_unstale":"2024-03-28T18:09:15.716206+0000","last_undegraded":"2024-03-28T18:09:15.716206+0000","last_fullsized":"2024-03-28T18:09:15.716206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:37:22.359475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.007695+0000","last_change":"2024-03-28T18:08:20.021790+0000","last_active":"2024-03-28T18:09:44.007695+0000","last_peered":"2024-03-28T18:09:44.007695+0000","last_clean":"2024-03-28T18:09:44.007695+0000","last_became_active":"2024-03-28T18:08:20.021031+0000","last_became_peered":"2024-03-28T18:08:20.021031+0000","last_unstale":"2024-03-28T18:09:44.007695+0000","last_undegraded":"2024-03-28T18:09:44.007695+0000","last_fullsized":"2024-03-28T18:09:44.007695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:59:50.897692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887257+0000","last_change":"2024-03-28T18:08:19.481576+0000","last_active":"2024-03-28T18:09:15.887257+0000","last_peered":"2024-03-28T18:09:15.887257+0000","last_clean":"2024-03-28T18:09:15.887257+0000","last_became_active":"2024-03-28T18:08:19.481226+0000","last_became_peered":"2024-03-28T18:08:19.481226+0000","last_unstale":"2024-03-28T18:09:15.887257+0000","last_undegraded":"2024-03-28T18:09:15.887257+0000","last_fullsized":"2024-03-28T18:09:15.887257+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:54:44.677343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.013852+0000","last_change":"2024-03-28T18:09:15.890672+0000","last_active":"2024-03-28T18:09:44.013852+0000","last_peered":"2024-03-28T18:09:44.013852+0000","last_clean":"2024-03-28T18:09:44.013852+0000","last_became_active":"2024-03-28T18:09:15.890342+0000","last_became_peered":"2024-03-28T18:09:15.890342+0000","last_unstale":"2024-03-28T18:09:44.013852+0000","last_undegraded":"2024-03-28T18:09:44.013852+0000","last_fullsized":"2024-03-28T18:09:44.013852+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:08.268067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'34","reported_seq":354,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:49.577466+0000","last_change":"2024-03-28T18:08:19.477559+0000","last_active":"2024-03-28T18:10:49.577466+0000","last_peered":"2024-03-28T18:10:49.577466+0000","last_clean":"2024-03-28T18:10:49.577466+0000","last_became_active":"2024-03-28T18:08:19.477217+0000","last_became_peered":"2024-03-28T18:08:19.477217+0000","last_unstale":"2024-03-28T18:10:49.577466+0000","last_undegraded":"2024-03-28T18:10:49.577466+0000","last_fullsized":"2024-03-28T18:10:49.577466+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:37.071671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62451,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":300,"num_read_kb":211,"num_write":35,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.955611+0000","last_change":"2024-03-28T18:08:20.011303+0000","last_active":"2024-03-28T18:09:43.955611+0000","last_peered":"2024-03-28T18:09:43.955611+0000","last_clean":"2024-03-28T18:09:43.955611+0000","last_became_active":"2024-03-28T18:08:20.010997+0000","last_became_peered":"2024-03-28T18:08:20.010997+0000","last_unstale":"2024-03-28T18:09:43.955611+0000","last_undegraded":"2024-03-28T18:09:43.955611+0000","last_fullsized":"2024-03-28T18:09:43.955611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:10:56.036132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.718063+0000","last_change":"2024-03-28T18:08:19.477342+0000","last_active":"2024-03-28T18:09:15.718063+0000","last_peered":"2024-03-28T18:09:15.718063+0000","last_clean":"2024-03-28T18:09:15.718063+0000","last_became_active":"2024-03-28T18:08:19.476984+0000","last_became_peered":"2024-03-28T18:08:19.476984+0000","last_unstale":"2024-03-28T18:09:15.718063+0000","last_undegraded":"2024-03-28T18:09:15.718063+0000","last_fullsized":"2024-03-28T18:09:15.718063+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:15:42.606213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.017873+0000","last_change":"2024-03-28T18:08:20.030524+0000","last_active":"2024-03-28T18:09:44.017873+0000","last_peered":"2024-03-28T18:09:44.017873+0000","last_clean":"2024-03-28T18:09:44.017873+0000","last_became_active":"2024-03-28T18:08:20.030227+0000","last_became_peered":"2024-03-28T18:08:20.030227+0000","last_unstale":"2024-03-28T18:09:44.017873+0000","last_undegraded":"2024-03-28T18:09:44.017873+0000","last_fullsized":"2024-03-28T18:09:44.017873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:07:41.763739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.886737+0000","last_change":"2024-03-28T18:08:19.481169+0000","last_active":"2024-03-28T18:09:15.886737+0000","last_peered":"2024-03-28T18:09:15.886737+0000","last_clean":"2024-03-28T18:09:15.886737+0000","last_became_active":"2024-03-28T18:08:19.480634+0000","last_became_peered":"2024-03-28T18:08:19.480634+0000","last_unstale":"2024-03-28T18:09:15.886737+0000","last_undegraded":"2024-03-28T18:09:15.886737+0000","last_fullsized":"2024-03-28T18:09:15.886737+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:39.265215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.006975+0000","last_change":"2024-03-28T18:08:20.015163+0000","last_active":"2024-03-28T18:09:44.006975+0000","last_peered":"2024-03-28T18:09:44.006975+0000","last_clean":"2024-03-28T18:09:44.006975+0000","last_became_active":"2024-03-28T18:08:20.014765+0000","last_became_peered":"2024-03-28T18:08:20.014765+0000","last_unstale":"2024-03-28T18:09:44.006975+0000","last_undegraded":"2024-03-28T18:09:44.006975+0000","last_fullsized":"2024-03-28T18:09:44.006975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:51.035532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'3","reported_seq":169,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:47.626894+0000","last_change":"2024-03-28T18:08:19.479086+0000","last_active":"2024-03-28T18:10:47.626894+0000","last_peered":"2024-03-28T18:10:47.626894+0000","last_clean":"2024-03-28T18:10:47.626894+0000","last_became_active":"2024-03-28T18:08:19.478748+0000","last_became_peered":"2024-03-28T18:08:19.478748+0000","last_unstale":"2024-03-28T18:10:47.626894+0000","last_undegraded":"2024-03-28T18:10:47.626894+0000","last_fullsized":"2024-03-28T18:10:47.626894+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:46:59.566998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.973667+0000","last_change":"2024-03-28T18:09:15.888751+0000","last_active":"2024-03-28T18:09:43.973667+0000","last_peered":"2024-03-28T18:09:43.973667+0000","last_clean":"2024-03-28T18:09:43.973667+0000","last_became_active":"2024-03-28T18:09:15.888350+0000","last_became_peered":"2024-03-28T18:09:15.888350+0000","last_unstale":"2024-03-28T18:09:43.973667+0000","last_undegraded":"2024-03-28T18:09:43.973667+0000","last_fullsized":"2024-03-28T18:09:43.973667+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:38:05.822205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":60,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.733834+0000","last_change":"2024-03-28T18:09:15.733434+0000","last_active":"2024-03-28T18:09:15.733834+0000","last_peered":"2024-03-28T18:09:15.733834+0000","last_clean":"2024-03-28T18:09:15.733834+0000","last_became_active":"2024-03-28T18:09:15.722437+0000","last_became_peered":"2024-03-28T18:09:15.722437+0000","last_unstale":"2024-03-28T18:09:15.733834+0000","last_undegraded":"2024-03-28T18:09:15.733834+0000","last_fullsized":"2024-03-28T18:09:15.733834+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:35:18.313676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.971701+0000","last_change":"2024-03-28T18:08:20.007457+0000","last_active":"2024-03-28T18:09:43.971701+0000","last_peered":"2024-03-28T18:09:43.971701+0000","last_clean":"2024-03-28T18:09:43.971701+0000","last_became_active":"2024-03-28T18:08:20.007090+0000","last_became_peered":"2024-03-28T18:08:20.007090+0000","last_unstale":"2024-03-28T18:09:43.971701+0000","last_undegraded":"2024-03-28T18:09:43.971701+0000","last_fullsized":"2024-03-28T18:09:43.971701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:22:45.702706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716396+0000","last_change":"2024-03-28T18:08:19.475104+0000","last_active":"2024-03-28T18:09:15.716396+0000","last_peered":"2024-03-28T18:09:15.716396+0000","last_clean":"2024-03-28T18:09:15.716396+0000","last_became_active":"2024-03-28T18:08:19.474754+0000","last_became_peered":"2024-03-28T18:08:19.474754+0000","last_unstale":"2024-03-28T18:09:15.716396+0000","last_undegraded":"2024-03-28T18:09:15.716396+0000","last_fullsized":"2024-03-28T18:09:15.716396+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:19.085631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.968043+0000","last_change":"2024-03-28T18:08:20.009347+0000","last_active":"2024-03-28T18:09:43.968043+0000","last_peered":"2024-03-28T18:09:43.968043+0000","last_clean":"2024-03-28T18:09:43.968043+0000","last_became_active":"2024-03-28T18:08:20.009012+0000","last_became_peered":"2024-03-28T18:08:20.009012+0000","last_unstale":"2024-03-28T18:09:43.968043+0000","last_undegraded":"2024-03-28T18:09:43.968043+0000","last_fullsized":"2024-03-28T18:09:43.968043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:04:26.625830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.721956+0000","last_change":"2024-03-28T18:08:19.482414+0000","last_active":"2024-03-28T18:09:15.721956+0000","last_peered":"2024-03-28T18:09:15.721956+0000","last_clean":"2024-03-28T18:09:15.721956+0000","last_became_active":"2024-03-28T18:08:19.481830+0000","last_became_peered":"2024-03-28T18:08:19.481830+0000","last_unstale":"2024-03-28T18:09:15.721956+0000","last_undegraded":"2024-03-28T18:09:15.721956+0000","last_fullsized":"2024-03-28T18:09:15.721956+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:27:14.259355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.001824+0000","last_change":"2024-03-28T18:08:20.030687+0000","last_active":"2024-03-28T18:09:44.001824+0000","last_peered":"2024-03-28T18:09:44.001824+0000","last_clean":"2024-03-28T18:09:44.001824+0000","last_became_active":"2024-03-28T18:08:20.030308+0000","last_became_peered":"2024-03-28T18:08:20.030308+0000","last_unstale":"2024-03-28T18:09:44.001824+0000","last_undegraded":"2024-03-28T18:09:44.001824+0000","last_fullsized":"2024-03-28T18:09:44.001824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:31.745952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716000+0000","last_change":"2024-03-28T18:08:19.494446+0000","last_active":"2024-03-28T18:09:15.716000+0000","last_peered":"2024-03-28T18:09:15.716000+0000","last_clean":"2024-03-28T18:09:15.716000+0000","last_became_active":"2024-03-28T18:08:19.494036+0000","last_became_peered":"2024-03-28T18:08:19.494036+0000","last_unstale":"2024-03-28T18:09:15.716000+0000","last_undegraded":"2024-03-28T18:09:15.716000+0000","last_fullsized":"2024-03-28T18:09:15.716000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:42:53.660741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.009456+0000","last_change":"2024-03-28T18:08:20.025706+0000","last_active":"2024-03-28T18:09:44.009456+0000","last_peered":"2024-03-28T18:09:44.009456+0000","last_clean":"2024-03-28T18:09:44.009456+0000","last_became_active":"2024-03-28T18:08:20.025343+0000","last_became_peered":"2024-03-28T18:08:20.025343+0000","last_unstale":"2024-03-28T18:09:44.009456+0000","last_undegraded":"2024-03-28T18:09:44.009456+0000","last_fullsized":"2024-03-28T18:09:44.009456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:32:27.883940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.963975+0000","last_change":"2024-03-28T18:08:20.009636+0000","last_active":"2024-03-28T18:09:43.963975+0000","last_peered":"2024-03-28T18:09:43.963975+0000","last_clean":"2024-03-28T18:09:43.963975+0000","last_became_active":"2024-03-28T18:08:20.009178+0000","last_became_peered":"2024-03-28T18:08:20.009178+0000","last_unstale":"2024-03-28T18:09:43.963975+0000","last_undegraded":"2024-03-28T18:09:43.963975+0000","last_fullsized":"2024-03-28T18:09:43.963975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:28.461411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":169,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:48.628017+0000","last_change":"2024-03-28T18:08:19.501287+0000","last_active":"2024-03-28T18:10:48.628017+0000","last_peered":"2024-03-28T18:10:48.628017+0000","last_clean":"2024-03-28T18:10:48.628017+0000","last_became_active":"2024-03-28T18:08:19.500920+0000","last_became_peered":"2024-03-28T18:08:19.500920+0000","last_unstale":"2024-03-28T18:10:48.628017+0000","last_undegraded":"2024-03-28T18:10:48.628017+0000","last_fullsized":"2024-03-28T18:10:48.628017+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:29.882164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":145,"num_read_kb":145,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.013107+0000","last_change":"2024-03-28T18:08:20.020811+0000","last_active":"2024-03-28T18:09:44.013107+0000","last_peered":"2024-03-28T18:09:44.013107+0000","last_clean":"2024-03-28T18:09:44.013107+0000","last_became_active":"2024-03-28T18:08:20.020381+0000","last_became_peered":"2024-03-28T18:08:20.020381+0000","last_unstale":"2024-03-28T18:09:44.013107+0000","last_undegraded":"2024-03-28T18:09:44.013107+0000","last_fullsized":"2024-03-28T18:09:44.013107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:10:35.999858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887764+0000","last_change":"2024-03-28T18:08:19.478020+0000","last_active":"2024-03-28T18:09:15.887764+0000","last_peered":"2024-03-28T18:09:15.887764+0000","last_clean":"2024-03-28T18:09:15.887764+0000","last_became_active":"2024-03-28T18:08:19.477627+0000","last_became_peered":"2024-03-28T18:08:19.477627+0000","last_unstale":"2024-03-28T18:09:15.887764+0000","last_undegraded":"2024-03-28T18:09:15.887764+0000","last_fullsized":"2024-03-28T18:09:15.887764+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:00:01.649974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":354,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.715885+0000","last_change":"2024-03-28T18:06:58.750932+0000","last_active":"2024-03-28T18:09:15.715885+0000","last_peered":"2024-03-28T18:09:15.715885+0000","last_clean":"2024-03-28T18:09:15.715885+0000","last_became_active":"2024-03-28T18:06:58.750309+0000","last_became_peered":"2024-03-28T18:06:58.750309+0000","last_unstale":"2024-03-28T18:09:15.715885+0000","last_undegraded":"2024-03-28T18:09:15.715885+0000","last_fullsized":"2024-03-28T18:09:15.715885+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:03:00.832683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:03:00.832683+0000","last_clean_scrub_stamp":"2024-03-28T18:03:00.832683+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:46.001886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.002070+0000","last_change":"2024-03-28T18:08:20.009573+0000","last_active":"2024-03-28T18:09:44.002070+0000","last_peered":"2024-03-28T18:09:44.002070+0000","last_clean":"2024-03-28T18:09:44.002070+0000","last_became_active":"2024-03-28T18:08:20.009143+0000","last_became_peered":"2024-03-28T18:08:20.009143+0000","last_unstale":"2024-03-28T18:09:44.002070+0000","last_undegraded":"2024-03-28T18:09:44.002070+0000","last_fullsized":"2024-03-28T18:09:44.002070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:37:50.643356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716076+0000","last_change":"2024-03-28T18:08:19.483733+0000","last_active":"2024-03-28T18:09:15.716076+0000","last_peered":"2024-03-28T18:09:15.716076+0000","last_clean":"2024-03-28T18:09:15.716076+0000","last_became_active":"2024-03-28T18:08:19.483366+0000","last_became_peered":"2024-03-28T18:08:19.483366+0000","last_unstale":"2024-03-28T18:09:15.716076+0000","last_undegraded":"2024-03-28T18:09:15.716076+0000","last_fullsized":"2024-03-28T18:09:15.716076+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:29.192442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716038+0000","last_change":"2024-03-28T18:08:19.481844+0000","last_active":"2024-03-28T18:09:15.716038+0000","last_peered":"2024-03-28T18:09:15.716038+0000","last_clean":"2024-03-28T18:09:15.716038+0000","last_became_active":"2024-03-28T18:08:19.481430+0000","last_became_peered":"2024-03-28T18:08:19.481430+0000","last_unstale":"2024-03-28T18:09:15.716038+0000","last_undegraded":"2024-03-28T18:09:15.716038+0000","last_fullsized":"2024-03-28T18:09:15.716038+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:21:28.789883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.018278+0000","last_change":"2024-03-28T18:08:20.021037+0000","last_active":"2024-03-28T18:09:44.018278+0000","last_peered":"2024-03-28T18:09:44.018278+0000","last_clean":"2024-03-28T18:09:44.018278+0000","last_became_active":"2024-03-28T18:08:20.020680+0000","last_became_peered":"2024-03-28T18:08:20.020680+0000","last_unstale":"2024-03-28T18:09:44.018278+0000","last_undegraded":"2024-03-28T18:09:44.018278+0000","last_fullsized":"2024-03-28T18:09:44.018278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:08:28.825774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'11","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:28.677487+0000","last_change":"2024-03-28T18:09:15.737583+0000","last_active":"2024-03-28T18:10:28.677487+0000","last_peered":"2024-03-28T18:10:28.677487+0000","last_clean":"2024-03-28T18:10:28.677487+0000","last_became_active":"2024-03-28T18:09:15.726579+0000","last_became_peered":"2024-03-28T18:09:15.726579+0000","last_unstale":"2024-03-28T18:10:28.677487+0000","last_undegraded":"2024-03-28T18:10:28.677487+0000","last_fullsized":"2024-03-28T18:10:28.677487+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:03.380037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.011149+0000","last_change":"2024-03-28T18:08:20.024938+0000","last_active":"2024-03-28T18:09:44.011149+0000","last_peered":"2024-03-28T18:09:44.011149+0000","last_clean":"2024-03-28T18:09:44.011149+0000","last_became_active":"2024-03-28T18:08:20.024536+0000","last_became_peered":"2024-03-28T18:08:20.024536+0000","last_unstale":"2024-03-28T18:09:44.011149+0000","last_undegraded":"2024-03-28T18:09:44.011149+0000","last_fullsized":"2024-03-28T18:09:44.011149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:46:27.080005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.888043+0000","last_change":"2024-03-28T18:08:19.480310+0000","last_active":"2024-03-28T18:09:15.888043+0000","last_peered":"2024-03-28T18:09:15.888043+0000","last_clean":"2024-03-28T18:09:15.888043+0000","last_became_active":"2024-03-28T18:08:19.479919+0000","last_became_peered":"2024-03-28T18:08:19.479919+0000","last_unstale":"2024-03-28T18:09:15.888043+0000","last_undegraded":"2024-03-28T18:09:15.888043+0000","last_fullsized":"2024-03-28T18:09:15.888043+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:36:26.239515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.018864+0000","last_change":"2024-03-28T18:08:20.033974+0000","last_active":"2024-03-28T18:09:44.018864+0000","last_peered":"2024-03-28T18:09:44.018864+0000","last_clean":"2024-03-28T18:09:44.018864+0000","last_became_active":"2024-03-28T18:08:20.033544+0000","last_became_peered":"2024-03-28T18:08:20.033544+0000","last_unstale":"2024-03-28T18:09:44.018864+0000","last_undegraded":"2024-03-28T18:09:44.018864+0000","last_fullsized":"2024-03-28T18:09:44.018864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:45:09.617988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.718382+0000","last_change":"2024-03-28T18:08:19.482604+0000","last_active":"2024-03-28T18:09:15.718382+0000","last_peered":"2024-03-28T18:09:15.718382+0000","last_clean":"2024-03-28T18:09:15.718382+0000","last_became_active":"2024-03-28T18:08:19.482248+0000","last_became_peered":"2024-03-28T18:08:19.482248+0000","last_unstale":"2024-03-28T18:09:15.718382+0000","last_undegraded":"2024-03-28T18:09:15.718382+0000","last_fullsized":"2024-03-28T18:09:15.718382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:23.199196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.947114+0000","last_change":"2024-03-28T18:08:20.011198+0000","last_active":"2024-03-28T18:09:43.947114+0000","last_peered":"2024-03-28T18:09:43.947114+0000","last_clean":"2024-03-28T18:09:43.947114+0000","last_became_active":"2024-03-28T18:08:20.010866+0000","last_became_peered":"2024-03-28T18:08:20.010866+0000","last_unstale":"2024-03-28T18:09:43.947114+0000","last_undegraded":"2024-03-28T18:09:43.947114+0000","last_fullsized":"2024-03-28T18:09:43.947114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:26.342858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.724859+0000","last_change":"2024-03-28T18:08:19.493830+0000","last_active":"2024-03-28T18:09:15.724859+0000","last_peered":"2024-03-28T18:09:15.724859+0000","last_clean":"2024-03-28T18:09:15.724859+0000","last_became_active":"2024-03-28T18:08:19.493447+0000","last_became_peered":"2024-03-28T18:08:19.493447+0000","last_unstale":"2024-03-28T18:09:15.724859+0000","last_undegraded":"2024-03-28T18:09:15.724859+0000","last_fullsized":"2024-03-28T18:09:15.724859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:33:29.797186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.008626+0000","last_change":"2024-03-28T18:08:20.025620+0000","last_active":"2024-03-28T18:09:44.008626+0000","last_peered":"2024-03-28T18:09:44.008626+0000","last_clean":"2024-03-28T18:09:44.008626+0000","last_became_active":"2024-03-28T18:08:20.025276+0000","last_became_peered":"2024-03-28T18:08:20.025276+0000","last_unstale":"2024-03-28T18:09:44.008626+0000","last_undegraded":"2024-03-28T18:09:44.008626+0000","last_fullsized":"2024-03-28T18:09:44.008626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:30:06.667707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:30.366681+0000","last_change":"2024-03-28T18:08:19.474204+0000","last_active":"2024-03-28T18:10:30.366681+0000","last_peered":"2024-03-28T18:10:30.366681+0000","last_clean":"2024-03-28T18:10:30.366681+0000","last_became_active":"2024-03-28T18:08:19.473688+0000","last_became_peered":"2024-03-28T18:08:19.473688+0000","last_unstale":"2024-03-28T18:10:30.366681+0000","last_undegraded":"2024-03-28T18:10:30.366681+0000","last_fullsized":"2024-03-28T18:10:30.366681+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:15:09.880974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.015204+0000","last_change":"2024-03-28T18:08:20.035920+0000","last_active":"2024-03-28T18:09:44.015204+0000","last_peered":"2024-03-28T18:09:44.015204+0000","last_clean":"2024-03-28T18:09:44.015204+0000","last_became_active":"2024-03-28T18:08:20.035600+0000","last_became_peered":"2024-03-28T18:08:20.035600+0000","last_unstale":"2024-03-28T18:09:44.015204+0000","last_undegraded":"2024-03-28T18:09:44.015204+0000","last_fullsized":"2024-03-28T18:09:44.015204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:41.280448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.715581+0000","last_change":"2024-03-28T18:08:19.481305+0000","last_active":"2024-03-28T18:09:15.715581+0000","last_peered":"2024-03-28T18:09:15.715581+0000","last_clean":"2024-03-28T18:09:15.715581+0000","last_became_active":"2024-03-28T18:08:19.480978+0000","last_became_peered":"2024-03-28T18:08:19.480978+0000","last_unstale":"2024-03-28T18:09:15.715581+0000","last_undegraded":"2024-03-28T18:09:15.715581+0000","last_fullsized":"2024-03-28T18:09:15.715581+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:14:06.464967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.002331+0000","last_change":"2024-03-28T18:08:20.017660+0000","last_active":"2024-03-28T18:09:44.002331+0000","last_peered":"2024-03-28T18:09:44.002331+0000","last_clean":"2024-03-28T18:09:44.002331+0000","last_became_active":"2024-03-28T18:08:20.016083+0000","last_became_peered":"2024-03-28T18:08:20.016083+0000","last_unstale":"2024-03-28T18:09:44.002331+0000","last_undegraded":"2024-03-28T18:09:44.002331+0000","last_fullsized":"2024-03-28T18:09:44.002331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:33:12.104274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.718320+0000","last_change":"2024-03-28T18:08:19.479855+0000","last_active":"2024-03-28T18:09:15.718320+0000","last_peered":"2024-03-28T18:09:15.718320+0000","last_clean":"2024-03-28T18:09:15.718320+0000","last_became_active":"2024-03-28T18:08:19.479429+0000","last_became_peered":"2024-03-28T18:08:19.479429+0000","last_unstale":"2024-03-28T18:09:15.718320+0000","last_undegraded":"2024-03-28T18:09:15.718320+0000","last_fullsized":"2024-03-28T18:09:15.718320+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:06:23.286970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.978524+0000","last_change":"2024-03-28T18:08:20.015419+0000","last_active":"2024-03-28T18:09:43.978524+0000","last_peered":"2024-03-28T18:09:43.978524+0000","last_clean":"2024-03-28T18:09:43.978524+0000","last_became_active":"2024-03-28T18:08:20.015011+0000","last_became_peered":"2024-03-28T18:08:20.015011+0000","last_unstale":"2024-03-28T18:09:43.978524+0000","last_undegraded":"2024-03-28T18:09:43.978524+0000","last_fullsized":"2024-03-28T18:09:43.978524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:44:46.858352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887142+0000","last_change":"2024-03-28T18:08:19.495541+0000","last_active":"2024-03-28T18:09:15.887142+0000","last_peered":"2024-03-28T18:09:15.887142+0000","last_clean":"2024-03-28T18:09:15.887142+0000","last_became_active":"2024-03-28T18:08:19.495173+0000","last_became_peered":"2024-03-28T18:08:19.495173+0000","last_unstale":"2024-03-28T18:09:15.887142+0000","last_undegraded":"2024-03-28T18:09:15.887142+0000","last_fullsized":"2024-03-28T18:09:15.887142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:42:06.423107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.981192+0000","last_change":"2024-03-28T18:08:20.018582+0000","last_active":"2024-03-28T18:09:43.981192+0000","last_peered":"2024-03-28T18:09:43.981192+0000","last_clean":"2024-03-28T18:09:43.981192+0000","last_became_active":"2024-03-28T18:08:20.017936+0000","last_became_peered":"2024-03-28T18:08:20.017936+0000","last_unstale":"2024-03-28T18:09:43.981192+0000","last_undegraded":"2024-03-28T18:09:43.981192+0000","last_fullsized":"2024-03-28T18:09:43.981192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:05:49.510246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.889005+0000","last_change":"2024-03-28T18:08:19.497049+0000","last_active":"2024-03-28T18:09:15.889005+0000","last_peered":"2024-03-28T18:09:15.889005+0000","last_clean":"2024-03-28T18:09:15.889005+0000","last_became_active":"2024-03-28T18:08:19.496623+0000","last_became_peered":"2024-03-28T18:08:19.496623+0000","last_unstale":"2024-03-28T18:09:15.889005+0000","last_undegraded":"2024-03-28T18:09:15.889005+0000","last_fullsized":"2024-03-28T18:09:15.889005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:55:07.842439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.978646+0000","last_change":"2024-03-28T18:08:20.017419+0000","last_active":"2024-03-28T18:09:43.978646+0000","last_peered":"2024-03-28T18:09:43.978646+0000","last_clean":"2024-03-28T18:09:43.978646+0000","last_became_active":"2024-03-28T18:08:20.015916+0000","last_became_peered":"2024-03-28T18:08:20.015916+0000","last_unstale":"2024-03-28T18:09:43.978646+0000","last_undegraded":"2024-03-28T18:09:43.978646+0000","last_fullsized":"2024-03-28T18:09:43.978646+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:10:55.927794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.964132+0000","last_change":"2024-03-28T18:08:20.008093+0000","last_active":"2024-03-28T18:09:43.964132+0000","last_peered":"2024-03-28T18:09:43.964132+0000","last_clean":"2024-03-28T18:09:43.964132+0000","last_became_active":"2024-03-28T18:08:20.007801+0000","last_became_peered":"2024-03-28T18:08:20.007801+0000","last_unstale":"2024-03-28T18:09:43.964132+0000","last_undegraded":"2024-03-28T18:09:43.964132+0000","last_fullsized":"2024-03-28T18:09:43.964132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:58.976859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.967834+0000","last_change":"2024-03-28T18:08:20.012771+0000","last_active":"2024-03-28T18:09:43.967834+0000","last_peered":"2024-03-28T18:09:43.967834+0000","last_clean":"2024-03-28T18:09:43.967834+0000","last_became_active":"2024-03-28T18:08:20.012456+0000","last_became_peered":"2024-03-28T18:08:20.012456+0000","last_unstale":"2024-03-28T18:09:43.967834+0000","last_undegraded":"2024-03-28T18:09:43.967834+0000","last_fullsized":"2024-03-28T18:09:43.967834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:56.428332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.013650+0000","last_change":"2024-03-28T18:08:20.034872+0000","last_active":"2024-03-28T18:09:44.013650+0000","last_peered":"2024-03-28T18:09:44.013650+0000","last_clean":"2024-03-28T18:09:44.013650+0000","last_became_active":"2024-03-28T18:08:20.034614+0000","last_became_peered":"2024-03-28T18:08:20.034614+0000","last_unstale":"2024-03-28T18:09:44.013650+0000","last_undegraded":"2024-03-28T18:09:44.013650+0000","last_fullsized":"2024-03-28T18:09:44.013650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:35.658186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.018419+0000","last_change":"2024-03-28T18:08:20.033119+0000","last_active":"2024-03-28T18:09:44.018419+0000","last_peered":"2024-03-28T18:09:44.018419+0000","last_clean":"2024-03-28T18:09:44.018419+0000","last_became_active":"2024-03-28T18:08:20.030694+0000","last_became_peered":"2024-03-28T18:08:20.030694+0000","last_unstale":"2024-03-28T18:09:44.018419+0000","last_undegraded":"2024-03-28T18:09:44.018419+0000","last_fullsized":"2024-03-28T18:09:44.018419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:51:50.082993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.020461+0000","last_change":"2024-03-28T18:08:20.025725+0000","last_active":"2024-03-28T18:09:44.020461+0000","last_peered":"2024-03-28T18:09:44.020461+0000","last_clean":"2024-03-28T18:09:44.020461+0000","last_became_active":"2024-03-28T18:08:20.024929+0000","last_became_peered":"2024-03-28T18:08:20.024929+0000","last_unstale":"2024-03-28T18:09:44.020461+0000","last_undegraded":"2024-03-28T18:09:44.020461+0000","last_fullsized":"2024-03-28T18:09:44.020461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:04.894246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.960946+0000","last_change":"2024-03-28T18:08:20.005293+0000","last_active":"2024-03-28T18:09:43.960946+0000","last_peered":"2024-03-28T18:09:43.960946+0000","last_clean":"2024-03-28T18:09:43.960946+0000","last_became_active":"2024-03-28T18:08:20.004938+0000","last_became_peered":"2024-03-28T18:08:20.004938+0000","last_unstale":"2024-03-28T18:09:43.960946+0000","last_undegraded":"2024-03-28T18:09:43.960946+0000","last_fullsized":"2024-03-28T18:09:43.960946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:36.100810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.971806+0000","last_change":"2024-03-28T18:08:20.008741+0000","last_active":"2024-03-28T18:09:43.971806+0000","last_peered":"2024-03-28T18:09:43.971806+0000","last_clean":"2024-03-28T18:09:43.971806+0000","last_became_active":"2024-03-28T18:08:20.008411+0000","last_became_peered":"2024-03-28T18:08:20.008411+0000","last_unstale":"2024-03-28T18:09:43.971806+0000","last_undegraded":"2024-03-28T18:09:43.971806+0000","last_fullsized":"2024-03-28T18:09:43.971806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:28:52.276278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.997108+0000","last_change":"2024-03-28T18:08:20.028486+0000","last_active":"2024-03-28T18:09:43.997108+0000","last_peered":"2024-03-28T18:09:43.997108+0000","last_clean":"2024-03-28T18:09:43.997108+0000","last_became_active":"2024-03-28T18:08:20.028121+0000","last_became_peered":"2024-03-28T18:08:20.028121+0000","last_unstale":"2024-03-28T18:09:43.997108+0000","last_undegraded":"2024-03-28T18:09:43.997108+0000","last_fullsized":"2024-03-28T18:09:43.997108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:55:26.828305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.970707+0000","last_change":"2024-03-28T18:08:20.028085+0000","last_active":"2024-03-28T18:09:43.970707+0000","last_peered":"2024-03-28T18:09:43.970707+0000","last_clean":"2024-03-28T18:09:43.970707+0000","last_became_active":"2024-03-28T18:08:20.027721+0000","last_became_peered":"2024-03-28T18:08:20.027721+0000","last_unstale":"2024-03-28T18:09:43.970707+0000","last_undegraded":"2024-03-28T18:09:43.970707+0000","last_fullsized":"2024-03-28T18:09:43.970707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:31:36.674002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.012065+0000","last_change":"2024-03-28T18:08:20.023605+0000","last_active":"2024-03-28T18:09:44.012065+0000","last_peered":"2024-03-28T18:09:44.012065+0000","last_clean":"2024-03-28T18:09:44.012065+0000","last_became_active":"2024-03-28T18:08:20.018217+0000","last_became_peered":"2024-03-28T18:08:20.018217+0000","last_unstale":"2024-03-28T18:09:44.012065+0000","last_undegraded":"2024-03-28T18:09:44.012065+0000","last_fullsized":"2024-03-28T18:09:44.012065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:43.717751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.014919+0000","last_change":"2024-03-28T18:08:20.023072+0000","last_active":"2024-03-28T18:09:44.014919+0000","last_peered":"2024-03-28T18:09:44.014919+0000","last_clean":"2024-03-28T18:09:44.014919+0000","last_became_active":"2024-03-28T18:08:20.022690+0000","last_became_peered":"2024-03-28T18:08:20.022690+0000","last_unstale":"2024-03-28T18:09:44.014919+0000","last_undegraded":"2024-03-28T18:09:44.014919+0000","last_fullsized":"2024-03-28T18:09:44.014919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:21.391415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.018990+0000","last_change":"2024-03-28T18:08:20.024261+0000","last_active":"2024-03-28T18:09:44.018990+0000","last_peered":"2024-03-28T18:09:44.018990+0000","last_clean":"2024-03-28T18:09:44.018990+0000","last_became_active":"2024-03-28T18:08:20.023792+0000","last_became_peered":"2024-03-28T18:08:20.023792+0000","last_unstale":"2024-03-28T18:09:44.018990+0000","last_undegraded":"2024-03-28T18:09:44.018990+0000","last_fullsized":"2024-03-28T18:09:44.018990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:23:08.030982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.968857+0000","last_change":"2024-03-28T18:09:15.720113+0000","last_active":"2024-03-28T18:09:43.968857+0000","last_peered":"2024-03-28T18:09:43.968857+0000","last_clean":"2024-03-28T18:09:43.968857+0000","last_became_active":"2024-03-28T18:09:15.719739+0000","last_became_peered":"2024-03-28T18:09:15.719739+0000","last_unstale":"2024-03-28T18:09:43.968857+0000","last_undegraded":"2024-03-28T18:09:43.968857+0000","last_fullsized":"2024-03-28T18:09:43.968857+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:09:38.179780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":248983,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66899,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":901,"num_read_kb":522,"num_write":174,"num_write_kb":121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":694,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":205011,"data_compressed":5634,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383725,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":456779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645362,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":1110129,"data_compressed":37917,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808888,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070526,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332164,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":1048148,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495689,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":455822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":522492,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738476,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":457093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411424,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1024000,"data_stored":520142,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182491,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":456133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920854,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":456259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757327,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":1048414,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34394,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34394,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":66877,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64527,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64029,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:10:55.062 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:10:55.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.301+0000 7f95e4c00640 1 -- 172.21.15.37:0/1636618896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95e01028b0 msgr2=0x7f95e0102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:55.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.301+0000 7f95e4c00640 1 --2- 172.21.15.37:0/1636618896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95e01028b0 0x7f95e0102d30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f95cc009900 tx=0x7f95cc02bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:55.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.301+0000 7f95e4c00640 1 -- 172.21.15.37:0/1636618896 shutdown_connections 2024-03-28T18:10:55.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.301+0000 7f95e4c00640 1 --2- 172.21.15.37:0/1636618896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95e01028b0 0x7f95e0102d30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:55.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.301+0000 7f95e4c00640 1 --2- 172.21.15.37:0/1636618896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95e0103270 0x7f95e0109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:55.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.301+0000 7f95e4c00640 1 --2- 172.21.15.37:0/1636618896 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95e01016c0 0x7f95e0101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:55.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.301+0000 7f95e4c00640 1 -- 172.21.15.37:0/1636618896 >> 172.21.15.37:0/1636618896 conn(0x7f95e00fcc50 msgr2=0x7f95e00ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:55.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.301+0000 7f95e4c00640 1 -- 172.21.15.37:0/1636618896 shutdown_connections 2024-03-28T18:10:55.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.301+0000 7f95e4c00640 1 -- 172.21.15.37:0/1636618896 wait complete. 2024-03-28T18:10:55.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.302+0000 7f95e4c00640 1 Processor -- start 2024-03-28T18:10:55.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.303+0000 7f95e4c00640 1 -- start start 2024-03-28T18:10:55.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.303+0000 7f95e4c00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95e01016c0 0x7f95e01979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:55.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.303+0000 7f95e4c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95e01028b0 0x7f95e0197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:55.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.303+0000 7f95e4c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95e0103270 0x7f95e019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:55.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.303+0000 7f95e4c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e010d370 con 0x7f95e0103270 2024-03-28T18:10:55.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.303+0000 7f95e4c00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e010d1f0 con 0x7f95e01028b0 2024-03-28T18:10:55.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.303+0000 7f95e4c00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e010d4f0 con 0x7f95e01016c0 2024-03-28T18:10:55.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.303+0000 7f95dde00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95e01016c0 0x7f95e01979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:55.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.303+0000 7f95dde00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95e01016c0 0x7f95e01979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55814/0 (socket says 172.21.15.37:55814) 2024-03-28T18:10:55.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.303+0000 7f95dde00640 1 -- 172.21.15.37:0/3530645810 learned_addr learned my addr 172.21.15.37:0/3530645810 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.303+0000 7f95dd400640 1 --2- 172.21.15.37:0/3530645810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95e01028b0 0x7f95e0197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.304+0000 7f95de800640 1 --2- 172.21.15.37:0/3530645810 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95e0103270 0x7f95e019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.304+0000 7f95dde00640 1 -- 172.21.15.37:0/3530645810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95e01028b0 msgr2=0x7f95e0197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.304+0000 7f95dde00640 1 --2- 172.21.15.37:0/3530645810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95e01028b0 0x7f95e0197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.304+0000 7f95dde00640 1 -- 172.21.15.37:0/3530645810 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95e0103270 msgr2=0x7f95e019ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.304+0000 7f95dde00640 1 --2- 172.21.15.37:0/3530645810 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95e0103270 0x7f95e019ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.304+0000 7f95dde00640 1 -- 172.21.15.37:0/3530645810 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e019f600 con 0x7f95e01016c0 2024-03-28T18:10:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.304+0000 7f95de800640 1 --2- 172.21.15.37:0/3530645810 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95e0103270 0x7f95e019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:10:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.304+0000 7f95dde00640 1 --2- 172.21.15.37:0/3530645810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95e01016c0 0x7f95e01979e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f95d000d950 tx=0x7f95d000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.304+0000 7f95d6a00640 1 -- 172.21.15.37:0/3530645810 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d0017020 con 0x7f95e01016c0 2024-03-28T18:10:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.304+0000 7f95d6a00640 1 -- 172.21.15.37:0/3530645810 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95d0009c30 con 0x7f95e01016c0 2024-03-28T18:10:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.304+0000 7f95d6a00640 1 -- 172.21.15.37:0/3530645810 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d000f720 con 0x7f95e01016c0 2024-03-28T18:10:55.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.304+0000 7f95e4c00640 1 -- 172.21.15.37:0/3530645810 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e0073930 con 0x7f95e01016c0 2024-03-28T18:10:55.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.305+0000 7f95e4c00640 1 -- 172.21.15.37:0/3530645810 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e0073e50 con 0x7f95e01016c0 2024-03-28T18:10:55.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.306+0000 7f95e4c00640 1 -- 172.21.15.37:0/3530645810 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95e01089e0 con 0x7f95e01016c0 2024-03-28T18:10:55.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.307+0000 7f95d6a00640 1 -- 172.21.15.37:0/3530645810 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f95d001a020 con 0x7f95e01016c0 2024-03-28T18:10:55.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.310+0000 7f95d6a00640 1 --2- 172.21.15.37:0/3530645810 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f95c40a2800 0x7f95c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:55.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.310+0000 7f95d6a00640 1 -- 172.21.15.37:0/3530645810 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f95d00cea70 con 0x7f95e01016c0 2024-03-28T18:10:55.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.310+0000 7f95dd400640 1 --2- 172.21.15.37:0/3530645810 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f95c40a2800 0x7f95c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:55.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.311+0000 7f95dd400640 1 --2- 172.21.15.37:0/3530645810 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f95c40a2800 0x7f95c40a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f95cc02c240 tx=0x7f95cc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:55.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.312+0000 7f95d6a00640 1 -- 172.21.15.37:0/3530645810 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f95d0021020 con 0x7f95e01016c0 2024-03-28T18:10:55.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.527+0000 7f95e4c00640 1 -- 172.21.15.37:0/3530645810 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f95e0061ae0 con 0x7f95c40a2800 2024-03-28T18:10:55.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.551+0000 7f95d6a00640 1 -- 172.21.15.37:0/3530645810 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259036 (secure 0 0 0) 0x7f95e0061ae0 con 0x7f95c40a2800 2024-03-28T18:10:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:55.557 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-03-28T18:10:55.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.558+0000 7f95e4c00640 1 -- 172.21.15.37:0/3530645810 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f95c40a2800 msgr2=0x7f95c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:55.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.558+0000 7f95e4c00640 1 --2- 172.21.15.37:0/3530645810 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f95c40a2800 0x7f95c40a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f95cc02c240 tx=0x7f95cc02df90 comp rx=0 tx=0).stop 2024-03-28T18:10:55.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.558+0000 7f95e4c00640 1 -- 172.21.15.37:0/3530645810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95e01016c0 msgr2=0x7f95e01979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:55.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.558+0000 7f95e4c00640 1 --2- 172.21.15.37:0/3530645810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95e01016c0 0x7f95e01979e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f95d000d950 tx=0x7f95d000de10 comp rx=0 tx=0).stop 2024-03-28T18:10:55.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.558+0000 7f95e4c00640 1 -- 172.21.15.37:0/3530645810 shutdown_connections 2024-03-28T18:10:55.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.558+0000 7f95e4c00640 1 --2- 172.21.15.37:0/3530645810 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f95c40a2800 0x7f95c40a4cc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:55.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.558+0000 7f95e4c00640 1 --2- 172.21.15.37:0/3530645810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95e01028b0 0x7f95e0197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:55.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.558+0000 7f95e4c00640 1 --2- 172.21.15.37:0/3530645810 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95e0103270 0x7f95e019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:55.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.558+0000 7f95e4c00640 1 --2- 172.21.15.37:0/3530645810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95e01016c0 0x7f95e01979e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:55.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.558+0000 7f95e4c00640 1 -- 172.21.15.37:0/3530645810 >> 172.21.15.37:0/3530645810 conn(0x7f95e00fcc50 msgr2=0x7f95e0105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:55.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.558+0000 7f95e4c00640 1 -- 172.21.15.37:0/3530645810 shutdown_connections 2024-03-28T18:10:55.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.558+0000 7f95e4c00640 1 -- 172.21.15.37:0/3530645810 wait complete. 2024-03-28T18:10:55.579 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-03-28T18:10:54.014176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657501,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":1037,"num_read_kb":641,"num_write":357,"num_write_kb":3061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1181430,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":788,"ondisk_log_size":788,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377364,"kb_used_data":11572,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564460,"statfs":{"total":1151940427776,"available":1151554007040,"internally_reserved":0,"allocated":11849728,"data_stored":7444197,"data_compressed":113751,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":612,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":32,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004862"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.005929+0000","last_change":"2024-03-28T18:08:20.025576+0000","last_active":"2024-03-28T18:09:44.005929+0000","last_peered":"2024-03-28T18:09:44.005929+0000","last_clean":"2024-03-28T18:09:44.005929+0000","last_became_active":"2024-03-28T18:08:20.025177+0000","last_became_peered":"2024-03-28T18:08:20.025177+0000","last_unstale":"2024-03-28T18:09:44.005929+0000","last_undegraded":"2024-03-28T18:09:44.005929+0000","last_fullsized":"2024-03-28T18:09:44.005929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:14.521170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.018719+0000","last_change":"2024-03-28T18:08:20.027418+0000","last_active":"2024-03-28T18:09:44.018719+0000","last_peered":"2024-03-28T18:09:44.018719+0000","last_clean":"2024-03-28T18:09:44.018719+0000","last_became_active":"2024-03-28T18:08:20.026731+0000","last_became_peered":"2024-03-28T18:08:20.026731+0000","last_unstale":"2024-03-28T18:09:44.018719+0000","last_undegraded":"2024-03-28T18:09:44.018719+0000","last_fullsized":"2024-03-28T18:09:44.018719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:52:58.328738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.001347+0000","last_change":"2024-03-28T18:08:20.029821+0000","last_active":"2024-03-28T18:09:44.001347+0000","last_peered":"2024-03-28T18:09:44.001347+0000","last_clean":"2024-03-28T18:09:44.001347+0000","last_became_active":"2024-03-28T18:08:20.028245+0000","last_became_peered":"2024-03-28T18:08:20.028245+0000","last_unstale":"2024-03-28T18:09:44.001347+0000","last_undegraded":"2024-03-28T18:09:44.001347+0000","last_fullsized":"2024-03-28T18:09:44.001347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:44.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.966606+0000","last_change":"2024-03-28T18:08:20.015682+0000","last_active":"2024-03-28T18:09:43.966606+0000","last_peered":"2024-03-28T18:09:43.966606+0000","last_clean":"2024-03-28T18:09:43.966606+0000","last_became_active":"2024-03-28T18:08:20.015303+0000","last_became_peered":"2024-03-28T18:08:20.015303+0000","last_unstale":"2024-03-28T18:09:43.966606+0000","last_undegraded":"2024-03-28T18:09:43.966606+0000","last_fullsized":"2024-03-28T18:09:43.966606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:57.456052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.998456+0000","last_change":"2024-03-28T18:08:20.020970+0000","last_active":"2024-03-28T18:09:43.998456+0000","last_peered":"2024-03-28T18:09:43.998456+0000","last_clean":"2024-03-28T18:09:43.998456+0000","last_became_active":"2024-03-28T18:08:20.020646+0000","last_became_peered":"2024-03-28T18:08:20.020646+0000","last_unstale":"2024-03-28T18:09:43.998456+0000","last_undegraded":"2024-03-28T18:09:43.998456+0000","last_fullsized":"2024-03-28T18:09:43.998456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:57:20.422546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.970526+0000","last_change":"2024-03-28T18:08:20.022395+0000","last_active":"2024-03-28T18:09:43.970526+0000","last_peered":"2024-03-28T18:09:43.970526+0000","last_clean":"2024-03-28T18:09:43.970526+0000","last_became_active":"2024-03-28T18:08:20.022022+0000","last_became_peered":"2024-03-28T18:08:20.022022+0000","last_unstale":"2024-03-28T18:09:43.970526+0000","last_undegraded":"2024-03-28T18:09:43.970526+0000","last_fullsized":"2024-03-28T18:09:43.970526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:32.590751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.972192+0000","last_change":"2024-03-28T18:08:20.019990+0000","last_active":"2024-03-28T18:09:43.972192+0000","last_peered":"2024-03-28T18:09:43.972192+0000","last_clean":"2024-03-28T18:09:43.972192+0000","last_became_active":"2024-03-28T18:08:20.019618+0000","last_became_peered":"2024-03-28T18:08:20.019618+0000","last_unstale":"2024-03-28T18:09:43.972192+0000","last_undegraded":"2024-03-28T18:09:43.972192+0000","last_fullsized":"2024-03-28T18:09:43.972192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:48.442579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.968407+0000","last_change":"2024-03-28T18:09:15.889503+0000","last_active":"2024-03-28T18:09:43.968407+0000","last_peered":"2024-03-28T18:09:43.968407+0000","last_clean":"2024-03-28T18:09:43.968407+0000","last_became_active":"2024-03-28T18:09:15.888272+0000","last_became_peered":"2024-03-28T18:09:15.888272+0000","last_unstale":"2024-03-28T18:09:43.968407+0000","last_undegraded":"2024-03-28T18:09:43.968407+0000","last_fullsized":"2024-03-28T18:09:43.968407+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:49.222646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.980289+0000","last_change":"2024-03-28T18:08:20.009732+0000","last_active":"2024-03-28T18:09:43.980289+0000","last_peered":"2024-03-28T18:09:43.980289+0000","last_clean":"2024-03-28T18:09:43.980289+0000","last_became_active":"2024-03-28T18:08:20.009288+0000","last_became_peered":"2024-03-28T18:08:20.009288+0000","last_unstale":"2024-03-28T18:09:43.980289+0000","last_undegraded":"2024-03-28T18:09:43.980289+0000","last_fullsized":"2024-03-28T18:09:43.980289+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:20:54.917964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.964990+0000","last_change":"2024-03-28T18:08:20.009687+0000","last_active":"2024-03-28T18:09:43.964990+0000","last_peered":"2024-03-28T18:09:43.964990+0000","last_clean":"2024-03-28T18:09:43.964990+0000","last_became_active":"2024-03-28T18:08:20.008364+0000","last_became_peered":"2024-03-28T18:08:20.008364+0000","last_unstale":"2024-03-28T18:09:43.964990+0000","last_undegraded":"2024-03-28T18:09:43.964990+0000","last_fullsized":"2024-03-28T18:09:43.964990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:08.419202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.978300+0000","last_change":"2024-03-28T18:08:20.011508+0000","last_active":"2024-03-28T18:09:43.978300+0000","last_peered":"2024-03-28T18:09:43.978300+0000","last_clean":"2024-03-28T18:09:43.978300+0000","last_became_active":"2024-03-28T18:08:20.011117+0000","last_became_peered":"2024-03-28T18:08:20.011117+0000","last_unstale":"2024-03-28T18:09:43.978300+0000","last_undegraded":"2024-03-28T18:09:43.978300+0000","last_fullsized":"2024-03-28T18:09:43.978300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:06:19.278931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.966450+0000","last_change":"2024-03-28T18:08:20.011403+0000","last_active":"2024-03-28T18:09:43.966450+0000","last_peered":"2024-03-28T18:09:43.966450+0000","last_clean":"2024-03-28T18:09:43.966450+0000","last_became_active":"2024-03-28T18:08:20.011025+0000","last_became_peered":"2024-03-28T18:08:20.011025+0000","last_unstale":"2024-03-28T18:09:43.966450+0000","last_undegraded":"2024-03-28T18:09:43.966450+0000","last_fullsized":"2024-03-28T18:09:43.966450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:13.599432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.005479+0000","last_change":"2024-03-28T18:08:20.001664+0000","last_active":"2024-03-28T18:09:44.005479+0000","last_peered":"2024-03-28T18:09:44.005479+0000","last_clean":"2024-03-28T18:09:44.005479+0000","last_became_active":"2024-03-28T18:08:20.000910+0000","last_became_peered":"2024-03-28T18:08:20.000910+0000","last_unstale":"2024-03-28T18:09:44.005479+0000","last_undegraded":"2024-03-28T18:09:44.005479+0000","last_fullsized":"2024-03-28T18:09:44.005479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:00:23.796302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.008215+0000","last_change":"2024-03-28T18:08:20.027174+0000","last_active":"2024-03-28T18:09:44.008215+0000","last_peered":"2024-03-28T18:09:44.008215+0000","last_clean":"2024-03-28T18:09:44.008215+0000","last_became_active":"2024-03-28T18:08:20.026113+0000","last_became_peered":"2024-03-28T18:08:20.026113+0000","last_unstale":"2024-03-28T18:09:44.008215+0000","last_undegraded":"2024-03-28T18:09:44.008215+0000","last_fullsized":"2024-03-28T18:09:44.008215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:55:00.061696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.010537+0000","last_change":"2024-03-28T18:08:20.020060+0000","last_active":"2024-03-28T18:09:44.010537+0000","last_peered":"2024-03-28T18:09:44.010537+0000","last_clean":"2024-03-28T18:09:44.010537+0000","last_became_active":"2024-03-28T18:08:20.019691+0000","last_became_peered":"2024-03-28T18:08:20.019691+0000","last_unstale":"2024-03-28T18:09:44.010537+0000","last_undegraded":"2024-03-28T18:09:44.010537+0000","last_fullsized":"2024-03-28T18:09:44.010537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:27.889737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.003319+0000","last_change":"2024-03-28T18:08:20.024239+0000","last_active":"2024-03-28T18:09:44.003319+0000","last_peered":"2024-03-28T18:09:44.003319+0000","last_clean":"2024-03-28T18:09:44.003319+0000","last_became_active":"2024-03-28T18:08:20.023874+0000","last_became_peered":"2024-03-28T18:08:20.023874+0000","last_unstale":"2024-03-28T18:09:44.003319+0000","last_undegraded":"2024-03-28T18:09:44.003319+0000","last_fullsized":"2024-03-28T18:09:44.003319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:05:32.903195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.008671+0000","last_change":"2024-03-28T18:08:20.033322+0000","last_active":"2024-03-28T18:09:44.008671+0000","last_peered":"2024-03-28T18:09:44.008671+0000","last_clean":"2024-03-28T18:09:44.008671+0000","last_became_active":"2024-03-28T18:08:20.033048+0000","last_became_peered":"2024-03-28T18:08:20.033048+0000","last_unstale":"2024-03-28T18:09:44.008671+0000","last_undegraded":"2024-03-28T18:09:44.008671+0000","last_fullsized":"2024-03-28T18:09:44.008671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:18:59.234141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.986420+0000","last_change":"2024-03-28T18:08:20.022093+0000","last_active":"2024-03-28T18:09:43.986420+0000","last_peered":"2024-03-28T18:09:43.986420+0000","last_clean":"2024-03-28T18:09:43.986420+0000","last_became_active":"2024-03-28T18:08:20.021411+0000","last_became_peered":"2024-03-28T18:08:20.021411+0000","last_unstale":"2024-03-28T18:09:43.986420+0000","last_undegraded":"2024-03-28T18:09:43.986420+0000","last_fullsized":"2024-03-28T18:09:43.986420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:09:40.366628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.002040+0000","last_change":"2024-03-28T18:08:20.028893+0000","last_active":"2024-03-28T18:09:44.002040+0000","last_peered":"2024-03-28T18:09:44.002040+0000","last_clean":"2024-03-28T18:09:44.002040+0000","last_became_active":"2024-03-28T18:08:20.028634+0000","last_became_peered":"2024-03-28T18:08:20.028634+0000","last_unstale":"2024-03-28T18:09:44.002040+0000","last_undegraded":"2024-03-28T18:09:44.002040+0000","last_fullsized":"2024-03-28T18:09:44.002040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:32:27.454406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":46,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.734956+0000","last_change":"2024-03-28T18:09:15.734404+0000","last_active":"2024-03-28T18:09:15.734956+0000","last_peered":"2024-03-28T18:09:15.734956+0000","last_clean":"2024-03-28T18:09:15.734956+0000","last_became_active":"2024-03-28T18:09:15.723870+0000","last_became_peered":"2024-03-28T18:09:15.723870+0000","last_unstale":"2024-03-28T18:09:15.734956+0000","last_undegraded":"2024-03-28T18:09:15.734956+0000","last_fullsized":"2024-03-28T18:09:15.734956+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:32.736966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.004329+0000","last_change":"2024-03-28T18:08:20.027855+0000","last_active":"2024-03-28T18:09:44.004329+0000","last_peered":"2024-03-28T18:09:44.004329+0000","last_clean":"2024-03-28T18:09:44.004329+0000","last_became_active":"2024-03-28T18:08:20.027565+0000","last_became_peered":"2024-03-28T18:08:20.027565+0000","last_unstale":"2024-03-28T18:09:44.004329+0000","last_undegraded":"2024-03-28T18:09:44.004329+0000","last_fullsized":"2024-03-28T18:09:44.004329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:36:37.596573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.725789+0000","last_change":"2024-03-28T18:08:19.499055+0000","last_active":"2024-03-28T18:09:15.725789+0000","last_peered":"2024-03-28T18:09:15.725789+0000","last_clean":"2024-03-28T18:09:15.725789+0000","last_became_active":"2024-03-28T18:08:19.497964+0000","last_became_peered":"2024-03-28T18:08:19.497964+0000","last_unstale":"2024-03-28T18:09:15.725789+0000","last_undegraded":"2024-03-28T18:09:15.725789+0000","last_fullsized":"2024-03-28T18:09:15.725789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:38:45.620406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.012271+0000","last_change":"2024-03-28T18:08:20.028967+0000","last_active":"2024-03-28T18:09:44.012271+0000","last_peered":"2024-03-28T18:09:44.012271+0000","last_clean":"2024-03-28T18:09:44.012271+0000","last_became_active":"2024-03-28T18:08:20.028607+0000","last_became_peered":"2024-03-28T18:08:20.028607+0000","last_unstale":"2024-03-28T18:09:44.012271+0000","last_undegraded":"2024-03-28T18:09:44.012271+0000","last_fullsized":"2024-03-28T18:09:44.012271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:21:23.994077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.893027+0000","last_change":"2024-03-28T18:09:15.892649+0000","last_active":"2024-03-28T18:09:15.893027+0000","last_peered":"2024-03-28T18:09:15.893027+0000","last_clean":"2024-03-28T18:09:15.893027+0000","last_became_active":"2024-03-28T18:09:15.888520+0000","last_became_peered":"2024-03-28T18:09:15.888520+0000","last_unstale":"2024-03-28T18:09:15.893027+0000","last_undegraded":"2024-03-28T18:09:15.893027+0000","last_fullsized":"2024-03-28T18:09:15.893027+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:34.085738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.013841+0000","last_change":"2024-03-28T18:08:20.009878+0000","last_active":"2024-03-28T18:09:44.013841+0000","last_peered":"2024-03-28T18:09:44.013841+0000","last_clean":"2024-03-28T18:09:44.013841+0000","last_became_active":"2024-03-28T18:08:20.009534+0000","last_became_peered":"2024-03-28T18:08:20.009534+0000","last_unstale":"2024-03-28T18:09:44.013841+0000","last_undegraded":"2024-03-28T18:09:44.013841+0000","last_fullsized":"2024-03-28T18:09:44.013841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:58.975854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716112+0000","last_change":"2024-03-28T18:08:19.472613+0000","last_active":"2024-03-28T18:09:15.716112+0000","last_peered":"2024-03-28T18:09:15.716112+0000","last_clean":"2024-03-28T18:09:15.716112+0000","last_became_active":"2024-03-28T18:08:19.472186+0000","last_became_peered":"2024-03-28T18:08:19.472186+0000","last_unstale":"2024-03-28T18:09:15.716112+0000","last_undegraded":"2024-03-28T18:09:15.716112+0000","last_fullsized":"2024-03-28T18:09:15.716112+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:50.807609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.974502+0000","last_change":"2024-03-28T18:08:20.008242+0000","last_active":"2024-03-28T18:09:43.974502+0000","last_peered":"2024-03-28T18:09:43.974502+0000","last_clean":"2024-03-28T18:09:43.974502+0000","last_became_active":"2024-03-28T18:08:20.007942+0000","last_became_peered":"2024-03-28T18:08:20.007942+0000","last_unstale":"2024-03-28T18:09:43.974502+0000","last_undegraded":"2024-03-28T18:09:43.974502+0000","last_fullsized":"2024-03-28T18:09:43.974502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:23.288978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'8","reported_seq":179,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:50.579007+0000","last_change":"2024-03-28T18:08:19.480869+0000","last_active":"2024-03-28T18:10:50.579007+0000","last_peered":"2024-03-28T18:10:50.579007+0000","last_clean":"2024-03-28T18:10:50.579007+0000","last_became_active":"2024-03-28T18:08:19.480441+0000","last_became_peered":"2024-03-28T18:08:19.480441+0000","last_unstale":"2024-03-28T18:10:50.579007+0000","last_undegraded":"2024-03-28T18:10:50.579007+0000","last_fullsized":"2024-03-28T18:10:50.579007+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:26.640605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1460,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":151,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.960050+0000","last_change":"2024-03-28T18:08:20.021530+0000","last_active":"2024-03-28T18:09:43.960050+0000","last_peered":"2024-03-28T18:09:43.960050+0000","last_clean":"2024-03-28T18:09:43.960050+0000","last_became_active":"2024-03-28T18:08:20.021179+0000","last_became_peered":"2024-03-28T18:08:20.021179+0000","last_unstale":"2024-03-28T18:09:43.960050+0000","last_undegraded":"2024-03-28T18:09:43.960050+0000","last_fullsized":"2024-03-28T18:09:43.960050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:43:07.300300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887313+0000","last_change":"2024-03-28T18:08:19.494597+0000","last_active":"2024-03-28T18:09:15.887313+0000","last_peered":"2024-03-28T18:09:15.887313+0000","last_clean":"2024-03-28T18:09:15.887313+0000","last_became_active":"2024-03-28T18:08:19.494178+0000","last_became_peered":"2024-03-28T18:08:19.494178+0000","last_unstale":"2024-03-28T18:09:15.887313+0000","last_undegraded":"2024-03-28T18:09:15.887313+0000","last_fullsized":"2024-03-28T18:09:15.887313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:08:10.257768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.991904+0000","last_change":"2024-03-28T18:08:20.021062+0000","last_active":"2024-03-28T18:09:43.991904+0000","last_peered":"2024-03-28T18:09:43.991904+0000","last_clean":"2024-03-28T18:09:43.991904+0000","last_became_active":"2024-03-28T18:08:20.020758+0000","last_became_peered":"2024-03-28T18:08:20.020758+0000","last_unstale":"2024-03-28T18:09:43.991904+0000","last_undegraded":"2024-03-28T18:09:43.991904+0000","last_fullsized":"2024-03-28T18:09:43.991904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:47:39.656513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.889759+0000","last_change":"2024-03-28T18:08:19.480189+0000","last_active":"2024-03-28T18:09:15.889759+0000","last_peered":"2024-03-28T18:09:15.889759+0000","last_clean":"2024-03-28T18:09:15.889759+0000","last_became_active":"2024-03-28T18:08:19.479812+0000","last_became_peered":"2024-03-28T18:08:19.479812+0000","last_unstale":"2024-03-28T18:09:15.889759+0000","last_undegraded":"2024-03-28T18:09:15.889759+0000","last_fullsized":"2024-03-28T18:09:15.889759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:46.408565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.007167+0000","last_change":"2024-03-28T18:08:20.023635+0000","last_active":"2024-03-28T18:09:44.007167+0000","last_peered":"2024-03-28T18:09:44.007167+0000","last_clean":"2024-03-28T18:09:44.007167+0000","last_became_active":"2024-03-28T18:08:20.023199+0000","last_became_peered":"2024-03-28T18:08:20.023199+0000","last_unstale":"2024-03-28T18:09:44.007167+0000","last_undegraded":"2024-03-28T18:09:44.007167+0000","last_fullsized":"2024-03-28T18:09:44.007167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:12.477899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.886913+0000","last_change":"2024-03-28T18:08:19.480858+0000","last_active":"2024-03-28T18:09:15.886913+0000","last_peered":"2024-03-28T18:09:15.886913+0000","last_clean":"2024-03-28T18:09:15.886913+0000","last_became_active":"2024-03-28T18:08:19.480418+0000","last_became_peered":"2024-03-28T18:08:19.480418+0000","last_unstale":"2024-03-28T18:09:15.886913+0000","last_undegraded":"2024-03-28T18:09:15.886913+0000","last_fullsized":"2024-03-28T18:09:15.886913+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:59:57.788915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.019886+0000","last_change":"2024-03-28T18:08:20.031719+0000","last_active":"2024-03-28T18:09:44.019886+0000","last_peered":"2024-03-28T18:09:44.019886+0000","last_clean":"2024-03-28T18:09:44.019886+0000","last_became_active":"2024-03-28T18:08:20.029482+0000","last_became_peered":"2024-03-28T18:08:20.029482+0000","last_unstale":"2024-03-28T18:09:44.019886+0000","last_undegraded":"2024-03-28T18:09:44.019886+0000","last_fullsized":"2024-03-28T18:09:44.019886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:13.226596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'5","reported_seq":170,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:48.619711+0000","last_change":"2024-03-28T18:08:19.483346+0000","last_active":"2024-03-28T18:10:48.619711+0000","last_peered":"2024-03-28T18:10:48.619711+0000","last_clean":"2024-03-28T18:10:48.619711+0000","last_became_active":"2024-03-28T18:08:19.483016+0000","last_became_peered":"2024-03-28T18:08:19.483016+0000","last_unstale":"2024-03-28T18:10:48.619711+0000","last_undegraded":"2024-03-28T18:10:48.619711+0000","last_fullsized":"2024-03-28T18:10:48.619711+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:34:33.008477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":145,"num_read_kb":145,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.984960+0000","last_change":"2024-03-28T18:08:20.020478+0000","last_active":"2024-03-28T18:09:43.984960+0000","last_peered":"2024-03-28T18:09:43.984960+0000","last_clean":"2024-03-28T18:09:43.984960+0000","last_became_active":"2024-03-28T18:08:20.020054+0000","last_became_peered":"2024-03-28T18:08:20.020054+0000","last_unstale":"2024-03-28T18:09:43.984960+0000","last_undegraded":"2024-03-28T18:09:43.984960+0000","last_fullsized":"2024-03-28T18:09:43.984960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:13:43.655549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716206+0000","last_change":"2024-03-28T18:08:19.479386+0000","last_active":"2024-03-28T18:09:15.716206+0000","last_peered":"2024-03-28T18:09:15.716206+0000","last_clean":"2024-03-28T18:09:15.716206+0000","last_became_active":"2024-03-28T18:08:19.478985+0000","last_became_peered":"2024-03-28T18:08:19.478985+0000","last_unstale":"2024-03-28T18:09:15.716206+0000","last_undegraded":"2024-03-28T18:09:15.716206+0000","last_fullsized":"2024-03-28T18:09:15.716206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:37:22.359475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.007695+0000","last_change":"2024-03-28T18:08:20.021790+0000","last_active":"2024-03-28T18:09:44.007695+0000","last_peered":"2024-03-28T18:09:44.007695+0000","last_clean":"2024-03-28T18:09:44.007695+0000","last_became_active":"2024-03-28T18:08:20.021031+0000","last_became_peered":"2024-03-28T18:08:20.021031+0000","last_unstale":"2024-03-28T18:09:44.007695+0000","last_undegraded":"2024-03-28T18:09:44.007695+0000","last_fullsized":"2024-03-28T18:09:44.007695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:59:50.897692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887257+0000","last_change":"2024-03-28T18:08:19.481576+0000","last_active":"2024-03-28T18:09:15.887257+0000","last_peered":"2024-03-28T18:09:15.887257+0000","last_clean":"2024-03-28T18:09:15.887257+0000","last_became_active":"2024-03-28T18:08:19.481226+0000","last_became_peered":"2024-03-28T18:08:19.481226+0000","last_unstale":"2024-03-28T18:09:15.887257+0000","last_undegraded":"2024-03-28T18:09:15.887257+0000","last_fullsized":"2024-03-28T18:09:15.887257+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:54:44.677343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.013852+0000","last_change":"2024-03-28T18:09:15.890672+0000","last_active":"2024-03-28T18:09:44.013852+0000","last_peered":"2024-03-28T18:09:44.013852+0000","last_clean":"2024-03-28T18:09:44.013852+0000","last_became_active":"2024-03-28T18:09:15.890342+0000","last_became_peered":"2024-03-28T18:09:15.890342+0000","last_unstale":"2024-03-28T18:09:44.013852+0000","last_undegraded":"2024-03-28T18:09:44.013852+0000","last_fullsized":"2024-03-28T18:09:44.013852+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:08.268067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'34","reported_seq":354,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:49.577466+0000","last_change":"2024-03-28T18:08:19.477559+0000","last_active":"2024-03-28T18:10:49.577466+0000","last_peered":"2024-03-28T18:10:49.577466+0000","last_clean":"2024-03-28T18:10:49.577466+0000","last_became_active":"2024-03-28T18:08:19.477217+0000","last_became_peered":"2024-03-28T18:08:19.477217+0000","last_unstale":"2024-03-28T18:10:49.577466+0000","last_undegraded":"2024-03-28T18:10:49.577466+0000","last_fullsized":"2024-03-28T18:10:49.577466+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:37.071671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62451,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":300,"num_read_kb":211,"num_write":35,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.955611+0000","last_change":"2024-03-28T18:08:20.011303+0000","last_active":"2024-03-28T18:09:43.955611+0000","last_peered":"2024-03-28T18:09:43.955611+0000","last_clean":"2024-03-28T18:09:43.955611+0000","last_became_active":"2024-03-28T18:08:20.010997+0000","last_became_peered":"2024-03-28T18:08:20.010997+0000","last_unstale":"2024-03-28T18:09:43.955611+0000","last_undegraded":"2024-03-28T18:09:43.955611+0000","last_fullsized":"2024-03-28T18:09:43.955611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:10:56.036132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.718063+0000","last_change":"2024-03-28T18:08:19.477342+0000","last_active":"2024-03-28T18:09:15.718063+0000","last_peered":"2024-03-28T18:09:15.718063+0000","last_clean":"2024-03-28T18:09:15.718063+0000","last_became_active":"2024-03-28T18:08:19.476984+0000","last_became_peered":"2024-03-28T18:08:19.476984+0000","last_unstale":"2024-03-28T18:09:15.718063+0000","last_undegraded":"2024-03-28T18:09:15.718063+0000","last_fullsized":"2024-03-28T18:09:15.718063+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:15:42.606213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.017873+0000","last_change":"2024-03-28T18:08:20.030524+0000","last_active":"2024-03-28T18:09:44.017873+0000","last_peered":"2024-03-28T18:09:44.017873+0000","last_clean":"2024-03-28T18:09:44.017873+0000","last_became_active":"2024-03-28T18:08:20.030227+0000","last_became_peered":"2024-03-28T18:08:20.030227+0000","last_unstale":"2024-03-28T18:09:44.017873+0000","last_undegraded":"2024-03-28T18:09:44.017873+0000","last_fullsized":"2024-03-28T18:09:44.017873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:07:41.763739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.886737+0000","last_change":"2024-03-28T18:08:19.481169+0000","last_active":"2024-03-28T18:09:15.886737+0000","last_peered":"2024-03-28T18:09:15.886737+0000","last_clean":"2024-03-28T18:09:15.886737+0000","last_became_active":"2024-03-28T18:08:19.480634+0000","last_became_peered":"2024-03-28T18:08:19.480634+0000","last_unstale":"2024-03-28T18:09:15.886737+0000","last_undegraded":"2024-03-28T18:09:15.886737+0000","last_fullsized":"2024-03-28T18:09:15.886737+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:39.265215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.006975+0000","last_change":"2024-03-28T18:08:20.015163+0000","last_active":"2024-03-28T18:09:44.006975+0000","last_peered":"2024-03-28T18:09:44.006975+0000","last_clean":"2024-03-28T18:09:44.006975+0000","last_became_active":"2024-03-28T18:08:20.014765+0000","last_became_peered":"2024-03-28T18:08:20.014765+0000","last_unstale":"2024-03-28T18:09:44.006975+0000","last_undegraded":"2024-03-28T18:09:44.006975+0000","last_fullsized":"2024-03-28T18:09:44.006975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:51.035532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'3","reported_seq":169,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:47.626894+0000","last_change":"2024-03-28T18:08:19.479086+0000","last_active":"2024-03-28T18:10:47.626894+0000","last_peered":"2024-03-28T18:10:47.626894+0000","last_clean":"2024-03-28T18:10:47.626894+0000","last_became_active":"2024-03-28T18:08:19.478748+0000","last_became_peered":"2024-03-28T18:08:19.478748+0000","last_unstale":"2024-03-28T18:10:47.626894+0000","last_undegraded":"2024-03-28T18:10:47.626894+0000","last_fullsized":"2024-03-28T18:10:47.626894+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:46:59.566998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.973667+0000","last_change":"2024-03-28T18:09:15.888751+0000","last_active":"2024-03-28T18:09:43.973667+0000","last_peered":"2024-03-28T18:09:43.973667+0000","last_clean":"2024-03-28T18:09:43.973667+0000","last_became_active":"2024-03-28T18:09:15.888350+0000","last_became_peered":"2024-03-28T18:09:15.888350+0000","last_unstale":"2024-03-28T18:09:43.973667+0000","last_undegraded":"2024-03-28T18:09:43.973667+0000","last_fullsized":"2024-03-28T18:09:43.973667+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:38:05.822205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":60,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.733834+0000","last_change":"2024-03-28T18:09:15.733434+0000","last_active":"2024-03-28T18:09:15.733834+0000","last_peered":"2024-03-28T18:09:15.733834+0000","last_clean":"2024-03-28T18:09:15.733834+0000","last_became_active":"2024-03-28T18:09:15.722437+0000","last_became_peered":"2024-03-28T18:09:15.722437+0000","last_unstale":"2024-03-28T18:09:15.733834+0000","last_undegraded":"2024-03-28T18:09:15.733834+0000","last_fullsized":"2024-03-28T18:09:15.733834+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:35:18.313676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.971701+0000","last_change":"2024-03-28T18:08:20.007457+0000","last_active":"2024-03-28T18:09:43.971701+0000","last_peered":"2024-03-28T18:09:43.971701+0000","last_clean":"2024-03-28T18:09:43.971701+0000","last_became_active":"2024-03-28T18:08:20.007090+0000","last_became_peered":"2024-03-28T18:08:20.007090+0000","last_unstale":"2024-03-28T18:09:43.971701+0000","last_undegraded":"2024-03-28T18:09:43.971701+0000","last_fullsized":"2024-03-28T18:09:43.971701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:22:45.702706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716396+0000","last_change":"2024-03-28T18:08:19.475104+0000","last_active":"2024-03-28T18:09:15.716396+0000","last_peered":"2024-03-28T18:09:15.716396+0000","last_clean":"2024-03-28T18:09:15.716396+0000","last_became_active":"2024-03-28T18:08:19.474754+0000","last_became_peered":"2024-03-28T18:08:19.474754+0000","last_unstale":"2024-03-28T18:09:15.716396+0000","last_undegraded":"2024-03-28T18:09:15.716396+0000","last_fullsized":"2024-03-28T18:09:15.716396+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:19.085631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.968043+0000","last_change":"2024-03-28T18:08:20.009347+0000","last_active":"2024-03-28T18:09:43.968043+0000","last_peered":"2024-03-28T18:09:43.968043+0000","last_clean":"2024-03-28T18:09:43.968043+0000","last_became_active":"2024-03-28T18:08:20.009012+0000","last_became_peered":"2024-03-28T18:08:20.009012+0000","last_unstale":"2024-03-28T18:09:43.968043+0000","last_undegraded":"2024-03-28T18:09:43.968043+0000","last_fullsized":"2024-03-28T18:09:43.968043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:04:26.625830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.721956+0000","last_change":"2024-03-28T18:08:19.482414+0000","last_active":"2024-03-28T18:09:15.721956+0000","last_peered":"2024-03-28T18:09:15.721956+0000","last_clean":"2024-03-28T18:09:15.721956+0000","last_became_active":"2024-03-28T18:08:19.481830+0000","last_became_peered":"2024-03-28T18:08:19.481830+0000","last_unstale":"2024-03-28T18:09:15.721956+0000","last_undegraded":"2024-03-28T18:09:15.721956+0000","last_fullsized":"2024-03-28T18:09:15.721956+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:27:14.259355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.001824+0000","last_change":"2024-03-28T18:08:20.030687+0000","last_active":"2024-03-28T18:09:44.001824+0000","last_peered":"2024-03-28T18:09:44.001824+0000","last_clean":"2024-03-28T18:09:44.001824+0000","last_became_active":"2024-03-28T18:08:20.030308+0000","last_became_peered":"2024-03-28T18:08:20.030308+0000","last_unstale":"2024-03-28T18:09:44.001824+0000","last_undegraded":"2024-03-28T18:09:44.001824+0000","last_fullsized":"2024-03-28T18:09:44.001824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:31.745952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716000+0000","last_change":"2024-03-28T18:08:19.494446+0000","last_active":"2024-03-28T18:09:15.716000+0000","last_peered":"2024-03-28T18:09:15.716000+0000","last_clean":"2024-03-28T18:09:15.716000+0000","last_became_active":"2024-03-28T18:08:19.494036+0000","last_became_peered":"2024-03-28T18:08:19.494036+0000","last_unstale":"2024-03-28T18:09:15.716000+0000","last_undegraded":"2024-03-28T18:09:15.716000+0000","last_fullsized":"2024-03-28T18:09:15.716000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:42:53.660741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.009456+0000","last_change":"2024-03-28T18:08:20.025706+0000","last_active":"2024-03-28T18:09:44.009456+0000","last_peered":"2024-03-28T18:09:44.009456+0000","last_clean":"2024-03-28T18:09:44.009456+0000","last_became_active":"2024-03-28T18:08:20.025343+0000","last_became_peered":"2024-03-28T18:08:20.025343+0000","last_unstale":"2024-03-28T18:09:44.009456+0000","last_undegraded":"2024-03-28T18:09:44.009456+0000","last_fullsized":"2024-03-28T18:09:44.009456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:32:27.883940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.963975+0000","last_change":"2024-03-28T18:08:20.009636+0000","last_active":"2024-03-28T18:09:43.963975+0000","last_peered":"2024-03-28T18:09:43.963975+0000","last_clean":"2024-03-28T18:09:43.963975+0000","last_became_active":"2024-03-28T18:08:20.009178+0000","last_became_peered":"2024-03-28T18:08:20.009178+0000","last_unstale":"2024-03-28T18:09:43.963975+0000","last_undegraded":"2024-03-28T18:09:43.963975+0000","last_fullsized":"2024-03-28T18:09:43.963975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:28.461411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":169,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:48.628017+0000","last_change":"2024-03-28T18:08:19.501287+0000","last_active":"2024-03-28T18:10:48.628017+0000","last_peered":"2024-03-28T18:10:48.628017+0000","last_clean":"2024-03-28T18:10:48.628017+0000","last_became_active":"2024-03-28T18:08:19.500920+0000","last_became_peered":"2024-03-28T18:08:19.500920+0000","last_unstale":"2024-03-28T18:10:48.628017+0000","last_undegraded":"2024-03-28T18:10:48.628017+0000","last_fullsized":"2024-03-28T18:10:48.628017+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:29.882164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":145,"num_read_kb":145,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.013107+0000","last_change":"2024-03-28T18:08:20.020811+0000","last_active":"2024-03-28T18:09:44.013107+0000","last_peered":"2024-03-28T18:09:44.013107+0000","last_clean":"2024-03-28T18:09:44.013107+0000","last_became_active":"2024-03-28T18:08:20.020381+0000","last_became_peered":"2024-03-28T18:08:20.020381+0000","last_unstale":"2024-03-28T18:09:44.013107+0000","last_undegraded":"2024-03-28T18:09:44.013107+0000","last_fullsized":"2024-03-28T18:09:44.013107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:10:35.999858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887764+0000","last_change":"2024-03-28T18:08:19.478020+0000","last_active":"2024-03-28T18:09:15.887764+0000","last_peered":"2024-03-28T18:09:15.887764+0000","last_clean":"2024-03-28T18:09:15.887764+0000","last_became_active":"2024-03-28T18:08:19.477627+0000","last_became_peered":"2024-03-28T18:08:19.477627+0000","last_unstale":"2024-03-28T18:09:15.887764+0000","last_undegraded":"2024-03-28T18:09:15.887764+0000","last_fullsized":"2024-03-28T18:09:15.887764+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:00:01.649974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":354,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.715885+0000","last_change":"2024-03-28T18:06:58.750932+0000","last_active":"2024-03-28T18:09:15.715885+0000","last_peered":"2024-03-28T18:09:15.715885+0000","last_clean":"2024-03-28T18:09:15.715885+0000","last_became_active":"2024-03-28T18:06:58.750309+0000","last_became_peered":"2024-03-28T18:06:58.750309+0000","last_unstale":"2024-03-28T18:09:15.715885+0000","last_undegraded":"2024-03-28T18:09:15.715885+0000","last_fullsized":"2024-03-28T18:09:15.715885+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:03:00.832683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:03:00.832683+0000","last_clean_scrub_stamp":"2024-03-28T18:03:00.832683+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:46.001886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.002070+0000","last_change":"2024-03-28T18:08:20.009573+0000","last_active":"2024-03-28T18:09:44.002070+0000","last_peered":"2024-03-28T18:09:44.002070+0000","last_clean":"2024-03-28T18:09:44.002070+0000","last_became_active":"2024-03-28T18:08:20.009143+0000","last_became_peered":"2024-03-28T18:08:20.009143+0000","last_unstale":"2024-03-28T18:09:44.002070+0000","last_undegraded":"2024-03-28T18:09:44.002070+0000","last_fullsized":"2024-03-28T18:09:44.002070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:37:50.643356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716076+0000","last_change":"2024-03-28T18:08:19.483733+0000","last_active":"2024-03-28T18:09:15.716076+0000","last_peered":"2024-03-28T18:09:15.716076+0000","last_clean":"2024-03-28T18:09:15.716076+0000","last_became_active":"2024-03-28T18:08:19.483366+0000","last_became_peered":"2024-03-28T18:08:19.483366+0000","last_unstale":"2024-03-28T18:09:15.716076+0000","last_undegraded":"2024-03-28T18:09:15.716076+0000","last_fullsized":"2024-03-28T18:09:15.716076+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:29.192442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716038+0000","last_change":"2024-03-28T18:08:19.481844+0000","last_active":"2024-03-28T18:09:15.716038+0000","last_peered":"2024-03-28T18:09:15.716038+0000","last_clean":"2024-03-28T18:09:15.716038+0000","last_became_active":"2024-03-28T18:08:19.481430+0000","last_became_peered":"2024-03-28T18:08:19.481430+0000","last_unstale":"2024-03-28T18:09:15.716038+0000","last_undegraded":"2024-03-28T18:09:15.716038+0000","last_fullsized":"2024-03-28T18:09:15.716038+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:21:28.789883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.018278+0000","last_change":"2024-03-28T18:08:20.021037+0000","last_active":"2024-03-28T18:09:44.018278+0000","last_peered":"2024-03-28T18:09:44.018278+0000","last_clean":"2024-03-28T18:09:44.018278+0000","last_became_active":"2024-03-28T18:08:20.020680+0000","last_became_peered":"2024-03-28T18:08:20.020680+0000","last_unstale":"2024-03-28T18:09:44.018278+0000","last_undegraded":"2024-03-28T18:09:44.018278+0000","last_fullsized":"2024-03-28T18:09:44.018278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:08:28.825774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'11","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:28.677487+0000","last_change":"2024-03-28T18:09:15.737583+0000","last_active":"2024-03-28T18:10:28.677487+0000","last_peered":"2024-03-28T18:10:28.677487+0000","last_clean":"2024-03-28T18:10:28.677487+0000","last_became_active":"2024-03-28T18:09:15.726579+0000","last_became_peered":"2024-03-28T18:09:15.726579+0000","last_unstale":"2024-03-28T18:10:28.677487+0000","last_undegraded":"2024-03-28T18:10:28.677487+0000","last_fullsized":"2024-03-28T18:10:28.677487+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:03.380037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.011149+0000","last_change":"2024-03-28T18:08:20.024938+0000","last_active":"2024-03-28T18:09:44.011149+0000","last_peered":"2024-03-28T18:09:44.011149+0000","last_clean":"2024-03-28T18:09:44.011149+0000","last_became_active":"2024-03-28T18:08:20.024536+0000","last_became_peered":"2024-03-28T18:08:20.024536+0000","last_unstale":"2024-03-28T18:09:44.011149+0000","last_undegraded":"2024-03-28T18:09:44.011149+0000","last_fullsized":"2024-03-28T18:09:44.011149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:46:27.080005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.888043+0000","last_change":"2024-03-28T18:08:19.480310+0000","last_active":"2024-03-28T18:09:15.888043+0000","last_peered":"2024-03-28T18:09:15.888043+0000","last_clean":"2024-03-28T18:09:15.888043+0000","last_became_active":"2024-03-28T18:08:19.479919+0000","last_became_peered":"2024-03-28T18:08:19.479919+0000","last_unstale":"2024-03-28T18:09:15.888043+0000","last_undegraded":"2024-03-28T18:09:15.888043+0000","last_fullsized":"2024-03-28T18:09:15.888043+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:36:26.239515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.018864+0000","last_change":"2024-03-28T18:08:20.033974+0000","last_active":"2024-03-28T18:09:44.018864+0000","last_peered":"2024-03-28T18:09:44.018864+0000","last_clean":"2024-03-28T18:09:44.018864+0000","last_became_active":"2024-03-28T18:08:20.033544+0000","last_became_peered":"2024-03-28T18:08:20.033544+0000","last_unstale":"2024-03-28T18:09:44.018864+0000","last_undegraded":"2024-03-28T18:09:44.018864+0000","last_fullsized":"2024-03-28T18:09:44.018864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:45:09.617988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.718382+0000","last_change":"2024-03-28T18:08:19.482604+0000","last_active":"2024-03-28T18:09:15.718382+0000","last_peered":"2024-03-28T18:09:15.718382+0000","last_clean":"2024-03-28T18:09:15.718382+0000","last_became_active":"2024-03-28T18:08:19.482248+0000","last_became_peered":"2024-03-28T18:08:19.482248+0000","last_unstale":"2024-03-28T18:09:15.718382+0000","last_undegraded":"2024-03-28T18:09:15.718382+0000","last_fullsized":"2024-03-28T18:09:15.718382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:23.199196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.947114+0000","last_change":"2024-03-28T18:08:20.011198+0000","last_active":"2024-03-28T18:09:43.947114+0000","last_peered":"2024-03-28T18:09:43.947114+0000","last_clean":"2024-03-28T18:09:43.947114+0000","last_became_active":"2024-03-28T18:08:20.010866+0000","last_became_peered":"2024-03-28T18:08:20.010866+0000","last_unstale":"2024-03-28T18:09:43.947114+0000","last_undegraded":"2024-03-28T18:09:43.947114+0000","last_fullsized":"2024-03-28T18:09:43.947114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:26.342858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.724859+0000","last_change":"2024-03-28T18:08:19.493830+0000","last_active":"2024-03-28T18:09:15.724859+0000","last_peered":"2024-03-28T18:09:15.724859+0000","last_clean":"2024-03-28T18:09:15.724859+0000","last_became_active":"2024-03-28T18:08:19.493447+0000","last_became_peered":"2024-03-28T18:08:19.493447+0000","last_unstale":"2024-03-28T18:09:15.724859+0000","last_undegraded":"2024-03-28T18:09:15.724859+0000","last_fullsized":"2024-03-28T18:09:15.724859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:33:29.797186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.008626+0000","last_change":"2024-03-28T18:08:20.025620+0000","last_active":"2024-03-28T18:09:44.008626+0000","last_peered":"2024-03-28T18:09:44.008626+0000","last_clean":"2024-03-28T18:09:44.008626+0000","last_became_active":"2024-03-28T18:08:20.025276+0000","last_became_peered":"2024-03-28T18:08:20.025276+0000","last_unstale":"2024-03-28T18:09:44.008626+0000","last_undegraded":"2024-03-28T18:09:44.008626+0000","last_fullsized":"2024-03-28T18:09:44.008626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:30:06.667707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:30.366681+0000","last_change":"2024-03-28T18:08:19.474204+0000","last_active":"2024-03-28T18:10:30.366681+0000","last_peered":"2024-03-28T18:10:30.366681+0000","last_clean":"2024-03-28T18:10:30.366681+0000","last_became_active":"2024-03-28T18:08:19.473688+0000","last_became_peered":"2024-03-28T18:08:19.473688+0000","last_unstale":"2024-03-28T18:10:30.366681+0000","last_undegraded":"2024-03-28T18:10:30.366681+0000","last_fullsized":"2024-03-28T18:10:30.366681+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:15:09.880974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.015204+0000","last_change":"2024-03-28T18:08:20.035920+0000","last_active":"2024-03-28T18:09:44.015204+0000","last_peered":"2024-03-28T18:09:44.015204+0000","last_clean":"2024-03-28T18:09:44.015204+0000","last_became_active":"2024-03-28T18:08:20.035600+0000","last_became_peered":"2024-03-28T18:08:20.035600+0000","last_unstale":"2024-03-28T18:09:44.015204+0000","last_undegraded":"2024-03-28T18:09:44.015204+0000","last_fullsized":"2024-03-28T18:09:44.015204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:41.280448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.715581+0000","last_change":"2024-03-28T18:08:19.481305+0000","last_active":"2024-03-28T18:09:15.715581+0000","last_peered":"2024-03-28T18:09:15.715581+0000","last_clean":"2024-03-28T18:09:15.715581+0000","last_became_active":"2024-03-28T18:08:19.480978+0000","last_became_peered":"2024-03-28T18:08:19.480978+0000","last_unstale":"2024-03-28T18:09:15.715581+0000","last_undegraded":"2024-03-28T18:09:15.715581+0000","last_fullsized":"2024-03-28T18:09:15.715581+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:14:06.464967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.002331+0000","last_change":"2024-03-28T18:08:20.017660+0000","last_active":"2024-03-28T18:09:44.002331+0000","last_peered":"2024-03-28T18:09:44.002331+0000","last_clean":"2024-03-28T18:09:44.002331+0000","last_became_active":"2024-03-28T18:08:20.016083+0000","last_became_peered":"2024-03-28T18:08:20.016083+0000","last_unstale":"2024-03-28T18:09:44.002331+0000","last_undegraded":"2024-03-28T18:09:44.002331+0000","last_fullsized":"2024-03-28T18:09:44.002331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:33:12.104274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.718320+0000","last_change":"2024-03-28T18:08:19.479855+0000","last_active":"2024-03-28T18:09:15.718320+0000","last_peered":"2024-03-28T18:09:15.718320+0000","last_clean":"2024-03-28T18:09:15.718320+0000","last_became_active":"2024-03-28T18:08:19.479429+0000","last_became_peered":"2024-03-28T18:08:19.479429+0000","last_unstale":"2024-03-28T18:09:15.718320+0000","last_undegraded":"2024-03-28T18:09:15.718320+0000","last_fullsized":"2024-03-28T18:09:15.718320+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:06:23.286970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.978524+0000","last_change":"2024-03-28T18:08:20.015419+0000","last_active":"2024-03-28T18:09:43.978524+0000","last_peered":"2024-03-28T18:09:43.978524+0000","last_clean":"2024-03-28T18:09:43.978524+0000","last_became_active":"2024-03-28T18:08:20.015011+0000","last_became_peered":"2024-03-28T18:08:20.015011+0000","last_unstale":"2024-03-28T18:09:43.978524+0000","last_undegraded":"2024-03-28T18:09:43.978524+0000","last_fullsized":"2024-03-28T18:09:43.978524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:44:46.858352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887142+0000","last_change":"2024-03-28T18:08:19.495541+0000","last_active":"2024-03-28T18:09:15.887142+0000","last_peered":"2024-03-28T18:09:15.887142+0000","last_clean":"2024-03-28T18:09:15.887142+0000","last_became_active":"2024-03-28T18:08:19.495173+0000","last_became_peered":"2024-03-28T18:08:19.495173+0000","last_unstale":"2024-03-28T18:09:15.887142+0000","last_undegraded":"2024-03-28T18:09:15.887142+0000","last_fullsized":"2024-03-28T18:09:15.887142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:42:06.423107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.981192+0000","last_change":"2024-03-28T18:08:20.018582+0000","last_active":"2024-03-28T18:09:43.981192+0000","last_peered":"2024-03-28T18:09:43.981192+0000","last_clean":"2024-03-28T18:09:43.981192+0000","last_became_active":"2024-03-28T18:08:20.017936+0000","last_became_peered":"2024-03-28T18:08:20.017936+0000","last_unstale":"2024-03-28T18:09:43.981192+0000","last_undegraded":"2024-03-28T18:09:43.981192+0000","last_fullsized":"2024-03-28T18:09:43.981192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:05:49.510246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.889005+0000","last_change":"2024-03-28T18:08:19.497049+0000","last_active":"2024-03-28T18:09:15.889005+0000","last_peered":"2024-03-28T18:09:15.889005+0000","last_clean":"2024-03-28T18:09:15.889005+0000","last_became_active":"2024-03-28T18:08:19.496623+0000","last_became_peered":"2024-03-28T18:08:19.496623+0000","last_unstale":"2024-03-28T18:09:15.889005+0000","last_undegraded":"2024-03-28T18:09:15.889005+0000","last_fullsized":"2024-03-28T18:09:15.889005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:55:07.842439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.978646+0000","last_change":"2024-03-28T18:08:20.017419+0000","last_active":"2024-03-28T18:09:43.978646+0000","last_peered":"2024-03-28T18:09:43.978646+0000","last_clean":"2024-03-28T18:09:43.978646+0000","last_became_active":"2024-03-28T18:08:20.015916+0000","last_became_peered":"2024-03-28T18:08:20.015916+0000","last_unstale":"2024-03-28T18:09:43.978646+0000","last_undegraded":"2024-03-28T18:09:43.978646+0000","last_fullsized":"2024-03-28T18:09:43.978646+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:10:55.927794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.964132+0000","last_change":"2024-03-28T18:08:20.008093+0000","last_active":"2024-03-28T18:09:43.964132+0000","last_peered":"2024-03-28T18:09:43.964132+0000","last_clean":"2024-03-28T18:09:43.964132+0000","last_became_active":"2024-03-28T18:08:20.007801+0000","last_became_peered":"2024-03-28T18:08:20.007801+0000","last_unstale":"2024-03-28T18:09:43.964132+0000","last_undegraded":"2024-03-28T18:09:43.964132+0000","last_fullsized":"2024-03-28T18:09:43.964132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:58.976859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.967834+0000","last_change":"2024-03-28T18:08:20.012771+0000","last_active":"2024-03-28T18:09:43.967834+0000","last_peered":"2024-03-28T18:09:43.967834+0000","last_clean":"2024-03-28T18:09:43.967834+0000","last_became_active":"2024-03-28T18:08:20.012456+0000","last_became_peered":"2024-03-28T18:08:20.012456+0000","last_unstale":"2024-03-28T18:09:43.967834+0000","last_undegraded":"2024-03-28T18:09:43.967834+0000","last_fullsized":"2024-03-28T18:09:43.967834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:56.428332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.013650+0000","last_change":"2024-03-28T18:08:20.034872+0000","last_active":"2024-03-28T18:09:44.013650+0000","last_peered":"2024-03-28T18:09:44.013650+0000","last_clean":"2024-03-28T18:09:44.013650+0000","last_became_active":"2024-03-28T18:08:20.034614+0000","last_became_peered":"2024-03-28T18:08:20.034614+0000","last_unstale":"2024-03-28T18:09:44.013650+0000","last_undegraded":"2024-03-28T18:09:44.013650+0000","last_fullsized":"2024-03-28T18:09:44.013650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:35.658186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.018419+0000","last_change":"2024-03-28T18:08:20.033119+0000","last_active":"2024-03-28T18:09:44.018419+0000","last_peered":"2024-03-28T18:09:44.018419+0000","last_clean":"2024-03-28T18:09:44.018419+0000","last_became_active":"2024-03-28T18:08:20.030694+0000","last_became_peered":"2024-03-28T18:08:20.030694+0000","last_unstale":"2024-03-28T18:09:44.018419+0000","last_undegraded":"2024-03-28T18:09:44.018419+0000","last_fullsized":"2024-03-28T18:09:44.018419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:51:50.082993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.020461+0000","last_change":"2024-03-28T18:08:20.025725+0000","last_active":"2024-03-28T18:09:44.020461+0000","last_peered":"2024-03-28T18:09:44.020461+0000","last_clean":"2024-03-28T18:09:44.020461+0000","last_became_active":"2024-03-28T18:08:20.024929+0000","last_became_peered":"2024-03-28T18:08:20.024929+0000","last_unstale":"2024-03-28T18:09:44.020461+0000","last_undegraded":"2024-03-28T18:09:44.020461+0000","last_fullsized":"2024-03-28T18:09:44.020461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:04.894246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.960946+0000","last_change":"2024-03-28T18:08:20.005293+0000","last_active":"2024-03-28T18:09:43.960946+0000","last_peered":"2024-03-28T18:09:43.960946+0000","last_clean":"2024-03-28T18:09:43.960946+0000","last_became_active":"2024-03-28T18:08:20.004938+0000","last_became_peered":"2024-03-28T18:08:20.004938+0000","last_unstale":"2024-03-28T18:09:43.960946+0000","last_undegraded":"2024-03-28T18:09:43.960946+0000","last_fullsized":"2024-03-28T18:09:43.960946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:36.100810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.971806+0000","last_change":"2024-03-28T18:08:20.008741+0000","last_active":"2024-03-28T18:09:43.971806+0000","last_peered":"2024-03-28T18:09:43.971806+0000","last_clean":"2024-03-28T18:09:43.971806+0000","last_became_active":"2024-03-28T18:08:20.008411+0000","last_became_peered":"2024-03-28T18:08:20.008411+0000","last_unstale":"2024-03-28T18:09:43.971806+0000","last_undegraded":"2024-03-28T18:09:43.971806+0000","last_fullsized":"2024-03-28T18:09:43.971806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:28:52.276278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.997108+0000","last_change":"2024-03-28T18:08:20.028486+0000","last_active":"2024-03-28T18:09:43.997108+0000","last_peered":"2024-03-28T18:09:43.997108+0000","last_clean":"2024-03-28T18:09:43.997108+0000","last_became_active":"2024-03-28T18:08:20.028121+0000","last_became_peered":"2024-03-28T18:08:20.028121+0000","last_unstale":"2024-03-28T18:09:43.997108+0000","last_undegraded":"2024-03-28T18:09:43.997108+0000","last_fullsized":"2024-03-28T18:09:43.997108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:55:26.828305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.970707+0000","last_change":"2024-03-28T18:08:20.028085+0000","last_active":"2024-03-28T18:09:43.970707+0000","last_peered":"2024-03-28T18:09:43.970707+0000","last_clean":"2024-03-28T18:09:43.970707+0000","last_became_active":"2024-03-28T18:08:20.027721+0000","last_became_peered":"2024-03-28T18:08:20.027721+0000","last_unstale":"2024-03-28T18:09:43.970707+0000","last_undegraded":"2024-03-28T18:09:43.970707+0000","last_fullsized":"2024-03-28T18:09:43.970707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:31:36.674002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.012065+0000","last_change":"2024-03-28T18:08:20.023605+0000","last_active":"2024-03-28T18:09:44.012065+0000","last_peered":"2024-03-28T18:09:44.012065+0000","last_clean":"2024-03-28T18:09:44.012065+0000","last_became_active":"2024-03-28T18:08:20.018217+0000","last_became_peered":"2024-03-28T18:08:20.018217+0000","last_unstale":"2024-03-28T18:09:44.012065+0000","last_undegraded":"2024-03-28T18:09:44.012065+0000","last_fullsized":"2024-03-28T18:09:44.012065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:43.717751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.014919+0000","last_change":"2024-03-28T18:08:20.023072+0000","last_active":"2024-03-28T18:09:44.014919+0000","last_peered":"2024-03-28T18:09:44.014919+0000","last_clean":"2024-03-28T18:09:44.014919+0000","last_became_active":"2024-03-28T18:08:20.022690+0000","last_became_peered":"2024-03-28T18:08:20.022690+0000","last_unstale":"2024-03-28T18:09:44.014919+0000","last_undegraded":"2024-03-28T18:09:44.014919+0000","last_fullsized":"2024-03-28T18:09:44.014919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:21.391415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.018990+0000","last_change":"2024-03-28T18:08:20.024261+0000","last_active":"2024-03-28T18:09:44.018990+0000","last_peered":"2024-03-28T18:09:44.018990+0000","last_clean":"2024-03-28T18:09:44.018990+0000","last_became_active":"2024-03-28T18:08:20.023792+0000","last_became_peered":"2024-03-28T18:08:20.023792+0000","last_unstale":"2024-03-28T18:09:44.018990+0000","last_undegraded":"2024-03-28T18:09:44.018990+0000","last_fullsized":"2024-03-28T18:09:44.018990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:23:08.030982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.968857+0000","last_change":"2024-03-28T18:09:15.720113+0000","last_active":"2024-03-28T18:09:43.968857+0000","last_peered":"2024-03-28T18:09:43.968857+0000","last_clean":"2024-03-28T18:09:43.968857+0000","last_became_active":"2024-03-28T18:09:15.719739+0000","last_became_peered":"2024-03-28T18:09:15.719739+0000","last_unstale":"2024-03-28T18:09:43.968857+0000","last_undegraded":"2024-03-28T18:09:43.968857+0000","last_fullsized":"2024-03-28T18:09:43.968857+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:09:38.179780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":248983,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66899,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":901,"num_read_kb":522,"num_write":174,"num_write_kb":121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":694,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":205011,"data_compressed":5634,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383725,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":456779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645362,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":1110129,"data_compressed":37917,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808888,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070526,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332164,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":1048148,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495689,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":455822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":522492,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738476,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":457093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411424,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1024000,"data_stored":520142,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182491,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":456133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920854,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":456259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757327,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":1048414,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34394,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34394,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":66877,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64527,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64029,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:10:55.583 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-28T18:10:55.583 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:10:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.855+0000 7f8596200640 1 -- 172.21.15.37:0/1661230443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85901071d0 msgr2=0x7f8590109620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.855+0000 7f8596200640 1 --2- 172.21.15.37:0/1661230443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85901071d0 0x7f8590109620 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8588009ca0 tx=0x7f858802c100 comp rx=0 tx=0).stop 2024-03-28T18:10:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.855+0000 7f8596200640 1 -- 172.21.15.37:0/1661230443 shutdown_connections 2024-03-28T18:10:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.855+0000 7f8596200640 1 --2- 172.21.15.37:0/1661230443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85901071d0 0x7f8590109620 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.855+0000 7f8596200640 1 --2- 172.21.15.37:0/1661230443 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85901048a0 0x7f8590106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.855+0000 7f8596200640 1 --2- 172.21.15.37:0/1661230443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8590101f70 0x7f8590104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.855+0000 7f8596200640 1 -- 172.21.15.37:0/1661230443 >> 172.21.15.37:0/1661230443 conn(0x7f85900fb980 msgr2=0x7f85900fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.855+0000 7f8596200640 1 -- 172.21.15.37:0/1661230443 shutdown_connections 2024-03-28T18:10:55.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.855+0000 7f8596200640 1 -- 172.21.15.37:0/1661230443 wait complete. 2024-03-28T18:10:55.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.856+0000 7f8596200640 1 Processor -- start 2024-03-28T18:10:55.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.857+0000 7f8596200640 1 -- start start 2024-03-28T18:10:55.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.857+0000 7f8596200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8590101f70 0x7f85901999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:55.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.857+0000 7f8596200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85901048a0 0x7f8590199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:55.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.857+0000 7f8596200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85901071d0 0x7f85901a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.857+0000 7f8596200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859010d180 con 0x7f8590101f70 2024-03-28T18:10:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.857+0000 7f8596200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859010d000 con 0x7f85901071d0 2024-03-28T18:10:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.857+0000 7f8596200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859010d300 con 0x7f85901048a0 2024-03-28T18:10:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.857+0000 7f858fe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85901048a0 0x7f8590199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.857+0000 7f8594e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8590101f70 0x7f85901999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.857+0000 7f8595800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85901071d0 0x7f85901a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.857+0000 7f8595800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85901071d0 0x7f85901a0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:36918/0 (socket says 172.21.15.37:36918) 2024-03-28T18:10:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.857+0000 7f8595800640 1 -- 172.21.15.37:0/1169816536 learned_addr learned my addr 172.21.15.37:0/1169816536 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.857+0000 7f858fe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85901048a0 0x7f8590199f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55822/0 (socket says 172.21.15.37:55822) 2024-03-28T18:10:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.858+0000 7f8595800640 1 -- 172.21.15.37:0/1169816536 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85901048a0 msgr2=0x7f8590199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.858+0000 7f8595800640 1 --2- 172.21.15.37:0/1169816536 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85901048a0 0x7f8590199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.858+0000 7f8595800640 1 -- 172.21.15.37:0/1169816536 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8590101f70 msgr2=0x7f85901999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.858+0000 7f8595800640 1 --2- 172.21.15.37:0/1169816536 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8590101f70 0x7f85901999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:55.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.858+0000 7f8595800640 1 -- 172.21.15.37:0/1169816536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85901a15f0 con 0x7f85901071d0 2024-03-28T18:10:55.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.858+0000 7f8595800640 1 --2- 172.21.15.37:0/1169816536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85901071d0 0x7f85901a0f80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f858802c640 tx=0x7f8588035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:55.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.858+0000 7f858d600640 1 -- 172.21.15.37:0/1169816536 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8588039040 con 0x7f85901071d0 2024-03-28T18:10:55.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.858+0000 7f858d600640 1 -- 172.21.15.37:0/1169816536 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8588044070 con 0x7f85901071d0 2024-03-28T18:10:55.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.858+0000 7f8596200640 1 -- 172.21.15.37:0/1169816536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85901a1880 con 0x7f85901071d0 2024-03-28T18:10:55.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.858+0000 7f858d600640 1 -- 172.21.15.37:0/1169816536 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85880043c0 con 0x7f85901071d0 2024-03-28T18:10:55.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.859+0000 7f8596200640 1 -- 172.21.15.37:0/1169816536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85901a1dc0 con 0x7f85901071d0 2024-03-28T18:10:55.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.860+0000 7f8596200640 1 -- 172.21.15.37:0/1169816536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8590108cc0 con 0x7f85901071d0 2024-03-28T18:10:55.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.861+0000 7f858d600640 1 -- 172.21.15.37:0/1169816536 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8588004560 con 0x7f85901071d0 2024-03-28T18:10:55.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.863+0000 7f858d600640 1 --2- 172.21.15.37:0/1169816536 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f85780a2880 0x7f85780a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:55.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.863+0000 7f858d600640 1 -- 172.21.15.37:0/1169816536 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f85880f0fe0 con 0x7f85901071d0 2024-03-28T18:10:55.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.863+0000 7f8594e00640 1 --2- 172.21.15.37:0/1169816536 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f85780a2880 0x7f85780a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:55.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.864+0000 7f8594e00640 1 --2- 172.21.15.37:0/1169816536 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f85780a2880 0x7f85780a4d40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8584009bd0 tx=0x7f8584005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:55.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:55.867+0000 7f858d600640 1 -- 172.21.15.37:0/1169816536 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f85880be030 con 0x7f85901071d0 2024-03-28T18:10:55.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:55 smithi037 ceph-mon[31460]: pgmap v397: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-28T18:10:55.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:55 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1038418910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:55.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:55 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:10:55.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:55 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:10:55.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:55 smithi037 ceph-mon[31460]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:55 smithi059 ceph-mon[36190]: pgmap v397: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-28T18:10:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:55 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1038418910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:55 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:10:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:55 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:10:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:55 smithi059 ceph-mon[36190]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:56.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:55 smithi170 ceph-mon[36280]: pgmap v397: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-28T18:10:56.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:55 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1038418910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:55 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:10:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:55 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:10:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:55 smithi170 ceph-mon[36280]: from='mgr.14152 172.21.15.37:0/1129254547' entity='mgr.x' 2024-03-28T18:10:56.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.071+0000 7f8596200640 1 -- 172.21.15.37:0/1169816536 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8590061ae0 con 0x7f85780a2880 2024-03-28T18:10:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.097+0000 7f858d600640 1 -- 172.21.15.37:0/1169816536 <== mgr.14152 v2:172.21.15.37:6800/521857560 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259038 (secure 0 0 0) 0x7f8590061ae0 con 0x7f85780a2880 2024-03-28T18:10:56.101 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:56.102 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-03-28T18:10:56.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.103+0000 7f8596200640 1 -- 172.21.15.37:0/1169816536 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f85780a2880 msgr2=0x7f85780a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:56.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.103+0000 7f8596200640 1 --2- 172.21.15.37:0/1169816536 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f85780a2880 0x7f85780a4d40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8584009bd0 tx=0x7f8584005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:56.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.103+0000 7f8596200640 1 -- 172.21.15.37:0/1169816536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85901071d0 msgr2=0x7f85901a0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:56.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.103+0000 7f8596200640 1 --2- 172.21.15.37:0/1169816536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85901071d0 0x7f85901a0f80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f858802c640 tx=0x7f8588035d10 comp rx=0 tx=0).stop 2024-03-28T18:10:56.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.104+0000 7f8596200640 1 -- 172.21.15.37:0/1169816536 shutdown_connections 2024-03-28T18:10:56.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.104+0000 7f8596200640 1 --2- 172.21.15.37:0/1169816536 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f85780a2880 0x7f85780a4d40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:56.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.104+0000 7f8596200640 1 --2- 172.21.15.37:0/1169816536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85901071d0 0x7f85901a0f80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:56.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.104+0000 7f8596200640 1 --2- 172.21.15.37:0/1169816536 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85901048a0 0x7f8590199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:56.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.104+0000 7f8596200640 1 --2- 172.21.15.37:0/1169816536 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8590101f70 0x7f85901999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:56.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.104+0000 7f8596200640 1 -- 172.21.15.37:0/1169816536 >> 172.21.15.37:0/1169816536 conn(0x7f85900fb980 msgr2=0x7f8590101650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:56.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.104+0000 7f8596200640 1 -- 172.21.15.37:0/1169816536 shutdown_connections 2024-03-28T18:10:56.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.104+0000 7f8596200640 1 -- 172.21.15.37:0/1169816536 wait complete. 2024-03-28T18:10:56.126 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2024-03-28T18:10:56.014833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661815,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":1068,"num_read_kb":661,"num_write":358,"num_write_kb":3066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1181430,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":789,"ondisk_log_size":789,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377372,"kb_used_data":11580,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564452,"statfs":{"total":1151940427776,"available":1151553998848,"internally_reserved":0,"allocated":11857920,"data_stored":7452825,"data_compressed":113751,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4926,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":93,"num_read_kb":52,"num_write":9,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004817"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.005929+0000","last_change":"2024-03-28T18:08:20.025576+0000","last_active":"2024-03-28T18:09:44.005929+0000","last_peered":"2024-03-28T18:09:44.005929+0000","last_clean":"2024-03-28T18:09:44.005929+0000","last_became_active":"2024-03-28T18:08:20.025177+0000","last_became_peered":"2024-03-28T18:08:20.025177+0000","last_unstale":"2024-03-28T18:09:44.005929+0000","last_undegraded":"2024-03-28T18:09:44.005929+0000","last_fullsized":"2024-03-28T18:09:44.005929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:14.521170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.018719+0000","last_change":"2024-03-28T18:08:20.027418+0000","last_active":"2024-03-28T18:09:44.018719+0000","last_peered":"2024-03-28T18:09:44.018719+0000","last_clean":"2024-03-28T18:09:44.018719+0000","last_became_active":"2024-03-28T18:08:20.026731+0000","last_became_peered":"2024-03-28T18:08:20.026731+0000","last_unstale":"2024-03-28T18:09:44.018719+0000","last_undegraded":"2024-03-28T18:09:44.018719+0000","last_fullsized":"2024-03-28T18:09:44.018719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:52:58.328738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.001347+0000","last_change":"2024-03-28T18:08:20.029821+0000","last_active":"2024-03-28T18:09:44.001347+0000","last_peered":"2024-03-28T18:09:44.001347+0000","last_clean":"2024-03-28T18:09:44.001347+0000","last_became_active":"2024-03-28T18:08:20.028245+0000","last_became_peered":"2024-03-28T18:08:20.028245+0000","last_unstale":"2024-03-28T18:09:44.001347+0000","last_undegraded":"2024-03-28T18:09:44.001347+0000","last_fullsized":"2024-03-28T18:09:44.001347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:44.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.966606+0000","last_change":"2024-03-28T18:08:20.015682+0000","last_active":"2024-03-28T18:09:43.966606+0000","last_peered":"2024-03-28T18:09:43.966606+0000","last_clean":"2024-03-28T18:09:43.966606+0000","last_became_active":"2024-03-28T18:08:20.015303+0000","last_became_peered":"2024-03-28T18:08:20.015303+0000","last_unstale":"2024-03-28T18:09:43.966606+0000","last_undegraded":"2024-03-28T18:09:43.966606+0000","last_fullsized":"2024-03-28T18:09:43.966606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:57.456052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.998456+0000","last_change":"2024-03-28T18:08:20.020970+0000","last_active":"2024-03-28T18:09:43.998456+0000","last_peered":"2024-03-28T18:09:43.998456+0000","last_clean":"2024-03-28T18:09:43.998456+0000","last_became_active":"2024-03-28T18:08:20.020646+0000","last_became_peered":"2024-03-28T18:08:20.020646+0000","last_unstale":"2024-03-28T18:09:43.998456+0000","last_undegraded":"2024-03-28T18:09:43.998456+0000","last_fullsized":"2024-03-28T18:09:43.998456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:57:20.422546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.970526+0000","last_change":"2024-03-28T18:08:20.022395+0000","last_active":"2024-03-28T18:09:43.970526+0000","last_peered":"2024-03-28T18:09:43.970526+0000","last_clean":"2024-03-28T18:09:43.970526+0000","last_became_active":"2024-03-28T18:08:20.022022+0000","last_became_peered":"2024-03-28T18:08:20.022022+0000","last_unstale":"2024-03-28T18:09:43.970526+0000","last_undegraded":"2024-03-28T18:09:43.970526+0000","last_fullsized":"2024-03-28T18:09:43.970526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:32.590751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.972192+0000","last_change":"2024-03-28T18:08:20.019990+0000","last_active":"2024-03-28T18:09:43.972192+0000","last_peered":"2024-03-28T18:09:43.972192+0000","last_clean":"2024-03-28T18:09:43.972192+0000","last_became_active":"2024-03-28T18:08:20.019618+0000","last_became_peered":"2024-03-28T18:08:20.019618+0000","last_unstale":"2024-03-28T18:09:43.972192+0000","last_undegraded":"2024-03-28T18:09:43.972192+0000","last_fullsized":"2024-03-28T18:09:43.972192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:48.442579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.968407+0000","last_change":"2024-03-28T18:09:15.889503+0000","last_active":"2024-03-28T18:09:43.968407+0000","last_peered":"2024-03-28T18:09:43.968407+0000","last_clean":"2024-03-28T18:09:43.968407+0000","last_became_active":"2024-03-28T18:09:15.888272+0000","last_became_peered":"2024-03-28T18:09:15.888272+0000","last_unstale":"2024-03-28T18:09:43.968407+0000","last_undegraded":"2024-03-28T18:09:43.968407+0000","last_fullsized":"2024-03-28T18:09:43.968407+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:49.222646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.980289+0000","last_change":"2024-03-28T18:08:20.009732+0000","last_active":"2024-03-28T18:09:43.980289+0000","last_peered":"2024-03-28T18:09:43.980289+0000","last_clean":"2024-03-28T18:09:43.980289+0000","last_became_active":"2024-03-28T18:08:20.009288+0000","last_became_peered":"2024-03-28T18:08:20.009288+0000","last_unstale":"2024-03-28T18:09:43.980289+0000","last_undegraded":"2024-03-28T18:09:43.980289+0000","last_fullsized":"2024-03-28T18:09:43.980289+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:20:54.917964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.964990+0000","last_change":"2024-03-28T18:08:20.009687+0000","last_active":"2024-03-28T18:09:43.964990+0000","last_peered":"2024-03-28T18:09:43.964990+0000","last_clean":"2024-03-28T18:09:43.964990+0000","last_became_active":"2024-03-28T18:08:20.008364+0000","last_became_peered":"2024-03-28T18:08:20.008364+0000","last_unstale":"2024-03-28T18:09:43.964990+0000","last_undegraded":"2024-03-28T18:09:43.964990+0000","last_fullsized":"2024-03-28T18:09:43.964990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:08.419202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.978300+0000","last_change":"2024-03-28T18:08:20.011508+0000","last_active":"2024-03-28T18:09:43.978300+0000","last_peered":"2024-03-28T18:09:43.978300+0000","last_clean":"2024-03-28T18:09:43.978300+0000","last_became_active":"2024-03-28T18:08:20.011117+0000","last_became_peered":"2024-03-28T18:08:20.011117+0000","last_unstale":"2024-03-28T18:09:43.978300+0000","last_undegraded":"2024-03-28T18:09:43.978300+0000","last_fullsized":"2024-03-28T18:09:43.978300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:06:19.278931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.966450+0000","last_change":"2024-03-28T18:08:20.011403+0000","last_active":"2024-03-28T18:09:43.966450+0000","last_peered":"2024-03-28T18:09:43.966450+0000","last_clean":"2024-03-28T18:09:43.966450+0000","last_became_active":"2024-03-28T18:08:20.011025+0000","last_became_peered":"2024-03-28T18:08:20.011025+0000","last_unstale":"2024-03-28T18:09:43.966450+0000","last_undegraded":"2024-03-28T18:09:43.966450+0000","last_fullsized":"2024-03-28T18:09:43.966450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:13.599432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.005479+0000","last_change":"2024-03-28T18:08:20.001664+0000","last_active":"2024-03-28T18:09:44.005479+0000","last_peered":"2024-03-28T18:09:44.005479+0000","last_clean":"2024-03-28T18:09:44.005479+0000","last_became_active":"2024-03-28T18:08:20.000910+0000","last_became_peered":"2024-03-28T18:08:20.000910+0000","last_unstale":"2024-03-28T18:09:44.005479+0000","last_undegraded":"2024-03-28T18:09:44.005479+0000","last_fullsized":"2024-03-28T18:09:44.005479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:00:23.796302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.008215+0000","last_change":"2024-03-28T18:08:20.027174+0000","last_active":"2024-03-28T18:09:44.008215+0000","last_peered":"2024-03-28T18:09:44.008215+0000","last_clean":"2024-03-28T18:09:44.008215+0000","last_became_active":"2024-03-28T18:08:20.026113+0000","last_became_peered":"2024-03-28T18:08:20.026113+0000","last_unstale":"2024-03-28T18:09:44.008215+0000","last_undegraded":"2024-03-28T18:09:44.008215+0000","last_fullsized":"2024-03-28T18:09:44.008215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:55:00.061696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.010537+0000","last_change":"2024-03-28T18:08:20.020060+0000","last_active":"2024-03-28T18:09:44.010537+0000","last_peered":"2024-03-28T18:09:44.010537+0000","last_clean":"2024-03-28T18:09:44.010537+0000","last_became_active":"2024-03-28T18:08:20.019691+0000","last_became_peered":"2024-03-28T18:08:20.019691+0000","last_unstale":"2024-03-28T18:09:44.010537+0000","last_undegraded":"2024-03-28T18:09:44.010537+0000","last_fullsized":"2024-03-28T18:09:44.010537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:27.889737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.003319+0000","last_change":"2024-03-28T18:08:20.024239+0000","last_active":"2024-03-28T18:09:44.003319+0000","last_peered":"2024-03-28T18:09:44.003319+0000","last_clean":"2024-03-28T18:09:44.003319+0000","last_became_active":"2024-03-28T18:08:20.023874+0000","last_became_peered":"2024-03-28T18:08:20.023874+0000","last_unstale":"2024-03-28T18:09:44.003319+0000","last_undegraded":"2024-03-28T18:09:44.003319+0000","last_fullsized":"2024-03-28T18:09:44.003319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:05:32.903195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.008671+0000","last_change":"2024-03-28T18:08:20.033322+0000","last_active":"2024-03-28T18:09:44.008671+0000","last_peered":"2024-03-28T18:09:44.008671+0000","last_clean":"2024-03-28T18:09:44.008671+0000","last_became_active":"2024-03-28T18:08:20.033048+0000","last_became_peered":"2024-03-28T18:08:20.033048+0000","last_unstale":"2024-03-28T18:09:44.008671+0000","last_undegraded":"2024-03-28T18:09:44.008671+0000","last_fullsized":"2024-03-28T18:09:44.008671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:18:59.234141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.986420+0000","last_change":"2024-03-28T18:08:20.022093+0000","last_active":"2024-03-28T18:09:43.986420+0000","last_peered":"2024-03-28T18:09:43.986420+0000","last_clean":"2024-03-28T18:09:43.986420+0000","last_became_active":"2024-03-28T18:08:20.021411+0000","last_became_peered":"2024-03-28T18:08:20.021411+0000","last_unstale":"2024-03-28T18:09:43.986420+0000","last_undegraded":"2024-03-28T18:09:43.986420+0000","last_fullsized":"2024-03-28T18:09:43.986420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:09:40.366628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.002040+0000","last_change":"2024-03-28T18:08:20.028893+0000","last_active":"2024-03-28T18:09:44.002040+0000","last_peered":"2024-03-28T18:09:44.002040+0000","last_clean":"2024-03-28T18:09:44.002040+0000","last_became_active":"2024-03-28T18:08:20.028634+0000","last_became_peered":"2024-03-28T18:08:20.028634+0000","last_unstale":"2024-03-28T18:09:44.002040+0000","last_undegraded":"2024-03-28T18:09:44.002040+0000","last_fullsized":"2024-03-28T18:09:44.002040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:32:27.454406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":46,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.734956+0000","last_change":"2024-03-28T18:09:15.734404+0000","last_active":"2024-03-28T18:09:15.734956+0000","last_peered":"2024-03-28T18:09:15.734956+0000","last_clean":"2024-03-28T18:09:15.734956+0000","last_became_active":"2024-03-28T18:09:15.723870+0000","last_became_peered":"2024-03-28T18:09:15.723870+0000","last_unstale":"2024-03-28T18:09:15.734956+0000","last_undegraded":"2024-03-28T18:09:15.734956+0000","last_fullsized":"2024-03-28T18:09:15.734956+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:32.736966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.004329+0000","last_change":"2024-03-28T18:08:20.027855+0000","last_active":"2024-03-28T18:09:44.004329+0000","last_peered":"2024-03-28T18:09:44.004329+0000","last_clean":"2024-03-28T18:09:44.004329+0000","last_became_active":"2024-03-28T18:08:20.027565+0000","last_became_peered":"2024-03-28T18:08:20.027565+0000","last_unstale":"2024-03-28T18:09:44.004329+0000","last_undegraded":"2024-03-28T18:09:44.004329+0000","last_fullsized":"2024-03-28T18:09:44.004329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:36:37.596573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.725789+0000","last_change":"2024-03-28T18:08:19.499055+0000","last_active":"2024-03-28T18:09:15.725789+0000","last_peered":"2024-03-28T18:09:15.725789+0000","last_clean":"2024-03-28T18:09:15.725789+0000","last_became_active":"2024-03-28T18:08:19.497964+0000","last_became_peered":"2024-03-28T18:08:19.497964+0000","last_unstale":"2024-03-28T18:09:15.725789+0000","last_undegraded":"2024-03-28T18:09:15.725789+0000","last_fullsized":"2024-03-28T18:09:15.725789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:38:45.620406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.012271+0000","last_change":"2024-03-28T18:08:20.028967+0000","last_active":"2024-03-28T18:09:44.012271+0000","last_peered":"2024-03-28T18:09:44.012271+0000","last_clean":"2024-03-28T18:09:44.012271+0000","last_became_active":"2024-03-28T18:08:20.028607+0000","last_became_peered":"2024-03-28T18:08:20.028607+0000","last_unstale":"2024-03-28T18:09:44.012271+0000","last_undegraded":"2024-03-28T18:09:44.012271+0000","last_fullsized":"2024-03-28T18:09:44.012271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:21:23.994077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.893027+0000","last_change":"2024-03-28T18:09:15.892649+0000","last_active":"2024-03-28T18:09:15.893027+0000","last_peered":"2024-03-28T18:09:15.893027+0000","last_clean":"2024-03-28T18:09:15.893027+0000","last_became_active":"2024-03-28T18:09:15.888520+0000","last_became_peered":"2024-03-28T18:09:15.888520+0000","last_unstale":"2024-03-28T18:09:15.893027+0000","last_undegraded":"2024-03-28T18:09:15.893027+0000","last_fullsized":"2024-03-28T18:09:15.893027+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:34.085738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.013841+0000","last_change":"2024-03-28T18:08:20.009878+0000","last_active":"2024-03-28T18:09:44.013841+0000","last_peered":"2024-03-28T18:09:44.013841+0000","last_clean":"2024-03-28T18:09:44.013841+0000","last_became_active":"2024-03-28T18:08:20.009534+0000","last_became_peered":"2024-03-28T18:08:20.009534+0000","last_unstale":"2024-03-28T18:09:44.013841+0000","last_undegraded":"2024-03-28T18:09:44.013841+0000","last_fullsized":"2024-03-28T18:09:44.013841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:58.975854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716112+0000","last_change":"2024-03-28T18:08:19.472613+0000","last_active":"2024-03-28T18:09:15.716112+0000","last_peered":"2024-03-28T18:09:15.716112+0000","last_clean":"2024-03-28T18:09:15.716112+0000","last_became_active":"2024-03-28T18:08:19.472186+0000","last_became_peered":"2024-03-28T18:08:19.472186+0000","last_unstale":"2024-03-28T18:09:15.716112+0000","last_undegraded":"2024-03-28T18:09:15.716112+0000","last_fullsized":"2024-03-28T18:09:15.716112+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:50.807609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.974502+0000","last_change":"2024-03-28T18:08:20.008242+0000","last_active":"2024-03-28T18:09:43.974502+0000","last_peered":"2024-03-28T18:09:43.974502+0000","last_clean":"2024-03-28T18:09:43.974502+0000","last_became_active":"2024-03-28T18:08:20.007942+0000","last_became_peered":"2024-03-28T18:08:20.007942+0000","last_unstale":"2024-03-28T18:09:43.974502+0000","last_undegraded":"2024-03-28T18:09:43.974502+0000","last_fullsized":"2024-03-28T18:09:43.974502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:23.288978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'8","reported_seq":184,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:54.583738+0000","last_change":"2024-03-28T18:08:19.480869+0000","last_active":"2024-03-28T18:10:54.583738+0000","last_peered":"2024-03-28T18:10:54.583738+0000","last_clean":"2024-03-28T18:10:54.583738+0000","last_became_active":"2024-03-28T18:08:19.480441+0000","last_became_peered":"2024-03-28T18:08:19.480441+0000","last_unstale":"2024-03-28T18:10:54.583738+0000","last_undegraded":"2024-03-28T18:10:54.583738+0000","last_fullsized":"2024-03-28T18:10:54.583738+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:26.640605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1460,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":156,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.960050+0000","last_change":"2024-03-28T18:08:20.021530+0000","last_active":"2024-03-28T18:09:43.960050+0000","last_peered":"2024-03-28T18:09:43.960050+0000","last_clean":"2024-03-28T18:09:43.960050+0000","last_became_active":"2024-03-28T18:08:20.021179+0000","last_became_peered":"2024-03-28T18:08:20.021179+0000","last_unstale":"2024-03-28T18:09:43.960050+0000","last_undegraded":"2024-03-28T18:09:43.960050+0000","last_fullsized":"2024-03-28T18:09:43.960050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:43:07.300300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887313+0000","last_change":"2024-03-28T18:08:19.494597+0000","last_active":"2024-03-28T18:09:15.887313+0000","last_peered":"2024-03-28T18:09:15.887313+0000","last_clean":"2024-03-28T18:09:15.887313+0000","last_became_active":"2024-03-28T18:08:19.494178+0000","last_became_peered":"2024-03-28T18:08:19.494178+0000","last_unstale":"2024-03-28T18:09:15.887313+0000","last_undegraded":"2024-03-28T18:09:15.887313+0000","last_fullsized":"2024-03-28T18:09:15.887313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:08:10.257768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.991904+0000","last_change":"2024-03-28T18:08:20.021062+0000","last_active":"2024-03-28T18:09:43.991904+0000","last_peered":"2024-03-28T18:09:43.991904+0000","last_clean":"2024-03-28T18:09:43.991904+0000","last_became_active":"2024-03-28T18:08:20.020758+0000","last_became_peered":"2024-03-28T18:08:20.020758+0000","last_unstale":"2024-03-28T18:09:43.991904+0000","last_undegraded":"2024-03-28T18:09:43.991904+0000","last_fullsized":"2024-03-28T18:09:43.991904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:47:39.656513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.889759+0000","last_change":"2024-03-28T18:08:19.480189+0000","last_active":"2024-03-28T18:09:15.889759+0000","last_peered":"2024-03-28T18:09:15.889759+0000","last_clean":"2024-03-28T18:09:15.889759+0000","last_became_active":"2024-03-28T18:08:19.479812+0000","last_became_peered":"2024-03-28T18:08:19.479812+0000","last_unstale":"2024-03-28T18:09:15.889759+0000","last_undegraded":"2024-03-28T18:09:15.889759+0000","last_fullsized":"2024-03-28T18:09:15.889759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:46.408565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.007167+0000","last_change":"2024-03-28T18:08:20.023635+0000","last_active":"2024-03-28T18:09:44.007167+0000","last_peered":"2024-03-28T18:09:44.007167+0000","last_clean":"2024-03-28T18:09:44.007167+0000","last_became_active":"2024-03-28T18:08:20.023199+0000","last_became_peered":"2024-03-28T18:08:20.023199+0000","last_unstale":"2024-03-28T18:09:44.007167+0000","last_undegraded":"2024-03-28T18:09:44.007167+0000","last_fullsized":"2024-03-28T18:09:44.007167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:12.477899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.886913+0000","last_change":"2024-03-28T18:08:19.480858+0000","last_active":"2024-03-28T18:09:15.886913+0000","last_peered":"2024-03-28T18:09:15.886913+0000","last_clean":"2024-03-28T18:09:15.886913+0000","last_became_active":"2024-03-28T18:08:19.480418+0000","last_became_peered":"2024-03-28T18:08:19.480418+0000","last_unstale":"2024-03-28T18:09:15.886913+0000","last_undegraded":"2024-03-28T18:09:15.886913+0000","last_fullsized":"2024-03-28T18:09:15.886913+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:59:57.788915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.019886+0000","last_change":"2024-03-28T18:08:20.031719+0000","last_active":"2024-03-28T18:09:44.019886+0000","last_peered":"2024-03-28T18:09:44.019886+0000","last_clean":"2024-03-28T18:09:44.019886+0000","last_became_active":"2024-03-28T18:08:20.029482+0000","last_became_peered":"2024-03-28T18:08:20.029482+0000","last_unstale":"2024-03-28T18:09:44.019886+0000","last_undegraded":"2024-03-28T18:09:44.019886+0000","last_fullsized":"2024-03-28T18:09:44.019886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:13.226596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'5","reported_seq":175,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:53.582089+0000","last_change":"2024-03-28T18:08:19.483346+0000","last_active":"2024-03-28T18:10:53.582089+0000","last_peered":"2024-03-28T18:10:53.582089+0000","last_clean":"2024-03-28T18:10:53.582089+0000","last_became_active":"2024-03-28T18:08:19.483016+0000","last_became_peered":"2024-03-28T18:08:19.483016+0000","last_unstale":"2024-03-28T18:10:53.582089+0000","last_undegraded":"2024-03-28T18:10:53.582089+0000","last_fullsized":"2024-03-28T18:10:53.582089+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:34:33.008477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":150,"num_read_kb":150,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.984960+0000","last_change":"2024-03-28T18:08:20.020478+0000","last_active":"2024-03-28T18:09:43.984960+0000","last_peered":"2024-03-28T18:09:43.984960+0000","last_clean":"2024-03-28T18:09:43.984960+0000","last_became_active":"2024-03-28T18:08:20.020054+0000","last_became_peered":"2024-03-28T18:08:20.020054+0000","last_unstale":"2024-03-28T18:09:43.984960+0000","last_undegraded":"2024-03-28T18:09:43.984960+0000","last_fullsized":"2024-03-28T18:09:43.984960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:13:43.655549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716206+0000","last_change":"2024-03-28T18:08:19.479386+0000","last_active":"2024-03-28T18:09:15.716206+0000","last_peered":"2024-03-28T18:09:15.716206+0000","last_clean":"2024-03-28T18:09:15.716206+0000","last_became_active":"2024-03-28T18:08:19.478985+0000","last_became_peered":"2024-03-28T18:08:19.478985+0000","last_unstale":"2024-03-28T18:09:15.716206+0000","last_undegraded":"2024-03-28T18:09:15.716206+0000","last_fullsized":"2024-03-28T18:09:15.716206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:37:22.359475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.007695+0000","last_change":"2024-03-28T18:08:20.021790+0000","last_active":"2024-03-28T18:09:44.007695+0000","last_peered":"2024-03-28T18:09:44.007695+0000","last_clean":"2024-03-28T18:09:44.007695+0000","last_became_active":"2024-03-28T18:08:20.021031+0000","last_became_peered":"2024-03-28T18:08:20.021031+0000","last_unstale":"2024-03-28T18:09:44.007695+0000","last_undegraded":"2024-03-28T18:09:44.007695+0000","last_fullsized":"2024-03-28T18:09:44.007695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:59:50.897692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887257+0000","last_change":"2024-03-28T18:08:19.481576+0000","last_active":"2024-03-28T18:09:15.887257+0000","last_peered":"2024-03-28T18:09:15.887257+0000","last_clean":"2024-03-28T18:09:15.887257+0000","last_became_active":"2024-03-28T18:08:19.481226+0000","last_became_peered":"2024-03-28T18:08:19.481226+0000","last_unstale":"2024-03-28T18:09:15.887257+0000","last_undegraded":"2024-03-28T18:09:15.887257+0000","last_fullsized":"2024-03-28T18:09:15.887257+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:54:44.677343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.013852+0000","last_change":"2024-03-28T18:09:15.890672+0000","last_active":"2024-03-28T18:09:44.013852+0000","last_peered":"2024-03-28T18:09:44.013852+0000","last_clean":"2024-03-28T18:09:44.013852+0000","last_became_active":"2024-03-28T18:09:15.890342+0000","last_became_peered":"2024-03-28T18:09:15.890342+0000","last_unstale":"2024-03-28T18:09:44.013852+0000","last_undegraded":"2024-03-28T18:09:44.013852+0000","last_fullsized":"2024-03-28T18:09:44.013852+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:08.268067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'35","reported_seq":366,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:54.583213+0000","last_change":"2024-03-28T18:08:19.477559+0000","last_active":"2024-03-28T18:10:54.583213+0000","last_peered":"2024-03-28T18:10:54.583213+0000","last_clean":"2024-03-28T18:10:54.583213+0000","last_became_active":"2024-03-28T18:08:19.477217+0000","last_became_peered":"2024-03-28T18:08:19.477217+0000","last_unstale":"2024-03-28T18:10:54.583213+0000","last_undegraded":"2024-03-28T18:10:54.583213+0000","last_fullsized":"2024-03-28T18:10:54.583213+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:37.071671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66765,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":311,"num_read_kb":221,"num_write":36,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.955611+0000","last_change":"2024-03-28T18:08:20.011303+0000","last_active":"2024-03-28T18:09:43.955611+0000","last_peered":"2024-03-28T18:09:43.955611+0000","last_clean":"2024-03-28T18:09:43.955611+0000","last_became_active":"2024-03-28T18:08:20.010997+0000","last_became_peered":"2024-03-28T18:08:20.010997+0000","last_unstale":"2024-03-28T18:09:43.955611+0000","last_undegraded":"2024-03-28T18:09:43.955611+0000","last_fullsized":"2024-03-28T18:09:43.955611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:10:56.036132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.718063+0000","last_change":"2024-03-28T18:08:19.477342+0000","last_active":"2024-03-28T18:09:15.718063+0000","last_peered":"2024-03-28T18:09:15.718063+0000","last_clean":"2024-03-28T18:09:15.718063+0000","last_became_active":"2024-03-28T18:08:19.476984+0000","last_became_peered":"2024-03-28T18:08:19.476984+0000","last_unstale":"2024-03-28T18:09:15.718063+0000","last_undegraded":"2024-03-28T18:09:15.718063+0000","last_fullsized":"2024-03-28T18:09:15.718063+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:15:42.606213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.017873+0000","last_change":"2024-03-28T18:08:20.030524+0000","last_active":"2024-03-28T18:09:44.017873+0000","last_peered":"2024-03-28T18:09:44.017873+0000","last_clean":"2024-03-28T18:09:44.017873+0000","last_became_active":"2024-03-28T18:08:20.030227+0000","last_became_peered":"2024-03-28T18:08:20.030227+0000","last_unstale":"2024-03-28T18:09:44.017873+0000","last_undegraded":"2024-03-28T18:09:44.017873+0000","last_fullsized":"2024-03-28T18:09:44.017873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:07:41.763739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.886737+0000","last_change":"2024-03-28T18:08:19.481169+0000","last_active":"2024-03-28T18:09:15.886737+0000","last_peered":"2024-03-28T18:09:15.886737+0000","last_clean":"2024-03-28T18:09:15.886737+0000","last_became_active":"2024-03-28T18:08:19.480634+0000","last_became_peered":"2024-03-28T18:08:19.480634+0000","last_unstale":"2024-03-28T18:09:15.886737+0000","last_undegraded":"2024-03-28T18:09:15.886737+0000","last_fullsized":"2024-03-28T18:09:15.886737+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:39.265215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.006975+0000","last_change":"2024-03-28T18:08:20.015163+0000","last_active":"2024-03-28T18:09:44.006975+0000","last_peered":"2024-03-28T18:09:44.006975+0000","last_clean":"2024-03-28T18:09:44.006975+0000","last_became_active":"2024-03-28T18:08:20.014765+0000","last_became_peered":"2024-03-28T18:08:20.014765+0000","last_unstale":"2024-03-28T18:09:44.006975+0000","last_undegraded":"2024-03-28T18:09:44.006975+0000","last_fullsized":"2024-03-28T18:09:44.006975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:51.035532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'3","reported_seq":174,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:52.634345+0000","last_change":"2024-03-28T18:08:19.479086+0000","last_active":"2024-03-28T18:10:52.634345+0000","last_peered":"2024-03-28T18:10:52.634345+0000","last_clean":"2024-03-28T18:10:52.634345+0000","last_became_active":"2024-03-28T18:08:19.478748+0000","last_became_peered":"2024-03-28T18:08:19.478748+0000","last_unstale":"2024-03-28T18:10:52.634345+0000","last_undegraded":"2024-03-28T18:10:52.634345+0000","last_fullsized":"2024-03-28T18:10:52.634345+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:46:59.566998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":152,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.973667+0000","last_change":"2024-03-28T18:09:15.888751+0000","last_active":"2024-03-28T18:09:43.973667+0000","last_peered":"2024-03-28T18:09:43.973667+0000","last_clean":"2024-03-28T18:09:43.973667+0000","last_became_active":"2024-03-28T18:09:15.888350+0000","last_became_peered":"2024-03-28T18:09:15.888350+0000","last_unstale":"2024-03-28T18:09:43.973667+0000","last_undegraded":"2024-03-28T18:09:43.973667+0000","last_fullsized":"2024-03-28T18:09:43.973667+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:38:05.822205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":60,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.733834+0000","last_change":"2024-03-28T18:09:15.733434+0000","last_active":"2024-03-28T18:09:15.733834+0000","last_peered":"2024-03-28T18:09:15.733834+0000","last_clean":"2024-03-28T18:09:15.733834+0000","last_became_active":"2024-03-28T18:09:15.722437+0000","last_became_peered":"2024-03-28T18:09:15.722437+0000","last_unstale":"2024-03-28T18:09:15.733834+0000","last_undegraded":"2024-03-28T18:09:15.733834+0000","last_fullsized":"2024-03-28T18:09:15.733834+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:35:18.313676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.971701+0000","last_change":"2024-03-28T18:08:20.007457+0000","last_active":"2024-03-28T18:09:43.971701+0000","last_peered":"2024-03-28T18:09:43.971701+0000","last_clean":"2024-03-28T18:09:43.971701+0000","last_became_active":"2024-03-28T18:08:20.007090+0000","last_became_peered":"2024-03-28T18:08:20.007090+0000","last_unstale":"2024-03-28T18:09:43.971701+0000","last_undegraded":"2024-03-28T18:09:43.971701+0000","last_fullsized":"2024-03-28T18:09:43.971701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:22:45.702706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716396+0000","last_change":"2024-03-28T18:08:19.475104+0000","last_active":"2024-03-28T18:09:15.716396+0000","last_peered":"2024-03-28T18:09:15.716396+0000","last_clean":"2024-03-28T18:09:15.716396+0000","last_became_active":"2024-03-28T18:08:19.474754+0000","last_became_peered":"2024-03-28T18:08:19.474754+0000","last_unstale":"2024-03-28T18:09:15.716396+0000","last_undegraded":"2024-03-28T18:09:15.716396+0000","last_fullsized":"2024-03-28T18:09:15.716396+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:19.085631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.968043+0000","last_change":"2024-03-28T18:08:20.009347+0000","last_active":"2024-03-28T18:09:43.968043+0000","last_peered":"2024-03-28T18:09:43.968043+0000","last_clean":"2024-03-28T18:09:43.968043+0000","last_became_active":"2024-03-28T18:08:20.009012+0000","last_became_peered":"2024-03-28T18:08:20.009012+0000","last_unstale":"2024-03-28T18:09:43.968043+0000","last_undegraded":"2024-03-28T18:09:43.968043+0000","last_fullsized":"2024-03-28T18:09:43.968043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:04:26.625830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.721956+0000","last_change":"2024-03-28T18:08:19.482414+0000","last_active":"2024-03-28T18:09:15.721956+0000","last_peered":"2024-03-28T18:09:15.721956+0000","last_clean":"2024-03-28T18:09:15.721956+0000","last_became_active":"2024-03-28T18:08:19.481830+0000","last_became_peered":"2024-03-28T18:08:19.481830+0000","last_unstale":"2024-03-28T18:09:15.721956+0000","last_undegraded":"2024-03-28T18:09:15.721956+0000","last_fullsized":"2024-03-28T18:09:15.721956+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:27:14.259355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.001824+0000","last_change":"2024-03-28T18:08:20.030687+0000","last_active":"2024-03-28T18:09:44.001824+0000","last_peered":"2024-03-28T18:09:44.001824+0000","last_clean":"2024-03-28T18:09:44.001824+0000","last_became_active":"2024-03-28T18:08:20.030308+0000","last_became_peered":"2024-03-28T18:08:20.030308+0000","last_unstale":"2024-03-28T18:09:44.001824+0000","last_undegraded":"2024-03-28T18:09:44.001824+0000","last_fullsized":"2024-03-28T18:09:44.001824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:31.745952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716000+0000","last_change":"2024-03-28T18:08:19.494446+0000","last_active":"2024-03-28T18:09:15.716000+0000","last_peered":"2024-03-28T18:09:15.716000+0000","last_clean":"2024-03-28T18:09:15.716000+0000","last_became_active":"2024-03-28T18:08:19.494036+0000","last_became_peered":"2024-03-28T18:08:19.494036+0000","last_unstale":"2024-03-28T18:09:15.716000+0000","last_undegraded":"2024-03-28T18:09:15.716000+0000","last_fullsized":"2024-03-28T18:09:15.716000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:42:53.660741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.009456+0000","last_change":"2024-03-28T18:08:20.025706+0000","last_active":"2024-03-28T18:09:44.009456+0000","last_peered":"2024-03-28T18:09:44.009456+0000","last_clean":"2024-03-28T18:09:44.009456+0000","last_became_active":"2024-03-28T18:08:20.025343+0000","last_became_peered":"2024-03-28T18:08:20.025343+0000","last_unstale":"2024-03-28T18:09:44.009456+0000","last_undegraded":"2024-03-28T18:09:44.009456+0000","last_fullsized":"2024-03-28T18:09:44.009456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:32:27.883940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.963975+0000","last_change":"2024-03-28T18:08:20.009636+0000","last_active":"2024-03-28T18:09:43.963975+0000","last_peered":"2024-03-28T18:09:43.963975+0000","last_clean":"2024-03-28T18:09:43.963975+0000","last_became_active":"2024-03-28T18:08:20.009178+0000","last_became_peered":"2024-03-28T18:08:20.009178+0000","last_unstale":"2024-03-28T18:09:43.963975+0000","last_undegraded":"2024-03-28T18:09:43.963975+0000","last_fullsized":"2024-03-28T18:09:43.963975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:28.461411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":174,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:53.635099+0000","last_change":"2024-03-28T18:08:19.501287+0000","last_active":"2024-03-28T18:10:53.635099+0000","last_peered":"2024-03-28T18:10:53.635099+0000","last_clean":"2024-03-28T18:10:53.635099+0000","last_became_active":"2024-03-28T18:08:19.500920+0000","last_became_peered":"2024-03-28T18:08:19.500920+0000","last_unstale":"2024-03-28T18:10:53.635099+0000","last_undegraded":"2024-03-28T18:10:53.635099+0000","last_fullsized":"2024-03-28T18:10:53.635099+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:29.882164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":150,"num_read_kb":150,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.013107+0000","last_change":"2024-03-28T18:08:20.020811+0000","last_active":"2024-03-28T18:09:44.013107+0000","last_peered":"2024-03-28T18:09:44.013107+0000","last_clean":"2024-03-28T18:09:44.013107+0000","last_became_active":"2024-03-28T18:08:20.020381+0000","last_became_peered":"2024-03-28T18:08:20.020381+0000","last_unstale":"2024-03-28T18:09:44.013107+0000","last_undegraded":"2024-03-28T18:09:44.013107+0000","last_fullsized":"2024-03-28T18:09:44.013107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:10:35.999858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887764+0000","last_change":"2024-03-28T18:08:19.478020+0000","last_active":"2024-03-28T18:09:15.887764+0000","last_peered":"2024-03-28T18:09:15.887764+0000","last_clean":"2024-03-28T18:09:15.887764+0000","last_became_active":"2024-03-28T18:08:19.477627+0000","last_became_peered":"2024-03-28T18:08:19.477627+0000","last_unstale":"2024-03-28T18:09:15.887764+0000","last_undegraded":"2024-03-28T18:09:15.887764+0000","last_fullsized":"2024-03-28T18:09:15.887764+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:00:01.649974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":354,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.715885+0000","last_change":"2024-03-28T18:06:58.750932+0000","last_active":"2024-03-28T18:09:15.715885+0000","last_peered":"2024-03-28T18:09:15.715885+0000","last_clean":"2024-03-28T18:09:15.715885+0000","last_became_active":"2024-03-28T18:06:58.750309+0000","last_became_peered":"2024-03-28T18:06:58.750309+0000","last_unstale":"2024-03-28T18:09:15.715885+0000","last_undegraded":"2024-03-28T18:09:15.715885+0000","last_fullsized":"2024-03-28T18:09:15.715885+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:03:00.832683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:03:00.832683+0000","last_clean_scrub_stamp":"2024-03-28T18:03:00.832683+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:28:46.001886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.002070+0000","last_change":"2024-03-28T18:08:20.009573+0000","last_active":"2024-03-28T18:09:44.002070+0000","last_peered":"2024-03-28T18:09:44.002070+0000","last_clean":"2024-03-28T18:09:44.002070+0000","last_became_active":"2024-03-28T18:08:20.009143+0000","last_became_peered":"2024-03-28T18:08:20.009143+0000","last_unstale":"2024-03-28T18:09:44.002070+0000","last_undegraded":"2024-03-28T18:09:44.002070+0000","last_fullsized":"2024-03-28T18:09:44.002070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:37:50.643356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716076+0000","last_change":"2024-03-28T18:08:19.483733+0000","last_active":"2024-03-28T18:09:15.716076+0000","last_peered":"2024-03-28T18:09:15.716076+0000","last_clean":"2024-03-28T18:09:15.716076+0000","last_became_active":"2024-03-28T18:08:19.483366+0000","last_became_peered":"2024-03-28T18:08:19.483366+0000","last_unstale":"2024-03-28T18:09:15.716076+0000","last_undegraded":"2024-03-28T18:09:15.716076+0000","last_fullsized":"2024-03-28T18:09:15.716076+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:29.192442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.716038+0000","last_change":"2024-03-28T18:08:19.481844+0000","last_active":"2024-03-28T18:09:15.716038+0000","last_peered":"2024-03-28T18:09:15.716038+0000","last_clean":"2024-03-28T18:09:15.716038+0000","last_became_active":"2024-03-28T18:08:19.481430+0000","last_became_peered":"2024-03-28T18:08:19.481430+0000","last_unstale":"2024-03-28T18:09:15.716038+0000","last_undegraded":"2024-03-28T18:09:15.716038+0000","last_fullsized":"2024-03-28T18:09:15.716038+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:21:28.789883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.018278+0000","last_change":"2024-03-28T18:08:20.021037+0000","last_active":"2024-03-28T18:09:44.018278+0000","last_peered":"2024-03-28T18:09:44.018278+0000","last_clean":"2024-03-28T18:09:44.018278+0000","last_became_active":"2024-03-28T18:08:20.020680+0000","last_became_peered":"2024-03-28T18:08:20.020680+0000","last_unstale":"2024-03-28T18:09:44.018278+0000","last_undegraded":"2024-03-28T18:09:44.018278+0000","last_fullsized":"2024-03-28T18:09:44.018278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:08:28.825774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'11","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:28.677487+0000","last_change":"2024-03-28T18:09:15.737583+0000","last_active":"2024-03-28T18:10:28.677487+0000","last_peered":"2024-03-28T18:10:28.677487+0000","last_clean":"2024-03-28T18:10:28.677487+0000","last_became_active":"2024-03-28T18:09:15.726579+0000","last_became_peered":"2024-03-28T18:09:15.726579+0000","last_unstale":"2024-03-28T18:10:28.677487+0000","last_undegraded":"2024-03-28T18:10:28.677487+0000","last_fullsized":"2024-03-28T18:10:28.677487+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:03.380037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.011149+0000","last_change":"2024-03-28T18:08:20.024938+0000","last_active":"2024-03-28T18:09:44.011149+0000","last_peered":"2024-03-28T18:09:44.011149+0000","last_clean":"2024-03-28T18:09:44.011149+0000","last_became_active":"2024-03-28T18:08:20.024536+0000","last_became_peered":"2024-03-28T18:08:20.024536+0000","last_unstale":"2024-03-28T18:09:44.011149+0000","last_undegraded":"2024-03-28T18:09:44.011149+0000","last_fullsized":"2024-03-28T18:09:44.011149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:46:27.080005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.888043+0000","last_change":"2024-03-28T18:08:19.480310+0000","last_active":"2024-03-28T18:09:15.888043+0000","last_peered":"2024-03-28T18:09:15.888043+0000","last_clean":"2024-03-28T18:09:15.888043+0000","last_became_active":"2024-03-28T18:08:19.479919+0000","last_became_peered":"2024-03-28T18:08:19.479919+0000","last_unstale":"2024-03-28T18:09:15.888043+0000","last_undegraded":"2024-03-28T18:09:15.888043+0000","last_fullsized":"2024-03-28T18:09:15.888043+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:36:26.239515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.018864+0000","last_change":"2024-03-28T18:08:20.033974+0000","last_active":"2024-03-28T18:09:44.018864+0000","last_peered":"2024-03-28T18:09:44.018864+0000","last_clean":"2024-03-28T18:09:44.018864+0000","last_became_active":"2024-03-28T18:08:20.033544+0000","last_became_peered":"2024-03-28T18:08:20.033544+0000","last_unstale":"2024-03-28T18:09:44.018864+0000","last_undegraded":"2024-03-28T18:09:44.018864+0000","last_fullsized":"2024-03-28T18:09:44.018864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:45:09.617988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.718382+0000","last_change":"2024-03-28T18:08:19.482604+0000","last_active":"2024-03-28T18:09:15.718382+0000","last_peered":"2024-03-28T18:09:15.718382+0000","last_clean":"2024-03-28T18:09:15.718382+0000","last_became_active":"2024-03-28T18:08:19.482248+0000","last_became_peered":"2024-03-28T18:08:19.482248+0000","last_unstale":"2024-03-28T18:09:15.718382+0000","last_undegraded":"2024-03-28T18:09:15.718382+0000","last_fullsized":"2024-03-28T18:09:15.718382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:23.199196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.947114+0000","last_change":"2024-03-28T18:08:20.011198+0000","last_active":"2024-03-28T18:09:43.947114+0000","last_peered":"2024-03-28T18:09:43.947114+0000","last_clean":"2024-03-28T18:09:43.947114+0000","last_became_active":"2024-03-28T18:08:20.010866+0000","last_became_peered":"2024-03-28T18:08:20.010866+0000","last_unstale":"2024-03-28T18:09:43.947114+0000","last_undegraded":"2024-03-28T18:09:43.947114+0000","last_fullsized":"2024-03-28T18:09:43.947114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:26.342858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.724859+0000","last_change":"2024-03-28T18:08:19.493830+0000","last_active":"2024-03-28T18:09:15.724859+0000","last_peered":"2024-03-28T18:09:15.724859+0000","last_clean":"2024-03-28T18:09:15.724859+0000","last_became_active":"2024-03-28T18:08:19.493447+0000","last_became_peered":"2024-03-28T18:08:19.493447+0000","last_unstale":"2024-03-28T18:09:15.724859+0000","last_undegraded":"2024-03-28T18:09:15.724859+0000","last_fullsized":"2024-03-28T18:09:15.724859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:33:29.797186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.008626+0000","last_change":"2024-03-28T18:08:20.025620+0000","last_active":"2024-03-28T18:09:44.008626+0000","last_peered":"2024-03-28T18:09:44.008626+0000","last_clean":"2024-03-28T18:09:44.008626+0000","last_became_active":"2024-03-28T18:08:20.025276+0000","last_became_peered":"2024-03-28T18:08:20.025276+0000","last_unstale":"2024-03-28T18:09:44.008626+0000","last_undegraded":"2024-03-28T18:09:44.008626+0000","last_fullsized":"2024-03-28T18:09:44.008626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:30:06.667707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:10:30.366681+0000","last_change":"2024-03-28T18:08:19.474204+0000","last_active":"2024-03-28T18:10:30.366681+0000","last_peered":"2024-03-28T18:10:30.366681+0000","last_clean":"2024-03-28T18:10:30.366681+0000","last_became_active":"2024-03-28T18:08:19.473688+0000","last_became_peered":"2024-03-28T18:08:19.473688+0000","last_unstale":"2024-03-28T18:10:30.366681+0000","last_undegraded":"2024-03-28T18:10:30.366681+0000","last_fullsized":"2024-03-28T18:10:30.366681+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:15:09.880974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.015204+0000","last_change":"2024-03-28T18:08:20.035920+0000","last_active":"2024-03-28T18:09:44.015204+0000","last_peered":"2024-03-28T18:09:44.015204+0000","last_clean":"2024-03-28T18:09:44.015204+0000","last_became_active":"2024-03-28T18:08:20.035600+0000","last_became_peered":"2024-03-28T18:08:20.035600+0000","last_unstale":"2024-03-28T18:09:44.015204+0000","last_undegraded":"2024-03-28T18:09:44.015204+0000","last_fullsized":"2024-03-28T18:09:44.015204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:41.280448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.715581+0000","last_change":"2024-03-28T18:08:19.481305+0000","last_active":"2024-03-28T18:09:15.715581+0000","last_peered":"2024-03-28T18:09:15.715581+0000","last_clean":"2024-03-28T18:09:15.715581+0000","last_became_active":"2024-03-28T18:08:19.480978+0000","last_became_peered":"2024-03-28T18:08:19.480978+0000","last_unstale":"2024-03-28T18:09:15.715581+0000","last_undegraded":"2024-03-28T18:09:15.715581+0000","last_fullsized":"2024-03-28T18:09:15.715581+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:14:06.464967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.002331+0000","last_change":"2024-03-28T18:08:20.017660+0000","last_active":"2024-03-28T18:09:44.002331+0000","last_peered":"2024-03-28T18:09:44.002331+0000","last_clean":"2024-03-28T18:09:44.002331+0000","last_became_active":"2024-03-28T18:08:20.016083+0000","last_became_peered":"2024-03-28T18:08:20.016083+0000","last_unstale":"2024-03-28T18:09:44.002331+0000","last_undegraded":"2024-03-28T18:09:44.002331+0000","last_fullsized":"2024-03-28T18:09:44.002331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:33:12.104274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.718320+0000","last_change":"2024-03-28T18:08:19.479855+0000","last_active":"2024-03-28T18:09:15.718320+0000","last_peered":"2024-03-28T18:09:15.718320+0000","last_clean":"2024-03-28T18:09:15.718320+0000","last_became_active":"2024-03-28T18:08:19.479429+0000","last_became_peered":"2024-03-28T18:08:19.479429+0000","last_unstale":"2024-03-28T18:09:15.718320+0000","last_undegraded":"2024-03-28T18:09:15.718320+0000","last_fullsized":"2024-03-28T18:09:15.718320+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:06:23.286970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.978524+0000","last_change":"2024-03-28T18:08:20.015419+0000","last_active":"2024-03-28T18:09:43.978524+0000","last_peered":"2024-03-28T18:09:43.978524+0000","last_clean":"2024-03-28T18:09:43.978524+0000","last_became_active":"2024-03-28T18:08:20.015011+0000","last_became_peered":"2024-03-28T18:08:20.015011+0000","last_unstale":"2024-03-28T18:09:43.978524+0000","last_undegraded":"2024-03-28T18:09:43.978524+0000","last_fullsized":"2024-03-28T18:09:43.978524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:44:46.858352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.887142+0000","last_change":"2024-03-28T18:08:19.495541+0000","last_active":"2024-03-28T18:09:15.887142+0000","last_peered":"2024-03-28T18:09:15.887142+0000","last_clean":"2024-03-28T18:09:15.887142+0000","last_became_active":"2024-03-28T18:08:19.495173+0000","last_became_peered":"2024-03-28T18:08:19.495173+0000","last_unstale":"2024-03-28T18:09:15.887142+0000","last_undegraded":"2024-03-28T18:09:15.887142+0000","last_fullsized":"2024-03-28T18:09:15.887142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:42:06.423107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.981192+0000","last_change":"2024-03-28T18:08:20.018582+0000","last_active":"2024-03-28T18:09:43.981192+0000","last_peered":"2024-03-28T18:09:43.981192+0000","last_clean":"2024-03-28T18:09:43.981192+0000","last_became_active":"2024-03-28T18:08:20.017936+0000","last_became_peered":"2024-03-28T18:08:20.017936+0000","last_unstale":"2024-03-28T18:09:43.981192+0000","last_undegraded":"2024-03-28T18:09:43.981192+0000","last_fullsized":"2024-03-28T18:09:43.981192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:05:49.510246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:15.889005+0000","last_change":"2024-03-28T18:08:19.497049+0000","last_active":"2024-03-28T18:09:15.889005+0000","last_peered":"2024-03-28T18:09:15.889005+0000","last_clean":"2024-03-28T18:09:15.889005+0000","last_became_active":"2024-03-28T18:08:19.496623+0000","last_became_peered":"2024-03-28T18:08:19.496623+0000","last_unstale":"2024-03-28T18:09:15.889005+0000","last_undegraded":"2024-03-28T18:09:15.889005+0000","last_fullsized":"2024-03-28T18:09:15.889005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:18.416060+0000","last_clean_scrub_stamp":"2024-03-28T18:08:18.416060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:55:07.842439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.978646+0000","last_change":"2024-03-28T18:08:20.017419+0000","last_active":"2024-03-28T18:09:43.978646+0000","last_peered":"2024-03-28T18:09:43.978646+0000","last_clean":"2024-03-28T18:09:43.978646+0000","last_became_active":"2024-03-28T18:08:20.015916+0000","last_became_peered":"2024-03-28T18:08:20.015916+0000","last_unstale":"2024-03-28T18:09:43.978646+0000","last_undegraded":"2024-03-28T18:09:43.978646+0000","last_fullsized":"2024-03-28T18:09:43.978646+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:10:55.927794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.964132+0000","last_change":"2024-03-28T18:08:20.008093+0000","last_active":"2024-03-28T18:09:43.964132+0000","last_peered":"2024-03-28T18:09:43.964132+0000","last_clean":"2024-03-28T18:09:43.964132+0000","last_became_active":"2024-03-28T18:08:20.007801+0000","last_became_peered":"2024-03-28T18:08:20.007801+0000","last_unstale":"2024-03-28T18:09:43.964132+0000","last_undegraded":"2024-03-28T18:09:43.964132+0000","last_fullsized":"2024-03-28T18:09:43.964132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:58.976859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.967834+0000","last_change":"2024-03-28T18:08:20.012771+0000","last_active":"2024-03-28T18:09:43.967834+0000","last_peered":"2024-03-28T18:09:43.967834+0000","last_clean":"2024-03-28T18:09:43.967834+0000","last_became_active":"2024-03-28T18:08:20.012456+0000","last_became_peered":"2024-03-28T18:08:20.012456+0000","last_unstale":"2024-03-28T18:09:43.967834+0000","last_undegraded":"2024-03-28T18:09:43.967834+0000","last_fullsized":"2024-03-28T18:09:43.967834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:56.428332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.013650+0000","last_change":"2024-03-28T18:08:20.034872+0000","last_active":"2024-03-28T18:09:44.013650+0000","last_peered":"2024-03-28T18:09:44.013650+0000","last_clean":"2024-03-28T18:09:44.013650+0000","last_became_active":"2024-03-28T18:08:20.034614+0000","last_became_peered":"2024-03-28T18:08:20.034614+0000","last_unstale":"2024-03-28T18:09:44.013650+0000","last_undegraded":"2024-03-28T18:09:44.013650+0000","last_fullsized":"2024-03-28T18:09:44.013650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:35.658186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.018419+0000","last_change":"2024-03-28T18:08:20.033119+0000","last_active":"2024-03-28T18:09:44.018419+0000","last_peered":"2024-03-28T18:09:44.018419+0000","last_clean":"2024-03-28T18:09:44.018419+0000","last_became_active":"2024-03-28T18:08:20.030694+0000","last_became_peered":"2024-03-28T18:08:20.030694+0000","last_unstale":"2024-03-28T18:09:44.018419+0000","last_undegraded":"2024-03-28T18:09:44.018419+0000","last_fullsized":"2024-03-28T18:09:44.018419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:51:50.082993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.020461+0000","last_change":"2024-03-28T18:08:20.025725+0000","last_active":"2024-03-28T18:09:44.020461+0000","last_peered":"2024-03-28T18:09:44.020461+0000","last_clean":"2024-03-28T18:09:44.020461+0000","last_became_active":"2024-03-28T18:08:20.024929+0000","last_became_peered":"2024-03-28T18:08:20.024929+0000","last_unstale":"2024-03-28T18:09:44.020461+0000","last_undegraded":"2024-03-28T18:09:44.020461+0000","last_fullsized":"2024-03-28T18:09:44.020461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:04.894246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.960946+0000","last_change":"2024-03-28T18:08:20.005293+0000","last_active":"2024-03-28T18:09:43.960946+0000","last_peered":"2024-03-28T18:09:43.960946+0000","last_clean":"2024-03-28T18:09:43.960946+0000","last_became_active":"2024-03-28T18:08:20.004938+0000","last_became_peered":"2024-03-28T18:08:20.004938+0000","last_unstale":"2024-03-28T18:09:43.960946+0000","last_undegraded":"2024-03-28T18:09:43.960946+0000","last_fullsized":"2024-03-28T18:09:43.960946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:36.100810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.971806+0000","last_change":"2024-03-28T18:08:20.008741+0000","last_active":"2024-03-28T18:09:43.971806+0000","last_peered":"2024-03-28T18:09:43.971806+0000","last_clean":"2024-03-28T18:09:43.971806+0000","last_became_active":"2024-03-28T18:08:20.008411+0000","last_became_peered":"2024-03-28T18:08:20.008411+0000","last_unstale":"2024-03-28T18:09:43.971806+0000","last_undegraded":"2024-03-28T18:09:43.971806+0000","last_fullsized":"2024-03-28T18:09:43.971806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:28:52.276278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.997108+0000","last_change":"2024-03-28T18:08:20.028486+0000","last_active":"2024-03-28T18:09:43.997108+0000","last_peered":"2024-03-28T18:09:43.997108+0000","last_clean":"2024-03-28T18:09:43.997108+0000","last_became_active":"2024-03-28T18:08:20.028121+0000","last_became_peered":"2024-03-28T18:08:20.028121+0000","last_unstale":"2024-03-28T18:09:43.997108+0000","last_undegraded":"2024-03-28T18:09:43.997108+0000","last_fullsized":"2024-03-28T18:09:43.997108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:55:26.828305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.970707+0000","last_change":"2024-03-28T18:08:20.028085+0000","last_active":"2024-03-28T18:09:43.970707+0000","last_peered":"2024-03-28T18:09:43.970707+0000","last_clean":"2024-03-28T18:09:43.970707+0000","last_became_active":"2024-03-28T18:08:20.027721+0000","last_became_peered":"2024-03-28T18:08:20.027721+0000","last_unstale":"2024-03-28T18:09:43.970707+0000","last_undegraded":"2024-03-28T18:09:43.970707+0000","last_fullsized":"2024-03-28T18:09:43.970707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:31:36.674002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.012065+0000","last_change":"2024-03-28T18:08:20.023605+0000","last_active":"2024-03-28T18:09:44.012065+0000","last_peered":"2024-03-28T18:09:44.012065+0000","last_clean":"2024-03-28T18:09:44.012065+0000","last_became_active":"2024-03-28T18:08:20.018217+0000","last_became_peered":"2024-03-28T18:08:20.018217+0000","last_unstale":"2024-03-28T18:09:44.012065+0000","last_undegraded":"2024-03-28T18:09:44.012065+0000","last_fullsized":"2024-03-28T18:09:44.012065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:43.717751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.014919+0000","last_change":"2024-03-28T18:08:20.023072+0000","last_active":"2024-03-28T18:09:44.014919+0000","last_peered":"2024-03-28T18:09:44.014919+0000","last_clean":"2024-03-28T18:09:44.014919+0000","last_became_active":"2024-03-28T18:08:20.022690+0000","last_became_peered":"2024-03-28T18:08:20.022690+0000","last_unstale":"2024-03-28T18:09:44.014919+0000","last_undegraded":"2024-03-28T18:09:44.014919+0000","last_fullsized":"2024-03-28T18:09:44.014919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:21.391415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:44.018990+0000","last_change":"2024-03-28T18:08:20.024261+0000","last_active":"2024-03-28T18:09:44.018990+0000","last_peered":"2024-03-28T18:09:44.018990+0000","last_clean":"2024-03-28T18:09:44.018990+0000","last_became_active":"2024-03-28T18:08:20.023792+0000","last_became_peered":"2024-03-28T18:08:20.023792+0000","last_unstale":"2024-03-28T18:09:44.018990+0000","last_undegraded":"2024-03-28T18:09:44.018990+0000","last_fullsized":"2024-03-28T18:09:44.018990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:23:08.030982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:09:43.968857+0000","last_change":"2024-03-28T18:09:15.720113+0000","last_active":"2024-03-28T18:09:43.968857+0000","last_peered":"2024-03-28T18:09:43.968857+0000","last_clean":"2024-03-28T18:09:43.968857+0000","last_became_active":"2024-03-28T18:09:15.719739+0000","last_became_peered":"2024-03-28T18:09:15.719739+0000","last_unstale":"2024-03-28T18:09:43.968857+0000","last_undegraded":"2024-03-28T18:09:43.968857+0000","last_fullsized":"2024-03-28T18:09:43.968857+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:08:19.447252+0000","last_clean_scrub_stamp":"2024-03-28T18:08:19.447252+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:09:38.179780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":248983,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71213,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":932,"num_read_kb":542,"num_write":175,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":694,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":213639,"data_compressed":5634,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383725,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":456779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645363,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1097728,"data_stored":1114443,"data_compressed":37917,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808888,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070526,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332164,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":1048148,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495690,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":455822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":522492,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738477,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":457093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411425,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1028096,"data_stored":524456,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182491,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":456133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920855,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":456259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757328,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":1048414,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34394,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34394,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36039,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":66877,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68841,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":68343,"data_compressed":1878,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:10:56.130 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:10:56.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.374+0000 7f0a54a00640 1 -- 172.21.15.37:0/3907496613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a500756e0 msgr2=0x7f0a50073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:56.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.374+0000 7f0a54a00640 1 --2- 172.21.15.37:0/3907496613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a500756e0 0x7f0a50073b30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0a38009900 tx=0x7f0a3802bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:56.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.374+0000 7f0a54a00640 1 -- 172.21.15.37:0/3907496613 shutdown_connections 2024-03-28T18:10:56.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.374+0000 7f0a54a00640 1 --2- 172.21.15.37:0/3907496613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a50074100 0x7f0a50074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:56.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.374+0000 7f0a54a00640 1 --2- 172.21.15.37:0/3907496613 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a501093d0 0x7f0a5010b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:56.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.374+0000 7f0a54a00640 1 --2- 172.21.15.37:0/3907496613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a500756e0 0x7f0a50073b30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:56.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.374+0000 7f0a54a00640 1 -- 172.21.15.37:0/3907496613 >> 172.21.15.37:0/3907496613 conn(0x7f0a500fca80 msgr2=0x7f0a500feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:56.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.374+0000 7f0a54a00640 1 -- 172.21.15.37:0/3907496613 shutdown_connections 2024-03-28T18:10:56.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.374+0000 7f0a54a00640 1 -- 172.21.15.37:0/3907496613 wait complete. 2024-03-28T18:10:56.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.376+0000 7f0a54a00640 1 Processor -- start 2024-03-28T18:10:56.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.376+0000 7f0a54a00640 1 -- start start 2024-03-28T18:10:56.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.376+0000 7f0a54a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a50074100 0x7f0a50199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:56.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.376+0000 7f0a54a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a500756e0 0x7f0a50199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:56.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.376+0000 7f0a54a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a501093d0 0x7f0a501a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:56.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.376+0000 7f0a54a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5010f3c0 con 0x7f0a50074100 2024-03-28T18:10:56.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.376+0000 7f0a54a00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5010f240 con 0x7f0a500756e0 2024-03-28T18:10:56.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.376+0000 7f0a54a00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5010f540 con 0x7f0a501093d0 2024-03-28T18:10:56.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.377+0000 7f0a4de00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a500756e0 0x7f0a50199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:56.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.377+0000 7f0a4de00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a500756e0 0x7f0a50199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:36936/0 (socket says 172.21.15.37:36936) 2024-03-28T18:10:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.377+0000 7f0a4de00640 1 -- 172.21.15.37:0/1814577519 learned_addr learned my addr 172.21.15.37:0/1814577519 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.377+0000 7f0a4f200640 1 --2- 172.21.15.37:0/1814577519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a501093d0 0x7f0a501a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.377+0000 7f0a4e800640 1 --2- 172.21.15.37:0/1814577519 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a50074100 0x7f0a50199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.377+0000 7f0a4de00640 1 -- 172.21.15.37:0/1814577519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a501093d0 msgr2=0x7f0a501a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.377+0000 7f0a4de00640 1 --2- 172.21.15.37:0/1814577519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a501093d0 0x7f0a501a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.377+0000 7f0a4de00640 1 -- 172.21.15.37:0/1814577519 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a50074100 msgr2=0x7f0a50199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.377+0000 7f0a4de00640 1 --2- 172.21.15.37:0/1814577519 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a50074100 0x7f0a50199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.377+0000 7f0a4de00640 1 -- 172.21.15.37:0/1814577519 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a501a1640 con 0x7f0a500756e0 2024-03-28T18:10:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.377+0000 7f0a4e800640 1 --2- 172.21.15.37:0/1814577519 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a50074100 0x7f0a50199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.377+0000 7f0a4de00640 1 --2- 172.21.15.37:0/1814577519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a500756e0 0x7f0a50199f60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0a4000eb90 tx=0x7f0a4000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.378+0000 7f0a47400640 1 -- 172.21.15.37:0/1814577519 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40004d20 con 0x7f0a500756e0 2024-03-28T18:10:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.378+0000 7f0a47400640 1 -- 172.21.15.37:0/1814577519 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a4000ee90 con 0x7f0a500756e0 2024-03-28T18:10:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.378+0000 7f0a54a00640 1 -- 172.21.15.37:0/1814577519 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a501a1930 con 0x7f0a500756e0 2024-03-28T18:10:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.378+0000 7f0a47400640 1 -- 172.21.15.37:0/1814577519 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40010830 con 0x7f0a500756e0 2024-03-28T18:10:56.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.378+0000 7f0a54a00640 1 -- 172.21.15.37:0/1814577519 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a501a1e80 con 0x7f0a500756e0 2024-03-28T18:10:56.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.379+0000 7f0a54a00640 1 -- 172.21.15.37:0/1814577519 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a50108100 con 0x7f0a500756e0 2024-03-28T18:10:56.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.381+0000 7f0a47400640 1 -- 172.21.15.37:0/1814577519 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0a40005020 con 0x7f0a500756e0 2024-03-28T18:10:56.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.382+0000 7f0a47400640 1 --2- 172.21.15.37:0/1814577519 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f0a340a2880 0x7f0a340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:56.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.382+0000 7f0a47400640 1 -- 172.21.15.37:0/1814577519 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f0a400cf110 con 0x7f0a500756e0 2024-03-28T18:10:56.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.382+0000 7f0a4e800640 1 --2- 172.21.15.37:0/1814577519 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f0a340a2880 0x7f0a340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:56.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.383+0000 7f0a4e800640 1 --2- 172.21.15.37:0/1814577519 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f0a340a2880 0x7f0a340a4d40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0a3802c240 tx=0x7f0a3802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:56.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.386+0000 7f0a47400640 1 -- 172.21.15.37:0/1814577519 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0a4009c160 con 0x7f0a500756e0 2024-03-28T18:10:56.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.593+0000 7f0a54a00640 1 -- 172.21.15.37:0/1814577519 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a500622a0 con 0x7f0a500756e0 2024-03-28T18:10:56.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.603+0000 7f0a47400640 1 -- 172.21.15.37:0/1814577519 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22575 (secure 0 0 0) 0x7f0a4009ba60 con 0x7f0a500756e0 2024-03-28T18:10:56.603 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:56.603 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":87,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","created":"2024-03-28T17:59:05.949847+0000","modified":"2024-03-28T18:09:15.709901+0000","last_up_change":"2024-03-28T18:07:21.021030+0000","last_in_change":"2024-03-28T18:07:01.416192+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:03:00.517967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:08:18.365827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:08:19.446282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db60618d-56f7-4130-8a90-84cbc480965b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6803","nonce":3562602035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6805","nonce":3562602035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6809","nonce":3562602035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6807","nonce":3562602035}]},"public_addr":"172.21.15.37:6803/3562602035","cluster_addr":"172.21.15.37:6805/3562602035","heartbeat_back_addr":"172.21.15.37:6809/3562602035","heartbeat_front_addr":"172.21.15.37:6807/3562602035","state":["exists","up"]},{"osd":1,"uuid":"7a4b7ad6-905e-4258-9574-693d4277dfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6801","nonce":3326537211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6803","nonce":3326537211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6807","nonce":3326537211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6805","nonce":3326537211}]},"public_addr":"172.21.15.59:6801/3326537211","cluster_addr":"172.21.15.59:6803/3326537211","heartbeat_back_addr":"172.21.15.59:6807/3326537211","heartbeat_front_addr":"172.21.15.59:6805/3326537211","state":["exists","up"]},{"osd":2,"uuid":"8b76ec1c-46bf-4336-89b5-12d82906e8b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6801","nonce":1269398427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6803","nonce":1269398427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6807","nonce":1269398427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6805","nonce":1269398427}]},"public_addr":"172.21.15.170:6801/1269398427","cluster_addr":"172.21.15.170:6803/1269398427","heartbeat_back_addr":"172.21.15.170:6807/1269398427","heartbeat_front_addr":"172.21.15.170:6805/1269398427","state":["exists","up"]},{"osd":3,"uuid":"66eca1da-2fbd-44e1-8bbf-5635a2c70ad5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6811","nonce":317805240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6813","nonce":317805240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6817","nonce":317805240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6815","nonce":317805240}]},"public_addr":"172.21.15.37:6811/317805240","cluster_addr":"172.21.15.37:6813/317805240","heartbeat_back_addr":"172.21.15.37:6817/317805240","heartbeat_front_addr":"172.21.15.37:6815/317805240","state":["exists","up"]},{"osd":4,"uuid":"4c877335-e42d-4d8e-9f48-a1d0a89ae438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6809","nonce":2410632623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6811","nonce":2410632623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6815","nonce":2410632623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6813","nonce":2410632623}]},"public_addr":"172.21.15.59:6809/2410632623","cluster_addr":"172.21.15.59:6811/2410632623","heartbeat_back_addr":"172.21.15.59:6815/2410632623","heartbeat_front_addr":"172.21.15.59:6813/2410632623","state":["exists","up"]},{"osd":5,"uuid":"b884842a-86e2-455b-9570-ff163b3bc0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6809","nonce":1218943899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6811","nonce":1218943899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6815","nonce":1218943899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6813","nonce":1218943899}]},"public_addr":"172.21.15.170:6809/1218943899","cluster_addr":"172.21.15.170:6811/1218943899","heartbeat_back_addr":"172.21.15.170:6815/1218943899","heartbeat_front_addr":"172.21.15.170:6813/1218943899","state":["exists","up"]},{"osd":6,"uuid":"249a1200-77f4-44ad-981f-1d352f732b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6819","nonce":1899652046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6821","nonce":1899652046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6825","nonce":1899652046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6823","nonce":1899652046}]},"public_addr":"172.21.15.37:6819/1899652046","cluster_addr":"172.21.15.37:6821/1899652046","heartbeat_back_addr":"172.21.15.37:6825/1899652046","heartbeat_front_addr":"172.21.15.37:6823/1899652046","state":["exists","up"]},{"osd":7,"uuid":"c73338e6-f505-4517-a84b-f35e04750a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6817","nonce":1343675690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6819","nonce":1343675690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6823","nonce":1343675690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6821","nonce":1343675690}]},"public_addr":"172.21.15.59:6817/1343675690","cluster_addr":"172.21.15.59:6819/1343675690","heartbeat_back_addr":"172.21.15.59:6823/1343675690","heartbeat_front_addr":"172.21.15.59:6821/1343675690","state":["exists","up"]},{"osd":8,"uuid":"149b8297-ff2d-4a88-9b1e-0def021cc516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6817","nonce":3235794556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6819","nonce":3235794556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6823","nonce":3235794556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6821","nonce":3235794556}]},"public_addr":"172.21.15.170:6817/3235794556","cluster_addr":"172.21.15.170:6819/3235794556","heartbeat_back_addr":"172.21.15.170:6823/3235794556","heartbeat_front_addr":"172.21.15.170:6821/3235794556","state":["exists","up"]},{"osd":9,"uuid":"55c27cc2-c771-44b7-90fe-8f0feca44162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6827","nonce":2307153008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6829","nonce":2307153008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6833","nonce":2307153008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6831","nonce":2307153008}]},"public_addr":"172.21.15.37:6827/2307153008","cluster_addr":"172.21.15.37:6829/2307153008","heartbeat_back_addr":"172.21.15.37:6833/2307153008","heartbeat_front_addr":"172.21.15.37:6831/2307153008","state":["exists","up"]},{"osd":10,"uuid":"c0595135-ab81-49d3-8c23-19f960c03e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6825","nonce":2113830275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6827","nonce":2113830275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6831","nonce":2113830275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6829","nonce":2113830275}]},"public_addr":"172.21.15.59:6825/2113830275","cluster_addr":"172.21.15.59:6827/2113830275","heartbeat_back_addr":"172.21.15.59:6831/2113830275","heartbeat_front_addr":"172.21.15.59:6829/2113830275","state":["exists","up"]},{"osd":11,"uuid":"5871e177-c17b-4637-a797-4fa5125920dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6825","nonce":3082439360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6827","nonce":3082439360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6831","nonce":3082439360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6829","nonce":3082439360}]},"public_addr":"172.21.15.170:6825/3082439360","cluster_addr":"172.21.15.170:6827/3082439360","heartbeat_back_addr":"172.21.15.170:6831/3082439360","heartbeat_front_addr":"172.21.15.170:6829/3082439360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:07.690545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:33.183396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:57.400263+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:30.597211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:57.259044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:22.627823+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:57.585863+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:25.000476+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:50.876688+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:25.859761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:53.563146+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:07:19.304115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3023379428":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1443710231":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6800/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6801/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6801/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6800/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1007625490":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/2895698584":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/497313034":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/3293943475":"2024-03-29T17:59:39.162038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:10:56.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.607+0000 7f0a54a00640 1 -- 172.21.15.37:0/1814577519 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f0a340a2880 msgr2=0x7f0a340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:56.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.607+0000 7f0a54a00640 1 --2- 172.21.15.37:0/1814577519 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f0a340a2880 0x7f0a340a4d40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0a3802c240 tx=0x7f0a3802df90 comp rx=0 tx=0).stop 2024-03-28T18:10:56.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.607+0000 7f0a54a00640 1 -- 172.21.15.37:0/1814577519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a500756e0 msgr2=0x7f0a50199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:56.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.607+0000 7f0a54a00640 1 --2- 172.21.15.37:0/1814577519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a500756e0 0x7f0a50199f60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0a4000eb90 tx=0x7f0a4000c6a0 comp rx=0 tx=0).stop 2024-03-28T18:10:56.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.607+0000 7f0a54a00640 1 -- 172.21.15.37:0/1814577519 shutdown_connections 2024-03-28T18:10:56.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.607+0000 7f0a54a00640 1 --2- 172.21.15.37:0/1814577519 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f0a340a2880 0x7f0a340a4d40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:56.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.607+0000 7f0a54a00640 1 --2- 172.21.15.37:0/1814577519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a500756e0 0x7f0a50199f60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:56.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.607+0000 7f0a54a00640 1 --2- 172.21.15.37:0/1814577519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a501093d0 0x7f0a501a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:56.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.607+0000 7f0a54a00640 1 --2- 172.21.15.37:0/1814577519 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a50074100 0x7f0a50199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:56.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.607+0000 7f0a54a00640 1 -- 172.21.15.37:0/1814577519 >> 172.21.15.37:0/1814577519 conn(0x7f0a500fca80 msgr2=0x7f0a501052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:56.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.607+0000 7f0a54a00640 1 -- 172.21.15.37:0/1814577519 shutdown_connections 2024-03-28T18:10:56.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.607+0000 7f0a54a00640 1 -- 172.21.15.37:0/1814577519 wait complete. 2024-03-28T18:10:56.628 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:03:00.517967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:08:18.365827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:08:19.446282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:10:56.629 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:10:56.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.851+0000 7f8f29000640 1 -- 172.21.15.37:0/1672823489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f24103150 msgr2=0x7f8f241099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:56.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.851+0000 7f8f29000640 1 --2- 172.21.15.37:0/1672823489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f24103150 0x7f8f241099e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009900 tx=0x7f8f1c02bc40 comp rx=0 tx=0).stop 2024-03-28T18:10:56.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.851+0000 7f8f29000640 1 -- 172.21.15.37:0/1672823489 shutdown_connections 2024-03-28T18:10:56.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.851+0000 7f8f29000640 1 --2- 172.21.15.37:0/1672823489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f24103150 0x7f8f241099e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:56.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.851+0000 7f8f29000640 1 --2- 172.21.15.37:0/1672823489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f24102790 0x7f8f24102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:56.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.851+0000 7f8f29000640 1 --2- 172.21.15.37:0/1672823489 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f241015a0 0x7f8f241019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:56.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.851+0000 7f8f29000640 1 -- 172.21.15.37:0/1672823489 >> 172.21.15.37:0/1672823489 conn(0x7f8f240fcb10 msgr2=0x7f8f240fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:56.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.851+0000 7f8f29000640 1 -- 172.21.15.37:0/1672823489 shutdown_connections 2024-03-28T18:10:56.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.851+0000 7f8f29000640 1 -- 172.21.15.37:0/1672823489 wait complete. 2024-03-28T18:10:56.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.852+0000 7f8f29000640 1 Processor -- start 2024-03-28T18:10:56.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.852+0000 7f8f29000640 1 -- start start 2024-03-28T18:10:56.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.853+0000 7f8f29000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f241015a0 0x7f8f24197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:56.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.853+0000 7f8f29000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f24102790 0x7f8f24197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:56.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.853+0000 7f8f29000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f24103150 0x7f8f2419edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:56.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.853+0000 7f8f29000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2410d250 con 0x7f8f24102790 2024-03-28T18:10:56.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.853+0000 7f8f29000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2410d0d0 con 0x7f8f241015a0 2024-03-28T18:10:56.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.853+0000 7f8f29000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2410d3d0 con 0x7f8f24103150 2024-03-28T18:10:56.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.853+0000 7f8f22a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f24102790 0x7f8f24197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:56.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.853+0000 7f8f23e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f24103150 0x7f8f2419edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:56.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.853+0000 7f8f23400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f241015a0 0x7f8f24197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:56.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.853+0000 7f8f22a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f24102790 0x7f8f24197d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60958/0 (socket says 172.21.15.37:60958) 2024-03-28T18:10:56.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.853+0000 7f8f22a00640 1 -- 172.21.15.37:0/2345110925 learned_addr learned my addr 172.21.15.37:0/2345110925 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:56.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.854+0000 7f8f23e00640 1 -- 172.21.15.37:0/2345110925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f241015a0 msgr2=0x7f8f24197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:56.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.854+0000 7f8f23e00640 1 --2- 172.21.15.37:0/2345110925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f241015a0 0x7f8f24197810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:56.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.854+0000 7f8f23e00640 1 -- 172.21.15.37:0/2345110925 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f24102790 msgr2=0x7f8f24197d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:56.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.854+0000 7f8f23e00640 1 --2- 172.21.15.37:0/2345110925 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f24102790 0x7f8f24197d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:56.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.854+0000 7f8f23e00640 1 -- 172.21.15.37:0/2345110925 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f2419f430 con 0x7f8f24103150 2024-03-28T18:10:56.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.854+0000 7f8f22a00640 1 --2- 172.21.15.37:0/2345110925 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f24102790 0x7f8f24197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:56.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.854+0000 7f8f23e00640 1 --2- 172.21.15.37:0/2345110925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f24103150 0x7f8f2419edc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c02c180 tx=0x7f8f1c005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:56.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.854+0000 7f8f1be00640 1 -- 172.21.15.37:0/2345110925 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c041070 con 0x7f8f24103150 2024-03-28T18:10:56.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.854+0000 7f8f1be00640 1 -- 172.21.15.37:0/2345110925 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f1c0044d0 con 0x7f8f24103150 2024-03-28T18:10:56.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.854+0000 7f8f29000640 1 -- 172.21.15.37:0/2345110925 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f2419f6c0 con 0x7f8f24103150 2024-03-28T18:10:56.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.854+0000 7f8f1be00640 1 -- 172.21.15.37:0/2345110925 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c034440 con 0x7f8f24103150 2024-03-28T18:10:56.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.855+0000 7f8f29000640 1 -- 172.21.15.37:0/2345110925 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f2419fb80 con 0x7f8f24103150 2024-03-28T18:10:56.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.856+0000 7f8f29000640 1 -- 172.21.15.37:0/2345110925 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f24108860 con 0x7f8f24103150 2024-03-28T18:10:56.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.859+0000 7f8f1be00640 1 -- 172.21.15.37:0/2345110925 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8f1c044070 con 0x7f8f24103150 2024-03-28T18:10:56.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.861+0000 7f8f1be00640 1 --2- 172.21.15.37:0/2345110925 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8f080a2800 0x7f8f080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:56.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.861+0000 7f8f1be00640 1 -- 172.21.15.37:0/2345110925 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f8f1c0f0900 con 0x7f8f24103150 2024-03-28T18:10:56.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.861+0000 7f8f23400640 1 --2- 172.21.15.37:0/2345110925 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8f080a2800 0x7f8f080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:56.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.861+0000 7f8f1be00640 1 -- 172.21.15.37:0/2345110925 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8f1c0bd300 con 0x7f8f24103150 2024-03-28T18:10:56.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:56.862+0000 7f8f23400640 1 --2- 172.21.15.37:0/2345110925 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8f080a2800 0x7f8f080a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8f241025f0 tx=0x7f8f14009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:56.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:56 smithi037 ceph-mon[31460]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:10:56.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:56 smithi037 ceph-mon[31460]: from='client.25288 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:10:57.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:56 smithi059 ceph-mon[36190]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:10:57.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:56 smithi059 ceph-mon[36190]: from='client.25288 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:10:57.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:56 smithi170 ceph-mon[36280]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:10:57.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:56 smithi170 ceph-mon[36280]: from='client.25288 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:10:57.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.065+0000 7f8f29000640 1 -- 172.21.15.37:0/2345110925 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8f2402d200 con 0x7f8f24103150 2024-03-28T18:10:57.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.065+0000 7f8f1be00640 1 -- 172.21.15.37:0/2345110925 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8f1c0c13a0 con 0x7f8f24103150 2024-03-28T18:10:57.065 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 1 2024-03-28T18:10:57.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.068+0000 7f8f29000640 1 -- 172.21.15.37:0/2345110925 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8f080a2800 msgr2=0x7f8f080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:57.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.068+0000 7f8f29000640 1 --2- 172.21.15.37:0/2345110925 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8f080a2800 0x7f8f080a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8f241025f0 tx=0x7f8f14009670 comp rx=0 tx=0).stop 2024-03-28T18:10:57.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.068+0000 7f8f29000640 1 -- 172.21.15.37:0/2345110925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f24103150 msgr2=0x7f8f2419edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:57.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.068+0000 7f8f29000640 1 --2- 172.21.15.37:0/2345110925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f24103150 0x7f8f2419edc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c02c180 tx=0x7f8f1c005d50 comp rx=0 tx=0).stop 2024-03-28T18:10:57.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.068+0000 7f8f29000640 1 -- 172.21.15.37:0/2345110925 shutdown_connections 2024-03-28T18:10:57.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.068+0000 7f8f29000640 1 --2- 172.21.15.37:0/2345110925 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f8f080a2800 0x7f8f080a4cc0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:57.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.068+0000 7f8f29000640 1 --2- 172.21.15.37:0/2345110925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f24103150 0x7f8f2419edc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:57.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.068+0000 7f8f29000640 1 --2- 172.21.15.37:0/2345110925 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f24102790 0x7f8f24197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:57.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.068+0000 7f8f29000640 1 --2- 172.21.15.37:0/2345110925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f241015a0 0x7f8f24197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:57.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.068+0000 7f8f29000640 1 -- 172.21.15.37:0/2345110925 >> 172.21.15.37:0/2345110925 conn(0x7f8f240fcb10 msgr2=0x7f8f24105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:57.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.068+0000 7f8f29000640 1 -- 172.21.15.37:0/2345110925 shutdown_connections 2024-03-28T18:10:57.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.068+0000 7f8f29000640 1 -- 172.21.15.37:0/2345110925 wait complete. 2024-03-28T18:10:57.087 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:10:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.312+0000 7f101f200640 1 -- 172.21.15.37:0/79427918 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10181037c0 msgr2=0x7f1018103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.312+0000 7f101f200640 1 --2- 172.21.15.37:0/79427918 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10181037c0 0x7f1018103be0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f100c009900 tx=0x7f100c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.312+0000 7f101f200640 1 -- 172.21.15.37:0/79427918 shutdown_connections 2024-03-28T18:10:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.313+0000 7f101f200640 1 --2- 172.21.15.37:0/79427918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1018105370 0x7f101810bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.313+0000 7f101f200640 1 --2- 172.21.15.37:0/79427918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10181049b0 0x7f1018104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.313+0000 7f101f200640 1 --2- 172.21.15.37:0/79427918 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10181037c0 0x7f1018103be0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.313+0000 7f101f200640 1 -- 172.21.15.37:0/79427918 >> 172.21.15.37:0/79427918 conn(0x7f10180fed50 msgr2=0x7f1018101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.313+0000 7f101f200640 1 -- 172.21.15.37:0/79427918 shutdown_connections 2024-03-28T18:10:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.313+0000 7f101f200640 1 -- 172.21.15.37:0/79427918 wait complete. 2024-03-28T18:10:57.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.314+0000 7f101f200640 1 Processor -- start 2024-03-28T18:10:57.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.314+0000 7f101f200640 1 -- start start 2024-03-28T18:10:57.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.314+0000 7f101f200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10181037c0 0x7f1018199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:57.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.314+0000 7f101f200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10181049b0 0x7f1018199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:57.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.314+0000 7f101f200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1018105370 0x7f10181a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:57.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.314+0000 7f101f200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101810f490 con 0x7f1018105370 2024-03-28T18:10:57.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.314+0000 7f101f200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101810f310 con 0x7f10181049b0 2024-03-28T18:10:57.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.314+0000 7f101f200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101810f610 con 0x7f10181037c0 2024-03-28T18:10:57.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.314+0000 7f101d400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10181049b0 0x7f1018199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:57.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.314+0000 7f101de00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10181037c0 0x7f1018199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:57.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.314+0000 7f101d400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10181049b0 0x7f1018199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:36970/0 (socket says 172.21.15.37:36970) 2024-03-28T18:10:57.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.314+0000 7f101d400640 1 -- 172.21.15.37:0/2181021888 learned_addr learned my addr 172.21.15.37:0/2181021888 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:57.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.314+0000 7f101e800640 1 --2- 172.21.15.37:0/2181021888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1018105370 0x7f10181a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:57.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.315+0000 7f101d400640 1 -- 172.21.15.37:0/2181021888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10181037c0 msgr2=0x7f1018199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:57.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.315+0000 7f101d400640 1 --2- 172.21.15.37:0/2181021888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10181037c0 0x7f1018199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:57.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.315+0000 7f101d400640 1 -- 172.21.15.37:0/2181021888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1018105370 msgr2=0x7f10181a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:57.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.315+0000 7f101d400640 1 --2- 172.21.15.37:0/2181021888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1018105370 0x7f10181a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:57.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.315+0000 7f101d400640 1 -- 172.21.15.37:0/2181021888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10181a16d0 con 0x7f10181049b0 2024-03-28T18:10:57.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.315+0000 7f101de00640 1 --2- 172.21.15.37:0/2181021888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10181037c0 0x7f1018199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:57.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.315+0000 7f101d400640 1 --2- 172.21.15.37:0/2181021888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10181049b0 0x7f1018199ff0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f100800e9f0 tx=0x7f100800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:57.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.315+0000 7f1012a00640 1 -- 172.21.15.37:0/2181021888 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100800cd80 con 0x7f10181049b0 2024-03-28T18:10:57.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.315+0000 7f1012a00640 1 -- 172.21.15.37:0/2181021888 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1008010910 con 0x7f10181049b0 2024-03-28T18:10:57.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.315+0000 7f1012a00640 1 -- 172.21.15.37:0/2181021888 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1008018990 con 0x7f10181049b0 2024-03-28T18:10:57.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.315+0000 7f101f200640 1 -- 172.21.15.37:0/2181021888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10181a19c0 con 0x7f10181049b0 2024-03-28T18:10:57.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.316+0000 7f101f200640 1 -- 172.21.15.37:0/2181021888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10181a1f60 con 0x7f10181049b0 2024-03-28T18:10:57.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.317+0000 7f101f200640 1 -- 172.21.15.37:0/2181021888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f101810aa80 con 0x7f10181049b0 2024-03-28T18:10:57.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.320+0000 7f1012a00640 1 -- 172.21.15.37:0/2181021888 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1008010ac0 con 0x7f10181049b0 2024-03-28T18:10:57.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.321+0000 7f1012a00640 1 --2- 172.21.15.37:0/2181021888 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f10000a2880 0x7f10000a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:57.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.321+0000 7f1012a00640 1 -- 172.21.15.37:0/2181021888 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f10080ced40 con 0x7f10181049b0 2024-03-28T18:10:57.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.321+0000 7f101de00640 1 --2- 172.21.15.37:0/2181021888 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f10000a2880 0x7f10000a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:57.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.322+0000 7f101de00640 1 --2- 172.21.15.37:0/2181021888 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f10000a2880 0x7f10000a4d40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f100c02c170 tx=0x7f100c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:57.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.325+0000 7f1012a00640 1 -- 172.21.15.37:0/2181021888 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f100809b740 con 0x7f10181049b0 2024-03-28T18:10:57.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.532+0000 7f101f200640 1 -- 172.21.15.37:0/2181021888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f10180622a0 con 0x7f10181049b0 2024-03-28T18:10:57.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.533+0000 7f1012a00640 1 -- 172.21.15.37:0/2181021888 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f100809f7e0 con 0x7f10181049b0 2024-03-28T18:10:57.533 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 32 2024-03-28T18:10:57.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.535+0000 7f101f200640 1 -- 172.21.15.37:0/2181021888 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f10000a2880 msgr2=0x7f10000a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:57.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.535+0000 7f101f200640 1 --2- 172.21.15.37:0/2181021888 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f10000a2880 0x7f10000a4d40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f100c02c170 tx=0x7f100c02df90 comp rx=0 tx=0).stop 2024-03-28T18:10:57.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.535+0000 7f101f200640 1 -- 172.21.15.37:0/2181021888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10181049b0 msgr2=0x7f1018199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:57.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.535+0000 7f101f200640 1 --2- 172.21.15.37:0/2181021888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10181049b0 0x7f1018199ff0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f100800e9f0 tx=0x7f100800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:10:57.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.536+0000 7f101f200640 1 -- 172.21.15.37:0/2181021888 shutdown_connections 2024-03-28T18:10:57.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.536+0000 7f101f200640 1 --2- 172.21.15.37:0/2181021888 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f10000a2880 0x7f10000a4d40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:57.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.536+0000 7f101f200640 1 --2- 172.21.15.37:0/2181021888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10181049b0 0x7f1018199ff0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:57.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.536+0000 7f101f200640 1 --2- 172.21.15.37:0/2181021888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1018105370 0x7f10181a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:57.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.536+0000 7f101f200640 1 --2- 172.21.15.37:0/2181021888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10181037c0 0x7f1018199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:57.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.536+0000 7f101f200640 1 -- 172.21.15.37:0/2181021888 >> 172.21.15.37:0/2181021888 conn(0x7f10180fed50 msgr2=0x7f1018107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:57.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.536+0000 7f101f200640 1 -- 172.21.15.37:0/2181021888 shutdown_connections 2024-03-28T18:10:57.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.536+0000 7f101f200640 1 -- 172.21.15.37:0/2181021888 wait complete. 2024-03-28T18:10:57.558 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:10:57.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.821+0000 7fb3dac00640 1 -- 172.21.15.37:0/3185427695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3d40ffe90 msgr2=0x7fb3d41002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:57.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.821+0000 7fb3dac00640 1 --2- 172.21.15.37:0/3185427695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3d40ffe90 0x7fb3d41002b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8009900 tx=0x7fb3c802bcc0 comp rx=0 tx=0).stop 2024-03-28T18:10:57.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.822+0000 7fb3dac00640 1 -- 172.21.15.37:0/3185427695 shutdown_connections 2024-03-28T18:10:57.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.822+0000 7fb3dac00640 1 --2- 172.21.15.37:0/3185427695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3d4101190 0x7fb3d4101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:57.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.822+0000 7fb3dac00640 1 --2- 172.21.15.37:0/3185427695 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3d41007f0 0x7fb3d4100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:57.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.822+0000 7fb3dac00640 1 --2- 172.21.15.37:0/3185427695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3d40ffe90 0x7fb3d41002b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:57.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.822+0000 7fb3dac00640 1 -- 172.21.15.37:0/3185427695 >> 172.21.15.37:0/3185427695 conn(0x7fb3d40fba10 msgr2=0x7fb3d40fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:57.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.822+0000 7fb3dac00640 1 -- 172.21.15.37:0/3185427695 shutdown_connections 2024-03-28T18:10:57.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.822+0000 7fb3dac00640 1 -- 172.21.15.37:0/3185427695 wait complete. 2024-03-28T18:10:57.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.823+0000 7fb3dac00640 1 Processor -- start 2024-03-28T18:10:57.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.823+0000 7fb3dac00640 1 -- start start 2024-03-28T18:10:57.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.823+0000 7fb3dac00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3d40ffe90 0x7fb3d4199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:57.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.823+0000 7fb3dac00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3d41007f0 0x7fb3d4199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:57.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.823+0000 7fb3dac00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3d4101190 0x7fb3d41a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:57.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.823+0000 7fb3dac00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d410f440 con 0x7fb3d41007f0 2024-03-28T18:10:57.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.823+0000 7fb3dac00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d410f2c0 con 0x7fb3d40ffe90 2024-03-28T18:10:57.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.823+0000 7fb3dac00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d410f5c0 con 0x7fb3d4101190 2024-03-28T18:10:57.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.823+0000 7fb3d9800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3d40ffe90 0x7fb3d4199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:57.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.823+0000 7fb3d9800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3d40ffe90 0x7fb3d4199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:36992/0 (socket says 172.21.15.37:36992) 2024-03-28T18:10:57.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.823+0000 7fb3d9800640 1 -- 172.21.15.37:0/1069014125 learned_addr learned my addr 172.21.15.37:0/1069014125 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:57.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.824+0000 7fb3da200640 1 --2- 172.21.15.37:0/1069014125 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3d4101190 0x7fb3d41a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:57.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.824+0000 7fb3d8e00640 1 --2- 172.21.15.37:0/1069014125 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3d41007f0 0x7fb3d4199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:57.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.824+0000 7fb3d9800640 1 -- 172.21.15.37:0/1069014125 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3d4101190 msgr2=0x7fb3d41a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:57.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.824+0000 7fb3d9800640 1 --2- 172.21.15.37:0/1069014125 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3d4101190 0x7fb3d41a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:57.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.824+0000 7fb3d9800640 1 -- 172.21.15.37:0/1069014125 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3d41007f0 msgr2=0x7fb3d4199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:57.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.824+0000 7fb3d9800640 1 --2- 172.21.15.37:0/1069014125 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3d41007f0 0x7fb3d4199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:57.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.824+0000 7fb3d9800640 1 -- 172.21.15.37:0/1069014125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d41a16a0 con 0x7fb3d40ffe90 2024-03-28T18:10:57.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.824+0000 7fb3d9800640 1 --2- 172.21.15.37:0/1069014125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3d40ffe90 0x7fb3d4199a80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8000c00 tx=0x7fb3c802c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:57.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.824+0000 7fb3d2000640 1 -- 172.21.15.37:0/1069014125 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8043020 con 0x7fb3d40ffe90 2024-03-28T18:10:57.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.825+0000 7fb3d2000640 1 -- 172.21.15.37:0/1069014125 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3c80054e0 con 0x7fb3d40ffe90 2024-03-28T18:10:57.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.825+0000 7fb3dac00640 1 -- 172.21.15.37:0/1069014125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d41a1930 con 0x7fb3d40ffe90 2024-03-28T18:10:57.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.825+0000 7fb3d2000640 1 -- 172.21.15.37:0/1069014125 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8003b20 con 0x7fb3d40ffe90 2024-03-28T18:10:57.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.825+0000 7fb3dac00640 1 -- 172.21.15.37:0/1069014125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d41a1ed0 con 0x7fb3d40ffe90 2024-03-28T18:10:57.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.826+0000 7fb3dac00640 1 -- 172.21.15.37:0/1069014125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d4068800 con 0x7fb3d40ffe90 2024-03-28T18:10:57.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.828+0000 7fb3d2000640 1 -- 172.21.15.37:0/1069014125 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb3c8046020 con 0x7fb3d40ffe90 2024-03-28T18:10:57.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.829+0000 7fb3d2000640 1 --2- 172.21.15.37:0/1069014125 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb3bc0a2880 0x7fb3bc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:57.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.829+0000 7fb3d2000640 1 -- 172.21.15.37:0/1069014125 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb3c80f09d0 con 0x7fb3d40ffe90 2024-03-28T18:10:57.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.829+0000 7fb3d8e00640 1 --2- 172.21.15.37:0/1069014125 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb3bc0a2880 0x7fb3bc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:57.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.830+0000 7fb3d8e00640 1 --2- 172.21.15.37:0/1069014125 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb3bc0a2880 0x7fb3bc0a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb3d419af90 tx=0x7fb3c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:57.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:57.833+0000 7fb3d2000640 1 -- 172.21.15.37:0/1069014125 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb3c804d020 con 0x7fb3d40ffe90 2024-03-28T18:10:57.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:57 smithi037 ceph-mon[31460]: pgmap v398: 97 pgs: 97 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-28T18:10:57.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:57 smithi037 ceph-mon[31460]: from='client.25265 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:10:57.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:57 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1814577519' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:57.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:57 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2345110925' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:57.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:57 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2181021888' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:57 smithi059 ceph-mon[36190]: pgmap v398: 97 pgs: 97 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-28T18:10:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:57 smithi059 ceph-mon[36190]: from='client.25265 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:10:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:57 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1814577519' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:57 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2345110925' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:57 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2181021888' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:58.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:57 smithi170 ceph-mon[36280]: pgmap v398: 97 pgs: 97 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-28T18:10:58.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:57 smithi170 ceph-mon[36280]: from='client.25265 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:10:58.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:57 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1814577519' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:58.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:57 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2345110925' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:10:58.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:57 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2181021888' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:10:58.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.086+0000 7fb3dac00640 1 -- 172.21.15.37:0/1069014125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb3d40622a0 con 0x7fb3d40ffe90 2024-03-28T18:10:58.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.087+0000 7fb3d2000640 1 -- 172.21.15.37:0/1069014125 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb3c80dbe20 con 0x7fb3d40ffe90 2024-03-28T18:10:58.086 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 64 2024-03-28T18:10:58.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.089+0000 7fb3dac00640 1 -- 172.21.15.37:0/1069014125 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb3bc0a2880 msgr2=0x7fb3bc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:58.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.089+0000 7fb3dac00640 1 --2- 172.21.15.37:0/1069014125 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb3bc0a2880 0x7fb3bc0a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb3d419af90 tx=0x7fb3c0005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:58.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.089+0000 7fb3dac00640 1 -- 172.21.15.37:0/1069014125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3d40ffe90 msgr2=0x7fb3d4199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:58.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.089+0000 7fb3dac00640 1 --2- 172.21.15.37:0/1069014125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3d40ffe90 0x7fb3d4199a80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8000c00 tx=0x7fb3c802c6e0 comp rx=0 tx=0).stop 2024-03-28T18:10:58.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.089+0000 7fb3dac00640 1 -- 172.21.15.37:0/1069014125 shutdown_connections 2024-03-28T18:10:58.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.089+0000 7fb3dac00640 1 --2- 172.21.15.37:0/1069014125 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fb3bc0a2880 0x7fb3bc0a4d40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:58.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.089+0000 7fb3dac00640 1 --2- 172.21.15.37:0/1069014125 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3d4101190 0x7fb3d41a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:58.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.089+0000 7fb3dac00640 1 --2- 172.21.15.37:0/1069014125 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3d41007f0 0x7fb3d4199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:58.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.089+0000 7fb3dac00640 1 --2- 172.21.15.37:0/1069014125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3d40ffe90 0x7fb3d4199a80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:58.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.089+0000 7fb3dac00640 1 -- 172.21.15.37:0/1069014125 >> 172.21.15.37:0/1069014125 conn(0x7fb3d40fba10 msgr2=0x7fb3d40fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:58.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.089+0000 7fb3dac00640 1 -- 172.21.15.37:0/1069014125 shutdown_connections 2024-03-28T18:10:58.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.089+0000 7fb3dac00640 1 -- 172.21.15.37:0/1069014125 wait complete. 2024-03-28T18:10:58.109 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:10:58.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.322+0000 7fe1c3a00640 1 -- 172.21.15.37:0/3538418832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1bc102140 msgr2=0x7fe1bc104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:58.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.322+0000 7fe1c3a00640 1 --2- 172.21.15.37:0/3538418832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1bc102140 0x7fe1bc104530 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0009900 tx=0x7fe1b002bc50 comp rx=0 tx=0).stop 2024-03-28T18:10:58.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.322+0000 7fe1c3a00640 1 -- 172.21.15.37:0/3538418832 shutdown_connections 2024-03-28T18:10:58.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.322+0000 7fe1c3a00640 1 --2- 172.21.15.37:0/3538418832 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe1bc1073a0 0x7fe1bc1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:58.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.322+0000 7fe1c3a00640 1 --2- 172.21.15.37:0/3538418832 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1bc104a70 0x7fe1bc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:58.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.322+0000 7fe1c3a00640 1 --2- 172.21.15.37:0/3538418832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1bc102140 0x7fe1bc104530 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:58.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.322+0000 7fe1c3a00640 1 -- 172.21.15.37:0/3538418832 >> 172.21.15.37:0/3538418832 conn(0x7fe1bc0fbb50 msgr2=0x7fe1bc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:58.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.322+0000 7fe1c3a00640 1 -- 172.21.15.37:0/3538418832 shutdown_connections 2024-03-28T18:10:58.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.322+0000 7fe1c3a00640 1 -- 172.21.15.37:0/3538418832 wait complete. 2024-03-28T18:10:58.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.324+0000 7fe1c3a00640 1 Processor -- start 2024-03-28T18:10:58.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.324+0000 7fe1c3a00640 1 -- start start 2024-03-28T18:10:58.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.324+0000 7fe1c3a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1bc102140 0x7fe1bc199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:58.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.324+0000 7fe1c3a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1bc104a70 0x7fe1bc19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:58.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.324+0000 7fe1c3a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe1bc1073a0 0x7fe1bc1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:58.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.324+0000 7fe1c3a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1bc10d390 con 0x7fe1bc1073a0 2024-03-28T18:10:58.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.324+0000 7fe1c3a00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1bc10d210 con 0x7fe1bc104a70 2024-03-28T18:10:58.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.324+0000 7fe1c3a00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1bc10d510 con 0x7fe1bc102140 2024-03-28T18:10:58.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.324+0000 7fe1c1000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1bc102140 0x7fe1bc199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:58.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.324+0000 7fe1bbe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1bc104a70 0x7fe1bc19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:58.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.324+0000 7fe1c1a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe1bc1073a0 0x7fe1bc1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:58.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.324+0000 7fe1bbe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1bc104a70 0x7fe1bc19a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:37004/0 (socket says 172.21.15.37:37004) 2024-03-28T18:10:58.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.324+0000 7fe1bbe00640 1 -- 172.21.15.37:0/173304552 learned_addr learned my addr 172.21.15.37:0/173304552 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:58.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.324+0000 7fe1c1000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1bc102140 0x7fe1bc199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55926/0 (socket says 172.21.15.37:55926) 2024-03-28T18:10:58.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.325+0000 7fe1bbe00640 1 -- 172.21.15.37:0/173304552 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1bc102140 msgr2=0x7fe1bc199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:58.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.325+0000 7fe1bbe00640 1 --2- 172.21.15.37:0/173304552 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1bc102140 0x7fe1bc199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:58.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.325+0000 7fe1bbe00640 1 -- 172.21.15.37:0/173304552 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe1bc1073a0 msgr2=0x7fe1bc1a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:58.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.325+0000 7fe1bbe00640 1 --2- 172.21.15.37:0/173304552 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe1bc1073a0 0x7fe1bc1a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:58.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.325+0000 7fe1bbe00640 1 -- 172.21.15.37:0/173304552 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1bc1a1800 con 0x7fe1bc104a70 2024-03-28T18:10:58.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.325+0000 7fe1c1000640 1 --2- 172.21.15.37:0/173304552 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1bc102140 0x7fe1bc199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:58.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.325+0000 7fe1bbe00640 1 --2- 172.21.15.37:0/173304552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1bc104a70 0x7fe1bc19a120 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800e9c0 tx=0x7fe1a800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:58.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.325+0000 7fe1b9600640 1 -- 172.21.15.37:0/173304552 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a800cd80 con 0x7fe1bc104a70 2024-03-28T18:10:58.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.325+0000 7fe1b9600640 1 -- 172.21.15.37:0/173304552 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe1a8005500 con 0x7fe1bc104a70 2024-03-28T18:10:58.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.325+0000 7fe1b9600640 1 -- 172.21.15.37:0/173304552 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8010690 con 0x7fe1bc104a70 2024-03-28T18:10:58.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.326+0000 7fe1c3a00640 1 -- 172.21.15.37:0/173304552 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1bc1a1af0 con 0x7fe1bc104a70 2024-03-28T18:10:58.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.326+0000 7fe1c3a00640 1 -- 172.21.15.37:0/173304552 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1bc1a1ff0 con 0x7fe1bc104a70 2024-03-28T18:10:58.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.327+0000 7fe1c3a00640 1 -- 172.21.15.37:0/173304552 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1bc108ef0 con 0x7fe1bc104a70 2024-03-28T18:10:58.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.329+0000 7fe1b9600640 1 -- 172.21.15.37:0/173304552 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe1a80046c0 con 0x7fe1bc104a70 2024-03-28T18:10:58.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.331+0000 7fe1b9600640 1 --2- 172.21.15.37:0/173304552 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe1a40a2880 0x7fe1a40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:58.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.331+0000 7fe1b9600640 1 -- 172.21.15.37:0/173304552 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe1a80ced90 con 0x7fe1bc104a70 2024-03-28T18:10:58.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.331+0000 7fe1c1000640 1 --2- 172.21.15.37:0/173304552 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe1a40a2880 0x7fe1a40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:58.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.332+0000 7fe1c1000640 1 --2- 172.21.15.37:0/173304552 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe1a40a2880 0x7fe1a40a4d40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe1b002c260 tx=0x7fe1b0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:58.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.334+0000 7fe1b9600640 1 -- 172.21.15.37:0/173304552 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe1a809bde0 con 0x7fe1bc104a70 2024-03-28T18:10:58.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.581+0000 7fe1c3a00640 1 -- 172.21.15.37:0/173304552 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1bc0622a0 con 0x7fe1bc104a70 2024-03-28T18:10:58.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.582+0000 7fe1b9600640 1 -- 172.21.15.37:0/173304552 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11383 (secure 0 0 0) 0x7fe1a809b6e0 con 0x7fe1bc104a70 2024-03-28T18:10:58.582 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:58.582 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1926270371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6838","nonce":1926270371},{"type":"v1","addr":"172.21.15.37:6839","nonce":1926270371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:08:19.968894+0000","modified":"2024-03-28T18:10:49.565429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24580,"mds_1":24569,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6837/4034607523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":4034607523},{"type":"v1","addr":"172.21.15.59:6837","nonce":4034607523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6839/1575518279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":1575518279},{"type":"v1","addr":"172.21.15.170:6839","nonce":1575518279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6839/2091629078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":2091629078},{"type":"v1","addr":"172.21.15.59:6839","nonce":2091629078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24580":{"gid":24580,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.37:6841/1923263622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6840","nonce":1923263622},{"type":"v1","addr":"172.21.15.37:6841","nonce":1923263622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24580,"qdb_cluster":[24580,24550,24569]},"id":1}]} 2024-03-28T18:10:58.582 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 23 2024-03-28T18:10:58.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.584+0000 7fe1c3a00640 1 -- 172.21.15.37:0/173304552 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe1a40a2880 msgr2=0x7fe1a40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:58.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.584+0000 7fe1c3a00640 1 --2- 172.21.15.37:0/173304552 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe1a40a2880 0x7fe1a40a4d40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe1b002c260 tx=0x7fe1b0000bc0 comp rx=0 tx=0).stop 2024-03-28T18:10:58.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.584+0000 7fe1c3a00640 1 -- 172.21.15.37:0/173304552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1bc104a70 msgr2=0x7fe1bc19a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:58.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.584+0000 7fe1c3a00640 1 --2- 172.21.15.37:0/173304552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1bc104a70 0x7fe1bc19a120 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800e9c0 tx=0x7fe1a800ee80 comp rx=0 tx=0).stop 2024-03-28T18:10:58.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.584+0000 7fe1c3a00640 1 -- 172.21.15.37:0/173304552 shutdown_connections 2024-03-28T18:10:58.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.584+0000 7fe1c3a00640 1 --2- 172.21.15.37:0/173304552 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fe1a40a2880 0x7fe1a40a4d40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:58.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.584+0000 7fe1c3a00640 1 --2- 172.21.15.37:0/173304552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1bc104a70 0x7fe1bc19a120 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:58.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.584+0000 7fe1c3a00640 1 --2- 172.21.15.37:0/173304552 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe1bc1073a0 0x7fe1bc1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:58.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.584+0000 7fe1c3a00640 1 --2- 172.21.15.37:0/173304552 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1bc102140 0x7fe1bc199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:58.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.585+0000 7fe1c3a00640 1 -- 172.21.15.37:0/173304552 >> 172.21.15.37:0/173304552 conn(0x7fe1bc0fbb50 msgr2=0x7fe1bc101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:58.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.585+0000 7fe1c3a00640 1 -- 172.21.15.37:0/173304552 shutdown_connections 2024-03-28T18:10:58.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.585+0000 7fe1c3a00640 1 -- 172.21.15.37:0/173304552 wait complete. 2024-03-28T18:10:58.603 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:10:58.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.857+0000 7f7ebec00640 1 -- 172.21.15.37:0/3263729468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eb81014e0 msgr2=0x7f7eb8101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:58.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.857+0000 7f7ebec00640 1 --2- 172.21.15.37:0/3263729468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eb81014e0 0x7f7eb8101900 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7eac009900 tx=0x7f7eac02bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:58.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.857+0000 7f7ebec00640 1 -- 172.21.15.37:0/3263729468 shutdown_connections 2024-03-28T18:10:58.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.857+0000 7f7ebec00640 1 --2- 172.21.15.37:0/3263729468 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7eb8103090 0x7f7eb8109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:58.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.857+0000 7f7ebec00640 1 --2- 172.21.15.37:0/3263729468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7eb81026d0 0x7f7eb8102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:58.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.857+0000 7f7ebec00640 1 --2- 172.21.15.37:0/3263729468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eb81014e0 0x7f7eb8101900 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:58.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.857+0000 7f7ebec00640 1 -- 172.21.15.37:0/3263729468 >> 172.21.15.37:0/3263729468 conn(0x7f7eb80fca30 msgr2=0x7f7eb80fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:58.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.858+0000 7f7ebec00640 1 -- 172.21.15.37:0/3263729468 shutdown_connections 2024-03-28T18:10:58.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.858+0000 7f7ebec00640 1 -- 172.21.15.37:0/3263729468 wait complete. 2024-03-28T18:10:58.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.859+0000 7f7ebec00640 1 Processor -- start 2024-03-28T18:10:58.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.859+0000 7f7ebec00640 1 -- start start 2024-03-28T18:10:58.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.860+0000 7f7ebec00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7eb81014e0 0x7f7eb8199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:58.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.860+0000 7f7ebec00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7eb81026d0 0x7f7eb8199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:58.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.860+0000 7f7ebec00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eb8103090 0x7f7eb81a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:58.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.860+0000 7f7ebec00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb810d190 con 0x7f7eb81014e0 2024-03-28T18:10:58.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.860+0000 7f7ebec00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb810d010 con 0x7f7eb8103090 2024-03-28T18:10:58.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.860+0000 7f7ebec00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb810d310 con 0x7f7eb81026d0 2024-03-28T18:10:58.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.860+0000 7f7ebd800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7eb81014e0 0x7f7eb8199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:58.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.860+0000 7f7ebd800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7eb81014e0 0x7f7eb8199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:32812/0 (socket says 172.21.15.37:32812) 2024-03-28T18:10:58.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.860+0000 7f7ebd800640 1 -- 172.21.15.37:0/2010747206 learned_addr learned my addr 172.21.15.37:0/2010747206 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:58.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.860+0000 7f7ebce00640 1 --2- 172.21.15.37:0/2010747206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7eb81026d0 0x7f7eb8199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:58.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.860+0000 7f7ebe200640 1 --2- 172.21.15.37:0/2010747206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eb8103090 0x7f7eb81a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:58.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.861+0000 7f7ebd800640 1 -- 172.21.15.37:0/2010747206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7eb81026d0 msgr2=0x7f7eb8199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:58.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.861+0000 7f7ebd800640 1 --2- 172.21.15.37:0/2010747206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7eb81026d0 0x7f7eb8199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:58.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.861+0000 7f7ebd800640 1 -- 172.21.15.37:0/2010747206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eb8103090 msgr2=0x7f7eb81a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:58.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.861+0000 7f7ebd800640 1 --2- 172.21.15.37:0/2010747206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eb8103090 0x7f7eb81a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:58.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.861+0000 7f7ebd800640 1 -- 172.21.15.37:0/2010747206 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb81a1670 con 0x7f7eb81014e0 2024-03-28T18:10:58.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.861+0000 7f7ebe200640 1 --2- 172.21.15.37:0/2010747206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eb8103090 0x7f7eb81a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:58.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.861+0000 7f7ebce00640 1 --2- 172.21.15.37:0/2010747206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7eb81026d0 0x7f7eb8199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:58.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.861+0000 7f7ebd800640 1 --2- 172.21.15.37:0/2010747206 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7eb81014e0 0x7f7eb8199aa0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7eac000c00 tx=0x7f7eac02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:58.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.861+0000 7f7eb6000640 1 -- 172.21.15.37:0/2010747206 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac043020 con 0x7f7eb81014e0 2024-03-28T18:10:58.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.861+0000 7f7ebec00640 1 -- 172.21.15.37:0/2010747206 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb81a1900 con 0x7f7eb81014e0 2024-03-28T18:10:58.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.861+0000 7f7eb6000640 1 -- 172.21.15.37:0/2010747206 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7eac0054e0 con 0x7f7eb81014e0 2024-03-28T18:10:58.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.861+0000 7f7eb6000640 1 -- 172.21.15.37:0/2010747206 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac003c40 con 0x7f7eb81014e0 2024-03-28T18:10:58.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.861+0000 7f7ebec00640 1 -- 172.21.15.37:0/2010747206 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb81a1ea0 con 0x7f7eb81014e0 2024-03-28T18:10:58.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.863+0000 7f7ebec00640 1 -- 172.21.15.37:0/2010747206 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb81a2070 con 0x7f7eb81014e0 2024-03-28T18:10:58.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.864+0000 7f7eb6000640 1 -- 172.21.15.37:0/2010747206 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7eac046020 con 0x7f7eb81014e0 2024-03-28T18:10:58.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.866+0000 7f7eb6000640 1 --2- 172.21.15.37:0/2010747206 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f7ea00a2800 0x7f7ea00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:58.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.866+0000 7f7eb6000640 1 -- 172.21.15.37:0/2010747206 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7eac0f0a50 con 0x7f7eb81014e0 2024-03-28T18:10:58.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.866+0000 7f7ebce00640 1 --2- 172.21.15.37:0/2010747206 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f7ea00a2800 0x7f7ea00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:58.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.867+0000 7f7ebce00640 1 --2- 172.21.15.37:0/2010747206 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f7ea00a2800 0x7f7ea00a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7eb819af60 tx=0x7f7ea4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:58.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:58.873+0000 7f7eb6000640 1 -- 172.21.15.37:0/2010747206 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7eac04d020 con 0x7f7eb81014e0 2024-03-28T18:10:58.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:58 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1069014125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:58.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:58 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/173304552' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:59.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:58 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1069014125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:58 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/173304552' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:59.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:58 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1069014125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:10:59.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:58 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/173304552' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:10:59.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.148+0000 7f7ebec00640 1 -- 172.21.15.37:0/2010747206 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7eb81a26b0 con 0x7f7eb81014e0 2024-03-28T18:10:59.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.150+0000 7f7eb6000640 1 -- 172.21.15.37:0/2010747206 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22575 (secure 0 0 0) 0x7f7eac0bdaa0 con 0x7f7eb81014e0 2024-03-28T18:10:59.150 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:10:59.150 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":87,"fsid":"b5ad257c-ed2c-11ee-95d0-87774f69a715","created":"2024-03-28T17:59:05.949847+0000","modified":"2024-03-28T18:09:15.709901+0000","last_up_change":"2024-03-28T18:07:21.021030+0000","last_in_change":"2024-03-28T18:07:01.416192+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:03:00.517967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:08:18.365827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:08:19.446282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db60618d-56f7-4130-8a90-84cbc480965b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6803","nonce":3562602035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6805","nonce":3562602035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6809","nonce":3562602035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3562602035},{"type":"v1","addr":"172.21.15.37:6807","nonce":3562602035}]},"public_addr":"172.21.15.37:6803/3562602035","cluster_addr":"172.21.15.37:6805/3562602035","heartbeat_back_addr":"172.21.15.37:6809/3562602035","heartbeat_front_addr":"172.21.15.37:6807/3562602035","state":["exists","up"]},{"osd":1,"uuid":"7a4b7ad6-905e-4258-9574-693d4277dfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6801","nonce":3326537211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6803","nonce":3326537211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6807","nonce":3326537211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3326537211},{"type":"v1","addr":"172.21.15.59:6805","nonce":3326537211}]},"public_addr":"172.21.15.59:6801/3326537211","cluster_addr":"172.21.15.59:6803/3326537211","heartbeat_back_addr":"172.21.15.59:6807/3326537211","heartbeat_front_addr":"172.21.15.59:6805/3326537211","state":["exists","up"]},{"osd":2,"uuid":"8b76ec1c-46bf-4336-89b5-12d82906e8b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6801","nonce":1269398427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6803","nonce":1269398427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6807","nonce":1269398427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1269398427},{"type":"v1","addr":"172.21.15.170:6805","nonce":1269398427}]},"public_addr":"172.21.15.170:6801/1269398427","cluster_addr":"172.21.15.170:6803/1269398427","heartbeat_back_addr":"172.21.15.170:6807/1269398427","heartbeat_front_addr":"172.21.15.170:6805/1269398427","state":["exists","up"]},{"osd":3,"uuid":"66eca1da-2fbd-44e1-8bbf-5635a2c70ad5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6811","nonce":317805240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6813","nonce":317805240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6817","nonce":317805240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":317805240},{"type":"v1","addr":"172.21.15.37:6815","nonce":317805240}]},"public_addr":"172.21.15.37:6811/317805240","cluster_addr":"172.21.15.37:6813/317805240","heartbeat_back_addr":"172.21.15.37:6817/317805240","heartbeat_front_addr":"172.21.15.37:6815/317805240","state":["exists","up"]},{"osd":4,"uuid":"4c877335-e42d-4d8e-9f48-a1d0a89ae438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6809","nonce":2410632623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6811","nonce":2410632623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6815","nonce":2410632623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":2410632623},{"type":"v1","addr":"172.21.15.59:6813","nonce":2410632623}]},"public_addr":"172.21.15.59:6809/2410632623","cluster_addr":"172.21.15.59:6811/2410632623","heartbeat_back_addr":"172.21.15.59:6815/2410632623","heartbeat_front_addr":"172.21.15.59:6813/2410632623","state":["exists","up"]},{"osd":5,"uuid":"b884842a-86e2-455b-9570-ff163b3bc0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6809","nonce":1218943899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6811","nonce":1218943899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6815","nonce":1218943899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1218943899},{"type":"v1","addr":"172.21.15.170:6813","nonce":1218943899}]},"public_addr":"172.21.15.170:6809/1218943899","cluster_addr":"172.21.15.170:6811/1218943899","heartbeat_back_addr":"172.21.15.170:6815/1218943899","heartbeat_front_addr":"172.21.15.170:6813/1218943899","state":["exists","up"]},{"osd":6,"uuid":"249a1200-77f4-44ad-981f-1d352f732b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6819","nonce":1899652046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6821","nonce":1899652046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6825","nonce":1899652046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":1899652046},{"type":"v1","addr":"172.21.15.37:6823","nonce":1899652046}]},"public_addr":"172.21.15.37:6819/1899652046","cluster_addr":"172.21.15.37:6821/1899652046","heartbeat_back_addr":"172.21.15.37:6825/1899652046","heartbeat_front_addr":"172.21.15.37:6823/1899652046","state":["exists","up"]},{"osd":7,"uuid":"c73338e6-f505-4517-a84b-f35e04750a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6817","nonce":1343675690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6819","nonce":1343675690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6823","nonce":1343675690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1343675690},{"type":"v1","addr":"172.21.15.59:6821","nonce":1343675690}]},"public_addr":"172.21.15.59:6817/1343675690","cluster_addr":"172.21.15.59:6819/1343675690","heartbeat_back_addr":"172.21.15.59:6823/1343675690","heartbeat_front_addr":"172.21.15.59:6821/1343675690","state":["exists","up"]},{"osd":8,"uuid":"149b8297-ff2d-4a88-9b1e-0def021cc516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6817","nonce":3235794556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6819","nonce":3235794556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6823","nonce":3235794556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3235794556},{"type":"v1","addr":"172.21.15.170:6821","nonce":3235794556}]},"public_addr":"172.21.15.170:6817/3235794556","cluster_addr":"172.21.15.170:6819/3235794556","heartbeat_back_addr":"172.21.15.170:6823/3235794556","heartbeat_front_addr":"172.21.15.170:6821/3235794556","state":["exists","up"]},{"osd":9,"uuid":"55c27cc2-c771-44b7-90fe-8f0feca44162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6827","nonce":2307153008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6829","nonce":2307153008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6833","nonce":2307153008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":2307153008},{"type":"v1","addr":"172.21.15.37:6831","nonce":2307153008}]},"public_addr":"172.21.15.37:6827/2307153008","cluster_addr":"172.21.15.37:6829/2307153008","heartbeat_back_addr":"172.21.15.37:6833/2307153008","heartbeat_front_addr":"172.21.15.37:6831/2307153008","state":["exists","up"]},{"osd":10,"uuid":"c0595135-ab81-49d3-8c23-19f960c03e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6825","nonce":2113830275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6827","nonce":2113830275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6831","nonce":2113830275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2113830275},{"type":"v1","addr":"172.21.15.59:6829","nonce":2113830275}]},"public_addr":"172.21.15.59:6825/2113830275","cluster_addr":"172.21.15.59:6827/2113830275","heartbeat_back_addr":"172.21.15.59:6831/2113830275","heartbeat_front_addr":"172.21.15.59:6829/2113830275","state":["exists","up"]},{"osd":11,"uuid":"5871e177-c17b-4637-a797-4fa5125920dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6825","nonce":3082439360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6827","nonce":3082439360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6831","nonce":3082439360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3082439360},{"type":"v1","addr":"172.21.15.170:6829","nonce":3082439360}]},"public_addr":"172.21.15.170:6825/3082439360","cluster_addr":"172.21.15.170:6827/3082439360","heartbeat_back_addr":"172.21.15.170:6831/3082439360","heartbeat_front_addr":"172.21.15.170:6829/3082439360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:07.690545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:33.183396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:02:57.400263+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:30.597211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:03:57.259044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:22.627823+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:04:57.585863+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:25.000476+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:05:50.876688+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:25.859761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:06:53.563146+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:07:19.304115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3023379428":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1443710231":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6800/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6801/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:6801/1052839239":"2024-03-29T17:59:39.162038+0000","172.21.15.37:6800/3459498509":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/1007625490":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/2895698584":"2024-03-29T17:59:39.162038+0000","172.21.15.37:0/497313034":"2024-03-29T18:00:13.809417+0000","172.21.15.37:0/3293943475":"2024-03-29T17:59:39.162038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:10:59.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.152+0000 7f7ebec00640 1 -- 172.21.15.37:0/2010747206 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f7ea00a2800 msgr2=0x7f7ea00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.152+0000 7f7ebec00640 1 --2- 172.21.15.37:0/2010747206 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f7ea00a2800 0x7f7ea00a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7eb819af60 tx=0x7f7ea4005f90 comp rx=0 tx=0).stop 2024-03-28T18:10:59.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.152+0000 7f7ebec00640 1 -- 172.21.15.37:0/2010747206 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7eb81014e0 msgr2=0x7f7eb8199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.152+0000 7f7ebec00640 1 --2- 172.21.15.37:0/2010747206 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7eb81014e0 0x7f7eb8199aa0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7eac000c00 tx=0x7f7eac02c650 comp rx=0 tx=0).stop 2024-03-28T18:10:59.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.152+0000 7f7ebec00640 1 -- 172.21.15.37:0/2010747206 shutdown_connections 2024-03-28T18:10:59.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.152+0000 7f7ebec00640 1 --2- 172.21.15.37:0/2010747206 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f7ea00a2800 0x7f7ea00a4cc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.152+0000 7f7ebec00640 1 --2- 172.21.15.37:0/2010747206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eb8103090 0x7f7eb81a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.152+0000 7f7ebec00640 1 --2- 172.21.15.37:0/2010747206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7eb81026d0 0x7f7eb8199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.152+0000 7f7ebec00640 1 --2- 172.21.15.37:0/2010747206 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7eb81014e0 0x7f7eb8199aa0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.153+0000 7f7ebec00640 1 -- 172.21.15.37:0/2010747206 >> 172.21.15.37:0/2010747206 conn(0x7f7eb80fca30 msgr2=0x7f7eb80738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:59.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.153+0000 7f7ebec00640 1 -- 172.21.15.37:0/2010747206 shutdown_connections 2024-03-28T18:10:59.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.153+0000 7f7ebec00640 1 -- 172.21.15.37:0/2010747206 wait complete. 2024-03-28T18:10:59.171 DEBUG:tasks.fwd_scrub:Yielding 2024-03-28T18:10:59.172 INFO:teuthology.run_tasks:Running task exec... 2024-03-28T18:10:59.181 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T18:10:59.181 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi037.front.sepia.ceph.com 2024-03-28T18:10:59.181 DEBUG:teuthology.orchestra.run.smithi037:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-28T18:10:59.182 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-28T18:10:59.182 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:10:59.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:10:59.182 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:10:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.371+0000 7efea8a00640 1 -- 172.21.15.37:0/3388927827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efea4104ae0 msgr2=0x7efea4104f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.371+0000 7efea8a00640 1 --2- 172.21.15.37:0/3388927827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efea4104ae0 0x7efea4104f60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efe90009900 tx=0x7efe9002bc40 comp rx=0 tx=0).stop 2024-03-28T18:10:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.371+0000 7efea8a00640 1 -- 172.21.15.37:0/3388927827 shutdown_connections 2024-03-28T18:10:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.371+0000 7efea8a00640 1 --2- 172.21.15.37:0/3388927827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efea4104ae0 0x7efea4104f60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.371+0000 7efea8a00640 1 --2- 172.21.15.37:0/3388927827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efea41054a0 0x7efea410bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.371+0000 7efea8a00640 1 --2- 172.21.15.37:0/3388927827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efea41038f0 0x7efea4103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.371+0000 7efea8a00640 1 -- 172.21.15.37:0/3388927827 >> 172.21.15.37:0/3388927827 conn(0x7efea40fee60 msgr2=0x7efea41012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.371+0000 7efea8a00640 1 -- 172.21.15.37:0/3388927827 shutdown_connections 2024-03-28T18:10:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.371+0000 7efea8a00640 1 -- 172.21.15.37:0/3388927827 wait complete. 2024-03-28T18:10:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.373+0000 7efea8a00640 1 Processor -- start 2024-03-28T18:10:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.373+0000 7efea8a00640 1 -- start start 2024-03-28T18:10:59.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.373+0000 7efea8a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efea41038f0 0x7efea4199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.373+0000 7efea8a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efea4104ae0 0x7efea419a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.373+0000 7efea8a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efea41054a0 0x7efea41a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.373+0000 7efea8a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea410f5a0 con 0x7efea41054a0 2024-03-28T18:10:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.373+0000 7efea8a00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea410f420 con 0x7efea41038f0 2024-03-28T18:10:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.373+0000 7efea8a00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea410f720 con 0x7efea4104ae0 2024-03-28T18:10:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.373+0000 7efea1e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efea41038f0 0x7efea4199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.373+0000 7efea1e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efea41038f0 0x7efea4199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:37032/0 (socket says 172.21.15.37:37032) 2024-03-28T18:10:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.373+0000 7efea1400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efea4104ae0 0x7efea419a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.374+0000 7efea1e00640 1 -- 172.21.15.37:0/701822511 learned_addr learned my addr 172.21.15.37:0/701822511 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.374+0000 7efea2800640 1 --2- 172.21.15.37:0/701822511 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efea41054a0 0x7efea41a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.374+0000 7efea1e00640 1 -- 172.21.15.37:0/701822511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efea4104ae0 msgr2=0x7efea419a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.374+0000 7efea1e00640 1 --2- 172.21.15.37:0/701822511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efea4104ae0 0x7efea419a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.374+0000 7efea1e00640 1 -- 172.21.15.37:0/701822511 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efea41054a0 msgr2=0x7efea41a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.374+0000 7efea1e00640 1 --2- 172.21.15.37:0/701822511 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efea41054a0 0x7efea41a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.374+0000 7efea1e00640 1 -- 172.21.15.37:0/701822511 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea41a1750 con 0x7efea41038f0 2024-03-28T18:10:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.374+0000 7efea1e00640 1 --2- 172.21.15.37:0/701822511 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efea41038f0 0x7efea4199b30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7efe9400e9c0 tx=0x7efe9400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.374+0000 7efe9aa00640 1 -- 172.21.15.37:0/701822511 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9400cd80 con 0x7efea41038f0 2024-03-28T18:10:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.374+0000 7efe9aa00640 1 -- 172.21.15.37:0/701822511 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe940035f0 con 0x7efea41038f0 2024-03-28T18:10:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.374+0000 7efe9aa00640 1 -- 172.21.15.37:0/701822511 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe94010690 con 0x7efea41038f0 2024-03-28T18:10:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.374+0000 7efea8a00640 1 -- 172.21.15.37:0/701822511 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea41a1a40 con 0x7efea41038f0 2024-03-28T18:10:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.375+0000 7efea8a00640 1 -- 172.21.15.37:0/701822511 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea41a1fb0 con 0x7efea41038f0 2024-03-28T18:10:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.376+0000 7efea8a00640 1 -- 172.21.15.37:0/701822511 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea410ab40 con 0x7efea41038f0 2024-03-28T18:10:59.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.377+0000 7efe9aa00640 1 -- 172.21.15.37:0/701822511 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7efe940037a0 con 0x7efea41038f0 2024-03-28T18:10:59.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.379+0000 7efe9aa00640 1 --2- 172.21.15.37:0/701822511 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7efe880a2920 0x7efe880a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.379+0000 7efe9aa00640 1 -- 172.21.15.37:0/701822511 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7efe940cee90 con 0x7efea41038f0 2024-03-28T18:10:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.379+0000 7efea1400640 1 --2- 172.21.15.37:0/701822511 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7efe880a2920 0x7efe880a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.380+0000 7efea1400640 1 --2- 172.21.15.37:0/701822511 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7efe880a2920 0x7efe880a4de0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7efe9002c250 tx=0x7efe9002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:59.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.383+0000 7efe9aa00640 1 -- 172.21.15.37:0/701822511 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efe9409bee0 con 0x7efea41038f0 2024-03-28T18:10:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.387+0000 7f3847600640 1 -- 172.21.15.37:0/1811201749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3840105490 msgr2=0x7f384010bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.387+0000 7f3847600640 1 --2- 172.21.15.37:0/1811201749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3840105490 0x7f384010bd20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3838009be0 tx=0x7f383802bef0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.387+0000 7f3847600640 1 -- 172.21.15.37:0/1811201749 shutdown_connections 2024-03-28T18:10:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.387+0000 7f3847600640 1 --2- 172.21.15.37:0/1811201749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3840105490 0x7f384010bd20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.387+0000 7f3847600640 1 --2- 172.21.15.37:0/1811201749 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3840104ad0 0x7f3840104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.387+0000 7f3847600640 1 --2- 172.21.15.37:0/1811201749 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38401038e0 0x7f3840103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.387+0000 7f3847600640 1 -- 172.21.15.37:0/1811201749 >> 172.21.15.37:0/1811201749 conn(0x7f38400fee50 msgr2=0x7f3840101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.387+0000 7f3847600640 1 -- 172.21.15.37:0/1811201749 shutdown_connections 2024-03-28T18:10:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.387+0000 7f3847600640 1 -- 172.21.15.37:0/1811201749 wait complete. 2024-03-28T18:10:59.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.388+0000 7f3847600640 1 Processor -- start 2024-03-28T18:10:59.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.388+0000 7f3847600640 1 -- start start 2024-03-28T18:10:59.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.388+0000 7f3847600640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38401038e0 0x7f3840199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.389+0000 7f3847600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3840104ad0 0x7f384019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.389+0000 7f3847600640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3840105490 0x7f38401a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.389+0000 7f3847600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384010f590 con 0x7f3840104ad0 2024-03-28T18:10:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.389+0000 7f3847600640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384010f410 con 0x7f38401038e0 2024-03-28T18:10:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.389+0000 7f3847600640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384010f710 con 0x7f3840105490 2024-03-28T18:10:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.389+0000 7f3844c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38401038e0 0x7f3840199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.389+0000 7f3844c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38401038e0 0x7f3840199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:37050/0 (socket says 172.21.15.37:37050) 2024-03-28T18:10:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.389+0000 7f3845600640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3840105490 0x7f38401a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.389+0000 7f3844c00640 1 -- 172.21.15.37:0/3917258091 learned_addr learned my addr 172.21.15.37:0/3917258091 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.389+0000 7f383fe00640 1 --2- 172.21.15.37:0/3917258091 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3840104ad0 0x7f384019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.389+0000 7f3844c00640 1 -- 172.21.15.37:0/3917258091 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3840105490 msgr2=0x7f38401a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.389+0000 7f3844c00640 1 --2- 172.21.15.37:0/3917258091 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3840105490 0x7f38401a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.389+0000 7f3844c00640 1 -- 172.21.15.37:0/3917258091 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3840104ad0 msgr2=0x7f384019a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.389+0000 7f3844c00640 1 --2- 172.21.15.37:0/3917258091 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3840104ad0 0x7f384019a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.389+0000 7f3844c00640 1 -- 172.21.15.37:0/3917258091 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38401a1830 con 0x7f38401038e0 2024-03-28T18:10:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.389+0000 7f383fe00640 1 --2- 172.21.15.37:0/3917258091 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3840104ad0 0x7f384019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.389+0000 7f3845600640 1 --2- 172.21.15.37:0/3917258091 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3840105490 0x7f38401a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.390+0000 7f3844c00640 1 --2- 172.21.15.37:0/3917258091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38401038e0 0x7f3840199c20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f383400e9a0 tx=0x7f383400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.390+0000 7f383d600640 1 -- 172.21.15.37:0/3917258091 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383400cd80 con 0x7f38401038e0 2024-03-28T18:10:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.390+0000 7f383d600640 1 -- 172.21.15.37:0/3917258091 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3834010910 con 0x7f38401038e0 2024-03-28T18:10:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.390+0000 7f3847600640 1 -- 172.21.15.37:0/3917258091 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38401a1b20 con 0x7f38401038e0 2024-03-28T18:10:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.390+0000 7f383d600640 1 -- 172.21.15.37:0/3917258091 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834018a30 con 0x7f38401038e0 2024-03-28T18:10:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.390+0000 7f3847600640 1 -- 172.21.15.37:0/3917258091 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38401a20c0 con 0x7f38401038e0 2024-03-28T18:10:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.395+0000 7f384a20f740 1 Processor -- start 2024-03-28T18:10:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.395+0000 7f384a20f740 1 -- start start 2024-03-28T18:10:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.395+0000 7f383d600640 1 -- 172.21.15.37:0/3917258091 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3834010ac0 con 0x7f38401038e0 2024-03-28T18:10:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.395+0000 7f384a20f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b930a0 0x564be0b934c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.395+0000 7f384a20f740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b93a90 0x564be0b98590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.395+0000 7f384a20f740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 0x564be0b9b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.395+0000 7f384a20f740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x564be0af8eb0 con 0x564be0b93a90 2024-03-28T18:10:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.395+0000 7f384a20f740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x564be0aa1a60 con 0x564be0b98c00 2024-03-28T18:10:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.395+0000 7f384a20f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564be0af4f50 con 0x564be0b930a0 2024-03-28T18:10:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.395+0000 7f3845600640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b930a0 0x564be0b934c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.396+0000 7f3844c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b93a90 0x564be0b98590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.396+0000 7f3845600640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b930a0 0x564be0b934c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55998/0 (socket says 172.21.15.37:55998) 2024-03-28T18:10:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.396+0000 7f3845600640 1 -- 172.21.15.37:0/831632499 learned_addr learned my addr 172.21.15.37:0/831632499 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.396+0000 7f383fe00640 1 --2- 172.21.15.37:0/831632499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 0x564be0b9b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.396+0000 7f3845600640 1 -- 172.21.15.37:0/831632499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 msgr2=0x564be0b9b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.396+0000 7f3845600640 1 --2- 172.21.15.37:0/831632499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 0x564be0b9b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.396+0000 7f3845600640 1 -- 172.21.15.37:0/831632499 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b93a90 msgr2=0x564be0b98590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.396+0000 7f3845600640 1 --2- 172.21.15.37:0/831632499 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b93a90 0x564be0b98590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.396+0000 7f3845600640 1 -- 172.21.15.37:0/831632499 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564be08d57e0 con 0x564be0b930a0 2024-03-28T18:10:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.396+0000 7f383fe00640 1 --2- 172.21.15.37:0/831632499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 0x564be0b9b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:10:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.396+0000 7f3845600640 1 --2- 172.21.15.37:0/831632499 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b930a0 0x564be0b934c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f383802c500 tx=0x7f3838033ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3ae8161fca9f5f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.397+0000 7f3831c00640 1 -- 172.21.15.37:0/831632499 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838043030 con 0x564be0b930a0 2024-03-28T18:10:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.397+0000 7f3831c00640 1 -- 172.21.15.37:0/831632499 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3838003970 con 0x564be0b930a0 2024-03-28T18:10:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.397+0000 7f3831c00640 1 -- 172.21.15.37:0/831632499 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383802cd30 con 0x564be0b930a0 2024-03-28T18:10:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.398+0000 7f383d600640 1 --2- 172.21.15.37:0/3917258091 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f38280a2880 0x7f38280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.398+0000 7f384a20f740 1 -- 172.21.15.37:0/831632499 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b930a0 msgr2=0x564be0b934c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.398+0000 7f384a20f740 1 --2- 172.21.15.37:0/831632499 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b930a0 0x564be0b934c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f383802c500 tx=0x7f3838033ef0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.398+0000 7f383d600640 1 -- 172.21.15.37:0/3917258091 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f38340cee70 con 0x7f38401038e0 2024-03-28T18:10:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.398+0000 7f3844c00640 1 --2- 172.21.15.37:0/3917258091 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f38280a2880 0x7f38280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.398+0000 7f384a20f740 1 -- 172.21.15.37:0/831632499 shutdown_connections 2024-03-28T18:10:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.398+0000 7f384a20f740 1 --2- 172.21.15.37:0/831632499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 0x564be0b9b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.398+0000 7f384a20f740 1 --2- 172.21.15.37:0/831632499 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b93a90 0x564be0b98590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.398+0000 7f384a20f740 1 --2- 172.21.15.37:0/831632499 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b930a0 0x564be0b934c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.398+0000 7f384a20f740 1 -- 172.21.15.37:0/831632499 >> 172.21.15.37:0/831632499 conn(0x564be0b8e610 msgr2=0x564be0b8ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.398+0000 7f384a20f740 1 -- 172.21.15.37:0/831632499 shutdown_connections 2024-03-28T18:10:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.398+0000 7f384a20f740 1 -- 172.21.15.37:0/831632499 wait complete. 2024-03-28T18:10:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.399+0000 7f3844c00640 1 --2- 172.21.15.37:0/3917258091 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f38280a2880 0x7f38280a4d40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f38340cccf0 tx=0x7f38340d2950 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.399+0000 7f384a20f740 1 -- start start 2024-03-28T18:10:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.399+0000 7f384a20f740 1 Processor -- start 2024-03-28T18:10:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.400+0000 7f384a20f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b930a0 0x564be0b9d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.400+0000 7f384a20f740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b93a90 0x564be0b9d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.400+0000 7f384a20f740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 0x564be0ba4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.400+0000 7f384a20f740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x564be0b9bb50 con 0x564be0b93a90 2024-03-28T18:10:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.400+0000 7f384a20f740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x564be0ab4eb0 con 0x564be0b98c00 2024-03-28T18:10:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.400+0000 7f384a20f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564be0b9bcd0 con 0x564be0b930a0 2024-03-28T18:10:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.400+0000 7f384a20f740 10 client.0 _ll_register_callbacks cb 0x564be0aa9150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:10:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.400+0000 7f3845600640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b930a0 0x564be0b9d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.400+0000 7f3845600640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b930a0 0x564be0b9d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56006/0 (socket says 172.21.15.37:56006) 2024-03-28T18:10:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.400+0000 7f3845600640 1 -- 172.21.15.37:0/3814534709 learned_addr learned my addr 172.21.15.37:0/3814534709 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.400+0000 7f383fe00640 1 --2- 172.21.15.37:0/3814534709 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b93a90 0x564be0b9d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.400+0000 7f3845600640 1 --2- 172.21.15.37:0/3814534709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 0x564be0ba4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.400+0000 7f3845600640 1 -- 172.21.15.37:0/3814534709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 msgr2=0x564be0ba4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.400+0000 7f3845600640 1 --2- 172.21.15.37:0/3814534709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 0x564be0ba4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.400+0000 7f3845600640 1 -- 172.21.15.37:0/3814534709 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b93a90 msgr2=0x564be0b9d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.400+0000 7f3845600640 1 --2- 172.21.15.37:0/3814534709 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b93a90 0x564be0b9d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.400+0000 7f3845600640 1 -- 172.21.15.37:0/3814534709 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564be0935f70 con 0x564be0b930a0 2024-03-28T18:10:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.401+0000 7f3845600640 1 --2- 172.21.15.37:0/3814534709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b930a0 0x564be0b9d290 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f383802ca40 tx=0x7f383802bef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6efe3156a8bb4099 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.401+0000 7f3826000640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:10:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.401+0000 7f3826000640 1 -- 172.21.15.37:0/3814534709 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838043040 con 0x564be0b930a0 2024-03-28T18:10:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.401+0000 7f3826000640 1 -- 172.21.15.37:0/3814534709 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3838033610 con 0x564be0b930a0 2024-03-28T18:10:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.401+0000 7f384a20f740 1 -- 172.21.15.37:0/3814534709 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3838003970 con 0x564be0b930a0 2024-03-28T18:10:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.401+0000 7f3826000640 1 -- 172.21.15.37:0/3814534709 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3838033810 con 0x564be0b930a0 2024-03-28T18:10:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.401+0000 7f384a20f740 10 client.25348 fetch_fsmap learned FSMap version 23 2024-03-28T18:10:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.401+0000 7f384a20f740 1 -- 172.21.15.37:0/3814534709 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564be07cd170 con 0x564be0b930a0 2024-03-28T18:10:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.402+0000 7f3826000640 1 -- 172.21.15.37:0/3814534709 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3838003970 con 0x564be0b930a0 2024-03-28T18:10:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.402+0000 7f3826000640 1 -- 172.21.15.37:0/3814534709 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f3838034430 con 0x564be0b930a0 2024-03-28T18:10:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.402+0000 7f384a20f740 10 client.25348 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:10:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.402+0000 7f384a20f740 10 client.25348 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:10:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.402+0000 7f384a20f740 20 client.25348 populate_metadata read hostname 'smithi037' 2024-03-28T18:10:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.402+0000 7f384a20f740 1 --2- 172.21.15.37:0/3814534709 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x564be0ba8d00 0x564be0bab0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.402+0000 7f384a20f740 4 client.25348 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:10:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.402+0000 7f384a20f740 1 -- 172.21.15.37:0/3814534709 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564be0b063f0 con 0x564be0ba8d00 2024-03-28T18:10:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.402+0000 7f383fe00640 1 --2- 172.21.15.37:0/3814534709 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x564be0ba8d00 0x564be0bab0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.403+0000 7f383fe00640 1 --2- 172.21.15.37:0/3814534709 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x564be0ba8d00 0x564be0bab0f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f26931d1e6eae93 server_cookie=8f791e8573a381cd in_seq=0 out_seq=0 2024-03-28T18:10:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.403+0000 7f3826000640 10 client.25348 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:10:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.403+0000 7f3826000640 1 -- 172.21.15.37:0/3814534709 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564be0b063f0 con 0x564be0ba8d00 2024-03-28T18:10:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.403+0000 7f3826000640 10 client.25348 handle_command_reply: tid=0 2024-03-28T18:10:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.404+0000 7f384a20f740 1 client.25348 shutdown 2024-03-28T18:10:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.404+0000 7f384a20f740 1 -- 172.21.15.37:0/3814534709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b930a0 msgr2=0x564be0b9d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.404+0000 7f384a20f740 1 --2- 172.21.15.37:0/3814534709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b930a0 0x564be0b9d290 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f383802ca40 tx=0x7f383802bef0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.404+0000 7f384a20f740 1 -- 172.21.15.37:0/3814534709 shutdown_connections 2024-03-28T18:10:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.404+0000 7f384a20f740 1 --2- 172.21.15.37:0/3814534709 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x564be0ba8d00 0x564be0bab0f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.404+0000 7f384a20f740 1 --2- 172.21.15.37:0/3814534709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 0x564be0ba4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.404+0000 7f384a20f740 1 --2- 172.21.15.37:0/3814534709 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b93a90 0x564be0b9d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.404+0000 7f3826000640 0 client.25348 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:10:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.404+0000 7f384a20f740 1 --2- 172.21.15.37:0/3814534709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b930a0 0x564be0b9d290 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.404+0000 7f384a20f740 1 -- 172.21.15.37:0/3814534709 >> 172.21.15.37:0/3814534709 conn(0x564be0b8e610 msgr2=0x564be0b8d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.404+0000 7f384a20f740 1 -- 172.21.15.37:0/3814534709 shutdown_connections 2024-03-28T18:10:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.404+0000 7f384a20f740 1 -- 172.21.15.37:0/3814534709 wait complete. 2024-03-28T18:10:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.404+0000 7f384a20f740 20 client.25348 trim_cache size 0 max 16384 2024-03-28T18:10:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.422+0000 7f384a20f740 1 Processor -- start 2024-03-28T18:10:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.422+0000 7f384a20f740 1 -- start start 2024-03-28T18:10:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.422+0000 7f384a20f740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b930a0 0x564be0b8d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.422+0000 7f384a20f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b93a90 0x564be0b8dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.422+0000 7f384a20f740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 0x564be0ba47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.422+0000 7f384a20f740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38401ad0a0 con 0x564be0b930a0 2024-03-28T18:10:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.422+0000 7f384a20f740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38401aaba0 con 0x564be0b98c00 2024-03-28T18:10:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.422+0000 7f384a20f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38401a8c40 con 0x564be0b93a90 2024-03-28T18:10:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.423+0000 7f3845600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b930a0 0x564be0b8d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.423+0000 7f383fe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b93a90 0x564be0b8dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.423+0000 7f383fe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b93a90 0x564be0b8dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56012/0 (socket says 172.21.15.37:56012) 2024-03-28T18:10:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.423+0000 7f3845600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b930a0 0x564be0b8d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:32892/0 (socket says 172.21.15.37:32892) 2024-03-28T18:10:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.423+0000 7f383fe00640 1 -- 172.21.15.37:0/3152552183 learned_addr learned my addr 172.21.15.37:0/3152552183 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.423+0000 7f3845600640 1 --2- 172.21.15.37:0/3152552183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 0x564be0ba47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.423+0000 7f383fe00640 1 -- 172.21.15.37:0/3152552183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 msgr2=0x564be0ba47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.423+0000 7f383fe00640 1 --2- 172.21.15.37:0/3152552183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 0x564be0ba47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.423+0000 7f383fe00640 1 -- 172.21.15.37:0/3152552183 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b930a0 msgr2=0x564be0b8d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.423+0000 7f383fe00640 1 --2- 172.21.15.37:0/3152552183 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b930a0 0x564be0b8d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.423+0000 7f383fe00640 1 -- 172.21.15.37:0/3152552183 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564be0bab680 con 0x564be0b93a90 2024-03-28T18:10:59.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.423+0000 7f383fe00640 1 --2- 172.21.15.37:0/3152552183 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b93a90 0x564be0b8dbf0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f382c000c00 tx=0x7f382c0097f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f6c4dee012fb2ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:59.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.424+0000 7f3831200640 1 -- 172.21.15.37:0/3152552183 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c024070 con 0x564be0b93a90 2024-03-28T18:10:59.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.424+0000 7f3831200640 1 -- 172.21.15.37:0/3152552183 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f382c01edb0 con 0x564be0b93a90 2024-03-28T18:10:59.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.424+0000 7f3831200640 1 -- 172.21.15.37:0/3152552183 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c0129b0 con 0x564be0b93a90 2024-03-28T18:10:59.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.424+0000 7f384a20f740 1 -- 172.21.15.37:0/3152552183 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b93a90 msgr2=0x564be0b8dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.424+0000 7f384a20f740 1 --2- 172.21.15.37:0/3152552183 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b93a90 0x564be0b8dbf0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f382c000c00 tx=0x7f382c0097f0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.424+0000 7f384a20f740 1 -- 172.21.15.37:0/3152552183 shutdown_connections 2024-03-28T18:10:59.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.424+0000 7f384a20f740 1 --2- 172.21.15.37:0/3152552183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 0x564be0ba47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.424+0000 7f384a20f740 1 --2- 172.21.15.37:0/3152552183 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b93a90 0x564be0b8dbf0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.424+0000 7f384a20f740 1 --2- 172.21.15.37:0/3152552183 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b930a0 0x564be0b8d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.424+0000 7f384a20f740 1 -- 172.21.15.37:0/3152552183 >> 172.21.15.37:0/3152552183 conn(0x564be0b8e610 msgr2=0x564be0b8cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:59.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.425+0000 7f384a20f740 1 -- 172.21.15.37:0/3152552183 shutdown_connections 2024-03-28T18:10:59.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.425+0000 7f384a20f740 1 -- 172.21.15.37:0/3152552183 wait complete. 2024-03-28T18:10:59.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.425+0000 7f384a20f740 1 -- start start 2024-03-28T18:10:59.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.425+0000 7f384a20f740 1 Processor -- start 2024-03-28T18:10:59.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.426+0000 7f384a20f740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b930a0 0x564be0ba3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.426+0000 7f384a20f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b93a90 0x564be0ba36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.426+0000 7f384a20f740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 0x564be0ba3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.426+0000 7f384a20f740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38401a9240 con 0x564be0b930a0 2024-03-28T18:10:59.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.426+0000 7f384a20f740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38401aa540 con 0x564be0b98c00 2024-03-28T18:10:59.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.426+0000 7f384a20f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38401a9880 con 0x564be0b93a90 2024-03-28T18:10:59.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.426+0000 7f384a20f740 10 client.0 _ll_register_callbacks cb 0x7f38340d2820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:10:59.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.426+0000 7f383fe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b93a90 0x564be0ba36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.426+0000 7f3845600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b930a0 0x564be0ba3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.426+0000 7f383fe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b93a90 0x564be0ba36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56024/0 (socket says 172.21.15.37:56024) 2024-03-28T18:10:59.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.426+0000 7f383fe00640 1 -- 172.21.15.37:0/3662650159 learned_addr learned my addr 172.21.15.37:0/3662650159 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:59.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.426+0000 7f3845600640 1 --2- 172.21.15.37:0/3662650159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 0x564be0ba3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.426+0000 7f383fe00640 1 -- 172.21.15.37:0/3662650159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 msgr2=0x564be0ba3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:59.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.426+0000 7f383fe00640 1 --2- 172.21.15.37:0/3662650159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 0x564be0ba3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.426+0000 7f383fe00640 1 -- 172.21.15.37:0/3662650159 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b930a0 msgr2=0x564be0ba3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:59.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.426+0000 7f383fe00640 1 --2- 172.21.15.37:0/3662650159 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b930a0 0x564be0ba3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.426+0000 7f383fe00640 1 -- 172.21.15.37:0/3662650159 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564be0c045d0 con 0x564be0b93a90 2024-03-28T18:10:59.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.427+0000 7f383fe00640 1 --2- 172.21.15.37:0/3662650159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b93a90 0x564be0ba36a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f382c009d30 tx=0x7f382c012520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c741f02056f4b01 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:59.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.427+0000 7f3826000640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:10:59.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.427+0000 7f3826000640 1 -- 172.21.15.37:0/3662650159 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c02f060 con 0x564be0b93a90 2024-03-28T18:10:59.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.427+0000 7f384a20f740 1 -- 172.21.15.37:0/3662650159 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f382c01edb0 con 0x564be0b93a90 2024-03-28T18:10:59.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.427+0000 7f3826000640 1 -- 172.21.15.37:0/3662650159 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f382c005440 con 0x564be0b93a90 2024-03-28T18:10:59.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.427+0000 7f3826000640 1 -- 172.21.15.37:0/3662650159 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f382c024030 con 0x564be0b93a90 2024-03-28T18:10:59.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.427+0000 7f384a20f740 10 client.25354 fetch_fsmap learned FSMap version 23 2024-03-28T18:10:59.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.427+0000 7f384a20f740 1 -- 172.21.15.37:0/3662650159 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564be0c05980 con 0x564be0b93a90 2024-03-28T18:10:59.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.427+0000 7f3826000640 1 -- 172.21.15.37:0/3662650159 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f382c01edb0 con 0x564be0b93a90 2024-03-28T18:10:59.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.428+0000 7f3826000640 1 -- 172.21.15.37:0/3662650159 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f382c0048c0 con 0x564be0b93a90 2024-03-28T18:10:59.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.428+0000 7f384a20f740 10 client.25354 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:10:59.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.428+0000 7f384a20f740 10 client.25354 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:10:59.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.428+0000 7f384a20f740 20 client.25354 populate_metadata read hostname 'smithi037' 2024-03-28T18:10:59.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.428+0000 7f384a20f740 1 --2- 172.21.15.37:0/3662650159 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x564be0c06270 0x564be0c08690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.428+0000 7f384a20f740 4 client.25354 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:10:59.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.428+0000 7f384a20f740 1 -- 172.21.15.37:0/3662650159 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x564be0b00740 con 0x564be0c06270 2024-03-28T18:10:59.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.428+0000 7f3845600640 1 --2- 172.21.15.37:0/3662650159 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x564be0c06270 0x564be0c08690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.429+0000 7f3845600640 1 --2- 172.21.15.37:0/3662650159 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x564be0c06270 0x564be0c08690 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef29cf86874b97d6 server_cookie=de86a5e939d5e6be in_seq=0 out_seq=0 2024-03-28T18:10:59.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.429+0000 7f3826000640 10 client.25354 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:10:59.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.431+0000 7f3826000640 1 -- 172.21.15.37:0/3662650159 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x564be0b00740 con 0x564be0c06270 2024-03-28T18:10:59.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.431+0000 7f3826000640 10 client.25354 handle_command_reply: tid=0 2024-03-28T18:10:59.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.431+0000 7f384a20f740 1 client.25354 shutdown 2024-03-28T18:10:59.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.431+0000 7f384a20f740 1 -- 172.21.15.37:0/3662650159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b93a90 msgr2=0x564be0ba36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.431+0000 7f384a20f740 1 --2- 172.21.15.37:0/3662650159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b93a90 0x564be0ba36a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f382c009d30 tx=0x7f382c012520 comp rx=0 tx=0).stop 2024-03-28T18:10:59.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.431+0000 7f384a20f740 1 -- 172.21.15.37:0/3662650159 shutdown_connections 2024-03-28T18:10:59.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.431+0000 7f384a20f740 1 --2- 172.21.15.37:0/3662650159 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x564be0c06270 0x564be0c08690 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.431+0000 7f384a20f740 1 --2- 172.21.15.37:0/3662650159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564be0b98c00 0x564be0ba3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.431+0000 7f384a20f740 1 --2- 172.21.15.37:0/3662650159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564be0b93a90 0x564be0ba36a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.431+0000 7f384a20f740 1 --2- 172.21.15.37:0/3662650159 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564be0b930a0 0x564be0ba3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.431+0000 7f384a20f740 1 -- 172.21.15.37:0/3662650159 >> 172.21.15.37:0/3662650159 conn(0x564be0b8e610 msgr2=0x564be0b8d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:59.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.431+0000 7f3826000640 0 client.25354 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:10:59.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.431+0000 7f384a20f740 1 -- 172.21.15.37:0/3662650159 shutdown_connections 2024-03-28T18:10:59.433 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:10:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "return_code": 0, 2024-03-28T18:10:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "scrub_tag": "e5d6d978-9bb9-47bc-bc78-951bb7802412", 2024-03-28T18:10:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "mode": "asynchronous" 2024-03-28T18:10:59.434 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:10:59.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.432+0000 7f384a20f740 1 -- 172.21.15.37:0/3662650159 wait complete. 2024-03-28T18:10:59.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.432+0000 7f384a20f740 20 client.25354 trim_cache size 0 max 16384 2024-03-28T18:10:59.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.432+0000 7f3847600640 1 -- 172.21.15.37:0/3917258091 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f38280a2880 msgr2=0x7f38280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.432+0000 7f3847600640 1 --2- 172.21.15.37:0/3917258091 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f38280a2880 0x7f38280a4d40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f38340cccf0 tx=0x7f38340d2950 comp rx=0 tx=0).stop 2024-03-28T18:10:59.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.432+0000 7f3847600640 1 -- 172.21.15.37:0/3917258091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38401038e0 msgr2=0x7f3840199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.432+0000 7f3847600640 1 --2- 172.21.15.37:0/3917258091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38401038e0 0x7f3840199c20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f383400e9a0 tx=0x7f383400ee60 comp rx=0 tx=0).stop 2024-03-28T18:10:59.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.433+0000 7f3847600640 1 -- 172.21.15.37:0/3917258091 shutdown_connections 2024-03-28T18:10:59.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.433+0000 7f3847600640 1 --2- 172.21.15.37:0/3917258091 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f38280a2880 0x7f38280a4d40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.433+0000 7f3847600640 1 --2- 172.21.15.37:0/3917258091 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3840105490 0x7f38401a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.433+0000 7f3847600640 1 --2- 172.21.15.37:0/3917258091 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3840104ad0 0x7f384019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.433+0000 7f3847600640 1 --2- 172.21.15.37:0/3917258091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38401038e0 0x7f3840199c20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.433+0000 7f3847600640 1 -- 172.21.15.37:0/3917258091 >> 172.21.15.37:0/3917258091 conn(0x7f38400fee50 msgr2=0x7f3840107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:59.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.433+0000 7f3847600640 1 -- 172.21.15.37:0/3917258091 shutdown_connections 2024-03-28T18:10:59.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.433+0000 7f3847600640 1 -- 172.21.15.37:0/3917258091 wait complete. 2024-03-28T18:10:59.449 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:10:59.449 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:10:59.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.640+0000 7ff947000640 1 -- 172.21.15.37:0/954206290 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9401038c0 msgr2=0x7ff940103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.640+0000 7ff947000640 1 --2- 172.21.15.37:0/954206290 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9401038c0 0x7ff940103ce0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff930009900 tx=0x7ff93002bc30 comp rx=0 tx=0).stop 2024-03-28T18:10:59.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.640+0000 7ff947000640 1 -- 172.21.15.37:0/954206290 shutdown_connections 2024-03-28T18:10:59.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.640+0000 7ff947000640 1 --2- 172.21.15.37:0/954206290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff940105470 0x7ff94010bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.640+0000 7ff947000640 1 --2- 172.21.15.37:0/954206290 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff940104ab0 0x7ff940104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.640+0000 7ff947000640 1 --2- 172.21.15.37:0/954206290 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9401038c0 0x7ff940103ce0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.640+0000 7ff947000640 1 -- 172.21.15.37:0/954206290 >> 172.21.15.37:0/954206290 conn(0x7ff9400fee50 msgr2=0x7ff940101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:59.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.640+0000 7ff947000640 1 -- 172.21.15.37:0/954206290 shutdown_connections 2024-03-28T18:10:59.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.640+0000 7ff947000640 1 -- 172.21.15.37:0/954206290 wait complete. 2024-03-28T18:10:59.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.641+0000 7ff947000640 1 Processor -- start 2024-03-28T18:10:59.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.642+0000 7ff947000640 1 -- start start 2024-03-28T18:10:59.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.642+0000 7ff947000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9401038c0 0x7ff940199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.642+0000 7ff947000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff940104ab0 0x7ff94019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.642+0000 7ff947000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff940105470 0x7ff9401a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.642+0000 7ff947000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94010f590 con 0x7ff9401038c0 2024-03-28T18:10:59.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.642+0000 7ff947000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94010f410 con 0x7ff940104ab0 2024-03-28T18:10:59.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.642+0000 7ff947000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94010f710 con 0x7ff940105470 2024-03-28T18:10:59.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.642+0000 7ff93fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9401038c0 0x7ff940199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.642+0000 7ff93f400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff940104ab0 0x7ff94019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.642+0000 7ff945000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff940105470 0x7ff9401a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.642+0000 7ff93f400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff940104ab0 0x7ff94019a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:37092/0 (socket says 172.21.15.37:37092) 2024-03-28T18:10:59.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.642+0000 7ff93fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9401038c0 0x7ff940199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:32924/0 (socket says 172.21.15.37:32924) 2024-03-28T18:10:59.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.642+0000 7ff93f400640 1 -- 172.21.15.37:0/1899060108 learned_addr learned my addr 172.21.15.37:0/1899060108 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:59.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.643+0000 7ff945000640 1 -- 172.21.15.37:0/1899060108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff940104ab0 msgr2=0x7ff94019a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.643+0000 7ff945000640 1 --2- 172.21.15.37:0/1899060108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff940104ab0 0x7ff94019a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.643+0000 7ff945000640 1 -- 172.21.15.37:0/1899060108 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9401038c0 msgr2=0x7ff940199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.643+0000 7ff945000640 1 --2- 172.21.15.37:0/1899060108 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9401038c0 0x7ff940199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.643+0000 7ff945000640 1 -- 172.21.15.37:0/1899060108 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9401a1830 con 0x7ff940105470 2024-03-28T18:10:59.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.643+0000 7ff93fe00640 1 --2- 172.21.15.37:0/1899060108 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9401038c0 0x7ff940199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:59.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.643+0000 7ff945000640 1 --2- 172.21.15.37:0/1899060108 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff940105470 0x7ff9401a1190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff938009fd0 tx=0x7ff93800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:59.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.643+0000 7ff93cc00640 1 -- 172.21.15.37:0/1899060108 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93800ce20 con 0x7ff940105470 2024-03-28T18:10:59.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.643+0000 7ff93cc00640 1 -- 172.21.15.37:0/1899060108 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff938005020 con 0x7ff940105470 2024-03-28T18:10:59.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.643+0000 7ff93cc00640 1 -- 172.21.15.37:0/1899060108 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff938010720 con 0x7ff940105470 2024-03-28T18:10:59.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.643+0000 7ff947000640 1 -- 172.21.15.37:0/1899060108 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9401a1b20 con 0x7ff940105470 2024-03-28T18:10:59.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.644+0000 7ff947000640 1 -- 172.21.15.37:0/1899060108 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9401a1ff0 con 0x7ff940105470 2024-03-28T18:10:59.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.647+0000 7ff949daa740 1 Processor -- start 2024-03-28T18:10:59.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.648+0000 7ff93cc00640 1 -- 172.21.15.37:0/1899060108 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff938005540 con 0x7ff940105470 2024-03-28T18:10:59.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.648+0000 7ff949daa740 1 -- start start 2024-03-28T18:10:59.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.648+0000 7ff949daa740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab65fce80 0x55dab65fd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.648+0000 7ff949daa740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab6602370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.649+0000 7ff949daa740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab66029e0 0x55dab6604e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.649+0000 7ff949daa740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dab6562250 con 0x55dab65fce80 2024-03-28T18:10:59.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.649+0000 7ff949daa740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dab650aa30 con 0x55dab66029e0 2024-03-28T18:10:59.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.649+0000 7ff949daa740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dab655e2f0 con 0x55dab65fd870 2024-03-28T18:10:59.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.649+0000 7ff93f400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab65fce80 0x55dab65fd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.649+0000 7ff93fe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab6602370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.649+0000 7ff93fe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab6602370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56054/0 (socket says 172.21.15.37:56054) 2024-03-28T18:10:59.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.649+0000 7ff93f400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab65fce80 0x55dab65fd2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:32936/0 (socket says 172.21.15.37:32936) 2024-03-28T18:10:59.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.649+0000 7ff93fe00640 1 -- 172.21.15.37:0/1766210877 learned_addr learned my addr 172.21.15.37:0/1766210877 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:59.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.649+0000 7ff93f400640 1 --2- 172.21.15.37:0/1766210877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab66029e0 0x55dab6604e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.649+0000 7ff93fe00640 1 -- 172.21.15.37:0/1766210877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab66029e0 msgr2=0x55dab6604e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:59.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.649+0000 7ff93fe00640 1 --2- 172.21.15.37:0/1766210877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab66029e0 0x55dab6604e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.649+0000 7ff93fe00640 1 -- 172.21.15.37:0/1766210877 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab65fce80 msgr2=0x55dab65fd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:59.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.649+0000 7ff93fe00640 1 --2- 172.21.15.37:0/1766210877 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab65fce80 0x55dab65fd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.649+0000 7ff93fe00640 1 -- 172.21.15.37:0/1766210877 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dab64f5690 con 0x55dab65fd870 2024-03-28T18:10:59.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.650+0000 7ff93fe00640 1 --2- 172.21.15.37:0/1766210877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab6602370 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff93002c240 tx=0x7ff930009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83eee19de08322e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:59.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.650+0000 7ff93cc00640 1 --2- 172.21.15.37:0/1899060108 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff9240a2880 0x7ff9240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.650+0000 7ff935600640 1 -- 172.21.15.37:0/1766210877 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff930045070 con 0x55dab65fd870 2024-03-28T18:10:59.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.650+0000 7ff935600640 1 -- 172.21.15.37:0/1766210877 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff93002cd30 con 0x55dab65fd870 2024-03-28T18:10:59.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.650+0000 7ff935600640 1 -- 172.21.15.37:0/1766210877 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9300050d0 con 0x55dab65fd870 2024-03-28T18:10:59.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.650+0000 7ff93cc00640 1 -- 172.21.15.37:0/1899060108 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff9380cefd0 con 0x7ff940105470 2024-03-28T18:10:59.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.650+0000 7ff93f400640 1 --2- 172.21.15.37:0/1899060108 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff9240a2880 0x7ff9240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.651+0000 7ff949daa740 1 -- 172.21.15.37:0/1766210877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 msgr2=0x55dab6602370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.651+0000 7ff949daa740 1 --2- 172.21.15.37:0/1766210877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab6602370 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff93002c240 tx=0x7ff930009560 comp rx=0 tx=0).stop 2024-03-28T18:10:59.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.651+0000 7ff949daa740 1 -- 172.21.15.37:0/1766210877 shutdown_connections 2024-03-28T18:10:59.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.651+0000 7ff949daa740 1 --2- 172.21.15.37:0/1766210877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab66029e0 0x55dab6604e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.651+0000 7ff93f400640 1 --2- 172.21.15.37:0/1899060108 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff9240a2880 0x7ff9240a4d40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x55dab65fe0d0 tx=0x7ff92c009570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:59.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.651+0000 7ff949daa740 1 --2- 172.21.15.37:0/1766210877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab6602370 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.651+0000 7ff949daa740 1 --2- 172.21.15.37:0/1766210877 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab65fce80 0x55dab65fd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.651+0000 7ff949daa740 1 -- 172.21.15.37:0/1766210877 >> 172.21.15.37:0/1766210877 conn(0x55dab65c5200 msgr2=0x55dab65f87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:59.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.651+0000 7ff949daa740 1 -- 172.21.15.37:0/1766210877 shutdown_connections 2024-03-28T18:10:59.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.651+0000 7ff949daa740 1 -- 172.21.15.37:0/1766210877 wait complete. 2024-03-28T18:10:59.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.652+0000 7ff949daa740 1 -- start start 2024-03-28T18:10:59.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.652+0000 7ff949daa740 1 Processor -- start 2024-03-28T18:10:59.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.652+0000 7ff949daa740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab65fce80 0x55dab6606df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.652+0000 7ff949daa740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab6607330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.652+0000 7ff949daa740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab66029e0 0x55dab660e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.652+0000 7ff949daa740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dab6605930 con 0x55dab65fce80 2024-03-28T18:10:59.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.652+0000 7ff949daa740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dab651de80 con 0x55dab66029e0 2024-03-28T18:10:59.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.652+0000 7ff949daa740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dab6605ab0 con 0x55dab65fd870 2024-03-28T18:10:59.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.652+0000 7ff949daa740 10 client.0 _ll_register_callbacks cb 0x55dab6530130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:10:59.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.652+0000 7ff93f400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab6607330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.653+0000 7ff93fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab65fce80 0x55dab6606df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.653+0000 7ff93f400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab6607330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56060/0 (socket says 172.21.15.37:56060) 2024-03-28T18:10:59.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.653+0000 7ff93f400640 1 -- 172.21.15.37:0/957514363 learned_addr learned my addr 172.21.15.37:0/957514363 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:59.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.653+0000 7ff945000640 1 --2- 172.21.15.37:0/957514363 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab66029e0 0x55dab660e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.653+0000 7ff945000640 1 -- 172.21.15.37:0/957514363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 msgr2=0x55dab6607330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:59.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.653+0000 7ff945000640 1 --2- 172.21.15.37:0/957514363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab6607330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.653+0000 7ff945000640 1 -- 172.21.15.37:0/957514363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab65fce80 msgr2=0x55dab6606df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:59.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.653+0000 7ff945000640 1 --2- 172.21.15.37:0/957514363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab65fce80 0x55dab6606df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.653+0000 7ff945000640 1 -- 172.21.15.37:0/957514363 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dab63407d0 con 0x55dab66029e0 2024-03-28T18:10:59.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.653+0000 7ff93fe00640 1 --2- 172.21.15.37:0/957514363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab65fce80 0x55dab6606df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:59.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.653+0000 7ff945000640 1 --2- 172.21.15.37:0/957514363 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab66029e0 0x55dab660e370 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff9380ccbd0 tx=0x7ff93809c120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf9571ecda820737 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:59.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.653+0000 7ff92a000640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:10:59.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.653+0000 7ff92a000640 1 -- 172.21.15.37:0/957514363 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9380a41c0 con 0x55dab66029e0 2024-03-28T18:10:59.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.653+0000 7ff92a000640 1 -- 172.21.15.37:0/957514363 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff9380a3d30 con 0x55dab66029e0 2024-03-28T18:10:59.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.653+0000 7ff949daa740 1 -- 172.21.15.37:0/957514363 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff93002cd30 con 0x55dab66029e0 2024-03-28T18:10:59.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.654+0000 7ff92a000640 1 -- 172.21.15.37:0/957514363 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9380a3f30 con 0x55dab66029e0 2024-03-28T18:10:59.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.654+0000 7ff949daa740 10 client.25337 fetch_fsmap learned FSMap version 23 2024-03-28T18:10:59.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.654+0000 7ff949daa740 1 -- 172.21.15.37:0/957514363 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55dab62763f0 con 0x55dab66029e0 2024-03-28T18:10:59.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.654+0000 7ff92a000640 1 -- 172.21.15.37:0/957514363 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff93002cd30 con 0x55dab66029e0 2024-03-28T18:10:59.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.654+0000 7ff92a000640 1 -- 172.21.15.37:0/957514363 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ff9380ba150 con 0x55dab66029e0 2024-03-28T18:10:59.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.654+0000 7ff949daa740 10 client.25337 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:10:59.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.654+0000 7ff949daa740 10 client.25337 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:10:59.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.654+0000 7ff949daa740 20 client.25337 populate_metadata read hostname 'smithi037' 2024-03-28T18:10:59.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.654+0000 7ff949daa740 1 --2- 172.21.15.37:0/957514363 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55dab6612bc0 0x55dab6615060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.654+0000 7ff949daa740 4 client.25337 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:10:59.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.654+0000 7ff949daa740 1 -- 172.21.15.37:0/957514363 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dab656f790 con 0x55dab6612bc0 2024-03-28T18:10:59.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.655+0000 7ff93fe00640 1 --2- 172.21.15.37:0/957514363 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55dab6612bc0 0x55dab6615060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.655+0000 7ff93fe00640 1 --2- 172.21.15.37:0/957514363 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55dab6612bc0 0x55dab6615060 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11881a02b6a71fdb server_cookie=5432375408d5573e in_seq=0 out_seq=0 2024-03-28T18:10:59.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.655+0000 7ff92a000640 10 client.25337 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:10:59.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.656+0000 7ff92a000640 1 -- 172.21.15.37:0/957514363 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55dab656f790 con 0x55dab6612bc0 2024-03-28T18:10:59.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.656+0000 7ff92a000640 10 client.25337 handle_command_reply: tid=0 2024-03-28T18:10:59.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.656+0000 7ff949daa740 1 client.25337 shutdown 2024-03-28T18:10:59.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.656+0000 7ff949daa740 1 -- 172.21.15.37:0/957514363 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab66029e0 msgr2=0x55dab660e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.656+0000 7ff949daa740 1 --2- 172.21.15.37:0/957514363 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab66029e0 0x55dab660e370 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff9380ccbd0 tx=0x7ff93809c120 comp rx=0 tx=0).stop 2024-03-28T18:10:59.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.656+0000 7ff949daa740 1 -- 172.21.15.37:0/957514363 shutdown_connections 2024-03-28T18:10:59.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.656+0000 7ff949daa740 1 --2- 172.21.15.37:0/957514363 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55dab6612bc0 0x55dab6615060 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.656+0000 7ff949daa740 1 --2- 172.21.15.37:0/957514363 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab66029e0 0x55dab660e370 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.656+0000 7ff949daa740 1 --2- 172.21.15.37:0/957514363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab6607330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.656+0000 7ff92a000640 0 client.25337 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:10:59.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.656+0000 7ff949daa740 1 --2- 172.21.15.37:0/957514363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab65fce80 0x55dab6606df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.656+0000 7ff949daa740 1 -- 172.21.15.37:0/957514363 >> 172.21.15.37:0/957514363 conn(0x55dab65c5200 msgr2=0x55dab65f7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:59.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.656+0000 7ff949daa740 1 -- 172.21.15.37:0/957514363 shutdown_connections 2024-03-28T18:10:59.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.656+0000 7ff949daa740 1 -- 172.21.15.37:0/957514363 wait complete. 2024-03-28T18:10:59.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.657+0000 7ff949daa740 20 client.25337 trim_cache size 0 max 16384 2024-03-28T18:10:59.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.662+0000 7efea8a00640 1 -- 172.21.15.37:0/701822511 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7efea40622a0 con 0x7efea41038f0 2024-03-28T18:10:59.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.684+0000 7ff949daa740 1 Processor -- start 2024-03-28T18:10:59.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.684+0000 7ff949daa740 1 -- start start 2024-03-28T18:10:59.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.684+0000 7ff949daa740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab65fce80 0x55dab6601e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.684+0000 7ff949daa740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab660e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.684+0000 7ff949daa740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab66029e0 0x55dab65fba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.684+0000 7ff949daa740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9401ad0a0 con 0x55dab66029e0 2024-03-28T18:10:59.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.684+0000 7ff949daa740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9401aaba0 con 0x55dab65fce80 2024-03-28T18:10:59.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.684+0000 7ff949daa740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9401a8c40 con 0x55dab65fd870 2024-03-28T18:10:59.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.685+0000 7ff93f400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab660e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.685+0000 7ff93fe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab65fce80 0x55dab6601e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.685+0000 7ff93f400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab660e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56062/0 (socket says 172.21.15.37:56062) 2024-03-28T18:10:59.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.685+0000 7ff93fe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab65fce80 0x55dab6601e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:37118/0 (socket says 172.21.15.37:37118) 2024-03-28T18:10:59.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.685+0000 7ff93f400640 1 -- 172.21.15.37:0/2635724946 learned_addr learned my addr 172.21.15.37:0/2635724946 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:59.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.685+0000 7ff945000640 1 --2- 172.21.15.37:0/2635724946 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab66029e0 0x55dab65fba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.685+0000 7ff93f400640 1 -- 172.21.15.37:0/2635724946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab65fce80 msgr2=0x55dab6601e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:59.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.685+0000 7ff93f400640 1 --2- 172.21.15.37:0/2635724946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab65fce80 0x55dab6601e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.685+0000 7ff93f400640 1 -- 172.21.15.37:0/2635724946 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab66029e0 msgr2=0x55dab65fba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:59.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.685+0000 7ff93f400640 1 --2- 172.21.15.37:0/2635724946 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab66029e0 0x55dab65fba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.685+0000 7ff93f400640 1 -- 172.21.15.37:0/2635724946 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dab62360f0 con 0x55dab65fd870 2024-03-28T18:10:59.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.685+0000 7ff93f400640 1 --2- 172.21.15.37:0/2635724946 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab660e330 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff92c005b50 tx=0x7ff92c015040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c62947d8e7b62b7b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:59.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.685+0000 7ff934c00640 1 -- 172.21.15.37:0/2635724946 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c005020 con 0x55dab65fd870 2024-03-28T18:10:59.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.686+0000 7ff934c00640 1 -- 172.21.15.37:0/2635724946 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff92c0047a0 con 0x55dab65fd870 2024-03-28T18:10:59.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.686+0000 7ff934c00640 1 -- 172.21.15.37:0/2635724946 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c023840 con 0x55dab65fd870 2024-03-28T18:10:59.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.686+0000 7ff949daa740 1 -- 172.21.15.37:0/2635724946 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 msgr2=0x55dab660e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.686+0000 7ff949daa740 1 --2- 172.21.15.37:0/2635724946 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab660e330 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff92c005b50 tx=0x7ff92c015040 comp rx=0 tx=0).stop 2024-03-28T18:10:59.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.686+0000 7ff949daa740 1 -- 172.21.15.37:0/2635724946 shutdown_connections 2024-03-28T18:10:59.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.686+0000 7ff949daa740 1 --2- 172.21.15.37:0/2635724946 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab66029e0 0x55dab65fba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.686+0000 7ff949daa740 1 --2- 172.21.15.37:0/2635724946 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab660e330 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.686+0000 7ff949daa740 1 --2- 172.21.15.37:0/2635724946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab65fce80 0x55dab6601e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.686+0000 7ff949daa740 1 -- 172.21.15.37:0/2635724946 >> 172.21.15.37:0/2635724946 conn(0x55dab65c5200 msgr2=0x55dab65feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:59.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.686+0000 7ff949daa740 1 -- 172.21.15.37:0/2635724946 shutdown_connections 2024-03-28T18:10:59.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.686+0000 7ff949daa740 1 -- 172.21.15.37:0/2635724946 wait complete. 2024-03-28T18:10:59.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.687+0000 7ff949daa740 1 -- start start 2024-03-28T18:10:59.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.687+0000 7ff949daa740 1 Processor -- start 2024-03-28T18:10:59.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.687+0000 7ff949daa740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab65fce80 0x55dab660be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.687+0000 7ff949daa740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab660c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.687+0000 7ff949daa740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab66029e0 0x55dab660d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.687+0000 7ff949daa740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9401a9240 con 0x55dab65fce80 2024-03-28T18:10:59.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.687+0000 7ff949daa740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9401aa540 con 0x55dab66029e0 2024-03-28T18:10:59.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.687+0000 7ff949daa740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9401a9880 con 0x55dab65fd870 2024-03-28T18:10:59.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.687+0000 7ff949daa740 10 client.0 _ll_register_callbacks cb 0x7ff938005830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:10:59.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.687+0000 7ff93f400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab660c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.688+0000 7ff93f400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab660c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56074/0 (socket says 172.21.15.37:56074) 2024-03-28T18:10:59.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.688+0000 7ff945000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab66029e0 0x55dab660d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.688+0000 7ff93f400640 1 -- 172.21.15.37:0/3234506847 learned_addr learned my addr 172.21.15.37:0/3234506847 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:10:59.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.688+0000 7ff93fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab65fce80 0x55dab660be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.688+0000 7ff945000640 1 -- 172.21.15.37:0/3234506847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 msgr2=0x55dab660c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:59.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.688+0000 7ff945000640 1 --2- 172.21.15.37:0/3234506847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab660c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.688+0000 7ff945000640 1 -- 172.21.15.37:0/3234506847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab65fce80 msgr2=0x55dab660be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:10:59.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.688+0000 7ff945000640 1 --2- 172.21.15.37:0/3234506847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab65fce80 0x55dab660be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.688+0000 7ff945000640 1 -- 172.21.15.37:0/3234506847 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dab660d9c0 con 0x55dab66029e0 2024-03-28T18:10:59.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.688+0000 7ff93f400640 1 --2- 172.21.15.37:0/3234506847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab660c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:10:59.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.688+0000 7ff945000640 1 --2- 172.21.15.37:0/3234506847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab66029e0 0x55dab660d3d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff93809bbe0 tx=0x7ff93809b270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=616131a9628381fb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:10:59.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.688+0000 7ff92a000640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:10:59.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.688+0000 7ff92a000640 1 -- 172.21.15.37:0/3234506847 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9380a41c0 con 0x55dab66029e0 2024-03-28T18:10:59.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.688+0000 7ff92a000640 1 -- 172.21.15.37:0/3234506847 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff9380a3d30 con 0x55dab66029e0 2024-03-28T18:10:59.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.688+0000 7ff949daa740 1 -- 172.21.15.37:0/3234506847 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff92c0047a0 con 0x55dab66029e0 2024-03-28T18:10:59.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.689+0000 7ff92a000640 1 -- 172.21.15.37:0/3234506847 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9380ba200 con 0x55dab66029e0 2024-03-28T18:10:59.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.689+0000 7ff949daa740 10 client.25343 fetch_fsmap learned FSMap version 23 2024-03-28T18:10:59.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.689+0000 7ff949daa740 1 -- 172.21.15.37:0/3234506847 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55dab666e8d0 con 0x55dab66029e0 2024-03-28T18:10:59.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.689+0000 7ff92a000640 1 -- 172.21.15.37:0/3234506847 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff92c0047a0 con 0x55dab66029e0 2024-03-28T18:10:59.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.689+0000 7ff92a000640 1 -- 172.21.15.37:0/3234506847 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ff9380a0a90 con 0x55dab66029e0 2024-03-28T18:10:59.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.689+0000 7ff949daa740 10 client.25343 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:10:59.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.689+0000 7ff949daa740 10 client.25343 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:10:59.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.689+0000 7ff949daa740 20 client.25343 populate_metadata read hostname 'smithi037' 2024-03-28T18:10:59.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.689+0000 7ff949daa740 1 --2- 172.21.15.37:0/3234506847 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55dab666f1d0 0x55dab66715f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:10:59.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.689+0000 7ff949daa740 4 client.25343 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:10:59.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.689+0000 7ff949daa740 1 -- 172.21.15.37:0/3234506847 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55dab6569ae0 con 0x55dab666f1d0 2024-03-28T18:10:59.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.689+0000 7ff93fe00640 1 --2- 172.21.15.37:0/3234506847 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55dab666f1d0 0x55dab66715f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:10:59.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.690+0000 7ff93fe00640 1 --2- 172.21.15.37:0/3234506847 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55dab666f1d0 0x55dab66715f0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12271c1cc2b14ba8 server_cookie=69d72ae133253cd9 in_seq=0 out_seq=0 2024-03-28T18:10:59.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.690+0000 7ff92a000640 10 client.25343 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:10:59.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.690+0000 7ff92a000640 1 -- 172.21.15.37:0/3234506847 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55dab6569ae0 con 0x55dab666f1d0 2024-03-28T18:10:59.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.690+0000 7ff92a000640 10 client.25343 handle_command_reply: tid=0 2024-03-28T18:10:59.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.690+0000 7ff949daa740 1 client.25343 shutdown 2024-03-28T18:10:59.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.690+0000 7ff949daa740 1 -- 172.21.15.37:0/3234506847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab66029e0 msgr2=0x55dab660d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.690+0000 7ff949daa740 1 --2- 172.21.15.37:0/3234506847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab66029e0 0x55dab660d3d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff93809bbe0 tx=0x7ff93809b270 comp rx=0 tx=0).stop 2024-03-28T18:10:59.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.690+0000 7ff949daa740 1 -- 172.21.15.37:0/3234506847 shutdown_connections 2024-03-28T18:10:59.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.690+0000 7ff949daa740 1 --2- 172.21.15.37:0/3234506847 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55dab666f1d0 0x55dab66715f0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.690+0000 7ff949daa740 1 --2- 172.21.15.37:0/3234506847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dab66029e0 0x55dab660d3d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.690+0000 7ff949daa740 1 --2- 172.21.15.37:0/3234506847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dab65fd870 0x55dab660c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.690+0000 7ff92a000640 0 client.25343 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:10:59.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.690+0000 7ff949daa740 1 --2- 172.21.15.37:0/3234506847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dab65fce80 0x55dab660be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.690+0000 7ff949daa740 1 -- 172.21.15.37:0/3234506847 >> 172.21.15.37:0/3234506847 conn(0x55dab65c5200 msgr2=0x55dab65feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:59.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.690+0000 7ff949daa740 1 -- 172.21.15.37:0/3234506847 shutdown_connections 2024-03-28T18:10:59.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.690+0000 7ff949daa740 1 -- 172.21.15.37:0/3234506847 wait complete. 2024-03-28T18:10:59.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.691+0000 7ff949daa740 20 client.25343 trim_cache size 0 max 16384 2024-03-28T18:10:59.695 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:10:59.695 INFO:teuthology.orchestra.run.smithi037.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:10:59.695 INFO:teuthology.orchestra.run.smithi037.stdout: "scrubs": { 2024-03-28T18:10:59.695 INFO:teuthology.orchestra.run.smithi037.stdout: "e5d6d978-9bb9-47bc-bc78-951bb7802412": { 2024-03-28T18:10:59.695 INFO:teuthology.orchestra.run.smithi037.stdout: "path": "/", 2024-03-28T18:10:59.695 INFO:teuthology.orchestra.run.smithi037.stdout: "tag": "e5d6d978-9bb9-47bc-bc78-951bb7802412", 2024-03-28T18:10:59.696 INFO:teuthology.orchestra.run.smithi037.stdout: "options": "recursive,force" 2024-03-28T18:10:59.696 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-03-28T18:10:59.696 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-03-28T18:10:59.696 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:10:59.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.691+0000 7ff947000640 1 -- 172.21.15.37:0/1899060108 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff9240a2880 msgr2=0x7ff9240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.691+0000 7ff947000640 1 --2- 172.21.15.37:0/1899060108 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff9240a2880 0x7ff9240a4d40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x55dab65fe0d0 tx=0x7ff92c009570 comp rx=0 tx=0).stop 2024-03-28T18:10:59.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.691+0000 7ff947000640 1 -- 172.21.15.37:0/1899060108 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff940105470 msgr2=0x7ff9401a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:10:59.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.691+0000 7ff947000640 1 --2- 172.21.15.37:0/1899060108 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff940105470 0x7ff9401a1190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff938009fd0 tx=0x7ff93800ef10 comp rx=0 tx=0).stop 2024-03-28T18:10:59.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.691+0000 7ff947000640 1 -- 172.21.15.37:0/1899060108 shutdown_connections 2024-03-28T18:10:59.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.691+0000 7ff947000640 1 --2- 172.21.15.37:0/1899060108 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7ff9240a2880 0x7ff9240a4d40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.691+0000 7ff947000640 1 --2- 172.21.15.37:0/1899060108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff940104ab0 0x7ff94019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.691+0000 7ff947000640 1 --2- 172.21.15.37:0/1899060108 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff940105470 0x7ff9401a1190 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.691+0000 7ff947000640 1 --2- 172.21.15.37:0/1899060108 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9401038c0 0x7ff940199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:10:59.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.691+0000 7ff947000640 1 -- 172.21.15.37:0/1899060108 >> 172.21.15.37:0/1899060108 conn(0x7ff9400fee50 msgr2=0x7ff940107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:10:59.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.691+0000 7ff947000640 1 -- 172.21.15.37:0/1899060108 shutdown_connections 2024-03-28T18:10:59.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:10:59.692+0000 7ff947000640 1 -- 172.21.15.37:0/1899060108 wait complete. 2024-03-28T18:10:59.705 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:59 smithi037 ceph-mon[31460]: pgmap v399: 97 pgs: 97 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-28T18:10:59.705 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:10:59 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2010747206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:10:59.707 INFO:tasks.cephfs.filesystem:scrub status for tag:e5d6d978-9bb9-47bc-bc78-951bb7802412 - {'path': '/', 'tag': 'e5d6d978-9bb9-47bc-bc78-951bb7802412', 'options': 'recursive,force'} 2024-03-28T18:10:59.936 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:59 smithi170 ceph-mon[36280]: pgmap v399: 97 pgs: 97 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-28T18:10:59.936 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:10:59 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2010747206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:11:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:59 smithi059 ceph-mon[36190]: pgmap v399: 97 pgs: 97 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-28T18:11:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:10:59 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2010747206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:11:00.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.624+0000 7efe9aa00640 1 -- 172.21.15.37:0/701822511 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7efe94097e50 con 0x7efea41038f0 2024-03-28T18:11:00.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.626+0000 7efe9aa00640 1 -- 172.21.15.37:0/701822511 <== mon.2 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7efe94064910 con 0x7efea41038f0 2024-03-28T18:11:00.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.634+0000 7efea8a00640 1 -- 172.21.15.37:0/701822511 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7efe880a2920 msgr2=0x7efe880a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:00.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.634+0000 7efea8a00640 1 --2- 172.21.15.37:0/701822511 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7efe880a2920 0x7efe880a4de0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7efe9002c250 tx=0x7efe9002cf90 comp rx=0 tx=0).stop 2024-03-28T18:11:00.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.634+0000 7efea8a00640 1 -- 172.21.15.37:0/701822511 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efea41038f0 msgr2=0x7efea4199b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:00.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.634+0000 7efea8a00640 1 --2- 172.21.15.37:0/701822511 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efea41038f0 0x7efea4199b30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7efe9400e9c0 tx=0x7efe9400ee80 comp rx=0 tx=0).stop 2024-03-28T18:11:00.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.634+0000 7efea8a00640 1 -- 172.21.15.37:0/701822511 shutdown_connections 2024-03-28T18:11:00.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.634+0000 7efea8a00640 1 --2- 172.21.15.37:0/701822511 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7efe880a2920 0x7efe880a4de0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:00.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.634+0000 7efea8a00640 1 --2- 172.21.15.37:0/701822511 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efea41054a0 0x7efea41a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:00.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.634+0000 7efea8a00640 1 --2- 172.21.15.37:0/701822511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efea4104ae0 0x7efea419a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:00.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.634+0000 7efea8a00640 1 --2- 172.21.15.37:0/701822511 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efea41038f0 0x7efea4199b30 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:00.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.634+0000 7efea8a00640 1 -- 172.21.15.37:0/701822511 >> 172.21.15.37:0/701822511 conn(0x7efea40fee60 msgr2=0x7efea4107d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:00.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.635+0000 7efea8a00640 1 -- 172.21.15.37:0/701822511 shutdown_connections 2024-03-28T18:11:00.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.635+0000 7efea8a00640 1 -- 172.21.15.37:0/701822511 wait complete. 2024-03-28T18:11:00.654 DEBUG:teuthology.orchestra.run.smithi037:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-28T18:11:00.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.841+0000 7ff014a00640 1 -- 172.21.15.37:0/4039502807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff010103900 msgr2=0x7ff010103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:00.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.841+0000 7ff014a00640 1 --2- 172.21.15.37:0/4039502807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff010103900 0x7ff010103d20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff000009900 tx=0x7ff00002bc30 comp rx=0 tx=0).stop 2024-03-28T18:11:00.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.842+0000 7ff014a00640 1 -- 172.21.15.37:0/4039502807 shutdown_connections 2024-03-28T18:11:00.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.842+0000 7ff014a00640 1 --2- 172.21.15.37:0/4039502807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0101054b0 0x7ff01010bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:00.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.842+0000 7ff014a00640 1 --2- 172.21.15.37:0/4039502807 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff010104af0 0x7ff010104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:00.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.842+0000 7ff014a00640 1 --2- 172.21.15.37:0/4039502807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff010103900 0x7ff010103d20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:00.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.842+0000 7ff014a00640 1 -- 172.21.15.37:0/4039502807 >> 172.21.15.37:0/4039502807 conn(0x7ff0100fee90 msgr2=0x7ff0101012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:00.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.842+0000 7ff014a00640 1 -- 172.21.15.37:0/4039502807 shutdown_connections 2024-03-28T18:11:00.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.842+0000 7ff014a00640 1 -- 172.21.15.37:0/4039502807 wait complete. 2024-03-28T18:11:00.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.843+0000 7ff014a00640 1 Processor -- start 2024-03-28T18:11:00.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.843+0000 7ff014a00640 1 -- start start 2024-03-28T18:11:00.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.843+0000 7ff014a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff010103900 0x7ff010199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:00.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.843+0000 7ff014a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff010104af0 0x7ff01019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:00.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.843+0000 7ff014a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0101054b0 0x7ff0101a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:00.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.843+0000 7ff014a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01010f5c0 con 0x7ff0101054b0 2024-03-28T18:11:00.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.843+0000 7ff014a00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01010f440 con 0x7ff010104af0 2024-03-28T18:11:00.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.843+0000 7ff014a00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01010f740 con 0x7ff010103900 2024-03-28T18:11:00.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.843+0000 7ff00de00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff010103900 0x7ff010199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:00.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.843+0000 7ff00de00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff010103900 0x7ff010199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56116/0 (socket says 172.21.15.37:56116) 2024-03-28T18:11:00.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.843+0000 7ff00d400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff010104af0 0x7ff01019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:00.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.843+0000 7ff00de00640 1 -- 172.21.15.37:0/4082991277 learned_addr learned my addr 172.21.15.37:0/4082991277 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:00.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.843+0000 7ff00e800640 1 --2- 172.21.15.37:0/4082991277 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0101054b0 0x7ff0101a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:00.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.844+0000 7ff00de00640 1 -- 172.21.15.37:0/4082991277 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff010104af0 msgr2=0x7ff01019a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:00.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.844+0000 7ff00de00640 1 --2- 172.21.15.37:0/4082991277 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff010104af0 0x7ff01019a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:00.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.844+0000 7ff00de00640 1 -- 172.21.15.37:0/4082991277 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0101054b0 msgr2=0x7ff0101a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:00.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.844+0000 7ff00de00640 1 --2- 172.21.15.37:0/4082991277 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0101054b0 0x7ff0101a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:00.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.844+0000 7ff00de00640 1 -- 172.21.15.37:0/4082991277 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0101a1810 con 0x7ff010103900 2024-03-28T18:11:00.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.844+0000 7ff00d400640 1 --2- 172.21.15.37:0/4082991277 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff010104af0 0x7ff01019a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:11:00.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.844+0000 7ff00de00640 1 --2- 172.21.15.37:0/4082991277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff010103900 0x7ff010199bf0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff000000c00 tx=0x7ff00002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:00.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.844+0000 7ff006a00640 1 -- 172.21.15.37:0/4082991277 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff000043020 con 0x7ff010103900 2024-03-28T18:11:00.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.845+0000 7ff006a00640 1 -- 172.21.15.37:0/4082991277 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0000054e0 con 0x7ff010103900 2024-03-28T18:11:00.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.845+0000 7ff014a00640 1 -- 172.21.15.37:0/4082991277 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0101a1aa0 con 0x7ff010103900 2024-03-28T18:11:00.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.845+0000 7ff006a00640 1 -- 172.21.15.37:0/4082991277 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff000003c40 con 0x7ff010103900 2024-03-28T18:11:00.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.845+0000 7ff014a00640 1 -- 172.21.15.37:0/4082991277 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0101a1ff0 con 0x7ff010103900 2024-03-28T18:11:00.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.846+0000 7ff014a00640 1 -- 172.21.15.37:0/4082991277 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff01010ab50 con 0x7ff010103900 2024-03-28T18:11:00.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.848+0000 7ff006a00640 1 -- 172.21.15.37:0/4082991277 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7ff000046020 con 0x7ff010103900 2024-03-28T18:11:00.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.849+0000 7ff006a00640 1 --2- 172.21.15.37:0/4082991277 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7feff40a28f0 0x7feff40a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:00.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.849+0000 7ff00d400640 1 -- 172.21.15.37:0/4082991277 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7feff40a28f0 msgr2=0x7feff40a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/521857560 2024-03-28T18:11:00.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.849+0000 7ff006a00640 1 -- 172.21.15.37:0/4082991277 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff0000f0ff0 con 0x7ff010103900 2024-03-28T18:11:00.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.849+0000 7ff00d400640 1 --2- 172.21.15.37:0/4082991277 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7feff40a28f0 0x7feff40a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:11:00.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:00.851+0000 7ff006a00640 1 -- 172.21.15.37:0/4082991277 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff00004d020 con 0x7ff010103900 2024-03-28T18:11:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:00 smithi059 ceph-mon[36190]: scrub queued for path: / 2024-03-28T18:11:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:00 smithi059 ceph-mon[36190]: scrub summary: idle+waiting paths [/] 2024-03-28T18:11:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:00 smithi059 ceph-mon[36190]: scrub summary: active paths [/] 2024-03-28T18:11:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:00 smithi059 ceph-mon[36190]: scrub summary: idle+waiting paths [/] 2024-03-28T18:11:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:00 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/701822511' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T18:11:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:00 smithi059 ceph-mon[36190]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T18:11:01.035 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: ignoring --setuser ceph since I am not root 2024-03-28T18:11:01.036 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: ignoring --setgroup ceph since I am not root 2024-03-28T18:11:01.036 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:00 smithi059 ceph-mgr[37580]: -- 172.21.15.59:0/1199238489 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55772e0f45a0 con 0x55772e0c8c00 2024-03-28T18:11:01.036 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:00 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: ignoring --setuser ceph since I am not root 2024-03-28T18:11:01.036 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:00 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: ignoring --setgroup ceph since I am not root 2024-03-28T18:11:01.036 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:00 smithi170 ceph-mgr[38311]: -- 172.21.15.170:0/3244121505 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55943fc725a0 con 0x55943fc47000 2024-03-28T18:11:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:00 smithi170 ceph-mon[36280]: scrub queued for path: / 2024-03-28T18:11:01.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:00 smithi170 ceph-mon[36280]: scrub summary: idle+waiting paths [/] 2024-03-28T18:11:01.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:00 smithi170 ceph-mon[36280]: scrub summary: active paths [/] 2024-03-28T18:11:01.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:00 smithi170 ceph-mon[36280]: scrub summary: idle+waiting paths [/] 2024-03-28T18:11:01.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:00 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/701822511' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T18:11:01.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:00 smithi170 ceph-mon[36280]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T18:11:01.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.050+0000 7ff00d400640 1 -- 172.21.15.37:0/4082991277 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7feff40a28f0 msgr2=0x7feff40a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/521857560 2024-03-28T18:11:01.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.050+0000 7ff00d400640 1 --2- 172.21.15.37:0/4082991277 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7feff40a28f0 0x7feff40a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T18:11:01.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.077+0000 7ff014a00640 1 -- 172.21.15.37:0/4082991277 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7ff0100622a0 con 0x7ff010103900 2024-03-28T18:11:01.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.085+0000 7ff006a00640 1 -- 172.21.15.37:0/4082991277 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v31)=0 v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff0000be0a0 con 0x7ff010103900 2024-03-28T18:11:01.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.088+0000 7ff014a00640 1 -- 172.21.15.37:0/4082991277 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7feff40a28f0 msgr2=0x7feff40a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:11:01.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.088+0000 7ff014a00640 1 --2- 172.21.15.37:0/4082991277 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7feff40a28f0 0x7feff40a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.088+0000 7ff014a00640 1 -- 172.21.15.37:0/4082991277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff010103900 msgr2=0x7ff010199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:01.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.088+0000 7ff014a00640 1 --2- 172.21.15.37:0/4082991277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff010103900 0x7ff010199bf0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff000000c00 tx=0x7ff00002c650 comp rx=0 tx=0).stop 2024-03-28T18:11:01.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.089+0000 7ff014a00640 1 -- 172.21.15.37:0/4082991277 shutdown_connections 2024-03-28T18:11:01.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.089+0000 7ff014a00640 1 --2- 172.21.15.37:0/4082991277 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7feff40a28f0 0x7feff40a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.089+0000 7ff014a00640 1 --2- 172.21.15.37:0/4082991277 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff010104af0 0x7ff01019a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.089+0000 7ff014a00640 1 --2- 172.21.15.37:0/4082991277 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0101054b0 0x7ff0101a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.089+0000 7ff014a00640 1 --2- 172.21.15.37:0/4082991277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff010103900 0x7ff010199bf0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.089+0000 7ff014a00640 1 -- 172.21.15.37:0/4082991277 >> 172.21.15.37:0/4082991277 conn(0x7ff0100fee90 msgr2=0x7ff010107d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:01.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.089+0000 7ff014a00640 1 -- 172.21.15.37:0/4082991277 shutdown_connections 2024-03-28T18:11:01.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.089+0000 7ff014a00640 1 -- 172.21.15.37:0/4082991277 wait complete. 2024-03-28T18:11:01.105 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:00 smithi037 ceph-mon[31460]: scrub queued for path: / 2024-03-28T18:11:01.106 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:00 smithi037 ceph-mon[31460]: scrub summary: idle+waiting paths [/] 2024-03-28T18:11:01.106 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:00 smithi037 ceph-mon[31460]: scrub summary: active paths [/] 2024-03-28T18:11:01.106 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:00 smithi037 ceph-mon[31460]: scrub summary: idle+waiting paths [/] 2024-03-28T18:11:01.106 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:00 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/701822511' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T18:11:01.106 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:00 smithi037 ceph-mon[31460]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T18:11:01.106 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: ignoring --setuser ceph since I am not root 2024-03-28T18:11:01.106 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: ignoring --setgroup ceph since I am not root 2024-03-28T18:11:01.107 DEBUG:teuthology.orchestra.run.smithi037:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-28T18:11:01.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.304+0000 7fd693000640 1 -- 172.21.15.37:0/325225132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd68c101510 msgr2=0x7fd68c101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:01.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.304+0000 7fd693000640 1 --2- 172.21.15.37:0/325225132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd68c101510 0x7fd68c101930 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd680009900 tx=0x7fd68002bc30 comp rx=0 tx=0).stop 2024-03-28T18:11:01.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.305+0000 7fd693000640 1 -- 172.21.15.37:0/325225132 shutdown_connections 2024-03-28T18:11:01.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.305+0000 7fd693000640 1 --2- 172.21.15.37:0/325225132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd68c1030c0 0x7fd68c109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.305+0000 7fd693000640 1 --2- 172.21.15.37:0/325225132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd68c102700 0x7fd68c102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.305+0000 7fd693000640 1 --2- 172.21.15.37:0/325225132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd68c101510 0x7fd68c101930 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.305+0000 7fd693000640 1 -- 172.21.15.37:0/325225132 >> 172.21.15.37:0/325225132 conn(0x7fd68c0fca80 msgr2=0x7fd68c0feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:01.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.305+0000 7fd693000640 1 -- 172.21.15.37:0/325225132 shutdown_connections 2024-03-28T18:11:01.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.305+0000 7fd693000640 1 -- 172.21.15.37:0/325225132 wait complete. 2024-03-28T18:11:01.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.306+0000 7fd693000640 1 Processor -- start 2024-03-28T18:11:01.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.307+0000 7fd693000640 1 -- start start 2024-03-28T18:11:01.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.307+0000 7fd693000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd68c101510 0x7fd68c075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:01.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.307+0000 7fd693000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd68c102700 0x7fd68c073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:01.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.307+0000 7fd693000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd68c1030c0 0x7fd68c073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:01.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.307+0000 7fd693000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c10d1f0 con 0x7fd68c101510 2024-03-28T18:11:01.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.307+0000 7fd693000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c10d070 con 0x7fd68c1030c0 2024-03-28T18:11:01.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.307+0000 7fd693000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c10d370 con 0x7fd68c102700 2024-03-28T18:11:01.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.307+0000 7fd691200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd68c102700 0x7fd68c073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:01.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.307+0000 7fd692600640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd68c1030c0 0x7fd68c073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:01.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.308+0000 7fd691c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd68c101510 0x7fd68c075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:01.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.308+0000 7fd691200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd68c102700 0x7fd68c073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56128/0 (socket says 172.21.15.37:56128) 2024-03-28T18:11:01.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.308+0000 7fd691200640 1 -- 172.21.15.37:0/2302763262 learned_addr learned my addr 172.21.15.37:0/2302763262 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:01.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.308+0000 7fd691200640 1 -- 172.21.15.37:0/2302763262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd68c1030c0 msgr2=0x7fd68c073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:01.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.308+0000 7fd691200640 1 --2- 172.21.15.37:0/2302763262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd68c1030c0 0x7fd68c073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.308+0000 7fd691200640 1 -- 172.21.15.37:0/2302763262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd68c101510 msgr2=0x7fd68c075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:01.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.308+0000 7fd691200640 1 --2- 172.21.15.37:0/2302763262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd68c101510 0x7fd68c075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.308+0000 7fd691200640 1 -- 172.21.15.37:0/2302763262 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68c074860 con 0x7fd68c102700 2024-03-28T18:11:01.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.308+0000 7fd692600640 1 --2- 172.21.15.37:0/2302763262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd68c1030c0 0x7fd68c073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:11:01.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.308+0000 7fd691200640 1 --2- 172.21.15.37:0/2302763262 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd68c102700 0x7fd68c073a90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd67800e9f0 tx=0x7fd67800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:01.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.309+0000 7fd68a000640 1 -- 172.21.15.37:0/2302763262 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67800cd80 con 0x7fd68c102700 2024-03-28T18:11:01.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.309+0000 7fd68a000640 1 -- 172.21.15.37:0/2302763262 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd678010910 con 0x7fd68c102700 2024-03-28T18:11:01.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.309+0000 7fd693000640 1 -- 172.21.15.37:0/2302763262 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68c1a1920 con 0x7fd68c102700 2024-03-28T18:11:01.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.309+0000 7fd68a000640 1 -- 172.21.15.37:0/2302763262 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd678018990 con 0x7fd68c102700 2024-03-28T18:11:01.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.309+0000 7fd693000640 1 -- 172.21.15.37:0/2302763262 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd68c1a1ec0 con 0x7fd68c102700 2024-03-28T18:11:01.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.310+0000 7fd693000640 1 -- 172.21.15.37:0/2302763262 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd68c108830 con 0x7fd68c102700 2024-03-28T18:11:01.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.314+0000 7fd68a000640 1 -- 172.21.15.37:0/2302763262 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7fd678010ac0 con 0x7fd68c102700 2024-03-28T18:11:01.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.315+0000 7fd68a000640 1 --2- 172.21.15.37:0/2302763262 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd6740a28c0 0x7fd6740a4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:01.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.315+0000 7fd68a000640 1 -- 172.21.15.37:0/2302763262 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd6780cf180 con 0x7fd68c102700 2024-03-28T18:11:01.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.315+0000 7fd691c00640 1 -- 172.21.15.37:0/2302763262 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd6740a28c0 msgr2=0x7fd6740a4d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/521857560 2024-03-28T18:11:01.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.315+0000 7fd691c00640 1 --2- 172.21.15.37:0/2302763262 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd6740a28c0 0x7fd6740a4d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:11:01.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.316+0000 7fd68a000640 1 -- 172.21.15.37:0/2302763262 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd67809bb80 con 0x7fd68c102700 2024-03-28T18:11:01.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.516+0000 7fd691c00640 1 -- 172.21.15.37:0/2302763262 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd6740a28c0 msgr2=0x7fd6740a4d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/521857560 2024-03-28T18:11:01.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.516+0000 7fd691c00640 1 --2- 172.21.15.37:0/2302763262 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd6740a28c0 0x7fd6740a4d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T18:11:01.534 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:01 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:01.214+0000 7f4a5cb79200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:11:01.534 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:01 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:01.310+0000 7f4a5cb79200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:11:01.535 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:01 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:01.213+0000 7f17c0d47200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:11:01.535 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:01 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:01.309+0000 7f17c0d47200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:11:01.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.553+0000 7fd693000640 1 -- 172.21.15.37:0/2302763262 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fd68c0622a0 con 0x7fd68c102700 2024-03-28T18:11:01.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.560+0000 7fd68a000640 1 -- 172.21.15.37:0/2302763262 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v32)=0 v32) v1 ==== 133+0+0 (secure 0 0 0) 0x7fd67809fc20 con 0x7fd68c102700 2024-03-28T18:11:01.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.564+0000 7fd693000640 1 -- 172.21.15.37:0/2302763262 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd6740a28c0 msgr2=0x7fd6740a4d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:11:01.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.564+0000 7fd693000640 1 --2- 172.21.15.37:0/2302763262 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd6740a28c0 0x7fd6740a4d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.564+0000 7fd693000640 1 -- 172.21.15.37:0/2302763262 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd68c102700 msgr2=0x7fd68c073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:01.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.564+0000 7fd693000640 1 --2- 172.21.15.37:0/2302763262 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd68c102700 0x7fd68c073a90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd67800e9f0 tx=0x7fd67800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.564+0000 7fd693000640 1 -- 172.21.15.37:0/2302763262 shutdown_connections 2024-03-28T18:11:01.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.564+0000 7fd693000640 1 --2- 172.21.15.37:0/2302763262 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7fd6740a28c0 0x7fd6740a4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.564+0000 7fd693000640 1 --2- 172.21.15.37:0/2302763262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd68c1030c0 0x7fd68c073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.564+0000 7fd693000640 1 --2- 172.21.15.37:0/2302763262 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd68c102700 0x7fd68c073a90 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.564+0000 7fd693000640 1 --2- 172.21.15.37:0/2302763262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd68c101510 0x7fd68c075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.564+0000 7fd693000640 1 -- 172.21.15.37:0/2302763262 >> 172.21.15.37:0/2302763262 conn(0x7fd68c0fca80 msgr2=0x7fd68c105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:01.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.564+0000 7fd693000640 1 -- 172.21.15.37:0/2302763262 shutdown_connections 2024-03-28T18:11:01.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.564+0000 7fd693000640 1 -- 172.21.15.37:0/2302763262 wait complete. 2024-03-28T18:11:01.592 DEBUG:teuthology.orchestra.run.smithi037:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-28T18:11:01.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.795+0000 7f5625a00640 1 -- 172.21.15.37:0/3813691884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56201028c0 msgr2=0x7f5620102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:01.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.795+0000 7f5625a00640 1 --2- 172.21.15.37:0/3813691884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56201028c0 0x7f5620102d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5608009900 tx=0x7f560802bc30 comp rx=0 tx=0).stop 2024-03-28T18:11:01.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.795+0000 7f5625a00640 1 -- 172.21.15.37:0/3813691884 shutdown_connections 2024-03-28T18:11:01.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.795+0000 7f5625a00640 1 --2- 172.21.15.37:0/3813691884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56201028c0 0x7f5620102d40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.795+0000 7f5625a00640 1 --2- 172.21.15.37:0/3813691884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5620103280 0x7f5620109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.795+0000 7f5625a00640 1 --2- 172.21.15.37:0/3813691884 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56201016d0 0x7f5620101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.795+0000 7f5625a00640 1 -- 172.21.15.37:0/3813691884 >> 172.21.15.37:0/3813691884 conn(0x7f56200fcc60 msgr2=0x7f56200ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:01.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.795+0000 7f5625a00640 1 -- 172.21.15.37:0/3813691884 shutdown_connections 2024-03-28T18:11:01.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.795+0000 7f5625a00640 1 -- 172.21.15.37:0/3813691884 wait complete. 2024-03-28T18:11:01.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.796+0000 7f5625a00640 1 Processor -- start 2024-03-28T18:11:01.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.797+0000 7f5625a00640 1 -- start start 2024-03-28T18:11:01.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.797+0000 7f5625a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56201016d0 0x7f5620197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:01.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.797+0000 7f5625a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56201028c0 0x7f5620197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:01.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.797+0000 7f5625a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5620103280 0x7f562019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:01.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.797+0000 7f5625a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562010d350 con 0x7f56201016d0 2024-03-28T18:11:01.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.797+0000 7f5625a00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562010d1d0 con 0x7f5620103280 2024-03-28T18:11:01.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.797+0000 7f5625a00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562010d4d0 con 0x7f56201028c0 2024-03-28T18:11:01.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.797+0000 7f5616000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56201028c0 0x7f5620197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:01.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.797+0000 7f561ea00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56201016d0 0x7f5620197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:01.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.797+0000 7f5616000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56201028c0 0x7f5620197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56138/0 (socket says 172.21.15.37:56138) 2024-03-28T18:11:01.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.797+0000 7f561ea00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56201016d0 0x7f5620197910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33026/0 (socket says 172.21.15.37:33026) 2024-03-28T18:11:01.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.797+0000 7f5616000640 1 -- 172.21.15.37:0/252872691 learned_addr learned my addr 172.21.15.37:0/252872691 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:01.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.797+0000 7f561f400640 1 --2- 172.21.15.37:0/252872691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5620103280 0x7f562019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:01.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.798+0000 7f5616000640 1 -- 172.21.15.37:0/252872691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5620103280 msgr2=0x7f562019eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:01.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.798+0000 7f5616000640 1 --2- 172.21.15.37:0/252872691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5620103280 0x7f562019eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.798+0000 7f5616000640 1 -- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56201016d0 msgr2=0x7f5620197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:01.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.798+0000 7f5616000640 1 --2- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56201016d0 0x7f5620197910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:01.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.798+0000 7f5616000640 1 -- 172.21.15.37:0/252872691 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562019f530 con 0x7f56201028c0 2024-03-28T18:11:01.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.798+0000 7f561ea00640 1 --2- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56201016d0 0x7f5620197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:11:01.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.798+0000 7f561f400640 1 --2- 172.21.15.37:0/252872691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5620103280 0x7f562019eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:11:01.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.798+0000 7f5616000640 1 --2- 172.21.15.37:0/252872691 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56201028c0 0x7f5620197e50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5608009230 tx=0x7f5608005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:01.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.799+0000 7f5617e00640 1 -- 172.21.15.37:0/252872691 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5608041070 con 0x7f56201028c0 2024-03-28T18:11:01.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.799+0000 7f5617e00640 1 -- 172.21.15.37:0/252872691 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56080044d0 con 0x7f56201028c0 2024-03-28T18:11:01.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.799+0000 7f5617e00640 1 -- 172.21.15.37:0/252872691 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5608034440 con 0x7f56201028c0 2024-03-28T18:11:01.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.799+0000 7f5625a00640 1 -- 172.21.15.37:0/252872691 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f562019f7c0 con 0x7f56201028c0 2024-03-28T18:11:01.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.799+0000 7f5625a00640 1 -- 172.21.15.37:0/252872691 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f562019fd00 con 0x7f56201028c0 2024-03-28T18:11:01.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.800+0000 7f5625a00640 1 -- 172.21.15.37:0/252872691 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5620108990 con 0x7f56201028c0 2024-03-28T18:11:01.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.803+0000 7f5617e00640 1 -- 172.21.15.37:0/252872691 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7f5608044070 con 0x7f56201028c0 2024-03-28T18:11:01.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.804+0000 7f5617e00640 1 --2- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56040a2970 0x7f56040a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:01.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.804+0000 7f561ea00640 1 -- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56040a2970 msgr2=0x7f56040a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/521857560 2024-03-28T18:11:01.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.805+0000 7f5617e00640 1 -- 172.21.15.37:0/252872691 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f56080f0630 con 0x7f56201028c0 2024-03-28T18:11:01.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.805+0000 7f561ea00640 1 --2- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56040a2970 0x7f56040a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:11:01.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:01.808+0000 7f5617e00640 1 -- 172.21.15.37:0/252872691 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f56080bd030 con 0x7f56201028c0 2024-03-28T18:11:01.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:01 smithi037 ceph-mon[31460]: scrub summary: idle 2024-03-28T18:11:01.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:01 smithi037 ceph-mon[31460]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T18:11:01.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:01 smithi037 ceph-mon[31460]: mgrmap e19: x(active, since 10m), standbys: y, z 2024-03-28T18:11:01.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:01 smithi037 ceph-mon[31460]: from='client.? ' entity='client.admin' 2024-03-28T18:11:01.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:01 smithi037 ceph-mon[31460]: from='client.? ' entity='client.admin' 2024-03-28T18:11:01.875 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:01 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:01.549+0000 7f105fef7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:11:01.875 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:01 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:01.690+0000 7f105fef7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:11:01.896 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:01 smithi059 ceph-mon[36190]: scrub summary: idle 2024-03-28T18:11:01.896 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:01 smithi059 ceph-mon[36190]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T18:11:01.896 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:01 smithi059 ceph-mon[36190]: mgrmap e19: x(active, since 10m), standbys: y, z 2024-03-28T18:11:01.896 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:01 smithi059 ceph-mon[36190]: from='client.? ' entity='client.admin' 2024-03-28T18:11:01.896 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:01 smithi059 ceph-mon[36190]: from='client.? ' entity='client.admin' 2024-03-28T18:11:01.978 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:01 smithi170 ceph-mon[36280]: scrub summary: idle 2024-03-28T18:11:01.978 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:01 smithi170 ceph-mon[36280]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T18:11:01.978 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:01 smithi170 ceph-mon[36280]: mgrmap e19: x(active, since 10m), standbys: y, z 2024-03-28T18:11:01.978 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:01 smithi170 ceph-mon[36280]: from='client.? ' entity='client.admin' 2024-03-28T18:11:01.978 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:01 smithi170 ceph-mon[36280]: from='client.? ' entity='client.admin' 2024-03-28T18:11:02.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:02.006+0000 7f561ea00640 1 -- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56040a2970 msgr2=0x7f56040a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/521857560 2024-03-28T18:11:02.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:02.006+0000 7f561ea00640 1 --2- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56040a2970 0x7f56040a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T18:11:02.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:02.054+0000 7f5625a00640 1 -- 172.21.15.37:0/252872691 --> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5620061ae0 con 0x7f56040a2970 2024-03-28T18:11:02.237 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:01 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:01.988+0000 7f4a5cb79200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:11:02.237 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:02 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:02.087+0000 7f4a5cb79200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:11:02.237 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:02 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:02.237+0000 7f4a5cb79200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:11:02.285 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:01 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:01.977+0000 7f17c0d47200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:11:02.285 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:02 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:02.071+0000 7f17c0d47200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:11:02.285 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:02 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:02.215+0000 7f17c0d47200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:11:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:02.407+0000 7f561ea00640 1 -- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56040a2970 msgr2=0x7f56040a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/521857560 2024-03-28T18:11:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:02.407+0000 7f561ea00640 1 --2- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56040a2970 0x7f56040a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T18:11:02.535 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:02 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:02.350+0000 7f4a5cb79200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:11:02.535 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:02 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:02.439+0000 7f4a5cb79200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:11:02.663 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:02 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:02.312+0000 7f17c0d47200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:11:02.663 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:02 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:02.396+0000 7f17c0d47200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:11:02.846 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:02.532+0000 7f105fef7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:11:02.846 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:02.659+0000 7f105fef7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:11:02.846 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:02.845+0000 7f105fef7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:11:02.979 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:02 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:02.716+0000 7f4a5cb79200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:11:02.979 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:02 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:02.830+0000 7f4a5cb79200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:11:02.979 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:02 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:02.919+0000 7f4a5cb79200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:11:03.002 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:02 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:02.662+0000 7f17c0d47200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:11:03.003 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:02 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:02.803+0000 7f17c0d47200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:11:03.003 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:02 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:02.907+0000 7f17c0d47200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:11:03.125 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:02.975+0000 7f105fef7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:11:03.125 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:03.090+0000 7f105fef7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:11:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:03.208+0000 7f561ea00640 1 -- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56040a2970 msgr2=0x7f56040a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/521857560 2024-03-28T18:11:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:03.208+0000 7f561ea00640 1 --2- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56040a2970 0x7f56040a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T18:11:03.285 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:03 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:03.029+0000 7f4a5cb79200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:11:03.285 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:03 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:03.211+0000 7f4a5cb79200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:11:03.285 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:03 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:03.002+0000 7f17c0d47200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:11:03.285 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:03 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:03.190+0000 7f17c0d47200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:11:03.730 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:03.444+0000 7f105fef7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:11:03.730 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:03.601+0000 7f105fef7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:11:03.730 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:03.729+0000 7f105fef7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:11:04.086 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:03.859+0000 7f105fef7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:11:04.086 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:04 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:04.085+0000 7f105fef7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:11:04.535 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:04 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:04.220+0000 7f4a5cb79200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:11:04.535 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:04 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:04.355+0000 7f4a5cb79200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:11:04.535 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:04 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:04.158+0000 7f17c0d47200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:11:04.535 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:04 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:04.287+0000 7f17c0d47200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:11:04.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:04.810+0000 7f561ea00640 1 -- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56040a2970 msgr2=0x7f56040a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/521857560 2024-03-28T18:11:04.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:04.810+0000 7f561ea00640 1 --2- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56040a2970 0x7f56040a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T18:11:04.912 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:04 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:04.626+0000 7f17c0d47200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:11:04.912 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:04 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:04.716+0000 7f17c0d47200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:11:04.994 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:04 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:04.698+0000 7f4a5cb79200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:11:04.994 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:04 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:04.791+0000 7f4a5cb79200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:11:04.994 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:04 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:04.994+0000 7f4a5cb79200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:11:05.285 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:05 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:05.185+0000 7f4a5cb79200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:11:05.285 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:04 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:04.912+0000 7f17c0d47200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:11:05.285 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:05 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:05.097+0000 7f17c0d47200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:11:05.648 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:05 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:05.381+0000 7f17c0d47200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:11:05.746 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:05 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:05.478+0000 7f4a5cb79200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:11:05.874 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:05 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:05.411+0000 7f105fef7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:11:05.875 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:05 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:05.604+0000 7f105fef7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:11:06.035 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:05 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:05.746+0000 7f4a5cb79200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:11:06.035 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:05 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:05.990+0000 7f4a5cb79200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:11:06.035 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:05 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:05.647+0000 7f17c0d47200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:11:06.035 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:05 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:05.885+0000 7f17c0d47200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:11:06.374 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:06 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:06.040+0000 7f105fef7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:11:06.375 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:06 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:06.164+0000 7f105fef7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:11:06.736 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:06 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:06.435+0000 7f105fef7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:11:06.736 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:06 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:06.734+0000 7f105fef7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:11:07.035 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:06 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:06.740+0000 7f17c0d47200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:11:07.285 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:06 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:06.824+0000 7f4a5cb79200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:11:07.510 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:07 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:07.170+0000 7f105fef7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:11:07.649 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:07.322+0000 7f17c0d47200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:11:07.649 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:07.420+0000 7f17c0d47200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:11:07.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:07.729+0000 7f5617e00640 1 -- 172.21.15.37:0/252872691 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mgrmap(e 20) v1 ==== 137307+0+0 (secure 0 0 0) 0x7f56080b96a0 con 0x7f56201028c0 2024-03-28T18:11:07.746 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:07 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:07.427+0000 7f4a5cb79200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:11:07.746 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:07 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:07.541+0000 7f4a5cb79200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:11:07.833 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:07 smithi037 ceph-mon[31460]: Standby manager daemon z restarted 2024-03-28T18:11:07.833 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:07 smithi037 ceph-mon[31460]: Standby manager daemon z started 2024-03-28T18:11:07.833 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:07 smithi037 ceph-mon[31460]: from='mgr.? 172.21.15.170:0/1484515216' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T18:11:07.833 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:07 smithi037 ceph-mon[31460]: from='mgr.? 172.21.15.170:0/1484515216' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:11:07.833 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:07 smithi037 ceph-mon[31460]: from='mgr.? 172.21.15.170:0/1484515216' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T18:11:07.834 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:07 smithi037 ceph-mon[31460]: from='mgr.? 172.21.15.170:0/1484515216' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:11:07.834 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:07 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:07.565+0000 7f105fef7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:11:08.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:08.012+0000 7f561ea00640 1 -- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56040a2970 msgr2=0x7f56040a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/521857560 2024-03-28T18:11:08.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:08.012+0000 7f561ea00640 1 --2- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56040a2970 0x7f56040a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-28T18:11:08.034 INFO:journalctl@ceph.mgr.y.smithi059.stdout:Mar 28 18:11:07 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-y[37576]: 2024-03-28T18:11:07.800+0000 7f4a5cb79200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:11:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:07 smithi059 ceph-mon[36190]: Standby manager daemon z restarted 2024-03-28T18:11:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:07 smithi059 ceph-mon[36190]: Standby manager daemon z started 2024-03-28T18:11:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:07 smithi059 ceph-mon[36190]: from='mgr.? 172.21.15.170:0/1484515216' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T18:11:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:07 smithi059 ceph-mon[36190]: from='mgr.? 172.21.15.170:0/1484515216' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:11:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:07 smithi059 ceph-mon[36190]: from='mgr.? 172.21.15.170:0/1484515216' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T18:11:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:07 smithi059 ceph-mon[36190]: from='mgr.? 172.21.15.170:0/1484515216' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:11:08.036 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 28 18:11:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-z[38307]: 2024-03-28T18:11:07.649+0000 7f17c0d47200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:11:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:07 smithi170 ceph-mon[36280]: Standby manager daemon z restarted 2024-03-28T18:11:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:07 smithi170 ceph-mon[36280]: Standby manager daemon z started 2024-03-28T18:11:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:07 smithi170 ceph-mon[36280]: from='mgr.? 172.21.15.170:0/1484515216' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T18:11:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:07 smithi170 ceph-mon[36280]: from='mgr.? 172.21.15.170:0/1484515216' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:11:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:07 smithi170 ceph-mon[36280]: from='mgr.? 172.21.15.170:0/1484515216' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T18:11:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:07 smithi170 ceph-mon[36280]: from='mgr.? 172.21.15.170:0/1484515216' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:11:08.124 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:07 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:07.924+0000 7f105fef7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:11:08.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:08.761+0000 7f5617e00640 1 -- 172.21.15.37:0/252872691 <== mon.1 v2:172.21.15.170:3300/0 8 ==== mgrmap(e 21) v1 ==== 137307+0+0 (secure 0 0 0) 0x7f56080ee160 con 0x7f56201028c0 2024-03-28T18:11:09.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:08 smithi059 ceph-mon[36190]: mgrmap e20: x(active, since 10m), standbys: y, z 2024-03-28T18:11:09.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:08 smithi059 ceph-mon[36190]: Standby manager daemon y restarted 2024-03-28T18:11:09.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:08 smithi059 ceph-mon[36190]: Standby manager daemon y started 2024-03-28T18:11:09.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:08 smithi059 ceph-mon[36190]: from='mgr.? 172.21.15.59:0/1831943644' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T18:11:09.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:08 smithi059 ceph-mon[36190]: from='mgr.? 172.21.15.59:0/1831943644' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:11:09.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:08 smithi059 ceph-mon[36190]: from='mgr.? 172.21.15.59:0/1831943644' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T18:11:09.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:08 smithi059 ceph-mon[36190]: from='mgr.? 172.21.15.59:0/1831943644' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:11:09.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:08 smithi170 ceph-mon[36280]: mgrmap e20: x(active, since 10m), standbys: y, z 2024-03-28T18:11:09.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:08 smithi170 ceph-mon[36280]: Standby manager daemon y restarted 2024-03-28T18:11:09.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:08 smithi170 ceph-mon[36280]: Standby manager daemon y started 2024-03-28T18:11:09.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:08 smithi170 ceph-mon[36280]: from='mgr.? 172.21.15.59:0/1831943644' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T18:11:09.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:08 smithi170 ceph-mon[36280]: from='mgr.? 172.21.15.59:0/1831943644' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:11:09.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:08 smithi170 ceph-mon[36280]: from='mgr.? 172.21.15.59:0/1831943644' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T18:11:09.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:08 smithi170 ceph-mon[36280]: from='mgr.? 172.21.15.59:0/1831943644' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:11:09.068 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:08 smithi037 ceph-mon[31460]: mgrmap e20: x(active, since 10m), standbys: y, z 2024-03-28T18:11:09.068 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:08 smithi037 ceph-mon[31460]: Standby manager daemon y restarted 2024-03-28T18:11:09.068 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:08 smithi037 ceph-mon[31460]: Standby manager daemon y started 2024-03-28T18:11:09.068 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:08 smithi037 ceph-mon[31460]: from='mgr.? 172.21.15.59:0/1831943644' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T18:11:09.068 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:08 smithi037 ceph-mon[31460]: from='mgr.? 172.21.15.59:0/1831943644' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:11:09.068 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:08 smithi037 ceph-mon[31460]: from='mgr.? 172.21.15.59:0/1831943644' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T18:11:09.068 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:08 smithi037 ceph-mon[31460]: from='mgr.? 172.21.15.59:0/1831943644' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:11:09.374 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:09 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:09.066+0000 7f105fef7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:11:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:09 smithi059 ceph-mon[36190]: mgrmap e21: x(active, since 10m), standbys: y, z 2024-03-28T18:11:10.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:09 smithi170 ceph-mon[36280]: mgrmap e21: x(active, since 10m), standbys: y, z 2024-03-28T18:11:10.040 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:09 smithi037 ceph-mon[31460]: mgrmap e21: x(active, since 10m), standbys: y, z 2024-03-28T18:11:10.040 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:09 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:09.901+0000 7f105fef7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:11:10.339 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:10.039+0000 7f105fef7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:11:10.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:10.362+0000 7f5617e00640 1 -- 172.21.15.37:0/252872691 <== mon.1 v2:172.21.15.170:3300/0 9 ==== mgrmap(e 22) v1 ==== 136975+0+0 (secure 0 0 0) 0x7f56080035f0 con 0x7f56201028c0 2024-03-28T18:11:10.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:10.362+0000 7f5617e00640 1 -- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56040a2970 msgr2=0x7f56040a4e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:11:10.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:10.362+0000 7f5617e00640 1 --2- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:6800/521857560,v1:172.21.15.37:6801/521857560] conn(0x7f56040a2970 0x7f56040a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:10.592 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:10.338+0000 7f105fef7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:11:10.876 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:10.710+0000 7f0fb6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:11:10.876 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:10.710+0000 7f0fb6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:11:10.876 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:10.710+0000 7f0fb6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:11:10.876 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:10.710+0000 7f0fb6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:11:10.876 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:10.710+0000 7f0fb6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:11:10.876 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:10.712+0000 7f0fb2800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:11:10.876 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:10.712+0000 7f0fb2800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:11:10.876 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:10.712+0000 7f0fb2800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:11:10.876 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:10.712+0000 7f0fb2800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:11:10.877 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:11:10.712+0000 7f0fb2800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:11:10.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: Active manager daemon x restarted 2024-03-28T18:11:10.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: Activating manager daemon x 2024-03-28T18:11:10.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T18:11:10.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: mgrmap e22: x(active, starting, since 0.019165s), standbys: y, z 2024-03-28T18:11:10.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:11:10.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:11:10.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:11:10.878 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:11:10.878 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:11:10.878 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:11:10.878 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:11:10.878 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:11:10.878 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:11:10.878 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:11:10.878 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:11:10.878 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:11:10.878 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:11:10.878 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:11:10.878 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:11:10.878 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T18:11:10.879 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T18:11:10.879 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T18:11:10.879 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:11:10.879 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:11:10.879 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:11:10.879 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:11:10.879 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:11:10.879 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:11:10.879 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:11:10.879 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:11:10.879 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:11:10.879 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:11:10.879 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:11:10.879 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:11:10.879 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T18:11:10.880 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T18:11:10.880 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T18:11:10.880 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: Manager daemon x is now available 2024-03-28T18:11:10.880 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:11:10.880 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:11:10.880 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T18:11:10.880 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T18:11:10.880 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T18:11:10.880 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:10 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T18:11:11.038 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: Active manager daemon x restarted 2024-03-28T18:11:11.038 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: Activating manager daemon x 2024-03-28T18:11:11.039 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T18:11:11.039 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: mgrmap e22: x(active, starting, since 0.019165s), standbys: y, z 2024-03-28T18:11:11.039 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:11:11.039 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:11:11.039 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:11:11.039 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:11:11.039 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:11:11.039 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:11:11.039 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:11:11.039 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:11:11.039 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:11:11.039 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:11:11.039 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:11:11.039 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:11:11.040 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:11:11.040 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:11:11.040 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:11:11.041 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T18:11:11.041 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T18:11:11.041 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T18:11:11.041 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:11:11.041 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:11:11.041 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:11:11.041 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:11:11.041 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:11:11.041 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:11:11.041 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:11:11.042 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:11:11.042 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:11:11.042 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:11:11.042 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:11:11.042 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:11:11.042 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T18:11:11.042 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T18:11:11.042 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T18:11:11.042 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: Manager daemon x is now available 2024-03-28T18:11:11.042 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:11:11.042 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:11:11.043 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T18:11:11.043 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T18:11:11.043 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T18:11:11.043 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:10 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T18:11:11.044 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: Active manager daemon x restarted 2024-03-28T18:11:11.044 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: Activating manager daemon x 2024-03-28T18:11:11.044 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T18:11:11.044 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: mgrmap e22: x(active, starting, since 0.019165s), standbys: y, z 2024-03-28T18:11:11.044 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:11:11.044 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:11:11.044 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:11:11.044 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:11:11.044 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:11:11.044 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:11:11.044 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:11:11.044 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:11:11.045 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:11:11.045 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:11:11.045 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:11:11.045 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:11:11.045 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:11:11.045 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:11:11.045 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:11:11.045 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T18:11:11.045 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T18:11:11.045 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T18:11:11.045 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:11:11.045 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:11:11.045 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:11:11.045 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:11:11.046 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:11:11.046 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:11:11.046 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:11:11.046 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:11:11.046 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:11:11.046 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:11:11.046 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:11:11.046 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:11:11.046 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T18:11:11.046 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T18:11:11.046 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T18:11:11.046 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: Manager daemon x is now available 2024-03-28T18:11:11.047 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:11:11.047 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:11:11.047 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T18:11:11.047 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T18:11:11.047 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T18:11:11.047 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:10 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T18:11:11.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.368+0000 7f5617e00640 1 -- 172.21.15.37:0/252872691 <== mon.1 v2:172.21.15.170:3300/0 10 ==== mgrmap(e 23) v1 ==== 137102+0+0 (secure 0 0 0) 0x7f56080b9f70 con 0x7f56201028c0 2024-03-28T18:11:11.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.369+0000 7f5617e00640 1 --2- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f56040686e0 0x7f5604068ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:11.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.369+0000 7f5617e00640 1 -- 172.21.15.37:0/252872691 --> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f560400ecb0 con 0x7f56040686e0 2024-03-28T18:11:11.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.372+0000 7f561ea00640 1 --2- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f56040686e0 0x7f5604068ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:11.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.378+0000 7f561ea00640 1 --2- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f56040686e0 0x7f5604068ac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5610009fd0 tx=0x7f5610009720 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:11.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.474+0000 7f5617e00640 1 -- 172.21.15.37:0/252872691 <== mgr.15207 v2:172.21.15.37:6800/683867216 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f560400ecb0 con 0x7f56040686e0 2024-03-28T18:11:11.475 INFO:teuthology.orchestra.run.smithi037.stdout:Schedule set for path / 2024-03-28T18:11:11.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.477+0000 7f5625a00640 1 -- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f56040686e0 msgr2=0x7f5604068ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:11.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.477+0000 7f5625a00640 1 --2- 172.21.15.37:0/252872691 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f56040686e0 0x7f5604068ac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5610009fd0 tx=0x7f5610009720 comp rx=0 tx=0).stop 2024-03-28T18:11:11.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.477+0000 7f5625a00640 1 -- 172.21.15.37:0/252872691 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56201028c0 msgr2=0x7f5620197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:11.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.477+0000 7f5625a00640 1 --2- 172.21.15.37:0/252872691 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56201028c0 0x7f5620197e50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5608009230 tx=0x7f5608005d50 comp rx=0 tx=0).stop 2024-03-28T18:11:11.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.477+0000 7f561f400640 1 -- 172.21.15.37:0/252872691 reap_dead start 2024-03-28T18:11:11.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.477+0000 7f5625a00640 1 -- 172.21.15.37:0/252872691 shutdown_connections 2024-03-28T18:11:11.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.477+0000 7f5625a00640 1 -- 172.21.15.37:0/252872691 >> 172.21.15.37:0/252872691 conn(0x7f56200fcc60 msgr2=0x7f5620105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:11.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.477+0000 7f5625a00640 1 -- 172.21.15.37:0/252872691 shutdown_connections 2024-03-28T18:11:11.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.477+0000 7f5625a00640 1 -- 172.21.15.37:0/252872691 wait complete. 2024-03-28T18:11:11.498 DEBUG:teuthology.orchestra.run.smithi037:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-28T18:11:11.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.683+0000 7fc1ad800640 1 -- 172.21.15.37:0/2112773779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a8103900 msgr2=0x7fc1a8103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:11.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.683+0000 7fc1ad800640 1 --2- 172.21.15.37:0/2112773779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a8103900 0x7fc1a8103d20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc198009ca0 tx=0x7fc19802c100 comp rx=0 tx=0).stop 2024-03-28T18:11:11.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.683+0000 7fc1ad800640 1 -- 172.21.15.37:0/2112773779 shutdown_connections 2024-03-28T18:11:11.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.683+0000 7fc1ad800640 1 --2- 172.21.15.37:0/2112773779 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1a81054b0 0x7fc1a810bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:11.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.683+0000 7fc1ad800640 1 --2- 172.21.15.37:0/2112773779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1a8104af0 0x7fc1a8104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:11.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.683+0000 7fc1ad800640 1 --2- 172.21.15.37:0/2112773779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a8103900 0x7fc1a8103d20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:11.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.683+0000 7fc1ad800640 1 -- 172.21.15.37:0/2112773779 >> 172.21.15.37:0/2112773779 conn(0x7fc1a80fee90 msgr2=0x7fc1a81012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:11.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.683+0000 7fc1ad800640 1 -- 172.21.15.37:0/2112773779 shutdown_connections 2024-03-28T18:11:11.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.683+0000 7fc1ad800640 1 -- 172.21.15.37:0/2112773779 wait complete. 2024-03-28T18:11:11.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.685+0000 7fc1ad800640 1 Processor -- start 2024-03-28T18:11:11.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.685+0000 7fc1ad800640 1 -- start start 2024-03-28T18:11:11.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.686+0000 7fc1ad800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a8103900 0x7fc1a8199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:11.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.686+0000 7fc1ad800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1a8104af0 0x7fc1a819a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:11.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.686+0000 7fc1ad800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1a81054b0 0x7fc1a81a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:11.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.686+0000 7fc1ad800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a810f550 con 0x7fc1a8104af0 2024-03-28T18:11:11.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.686+0000 7fc1ad800640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a810f3d0 con 0x7fc1a8103900 2024-03-28T18:11:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.686+0000 7fc1ad800640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a810f6d0 con 0x7fc1a81054b0 2024-03-28T18:11:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.686+0000 7fc1a6000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1a8104af0 0x7fc1a819a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.686+0000 7fc1a6a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a8103900 0x7fc1a8199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.686+0000 7fc1a6000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1a8104af0 0x7fc1a819a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45878/0 (socket says 172.21.15.37:45878) 2024-03-28T18:11:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.686+0000 7fc1a6000640 1 -- 172.21.15.37:0/3687821428 learned_addr learned my addr 172.21.15.37:0/3687821428 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.686+0000 7fc1a7400640 1 --2- 172.21.15.37:0/3687821428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1a81054b0 0x7fc1a81a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.686+0000 7fc1a6a00640 1 -- 172.21.15.37:0/3687821428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1a81054b0 msgr2=0x7fc1a81a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.686+0000 7fc1a6a00640 1 --2- 172.21.15.37:0/3687821428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1a81054b0 0x7fc1a81a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.686+0000 7fc1a6a00640 1 -- 172.21.15.37:0/3687821428 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1a8104af0 msgr2=0x7fc1a819a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.687+0000 7fc1a6a00640 1 --2- 172.21.15.37:0/3687821428 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1a8104af0 0x7fc1a819a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.687+0000 7fc1a6a00640 1 -- 172.21.15.37:0/3687821428 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a81a1790 con 0x7fc1a8103900 2024-03-28T18:11:11.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.687+0000 7fc1a6a00640 1 --2- 172.21.15.37:0/3687821428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a8103900 0x7fc1a8199b70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc198037040 tx=0x7fc198035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:11.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.687+0000 7fc19f400640 1 -- 172.21.15.37:0/3687821428 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc198039040 con 0x7fc1a8103900 2024-03-28T18:11:11.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.687+0000 7fc19f400640 1 -- 172.21.15.37:0/3687821428 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc198003970 con 0x7fc1a8103900 2024-03-28T18:11:11.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.687+0000 7fc19f400640 1 -- 172.21.15.37:0/3687821428 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1980044a0 con 0x7fc1a8103900 2024-03-28T18:11:11.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.687+0000 7fc1ad800640 1 -- 172.21.15.37:0/3687821428 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a81a1a20 con 0x7fc1a8103900 2024-03-28T18:11:11.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.688+0000 7fc1ad800640 1 -- 172.21.15.37:0/3687821428 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a81a1e90 con 0x7fc1a8103900 2024-03-28T18:11:11.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.689+0000 7fc1ad800640 1 -- 172.21.15.37:0/3687821428 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a810ac20 con 0x7fc1a8103900 2024-03-28T18:11:11.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.690+0000 7fc19f400640 1 -- 172.21.15.37:0/3687821428 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 23) v1 ==== 137102+0+0 (secure 0 0 0) 0x7fc198046030 con 0x7fc1a8103900 2024-03-28T18:11:11.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.692+0000 7fc19f400640 1 --2- 172.21.15.37:0/3687821428 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fc18c0a2730 0x7fc18c0a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:11.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.692+0000 7fc19f400640 1 -- 172.21.15.37:0/3687821428 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc1980f0340 con 0x7fc1a8103900 2024-03-28T18:11:11.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.692+0000 7fc1a6000640 1 --2- 172.21.15.37:0/3687821428 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fc18c0a2730 0x7fc18c0a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:11.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.693+0000 7fc1a6000640 1 --2- 172.21.15.37:0/3687821428 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fc18c0a2730 0x7fc18c0a4bf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc1a819b080 tx=0x7fc190005f90 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:11.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.696+0000 7fc19f400640 1 -- 172.21.15.37:0/3687821428 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc19804d060 con 0x7fc1a8103900 2024-03-28T18:11:11.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.915+0000 7fc1ad800640 1 -- 172.21.15.37:0/3687821428 --> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1a8061b80 con 0x7fc18c0a2730 2024-03-28T18:11:11.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.991+0000 7fc19f400640 1 -- 172.21.15.37:0/3687821428 <== mgr.15207 v2:172.21.15.37:6800/683867216 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fc1a8061b80 con 0x7fc18c0a2730 2024-03-28T18:11:11.993 INFO:teuthology.orchestra.run.smithi037.stdout:Retention added to path / 2024-03-28T18:11:11.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.994+0000 7fc1ad800640 1 -- 172.21.15.37:0/3687821428 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fc18c0a2730 msgr2=0x7fc18c0a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:11.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.994+0000 7fc1ad800640 1 --2- 172.21.15.37:0/3687821428 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fc18c0a2730 0x7fc18c0a4bf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc1a819b080 tx=0x7fc190005f90 comp rx=0 tx=0).stop 2024-03-28T18:11:11.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.994+0000 7fc1ad800640 1 -- 172.21.15.37:0/3687821428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a8103900 msgr2=0x7fc1a8199b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:11.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.994+0000 7fc1ad800640 1 --2- 172.21.15.37:0/3687821428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a8103900 0x7fc1a8199b70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc198037040 tx=0x7fc198035d10 comp rx=0 tx=0).stop 2024-03-28T18:11:11.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.994+0000 7fc1ad800640 1 -- 172.21.15.37:0/3687821428 shutdown_connections 2024-03-28T18:11:11.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.994+0000 7fc1ad800640 1 --2- 172.21.15.37:0/3687821428 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fc18c0a2730 0x7fc18c0a4bf0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:11.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.994+0000 7fc1ad800640 1 --2- 172.21.15.37:0/3687821428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1a81054b0 0x7fc1a81a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:11.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.994+0000 7fc1ad800640 1 --2- 172.21.15.37:0/3687821428 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1a8104af0 0x7fc1a819a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:11.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.994+0000 7fc1ad800640 1 --2- 172.21.15.37:0/3687821428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a8103900 0x7fc1a8199b70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:11.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.994+0000 7fc1ad800640 1 -- 172.21.15.37:0/3687821428 >> 172.21.15.37:0/3687821428 conn(0x7fc1a80fee90 msgr2=0x7fc1a8107df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:11.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.995+0000 7fc1ad800640 1 -- 172.21.15.37:0/3687821428 shutdown_connections 2024-03-28T18:11:11.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:11.995+0000 7fc1ad800640 1 -- 172.21.15.37:0/3687821428 wait complete. 2024-03-28T18:11:12.017 DEBUG:teuthology.orchestra.run.smithi037:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-28T18:11:12.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.207+0000 7fbd22e00640 1 -- 172.21.15.37:0/1609504431 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd1c101510 msgr2=0x7fbd1c101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:12.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.207+0000 7fbd22e00640 1 --2- 172.21.15.37:0/1609504431 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd1c101510 0x7fbd1c101930 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbd10009860 tx=0x7fbd1002bb70 comp rx=0 tx=0).stop 2024-03-28T18:11:12.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.207+0000 7fbd22e00640 1 -- 172.21.15.37:0/1609504431 shutdown_connections 2024-03-28T18:11:12.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.207+0000 7fbd22e00640 1 --2- 172.21.15.37:0/1609504431 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd1c1030c0 0x7fbd1c109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:12.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.207+0000 7fbd22e00640 1 --2- 172.21.15.37:0/1609504431 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd1c102700 0x7fbd1c102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:12.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.207+0000 7fbd22e00640 1 --2- 172.21.15.37:0/1609504431 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd1c101510 0x7fbd1c101930 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:12.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.207+0000 7fbd22e00640 1 -- 172.21.15.37:0/1609504431 >> 172.21.15.37:0/1609504431 conn(0x7fbd1c0fca80 msgr2=0x7fbd1c0feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:12.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.207+0000 7fbd22e00640 1 -- 172.21.15.37:0/1609504431 shutdown_connections 2024-03-28T18:11:12.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.207+0000 7fbd22e00640 1 -- 172.21.15.37:0/1609504431 wait complete. 2024-03-28T18:11:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.209+0000 7fbd22e00640 1 Processor -- start 2024-03-28T18:11:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.209+0000 7fbd22e00640 1 -- start start 2024-03-28T18:11:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.209+0000 7fbd22e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd1c101510 0x7fbd1c1977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.209+0000 7fbd22e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd1c102700 0x7fbd1c197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.209+0000 7fbd22e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd1c1030c0 0x7fbd1c19eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.209+0000 7fbd22e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1c10d170 con 0x7fbd1c101510 2024-03-28T18:11:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.209+0000 7fbd22e00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1c10cff0 con 0x7fbd1c1030c0 2024-03-28T18:11:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.209+0000 7fbd22e00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1c10d2f0 con 0x7fbd1c102700 2024-03-28T18:11:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.210+0000 7fbd21000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd1c102700 0x7fbd1c197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.210+0000 7fbd22400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd1c1030c0 0x7fbd1c19eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.210+0000 7fbd21000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd1c102700 0x7fbd1c197d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:42554/0 (socket says 172.21.15.37:42554) 2024-03-28T18:11:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.210+0000 7fbd21000640 1 -- 172.21.15.37:0/1870139082 learned_addr learned my addr 172.21.15.37:0/1870139082 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.210+0000 7fbd22400640 1 --2- 172.21.15.37:0/1870139082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd1c1030c0 0x7fbd1c19eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:44052/0 (socket says 172.21.15.37:44052) 2024-03-28T18:11:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.210+0000 7fbd21a00640 1 --2- 172.21.15.37:0/1870139082 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd1c101510 0x7fbd1c1977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.210+0000 7fbd21000640 1 -- 172.21.15.37:0/1870139082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd1c1030c0 msgr2=0x7fbd1c19eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.210+0000 7fbd21000640 1 --2- 172.21.15.37:0/1870139082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd1c1030c0 0x7fbd1c19eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.210+0000 7fbd21000640 1 -- 172.21.15.37:0/1870139082 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd1c101510 msgr2=0x7fbd1c1977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.210+0000 7fbd21000640 1 --2- 172.21.15.37:0/1870139082 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd1c101510 0x7fbd1c1977f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.210+0000 7fbd21000640 1 -- 172.21.15.37:0/1870139082 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd1c19f410 con 0x7fbd1c102700 2024-03-28T18:11:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.210+0000 7fbd22400640 1 --2- 172.21.15.37:0/1870139082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd1c1030c0 0x7fbd1c19eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:11:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.210+0000 7fbd21000640 1 --2- 172.21.15.37:0/1870139082 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd1c102700 0x7fbd1c197d30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800e9c0 tx=0x7fbd0800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:12.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.211+0000 7fbd1a000640 1 -- 172.21.15.37:0/1870139082 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0800ccc0 con 0x7fbd1c102700 2024-03-28T18:11:12.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.211+0000 7fbd1a000640 1 -- 172.21.15.37:0/1870139082 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd08005500 con 0x7fbd1c102700 2024-03-28T18:11:12.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.211+0000 7fbd22e00640 1 -- 172.21.15.37:0/1870139082 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd1c19f700 con 0x7fbd1c102700 2024-03-28T18:11:12.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.211+0000 7fbd1a000640 1 -- 172.21.15.37:0/1870139082 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd08010430 con 0x7fbd1c102700 2024-03-28T18:11:12.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.211+0000 7fbd22e00640 1 -- 172.21.15.37:0/1870139082 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd1c19fca0 con 0x7fbd1c102700 2024-03-28T18:11:12.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.212+0000 7fbd22e00640 1 -- 172.21.15.37:0/1870139082 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd1c108830 con 0x7fbd1c102700 2024-03-28T18:11:12.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.214+0000 7fbd1a000640 1 -- 172.21.15.37:0/1870139082 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 23) v1 ==== 137102+0+0 (secure 0 0 0) 0x7fbd080046c0 con 0x7fbd1c102700 2024-03-28T18:11:12.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.216+0000 7fbd1a000640 1 --2- 172.21.15.37:0/1870139082 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fbd040a2660 0x7fbd040a4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:12.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.216+0000 7fbd1a000640 1 -- 172.21.15.37:0/1870139082 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbd080ce9f0 con 0x7fbd1c102700 2024-03-28T18:11:12.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.216+0000 7fbd21a00640 1 --2- 172.21.15.37:0/1870139082 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fbd040a2660 0x7fbd040a4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:12.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.217+0000 7fbd21a00640 1 --2- 172.21.15.37:0/1870139082 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fbd040a2660 0x7fbd040a4b20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbd100349e0 tx=0x7fbd10034ab0 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:12.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.220+0000 7fbd1a000640 1 -- 172.21.15.37:0/1870139082 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbd0809bc50 con 0x7fbd1c102700 2024-03-28T18:11:12.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.469+0000 7fbd22e00640 1 -- 172.21.15.37:0/1870139082 --> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd1c061ae0 con 0x7fbd040a2660 2024-03-28T18:11:12.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.479+0000 7fbd1a000640 1 -- 172.21.15.37:0/1870139082 <== mgr.15207 v2:172.21.15.37:6800/683867216 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+298 (secure 0 0 0) 0x7fbd1c061ae0 con 0x7fbd040a2660 2024-03-28T18:11:12.481 INFO:teuthology.orchestra.run.smithi037.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-28T00:00:00", "created": "2024-03-28T18:11:11", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-28T18:11:12.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.484+0000 7fbd1a000640 1 -- 172.21.15.37:0/1870139082 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mgrmap(e 24) v1 ==== 137410+0+0 (secure 0 0 0) 0x7fbd08097bc0 con 0x7fbd1c102700 2024-03-28T18:11:12.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.484+0000 7fbd22e00640 1 -- 172.21.15.37:0/1870139082 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fbd040a2660 msgr2=0x7fbd040a4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:12.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.484+0000 7fbd22e00640 1 --2- 172.21.15.37:0/1870139082 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fbd040a2660 0x7fbd040a4b20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbd100349e0 tx=0x7fbd10034ab0 comp rx=0 tx=0).stop 2024-03-28T18:11:12.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.484+0000 7fbd22e00640 1 -- 172.21.15.37:0/1870139082 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd1c102700 msgr2=0x7fbd1c197d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:12.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.484+0000 7fbd22e00640 1 --2- 172.21.15.37:0/1870139082 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd1c102700 0x7fbd1c197d30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800e9c0 tx=0x7fbd0800ee80 comp rx=0 tx=0).stop 2024-03-28T18:11:12.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.485+0000 7fbd22e00640 1 -- 172.21.15.37:0/1870139082 shutdown_connections 2024-03-28T18:11:12.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.485+0000 7fbd22e00640 1 --2- 172.21.15.37:0/1870139082 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fbd040a2660 0x7fbd040a4b20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:12.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.485+0000 7fbd22e00640 1 --2- 172.21.15.37:0/1870139082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd1c1030c0 0x7fbd1c19eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:12.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.485+0000 7fbd22e00640 1 --2- 172.21.15.37:0/1870139082 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd1c102700 0x7fbd1c197d30 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:12.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.485+0000 7fbd22e00640 1 --2- 172.21.15.37:0/1870139082 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd1c101510 0x7fbd1c1977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:12.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.485+0000 7fbd22e00640 1 -- 172.21.15.37:0/1870139082 >> 172.21.15.37:0/1870139082 conn(0x7fbd1c0fca80 msgr2=0x7fbd1c105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:12.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.485+0000 7fbd22e00640 1 -- 172.21.15.37:0/1870139082 shutdown_connections 2024-03-28T18:11:12.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.485+0000 7fbd22e00640 1 -- 172.21.15.37:0/1870139082 wait complete. 2024-03-28T18:11:12.504 DEBUG:teuthology.orchestra.run.smithi037:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-28T18:11:12.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:12 smithi037 ceph-mon[31460]: mgrmap e23: x(active, since 1.02433s), standbys: y, z 2024-03-28T18:11:12.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:12 smithi037 ceph-mon[31460]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:11:12.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:12 smithi037 ceph-mon[31460]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:11:12.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:12 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:12.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:12 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:12.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.719+0000 7f648e800640 1 -- 172.21.15.37:0/2262611209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6488103720 msgr2=0x7f6488103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:12.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.719+0000 7f648e800640 1 --2- 172.21.15.37:0/2262611209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6488103720 0x7f6488103b40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f647c009900 tx=0x7f647c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:11:12.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.719+0000 7f648e800640 1 -- 172.21.15.37:0/2262611209 shutdown_connections 2024-03-28T18:11:12.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.719+0000 7f648e800640 1 --2- 172.21.15.37:0/2262611209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64881052d0 0x7f648810bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:12.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.719+0000 7f648e800640 1 --2- 172.21.15.37:0/2262611209 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6488104910 0x7f6488104d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:12.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.719+0000 7f648e800640 1 --2- 172.21.15.37:0/2262611209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6488103720 0x7f6488103b40 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:12.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.719+0000 7f648e800640 1 -- 172.21.15.37:0/2262611209 >> 172.21.15.37:0/2262611209 conn(0x7f64880fecb0 msgr2=0x7f64881010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:12.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.719+0000 7f648e800640 1 -- 172.21.15.37:0/2262611209 shutdown_connections 2024-03-28T18:11:12.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.719+0000 7f648e800640 1 -- 172.21.15.37:0/2262611209 wait complete. 2024-03-28T18:11:12.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.720+0000 7f648e800640 1 Processor -- start 2024-03-28T18:11:12.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.720+0000 7f648e800640 1 -- start start 2024-03-28T18:11:12.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.721+0000 7f648e800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6488103720 0x7f64881999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:12.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.721+0000 7f648e800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6488104910 0x7f6488199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:12.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.721+0000 7f648e800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64881052d0 0x7f64881a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:12.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.721+0000 7f648e800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648810f3f0 con 0x7f6488104910 2024-03-28T18:11:12.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.721+0000 7f648e800640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648810f270 con 0x7f6488103720 2024-03-28T18:11:12.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.721+0000 7f648e800640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648810f570 con 0x7f64881052d0 2024-03-28T18:11:12.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.721+0000 7f648d400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6488103720 0x7f64881999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:12.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.721+0000 7f648de00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64881052d0 0x7f64881a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:12.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.721+0000 7f648d400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6488103720 0x7f64881999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:44098/0 (socket says 172.21.15.37:44098) 2024-03-28T18:11:12.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.721+0000 7f648ca00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6488104910 0x7f6488199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:12.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.721+0000 7f648d400640 1 -- 172.21.15.37:0/195858591 learned_addr learned my addr 172.21.15.37:0/195858591 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:12.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.721+0000 7f648de00640 1 -- 172.21.15.37:0/195858591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6488103720 msgr2=0x7f64881999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:12.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.721+0000 7f648de00640 1 --2- 172.21.15.37:0/195858591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6488103720 0x7f64881999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:12.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.721+0000 7f648de00640 1 -- 172.21.15.37:0/195858591 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6488104910 msgr2=0x7f6488199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:12.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.721+0000 7f648de00640 1 --2- 172.21.15.37:0/195858591 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6488104910 0x7f6488199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:12.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.721+0000 7f648de00640 1 -- 172.21.15.37:0/195858591 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64881a1600 con 0x7f64881052d0 2024-03-28T18:11:12.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.722+0000 7f648de00640 1 --2- 172.21.15.37:0/195858591 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64881052d0 0x7f64881a0f90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6480009fd0 tx=0x7f648000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:12.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.722+0000 7f6486000640 1 -- 172.21.15.37:0/195858591 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648000ce20 con 0x7f64881052d0 2024-03-28T18:11:12.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.722+0000 7f6486000640 1 -- 172.21.15.37:0/195858591 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6480005020 con 0x7f64881052d0 2024-03-28T18:11:12.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.722+0000 7f6486000640 1 -- 172.21.15.37:0/195858591 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6480010720 con 0x7f64881052d0 2024-03-28T18:11:12.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.722+0000 7f648e800640 1 -- 172.21.15.37:0/195858591 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64881a18f0 con 0x7f64881052d0 2024-03-28T18:11:12.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.722+0000 7f648e800640 1 -- 172.21.15.37:0/195858591 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64881a1df0 con 0x7f64881052d0 2024-03-28T18:11:12.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.724+0000 7f648e800640 1 -- 172.21.15.37:0/195858591 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f648810a9e0 con 0x7f64881052d0 2024-03-28T18:11:12.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.725+0000 7f6486000640 1 -- 172.21.15.37:0/195858591 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 24) v1 ==== 137410+0+0 (secure 0 0 0) 0x7f6480005540 con 0x7f64881052d0 2024-03-28T18:11:12.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.726+0000 7f6486000640 1 --2- 172.21.15.37:0/195858591 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f64700a29e0 0x7f64700a4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:12.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.726+0000 7f6486000640 1 -- 172.21.15.37:0/195858591 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f64800cf430 con 0x7f64881052d0 2024-03-28T18:11:12.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.727+0000 7f648d400640 1 --2- 172.21.15.37:0/195858591 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f64700a29e0 0x7f64700a4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:12.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.727+0000 7f648d400640 1 --2- 172.21.15.37:0/195858591 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f64700a29e0 0x7f64700a4ea0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f647c006010 tx=0x7f647c02df90 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:12.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.730+0000 7f6486000640 1 -- 172.21.15.37:0/195858591 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f648009c160 con 0x7f64881052d0 2024-03-28T18:11:12.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:12 smithi059 ceph-mon[36190]: mgrmap e23: x(active, since 1.02433s), standbys: y, z 2024-03-28T18:11:12.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:12 smithi059 ceph-mon[36190]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:11:12.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:12 smithi059 ceph-mon[36190]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:11:12.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:12 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:12.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:12 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:12.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:12 smithi170 ceph-mon[36280]: mgrmap e23: x(active, since 1.02433s), standbys: y, z 2024-03-28T18:11:12.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:12 smithi170 ceph-mon[36280]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:11:12.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:12 smithi170 ceph-mon[36280]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:11:12.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:12 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:12.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:12 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:12.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.958+0000 7f648e800640 1 -- 172.21.15.37:0/195858591 --> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f6488061ae0 con 0x7f64700a29e0 2024-03-28T18:11:12.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.967+0000 7f6486000640 1 -- 172.21.15.37:0/195858591 <== mgr.15207 v2:172.21.15.37:6800/683867216 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f6488061ae0 con 0x7f64700a29e0 2024-03-28T18:11:12.968 INFO:teuthology.orchestra.run.smithi037.stdout:/ 1m 6m3h 2024-03-28T18:11:12.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.970+0000 7f648e800640 1 -- 172.21.15.37:0/195858591 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f64700a29e0 msgr2=0x7f64700a4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:12.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.970+0000 7f648e800640 1 --2- 172.21.15.37:0/195858591 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f64700a29e0 0x7f64700a4ea0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f647c006010 tx=0x7f647c02df90 comp rx=0 tx=0).stop 2024-03-28T18:11:12.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.970+0000 7f648e800640 1 -- 172.21.15.37:0/195858591 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64881052d0 msgr2=0x7f64881a0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:12.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.970+0000 7f648e800640 1 --2- 172.21.15.37:0/195858591 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64881052d0 0x7f64881a0f90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6480009fd0 tx=0x7f648000ef10 comp rx=0 tx=0).stop 2024-03-28T18:11:12.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.970+0000 7f648e800640 1 -- 172.21.15.37:0/195858591 shutdown_connections 2024-03-28T18:11:12.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.970+0000 7f648e800640 1 --2- 172.21.15.37:0/195858591 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f64700a29e0 0x7f64700a4ea0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:12.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.970+0000 7f648e800640 1 --2- 172.21.15.37:0/195858591 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64881052d0 0x7f64881a0f90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:12.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.970+0000 7f648e800640 1 --2- 172.21.15.37:0/195858591 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6488104910 0x7f6488199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:12.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.970+0000 7f648e800640 1 --2- 172.21.15.37:0/195858591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6488103720 0x7f64881999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:12.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.970+0000 7f648e800640 1 -- 172.21.15.37:0/195858591 >> 172.21.15.37:0/195858591 conn(0x7f64880fecb0 msgr2=0x7f6488107bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:12.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.970+0000 7f648e800640 1 -- 172.21.15.37:0/195858591 shutdown_connections 2024-03-28T18:11:12.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:12.970+0000 7f648e800640 1 -- 172.21.15.37:0/195858591 wait complete. 2024-03-28T18:11:12.995 DEBUG:teuthology.orchestra.run.smithi037:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-28T18:11:13.101 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-28T18:11:13.111 INFO:teuthology.run_tasks:Running task workunit... 2024-03-28T18:11:13.133 INFO:tasks.workunit:Pulling workunits from ref 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:11:13.134 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-28T18:11:13.134 INFO:tasks.workunit:timeout=3h 2024-03-28T18:11:13.134 INFO:tasks.workunit:cleanup=True 2024-03-28T18:11:13.134 DEBUG:teuthology.orchestra.run.smithi037:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-28T18:11:13.182 INFO:teuthology.orchestra.run.smithi037.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-28T18:11:13.182 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-28T18:11:13.182 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 76h/118d Inode: 1099511628283 Links: 2 2024-03-28T18:11:13.182 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:11:13.182 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-28T18:11:13.182 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-03-28 18:08:27.537865147 +0000 2024-03-28T18:11:13.182 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-03-28 18:08:27.537865147 +0000 2024-03-28T18:11:13.182 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-03-28 18:10:48.735221652 +0000 2024-03-28T18:11:13.182 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-03-28 18:08:27.537865147 +0000 2024-03-28T18:11:13.183 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-28T18:11:13.183 DEBUG:teuthology.orchestra.run.smithi037:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-28T18:11:13.312 DEBUG:teuthology.orchestra.run.smithi037:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:11:13.354 INFO:tasks.workunit.client.0.smithi037.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-28T18:11:13.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:13 smithi037 ceph-mon[31460]: [28/Mar/2024:18:11:11] ENGINE Bus STARTING 2024-03-28T18:11:13.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:13 smithi037 ceph-mon[31460]: [28/Mar/2024:18:11:12] ENGINE Serving on http://172.21.15.37:8765 2024-03-28T18:11:13.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:13 smithi037 ceph-mon[31460]: [28/Mar/2024:18:11:12] ENGINE Serving on https://172.21.15.37:7150 2024-03-28T18:11:13.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:13 smithi037 ceph-mon[31460]: [28/Mar/2024:18:11:12] ENGINE Bus STARTED 2024-03-28T18:11:13.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:13 smithi037 ceph-mon[31460]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:11:13.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:13 smithi037 ceph-mon[31460]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:11:13.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:13 smithi037 ceph-mon[31460]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-28T18:11:13.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:13 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:13.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:13 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:13.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:13 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:13.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:13 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:13 smithi059 ceph-mon[36190]: [28/Mar/2024:18:11:11] ENGINE Bus STARTING 2024-03-28T18:11:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:13 smithi059 ceph-mon[36190]: [28/Mar/2024:18:11:12] ENGINE Serving on http://172.21.15.37:8765 2024-03-28T18:11:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:13 smithi059 ceph-mon[36190]: [28/Mar/2024:18:11:12] ENGINE Serving on https://172.21.15.37:7150 2024-03-28T18:11:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:13 smithi059 ceph-mon[36190]: [28/Mar/2024:18:11:12] ENGINE Bus STARTED 2024-03-28T18:11:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:13 smithi059 ceph-mon[36190]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:11:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:13 smithi059 ceph-mon[36190]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:11:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:13 smithi059 ceph-mon[36190]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-28T18:11:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:13 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:13 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:13.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:13 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:13.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:13 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:13.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:13 smithi170 ceph-mon[36280]: [28/Mar/2024:18:11:11] ENGINE Bus STARTING 2024-03-28T18:11:13.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:13 smithi170 ceph-mon[36280]: [28/Mar/2024:18:11:12] ENGINE Serving on http://172.21.15.37:8765 2024-03-28T18:11:13.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:13 smithi170 ceph-mon[36280]: [28/Mar/2024:18:11:12] ENGINE Serving on https://172.21.15.37:7150 2024-03-28T18:11:13.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:13 smithi170 ceph-mon[36280]: [28/Mar/2024:18:11:12] ENGINE Bus STARTED 2024-03-28T18:11:13.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:13 smithi170 ceph-mon[36280]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:11:13.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:13 smithi170 ceph-mon[36280]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:11:13.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:13 smithi170 ceph-mon[36280]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-28T18:11:13.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:13 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:13.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:13 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:13.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:13 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:13.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:13 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='client.25459 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:11:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:14.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:14.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:14.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:14.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:14 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='client.25459 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:11:14.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:14.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:14.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:14.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:14.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:14.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:14.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:14 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='client.25459 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:11:14.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:14.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:14.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:14.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:14.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:14.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:14.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:14.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:14 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:15.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:15 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T18:11:15.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:15 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T18:11:15.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:15 smithi059 ceph-mon[36190]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:11:15.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:15 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T18:11:15.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:15 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T18:11:15.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:15 smithi059 ceph-mon[36190]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-28T18:11:15.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:15 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T18:11:15.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:15 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T18:11:15.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:15 smithi170 ceph-mon[36280]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:11:15.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:15 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T18:11:15.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:15 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T18:11:15.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:15 smithi170 ceph-mon[36280]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-28T18:11:15.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:15 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T18:11:15.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:15 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T18:11:15.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:15 smithi037 ceph-mon[31460]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:11:15.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:15 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T18:11:15.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:15 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T18:11:15.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:15 smithi037 ceph-mon[31460]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-28T18:11:17.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:17 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:17.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:17 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:17.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:17 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:17 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:17 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:17 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:17 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:17 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:17 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:17 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:17 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T18:11:17.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:17 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:11:17.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:17 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:11:17.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:17 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:11:17.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:17 smithi059 ceph-mon[36190]: Updating smithi037:/etc/ceph/ceph.conf 2024-03-28T18:11:17.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:17 smithi059 ceph-mon[36190]: Updating smithi059:/etc/ceph/ceph.conf 2024-03-28T18:11:17.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:17 smithi059 ceph-mon[36190]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-28T18:11:17.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:17 smithi059 ceph-mon[36190]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:11:17.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:17 smithi059 ceph-mon[36190]: Updating smithi059:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:11:17.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:17 smithi059 ceph-mon[36190]: Updating smithi170:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:11:17.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:17 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:17.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:17 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:17.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:17 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:17 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:17 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:17 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:17 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:17 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:17 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:17 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:17 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T18:11:17.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:17 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:11:17.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:17 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:11:17.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:17 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:11:17.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:17 smithi170 ceph-mon[36280]: Updating smithi037:/etc/ceph/ceph.conf 2024-03-28T18:11:17.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:17 smithi170 ceph-mon[36280]: Updating smithi059:/etc/ceph/ceph.conf 2024-03-28T18:11:17.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:17 smithi170 ceph-mon[36280]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-28T18:11:17.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:17 smithi170 ceph-mon[36280]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:11:17.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:17 smithi170 ceph-mon[36280]: Updating smithi059:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:11:17.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:17 smithi170 ceph-mon[36280]: Updating smithi170:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:11:17.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:17 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:17.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:17 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:17.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:17 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:17 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:17 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:17 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:17 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:17 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:17 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:17 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:11:17.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:17 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T18:11:17.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:17 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:11:17.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:17 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:11:17.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:17 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:11:17.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:17 smithi037 ceph-mon[31460]: Updating smithi037:/etc/ceph/ceph.conf 2024-03-28T18:11:17.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:17 smithi037 ceph-mon[31460]: Updating smithi059:/etc/ceph/ceph.conf 2024-03-28T18:11:17.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:17 smithi037 ceph-mon[31460]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-28T18:11:17.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:17 smithi037 ceph-mon[31460]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:11:17.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:17 smithi037 ceph-mon[31460]: Updating smithi059:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:11:17.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:17 smithi037 ceph-mon[31460]: Updating smithi170:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:11:18.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:18 smithi037 ceph-mon[31460]: Updating smithi059:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:11:18.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:18 smithi037 ceph-mon[31460]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:11:18.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:18 smithi037 ceph-mon[31460]: Updating smithi037:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:11:18.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:18 smithi037 ceph-mon[31460]: Updating smithi059:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:11:18.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:18 smithi037 ceph-mon[31460]: Updating smithi170:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:11:18.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:18 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:18.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:18 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:18.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:18 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:18.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:18 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:18.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:18 smithi037 ceph-mon[31460]: Updating smithi037:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:11:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:18 smithi059 ceph-mon[36190]: Updating smithi059:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:11:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:18 smithi059 ceph-mon[36190]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:11:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:18 smithi059 ceph-mon[36190]: Updating smithi037:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:11:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:18 smithi059 ceph-mon[36190]: Updating smithi059:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:11:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:18 smithi059 ceph-mon[36190]: Updating smithi170:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:11:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:18 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:18 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:18 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:18 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:18 smithi059 ceph-mon[36190]: Updating smithi037:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:11:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:18 smithi170 ceph-mon[36280]: Updating smithi059:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:11:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:18 smithi170 ceph-mon[36280]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:11:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:18 smithi170 ceph-mon[36280]: Updating smithi037:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:11:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:18 smithi170 ceph-mon[36280]: Updating smithi059:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:11:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:18 smithi170 ceph-mon[36280]: Updating smithi170:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:11:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:18 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:18 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:18 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:18 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:18 smithi170 ceph-mon[36280]: Updating smithi037:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:11:19.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:19 smithi037 ceph-mon[31460]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 133 KiB/s wr, 76 op/s 2024-03-28T18:11:19.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:19 smithi037 ceph-mon[31460]: Updating smithi037:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:11:19.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:19 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:19.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:19 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:19.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:19 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:19.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:19 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:11:19.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:19 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:11:19.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:19 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:11:19.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:19 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:19 smithi059 ceph-mon[36190]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 133 KiB/s wr, 76 op/s 2024-03-28T18:11:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:19 smithi059 ceph-mon[36190]: Updating smithi037:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:11:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:19 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:19 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:19 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:19 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:11:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:19 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:11:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:19 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:11:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:19 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:20.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:19 smithi170 ceph-mon[36280]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 133 KiB/s wr, 76 op/s 2024-03-28T18:11:20.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:19 smithi170 ceph-mon[36280]: Updating smithi037:/var/lib/ceph/b5ad257c-ed2c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:11:20.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:19 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:20.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:19 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:20.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:19 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:20.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:19 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:11:20.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:19 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:11:20.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:19 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:11:20.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:19 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:11:21.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:21 smithi037 ceph-mon[31460]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-28T18:11:22.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:21 smithi059 ceph-mon[36190]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-28T18:11:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:21 smithi170 ceph-mon[36280]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-28T18:11:23.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:23 smithi037 ceph-mon[31460]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-28T18:11:24.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:23 smithi059 ceph-mon[36190]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-28T18:11:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:23 smithi170 ceph-mon[36280]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-28T18:11:25.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:25 smithi037 ceph-mon[31460]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-28T18:11:26.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:25 smithi059 ceph-mon[36190]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-28T18:11:26.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:25 smithi170 ceph-mon[36280]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-28T18:11:27.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:27 smithi037 ceph-mon[31460]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-28T18:11:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:27 smithi059 ceph-mon[36190]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-28T18:11:28.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:27 smithi170 ceph-mon[36280]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-28T18:11:29.708 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:11:29.708 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:11:29.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:29 smithi037 ceph-mon[31460]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-03-28T18:11:29.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.950+0000 7f5a05a00640 1 -- 172.21.15.37:0/2185776018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a001026d0 msgr2=0x7f5a00102ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:29.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.950+0000 7f5a05a00640 1 --2- 172.21.15.37:0/2185776018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a001026d0 0x7f5a00102ab0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f59f0009ca0 tx=0x7f59f002c190 comp rx=0 tx=0).stop 2024-03-28T18:11:29.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.950+0000 7f5a05a00640 1 -- 172.21.15.37:0/2185776018 shutdown_connections 2024-03-28T18:11:29.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.950+0000 7f5a05a00640 1 --2- 172.21.15.37:0/2185776018 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a0010baf0 0x7f5a0010df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.950+0000 7f5a05a00640 1 --2- 172.21.15.37:0/2185776018 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a00102ff0 0x7f5a0010b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.950+0000 7f5a05a00640 1 --2- 172.21.15.37:0/2185776018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a001026d0 0x7f5a00102ab0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.950+0000 7f5a05a00640 1 -- 172.21.15.37:0/2185776018 >> 172.21.15.37:0/2185776018 conn(0x7f5a000fbef0 msgr2=0x7f5a000fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:29.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.950+0000 7f5a05a00640 1 -- 172.21.15.37:0/2185776018 shutdown_connections 2024-03-28T18:11:29.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.950+0000 7f5a05a00640 1 -- 172.21.15.37:0/2185776018 wait complete. 2024-03-28T18:11:29.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.951+0000 7f5a05a00640 1 Processor -- start 2024-03-28T18:11:29.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.952+0000 7f5a05a00640 1 -- start start 2024-03-28T18:11:29.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.952+0000 7f5a05a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a001026d0 0x7f5a0019a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:29.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.952+0000 7f5a05a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a00102ff0 0x7f5a0019a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:29.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.952+0000 7f5a05a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a0010baf0 0x7f5a0019e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:29.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.952+0000 7f5a05a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a001118b0 con 0x7f5a001026d0 2024-03-28T18:11:29.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.952+0000 7f5a05a00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a00111730 con 0x7f5a0010baf0 2024-03-28T18:11:29.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.952+0000 7f5a05a00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a00111a30 con 0x7f5a00102ff0 2024-03-28T18:11:29.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.952+0000 7f59ffe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a001026d0 0x7f5a0019a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:29.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.952+0000 7f59f7400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a00102ff0 0x7f5a0019a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:29.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.952+0000 7f5a05000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a0010baf0 0x7f5a0019e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:29.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.952+0000 7f59f7400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a00102ff0 0x7f5a0019a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:54824/0 (socket says 172.21.15.37:54824) 2024-03-28T18:11:29.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.952+0000 7f59f7400640 1 -- 172.21.15.37:0/3899521296 learned_addr learned my addr 172.21.15.37:0/3899521296 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:29.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.952+0000 7f59ffe00640 1 --2- 172.21.15.37:0/3899521296 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a001026d0 0x7f5a0019a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41902/0 (socket says 172.21.15.37:41902) 2024-03-28T18:11:29.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.953+0000 7f5a05000640 1 -- 172.21.15.37:0/3899521296 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a00102ff0 msgr2=0x7f5a0019a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:29.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.953+0000 7f5a05000640 1 --2- 172.21.15.37:0/3899521296 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a00102ff0 0x7f5a0019a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.953+0000 7f5a05000640 1 -- 172.21.15.37:0/3899521296 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a001026d0 msgr2=0x7f5a0019a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:29.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.953+0000 7f5a05000640 1 --2- 172.21.15.37:0/3899521296 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a001026d0 0x7f5a0019a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.953+0000 7f5a05000640 1 -- 172.21.15.37:0/3899521296 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0019f000 con 0x7f5a0010baf0 2024-03-28T18:11:29.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.953+0000 7f59f7400640 1 --2- 172.21.15.37:0/3899521296 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a00102ff0 0x7f5a0019a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:11:29.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.953+0000 7f5a05000640 1 --2- 172.21.15.37:0/3899521296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a0010baf0 0x7f5a0019e8d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f59f800e970 tx=0x7f59f800ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:29.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.953+0000 7f59fd600640 1 -- 172.21.15.37:0/3899521296 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f800ccc0 con 0x7f5a0010baf0 2024-03-28T18:11:29.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.953+0000 7f5a05a00640 1 -- 172.21.15.37:0/3899521296 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0019f2f0 con 0x7f5a0010baf0 2024-03-28T18:11:29.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.953+0000 7f59fd600640 1 -- 172.21.15.37:0/3899521296 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59f8005020 con 0x7f5a0010baf0 2024-03-28T18:11:29.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.953+0000 7f59fd600640 1 -- 172.21.15.37:0/3899521296 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f8004a00 con 0x7f5a0010baf0 2024-03-28T18:11:29.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.954+0000 7f5a05a00640 1 -- 172.21.15.37:0/3899521296 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a001a6b80 con 0x7f5a0010baf0 2024-03-28T18:11:29.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.957+0000 7f59fd600640 1 -- 172.21.15.37:0/3899521296 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 25) v1 ==== 137466+0+0 (secure 0 0 0) 0x7f59f80051d0 con 0x7f5a0010baf0 2024-03-28T18:11:29.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.957+0000 7f5a09dce740 1 Processor -- start 2024-03-28T18:11:29.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.957+0000 7f5a09dce740 1 -- start start 2024-03-28T18:11:29.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.957+0000 7f5a09dce740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc68edd0 0x556bbc68db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:29.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.957+0000 7f5a09dce740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc68e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:29.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.957+0000 7f5a09dce740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc697460 0x556bbc699850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:29.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.957+0000 7f5a09dce740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bbc5f2250 con 0x556bbc697460 2024-03-28T18:11:29.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.957+0000 7f5a09dce740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bbc59aa30 con 0x556bbc68e0f0 2024-03-28T18:11:29.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.957+0000 7f5a09dce740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bbc5ee2f0 con 0x556bbc68edd0 2024-03-28T18:11:29.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.958+0000 7f59ffe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc68e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:29.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.958+0000 7f59f7400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc68edd0 0x556bbc68db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:29.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.958+0000 7f59ffe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc68e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:56864/0 (socket says 172.21.15.37:56864) 2024-03-28T18:11:29.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.958+0000 7f59ffe00640 1 -- 172.21.15.37:0/2861273195 learned_addr learned my addr 172.21.15.37:0/2861273195 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:29.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.958+0000 7f59f7400640 1 --2- 172.21.15.37:0/2861273195 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc697460 0x556bbc699850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:29.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.958+0000 7f59ffe00640 1 -- 172.21.15.37:0/2861273195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc68edd0 msgr2=0x556bbc68db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:29.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.958+0000 7f59ffe00640 1 --2- 172.21.15.37:0/2861273195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc68edd0 0x556bbc68db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.958+0000 7f59ffe00640 1 -- 172.21.15.37:0/2861273195 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc697460 msgr2=0x556bbc699850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:29.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.958+0000 7f59ffe00640 1 --2- 172.21.15.37:0/2861273195 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc697460 0x556bbc699850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.958+0000 7f59ffe00640 1 -- 172.21.15.37:0/2861273195 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556bbc585690 con 0x556bbc68e0f0 2024-03-28T18:11:29.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.959+0000 7f59ffe00640 1 --2- 172.21.15.37:0/2861273195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc68e570 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f59f0004af0 tx=0x7f59f0005fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ee67b81dfd4a185 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:29.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.959+0000 7f59fd600640 1 --2- 172.21.15.37:0/3899521296 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f59e40a2a00 0x7f59e40a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:29.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.959+0000 7f59f5600640 1 -- 172.21.15.37:0/2861273195 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f0043070 con 0x556bbc68e0f0 2024-03-28T18:11:29.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.959+0000 7f59f5600640 1 -- 172.21.15.37:0/2861273195 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59f0003e50 con 0x556bbc68e0f0 2024-03-28T18:11:29.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.959+0000 7f59f5600640 1 -- 172.21.15.37:0/2861273195 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f0039040 con 0x556bbc68e0f0 2024-03-28T18:11:29.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.959+0000 7f59fd600640 1 -- 172.21.15.37:0/3899521296 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f59f80cef10 con 0x7f5a0010baf0 2024-03-28T18:11:29.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.959+0000 7f59f7400640 1 --2- 172.21.15.37:0/3899521296 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f59e40a2a00 0x7f59e40a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:29.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.960+0000 7f59f7400640 1 --2- 172.21.15.37:0/3899521296 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f59e40a2a00 0x7f59e40a4ec0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f59e8005fa0 tx=0x7f59e800b3d0 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:29.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.960+0000 7f5a09dce740 1 -- 172.21.15.37:0/2861273195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 msgr2=0x556bbc68e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:29.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.960+0000 7f5a09dce740 1 --2- 172.21.15.37:0/2861273195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc68e570 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f59f0004af0 tx=0x7f59f0005fc0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.960+0000 7f5a09dce740 1 -- 172.21.15.37:0/2861273195 shutdown_connections 2024-03-28T18:11:29.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.960+0000 7f5a09dce740 1 --2- 172.21.15.37:0/2861273195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc68e570 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.960+0000 7f5a09dce740 1 --2- 172.21.15.37:0/2861273195 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc697460 0x556bbc699850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.960+0000 7f5a09dce740 1 --2- 172.21.15.37:0/2861273195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc68edd0 0x556bbc68db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.960+0000 7f5a09dce740 1 -- 172.21.15.37:0/2861273195 >> 172.21.15.37:0/2861273195 conn(0x556bbc655200 msgr2=0x556bbc688b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:29.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.960+0000 7f5a09dce740 1 -- 172.21.15.37:0/2861273195 shutdown_connections 2024-03-28T18:11:29.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.960+0000 7f5a09dce740 1 -- 172.21.15.37:0/2861273195 wait complete. 2024-03-28T18:11:29.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.961+0000 7f5a09dce740 1 -- start start 2024-03-28T18:11:29.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.961+0000 7f5a09dce740 1 Processor -- start 2024-03-28T18:11:29.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.961+0000 7f5a09dce740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc696ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:29.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.961+0000 7f5a09dce740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc68edd0 0x556bbc693e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:29.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.962+0000 7f5a09dce740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc697460 0x556bbc6943b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:29.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.962+0000 7f5a09dce740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bbc69a270 con 0x556bbc68edd0 2024-03-28T18:11:29.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.962+0000 7f5a09dce740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bbc5ade80 con 0x556bbc68e0f0 2024-03-28T18:11:29.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.962+0000 7f5a09dce740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bbc69a3f0 con 0x556bbc697460 2024-03-28T18:11:29.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.962+0000 7f59ffe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc696ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:29.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.962+0000 7f5a05000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc697460 0x556bbc6943b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:29.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.962+0000 7f59f7400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc68edd0 0x556bbc693e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:29.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.962+0000 7f5a09dce740 10 client.0 _ll_register_callbacks cb 0x556bbc5c0130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:11:29.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.962+0000 7f59ffe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc696ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:56866/0 (socket says 172.21.15.37:56866) 2024-03-28T18:11:29.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.962+0000 7f5a05000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc697460 0x556bbc6943b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:54844/0 (socket says 172.21.15.37:54844) 2024-03-28T18:11:29.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.962+0000 7f59ffe00640 1 -- 172.21.15.37:0/2111704963 learned_addr learned my addr 172.21.15.37:0/2111704963 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:29.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.962+0000 7f5a05000640 1 -- 172.21.15.37:0/2111704963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 msgr2=0x556bbc696ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:29.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.962+0000 7f5a05000640 1 --2- 172.21.15.37:0/2111704963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc696ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.962+0000 7f5a05000640 1 -- 172.21.15.37:0/2111704963 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc68edd0 msgr2=0x556bbc693e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:29.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.962+0000 7f5a05000640 1 --2- 172.21.15.37:0/2111704963 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc68edd0 0x556bbc693e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.962+0000 7f5a05000640 1 -- 172.21.15.37:0/2111704963 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556bbc3d07d0 con 0x556bbc697460 2024-03-28T18:11:29.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.962+0000 7f59ffe00640 1 --2- 172.21.15.37:0/2111704963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc696ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:11:29.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.963+0000 7f5a05000640 1 --2- 172.21.15.37:0/2111704963 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc697460 0x556bbc6943b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f59f80cc570 tx=0x7f59f80055e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e7cfe568649d424 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:29.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.963+0000 7f59ee000640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:11:29.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.963+0000 7f59ee000640 1 -- 172.21.15.37:0/2111704963 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f801c040 con 0x556bbc697460 2024-03-28T18:11:29.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.963+0000 7f59ee000640 1 -- 172.21.15.37:0/2111704963 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f59f80a36e0 con 0x556bbc697460 2024-03-28T18:11:29.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.963+0000 7f5a09dce740 1 -- 172.21.15.37:0/2111704963 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f59f0003e50 con 0x556bbc697460 2024-03-28T18:11:29.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.963+0000 7f59ee000640 1 -- 172.21.15.37:0/2111704963 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59f80cca30 con 0x556bbc697460 2024-03-28T18:11:29.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.963+0000 7f5a09dce740 10 client.25471 fetch_fsmap learned FSMap version 23 2024-03-28T18:11:29.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.963+0000 7f5a09dce740 1 -- 172.21.15.37:0/2111704963 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556bbc3063f0 con 0x556bbc697460 2024-03-28T18:11:29.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.963+0000 7f59ee000640 1 -- 172.21.15.37:0/2111704963 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59f0003e50 con 0x556bbc697460 2024-03-28T18:11:29.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.964+0000 7f59ee000640 1 -- 172.21.15.37:0/2111704963 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f59f8004ba0 con 0x556bbc697460 2024-03-28T18:11:29.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.964+0000 7f5a09dce740 10 client.25471 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:11:29.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.964+0000 7f5a09dce740 10 client.25471 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:11:29.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.964+0000 7f5a09dce740 20 client.25471 populate_metadata read hostname 'smithi037' 2024-03-28T18:11:29.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.964+0000 7f5a09dce740 1 --2- 172.21.15.37:0/2111704963 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556bbc6a72d0 0x556bbc6a96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:29.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.964+0000 7f5a09dce740 4 client.25471 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:11:29.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.964+0000 7f5a09dce740 1 -- 172.21.15.37:0/2111704963 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556bbc5ff790 con 0x556bbc6a72d0 2024-03-28T18:11:29.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.964+0000 7f59ffe00640 1 --2- 172.21.15.37:0/2111704963 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556bbc6a72d0 0x556bbc6a96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:29.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.965+0000 7f59ffe00640 1 --2- 172.21.15.37:0/2111704963 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556bbc6a72d0 0x556bbc6a96e0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=931e672315931b91 server_cookie=9d1f45e9ebd3605a in_seq=0 out_seq=0 2024-03-28T18:11:29.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.965+0000 7f59ee000640 10 client.25471 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:11:29.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.965+0000 7f59ee000640 1 -- 172.21.15.37:0/2111704963 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556bbc5ff790 con 0x556bbc6a72d0 2024-03-28T18:11:29.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.965+0000 7f59ee000640 10 client.25471 handle_command_reply: tid=0 2024-03-28T18:11:29.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.965+0000 7f5a09dce740 1 client.25471 shutdown 2024-03-28T18:11:29.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.966+0000 7f5a09dce740 1 -- 172.21.15.37:0/2111704963 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc697460 msgr2=0x556bbc6943b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:29.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.966+0000 7f5a09dce740 1 --2- 172.21.15.37:0/2111704963 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc697460 0x556bbc6943b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f59f80cc570 tx=0x7f59f80055e0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.966+0000 7f5a09dce740 1 -- 172.21.15.37:0/2111704963 shutdown_connections 2024-03-28T18:11:29.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.966+0000 7f5a09dce740 1 --2- 172.21.15.37:0/2111704963 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556bbc6a72d0 0x556bbc6a96e0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.966+0000 7f5a09dce740 1 --2- 172.21.15.37:0/2111704963 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc697460 0x556bbc6943b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.966+0000 7f5a09dce740 1 --2- 172.21.15.37:0/2111704963 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc68edd0 0x556bbc693e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.966+0000 7f59ee000640 0 client.25471 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:11:29.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.966+0000 7f5a09dce740 1 --2- 172.21.15.37:0/2111704963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc696ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.966+0000 7f5a09dce740 1 -- 172.21.15.37:0/2111704963 >> 172.21.15.37:0/2111704963 conn(0x556bbc655200 msgr2=0x556bbc68a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:29.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.966+0000 7f5a09dce740 1 -- 172.21.15.37:0/2111704963 shutdown_connections 2024-03-28T18:11:29.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.966+0000 7f5a09dce740 1 -- 172.21.15.37:0/2111704963 wait complete. 2024-03-28T18:11:29.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.966+0000 7f5a09dce740 20 client.25471 trim_cache size 0 max 16384 2024-03-28T18:11:29.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.984+0000 7f5a09dce740 1 Processor -- start 2024-03-28T18:11:29.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.984+0000 7f5a09dce740 1 -- start start 2024-03-28T18:11:29.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.984+0000 7f5a09dce740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc698850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:29.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.984+0000 7f5a09dce740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc68edd0 0x556bbc698d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:29.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.984+0000 7f5a09dce740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc697460 0x556bbc6965a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:29.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.984+0000 7f5a09dce740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a001b1750 con 0x556bbc68edd0 2024-03-28T18:11:29.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.984+0000 7f5a09dce740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a001af250 con 0x556bbc68e0f0 2024-03-28T18:11:29.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.984+0000 7f5a09dce740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a001ad2f0 con 0x556bbc697460 2024-03-28T18:11:29.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.984+0000 7f59ffe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc698850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:29.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.984+0000 7f59ffe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc698850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:56878/0 (socket says 172.21.15.37:56878) 2024-03-28T18:11:29.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.984+0000 7f59ffe00640 1 -- 172.21.15.37:0/1950636063 learned_addr learned my addr 172.21.15.37:0/1950636063 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:29.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.984+0000 7f59f7400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc68edd0 0x556bbc698d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:29.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.985+0000 7f5a05000640 1 --2- 172.21.15.37:0/1950636063 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc697460 0x556bbc6965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:29.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.985+0000 7f59ffe00640 1 -- 172.21.15.37:0/1950636063 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc697460 msgr2=0x556bbc6965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:29.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.985+0000 7f59ffe00640 1 --2- 172.21.15.37:0/1950636063 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc697460 0x556bbc6965a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.985+0000 7f59ffe00640 1 -- 172.21.15.37:0/1950636063 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc68edd0 msgr2=0x556bbc698d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:29.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.985+0000 7f59ffe00640 1 --2- 172.21.15.37:0/1950636063 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc68edd0 0x556bbc698d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.985+0000 7f59ffe00640 1 -- 172.21.15.37:0/1950636063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556bbc2c60f0 con 0x556bbc68e0f0 2024-03-28T18:11:29.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.985+0000 7f59f7400640 1 --2- 172.21.15.37:0/1950636063 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc68edd0 0x556bbc698d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:11:29.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.985+0000 7f59ffe00640 1 --2- 172.21.15.37:0/1950636063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc698850 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f59f0004660 tx=0x7f59f002c190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e21adfdb1c55805e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:29.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.985+0000 7f59f4c00640 1 -- 172.21.15.37:0/1950636063 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f0057070 con 0x556bbc68e0f0 2024-03-28T18:11:29.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.985+0000 7f59f4c00640 1 -- 172.21.15.37:0/1950636063 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59f0032340 con 0x556bbc68e0f0 2024-03-28T18:11:29.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.985+0000 7f59f4c00640 1 -- 172.21.15.37:0/1950636063 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f0039040 con 0x556bbc68e0f0 2024-03-28T18:11:29.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.986+0000 7f5a09dce740 1 -- 172.21.15.37:0/1950636063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 msgr2=0x556bbc698850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:29.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.986+0000 7f5a09dce740 1 --2- 172.21.15.37:0/1950636063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc698850 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f59f0004660 tx=0x7f59f002c190 comp rx=0 tx=0).stop 2024-03-28T18:11:29.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.986+0000 7f5a09dce740 1 -- 172.21.15.37:0/1950636063 shutdown_connections 2024-03-28T18:11:29.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.986+0000 7f5a09dce740 1 --2- 172.21.15.37:0/1950636063 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc697460 0x556bbc6965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.986+0000 7f5a09dce740 1 --2- 172.21.15.37:0/1950636063 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc68edd0 0x556bbc698d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.986+0000 7f5a09dce740 1 --2- 172.21.15.37:0/1950636063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc698850 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.986+0000 7f5a09dce740 1 -- 172.21.15.37:0/1950636063 >> 172.21.15.37:0/1950636063 conn(0x556bbc655200 msgr2=0x556bbc692780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:29.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.986+0000 7f5a09dce740 1 -- 172.21.15.37:0/1950636063 shutdown_connections 2024-03-28T18:11:29.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.987+0000 7f5a09dce740 1 -- 172.21.15.37:0/1950636063 wait complete. 2024-03-28T18:11:29.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.987+0000 7f5a09dce740 1 -- start start 2024-03-28T18:11:29.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.987+0000 7f5a09dce740 1 Processor -- start 2024-03-28T18:11:29.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.988+0000 7f5a09dce740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc690c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:29.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.988+0000 7f5a09dce740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc68edd0 0x556bbc6911b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:29.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.988+0000 7f5a09dce740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc697460 0x556bbc701b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:29.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.988+0000 7f5a09dce740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a001ad8f0 con 0x556bbc68edd0 2024-03-28T18:11:29.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.988+0000 7f5a09dce740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a001aebf0 con 0x556bbc68e0f0 2024-03-28T18:11:29.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.988+0000 7f5a09dce740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a001adf30 con 0x556bbc697460 2024-03-28T18:11:29.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.988+0000 7f5a09dce740 10 client.0 _ll_register_callbacks cb 0x7f59f80cc400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:11:29.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.988+0000 7f59ffe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc690c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:29.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.988+0000 7f59ffe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc690c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:56884/0 (socket says 172.21.15.37:56884) 2024-03-28T18:11:29.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.988+0000 7f59ffe00640 1 -- 172.21.15.37:0/2818170522 learned_addr learned my addr 172.21.15.37:0/2818170522 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:29.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.988+0000 7f59f7400640 1 --2- 172.21.15.37:0/2818170522 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc68edd0 0x556bbc6911b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:29.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.988+0000 7f5a05000640 1 --2- 172.21.15.37:0/2818170522 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc697460 0x556bbc701b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:29.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.989+0000 7f59ffe00640 1 -- 172.21.15.37:0/2818170522 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc697460 msgr2=0x556bbc701b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:29.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.989+0000 7f59ffe00640 1 --2- 172.21.15.37:0/2818170522 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc697460 0x556bbc701b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.989+0000 7f59ffe00640 1 -- 172.21.15.37:0/2818170522 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc68edd0 msgr2=0x556bbc6911b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:29.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.989+0000 7f59ffe00640 1 --2- 172.21.15.37:0/2818170522 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc68edd0 0x556bbc6911b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.989+0000 7f59ffe00640 1 -- 172.21.15.37:0/2818170522 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556bbc7020e0 con 0x556bbc68e0f0 2024-03-28T18:11:29.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.989+0000 7f5a05000640 1 --2- 172.21.15.37:0/2818170522 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc697460 0x556bbc701b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:11:29.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.989+0000 7f59f7400640 1 --2- 172.21.15.37:0/2818170522 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc68edd0 0x556bbc6911b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:11:29.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.989+0000 7f59ffe00640 1 --2- 172.21.15.37:0/2818170522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc690c70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f59f002c6d0 tx=0x7f59f00046b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90da2961823f3acf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:29.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.989+0000 7f59ee000640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:11:29.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.989+0000 7f59ee000640 1 -- 172.21.15.37:0/2818170522 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f0057070 con 0x556bbc68e0f0 2024-03-28T18:11:29.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.989+0000 7f5a09dce740 1 -- 172.21.15.37:0/2818170522 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f59f0032340 con 0x556bbc68e0f0 2024-03-28T18:11:29.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.989+0000 7f59ee000640 1 -- 172.21.15.37:0/2818170522 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f59f0003f00 con 0x556bbc68e0f0 2024-03-28T18:11:29.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.989+0000 7f59ee000640 1 -- 172.21.15.37:0/2818170522 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59f0039070 con 0x556bbc68e0f0 2024-03-28T18:11:29.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.989+0000 7f5a09dce740 10 client.25451 fetch_fsmap learned FSMap version 23 2024-03-28T18:11:29.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.989+0000 7f5a09dce740 1 -- 172.21.15.37:0/2818170522 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556bbc7033b0 con 0x556bbc68e0f0 2024-03-28T18:11:29.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.990+0000 7f59ee000640 1 -- 172.21.15.37:0/2818170522 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59f0032340 con 0x556bbc68e0f0 2024-03-28T18:11:29.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.990+0000 7f59ee000640 1 -- 172.21.15.37:0/2818170522 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f59f0051600 con 0x556bbc68e0f0 2024-03-28T18:11:29.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.990+0000 7f5a09dce740 10 client.25451 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:11:29.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.990+0000 7f5a09dce740 10 client.25451 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:11:29.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.990+0000 7f5a09dce740 20 client.25451 populate_metadata read hostname 'smithi037' 2024-03-28T18:11:29.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.990+0000 7f5a09dce740 1 --2- 172.21.15.37:0/2818170522 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556bbc703630 0x556bbc705a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:29.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.990+0000 7f5a09dce740 4 client.25451 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:11:29.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.990+0000 7f5a09dce740 1 -- 172.21.15.37:0/2818170522 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556bbc5f9ae0 con 0x556bbc703630 2024-03-28T18:11:29.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.990+0000 7f59f7400640 1 --2- 172.21.15.37:0/2818170522 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556bbc703630 0x556bbc705a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:29.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.991+0000 7f59ee000640 10 client.25451 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:11:29.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.991+0000 7f59f7400640 1 --2- 172.21.15.37:0/2818170522 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556bbc703630 0x556bbc705a20 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df629d3508f74798 server_cookie=b5fd43a134d3f90d in_seq=0 out_seq=0 2024-03-28T18:11:29.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.991+0000 7f59ee000640 1 -- 172.21.15.37:0/2818170522 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x556bbc5f9ae0 con 0x556bbc703630 2024-03-28T18:11:29.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.991+0000 7f59ee000640 10 client.25451 handle_command_reply: tid=0 2024-03-28T18:11:29.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.991+0000 7f5a09dce740 1 client.25451 shutdown 2024-03-28T18:11:29.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.991+0000 7f5a09dce740 1 -- 172.21.15.37:0/2818170522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 msgr2=0x556bbc690c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:29.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.991+0000 7f5a09dce740 1 --2- 172.21.15.37:0/2818170522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc690c70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f59f002c6d0 tx=0x7f59f00046b0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.991+0000 7f5a09dce740 1 -- 172.21.15.37:0/2818170522 shutdown_connections 2024-03-28T18:11:29.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.991+0000 7f5a09dce740 1 --2- 172.21.15.37:0/2818170522 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556bbc703630 0x556bbc705a20 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.991+0000 7f5a09dce740 1 --2- 172.21.15.37:0/2818170522 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556bbc697460 0x556bbc701b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.991+0000 7f5a09dce740 1 --2- 172.21.15.37:0/2818170522 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556bbc68edd0 0x556bbc6911b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.992+0000 7f59ee000640 0 client.25451 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:11:29.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.992+0000 7f5a09dce740 1 --2- 172.21.15.37:0/2818170522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556bbc68e0f0 0x556bbc690c70 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.992+0000 7f5a09dce740 1 -- 172.21.15.37:0/2818170522 >> 172.21.15.37:0/2818170522 conn(0x556bbc655200 msgr2=0x556bbc687bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:29.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.992+0000 7f5a09dce740 1 -- 172.21.15.37:0/2818170522 shutdown_connections 2024-03-28T18:11:29.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.992+0000 7f5a09dce740 1 -- 172.21.15.37:0/2818170522 wait complete. 2024-03-28T18:11:29.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.992+0000 7f5a09dce740 20 client.25451 trim_cache size 0 max 16384 2024-03-28T18:11:29.996 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:11:29.996 INFO:teuthology.orchestra.run.smithi037.stdout: "status": "no active scrubs running", 2024-03-28T18:11:29.996 INFO:teuthology.orchestra.run.smithi037.stdout: "scrubs": {} 2024-03-28T18:11:29.996 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:11:29.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.992+0000 7f5a05a00640 1 -- 172.21.15.37:0/3899521296 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f59e40a2a00 msgr2=0x7f59e40a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:29.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.992+0000 7f5a05a00640 1 --2- 172.21.15.37:0/3899521296 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f59e40a2a00 0x7f59e40a4ec0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f59e8005fa0 tx=0x7f59e800b3d0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.992+0000 7f5a05a00640 1 -- 172.21.15.37:0/3899521296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a0010baf0 msgr2=0x7f5a0019e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:29.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.992+0000 7f5a05a00640 1 --2- 172.21.15.37:0/3899521296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a0010baf0 0x7f5a0019e8d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f59f800e970 tx=0x7f59f800ee30 comp rx=0 tx=0).stop 2024-03-28T18:11:29.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.993+0000 7f5a05a00640 1 -- 172.21.15.37:0/3899521296 shutdown_connections 2024-03-28T18:11:29.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.993+0000 7f5a05a00640 1 --2- 172.21.15.37:0/3899521296 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f59e40a2a00 0x7f59e40a4ec0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.993+0000 7f5a05a00640 1 --2- 172.21.15.37:0/3899521296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a0010baf0 0x7f5a0019e8d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.993+0000 7f5a05a00640 1 --2- 172.21.15.37:0/3899521296 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a00102ff0 0x7f5a0019a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.993+0000 7f5a05a00640 1 --2- 172.21.15.37:0/3899521296 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a001026d0 0x7f5a0019a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:29.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.993+0000 7f5a05a00640 1 -- 172.21.15.37:0/3899521296 >> 172.21.15.37:0/3899521296 conn(0x7f5a000fbef0 msgr2=0x7f5a000fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:29.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.993+0000 7f5a05a00640 1 -- 172.21.15.37:0/3899521296 shutdown_connections 2024-03-28T18:11:29.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:29.993+0000 7f5a05a00640 1 -- 172.21.15.37:0/3899521296 wait complete. 2024-03-28T18:11:30.014 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:11:30.014 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:11:30.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:29 smithi170 ceph-mon[36280]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-03-28T18:11:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:29 smithi059 ceph-mon[36190]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-03-28T18:11:30.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.246+0000 7fac7a800640 1 -- 172.21.15.37:0/1078105178 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac741023b0 msgr2=0x7fac74068800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:30.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.246+0000 7fac7a800640 1 --2- 172.21.15.37:0/1078105178 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac741023b0 0x7fac74068800 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fac68009ca0 tx=0x7fac6802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.246+0000 7fac7a800640 1 -- 172.21.15.37:0/1078105178 shutdown_connections 2024-03-28T18:11:30.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.246+0000 7fac7a800640 1 --2- 172.21.15.37:0/1078105178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac7410dd50 0x7fac74110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.246+0000 7fac7a800640 1 --2- 172.21.15.37:0/1078105178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac74068d40 0x7fac740691a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.246+0000 7fac7a800640 1 --2- 172.21.15.37:0/1078105178 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac741023b0 0x7fac74068800 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.246+0000 7fac7a800640 1 -- 172.21.15.37:0/1078105178 >> 172.21.15.37:0/1078105178 conn(0x7fac74075fb0 msgr2=0x7fac740763c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:30.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.247+0000 7fac7a800640 1 -- 172.21.15.37:0/1078105178 shutdown_connections 2024-03-28T18:11:30.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.247+0000 7fac7a800640 1 -- 172.21.15.37:0/1078105178 wait complete. 2024-03-28T18:11:30.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.248+0000 7fac7a800640 1 Processor -- start 2024-03-28T18:11:30.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.248+0000 7fac7a800640 1 -- start start 2024-03-28T18:11:30.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.248+0000 7fac7a800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac74068d40 0x7fac74199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.248+0000 7fac7a800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac741023b0 0x7fac7419a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.248+0000 7fac7a800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac7410dd50 0x7fac7419e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.248+0000 7fac7a800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac74113aa0 con 0x7fac741023b0 2024-03-28T18:11:30.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.248+0000 7fac7a800640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac74113920 con 0x7fac7410dd50 2024-03-28T18:11:30.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.248+0000 7fac7a800640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac74113c20 con 0x7fac74068d40 2024-03-28T18:11:30.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.248+0000 7fac79400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac74068d40 0x7fac74199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.248+0000 7fac79400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac74068d40 0x7fac74199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:54888/0 (socket says 172.21.15.37:54888) 2024-03-28T18:11:30.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.248+0000 7fac79400640 1 -- 172.21.15.37:0/1346941108 learned_addr learned my addr 172.21.15.37:0/1346941108 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:30.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.248+0000 7fac79e00640 1 --2- 172.21.15.37:0/1346941108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac7410dd50 0x7fac7419e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.249+0000 7fac78a00640 1 --2- 172.21.15.37:0/1346941108 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac741023b0 0x7fac7419a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.249+0000 7fac79400640 1 -- 172.21.15.37:0/1346941108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac7410dd50 msgr2=0x7fac7419e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:30.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.249+0000 7fac79400640 1 --2- 172.21.15.37:0/1346941108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac7410dd50 0x7fac7419e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.249+0000 7fac79400640 1 -- 172.21.15.37:0/1346941108 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac741023b0 msgr2=0x7fac7419a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:30.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.249+0000 7fac79400640 1 --2- 172.21.15.37:0/1346941108 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac741023b0 0x7fac7419a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.249+0000 7fac79400640 1 -- 172.21.15.37:0/1346941108 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac7419ef50 con 0x7fac74068d40 2024-03-28T18:11:30.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.249+0000 7fac78a00640 1 --2- 172.21.15.37:0/1346941108 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac741023b0 0x7fac7419a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:11:30.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.249+0000 7fac79400640 1 --2- 172.21.15.37:0/1346941108 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac74068d40 0x7fac74199f60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fac68004af0 tx=0x7fac68032f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:30.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.249+0000 7fac72000640 1 -- 172.21.15.37:0/1346941108 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac68041020 con 0x7fac74068d40 2024-03-28T18:11:30.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.249+0000 7fac72000640 1 -- 172.21.15.37:0/1346941108 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac68034d90 con 0x7fac74068d40 2024-03-28T18:11:30.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.249+0000 7fac7a800640 1 -- 172.21.15.37:0/1346941108 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac7419f1e0 con 0x7fac74068d40 2024-03-28T18:11:30.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.249+0000 7fac72000640 1 -- 172.21.15.37:0/1346941108 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac68039040 con 0x7fac74068d40 2024-03-28T18:11:30.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.250+0000 7fac7a800640 1 -- 172.21.15.37:0/1346941108 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac741a6ad0 con 0x7fac74068d40 2024-03-28T18:11:30.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.251+0000 7fac7a800640 1 -- 172.21.15.37:0/1346941108 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac7419b2b0 con 0x7fac74068d40 2024-03-28T18:11:30.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.253+0000 7fac72000640 1 -- 172.21.15.37:0/1346941108 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 25) v1 ==== 137466+0+0 (secure 0 0 0) 0x7fac68044020 con 0x7fac74068d40 2024-03-28T18:11:30.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.254+0000 7fac72000640 1 --2- 172.21.15.37:0/1346941108 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fac5c0a2a00 0x7fac5c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.254+0000 7fac72000640 1 -- 172.21.15.37:0/1346941108 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fac680f10e0 con 0x7fac74068d40 2024-03-28T18:11:30.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.254+0000 7fac78a00640 1 --2- 172.21.15.37:0/1346941108 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fac5c0a2a00 0x7fac5c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.255+0000 7fac78a00640 1 --2- 172.21.15.37:0/1346941108 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fac5c0a2a00 0x7fac5c0a4ec0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fac60005d00 tx=0x7fac60009550 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:30.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.261+0000 7fac72000640 1 -- 172.21.15.37:0/1346941108 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fac6804b020 con 0x7fac74068d40 2024-03-28T18:11:30.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.506+0000 7fac7a800640 1 -- 172.21.15.37:0/1346941108 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac741a6d40 con 0x7fac74068d40 2024-03-28T18:11:30.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.508+0000 7fac72000640 1 -- 172.21.15.37:0/1346941108 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11383 (secure 0 0 0) 0x7fac680dd030 con 0x7fac74068d40 2024-03-28T18:11:30.508 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:11:30.508 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1926270371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6838","nonce":1926270371},{"type":"v1","addr":"172.21.15.37:6839","nonce":1926270371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:08:19.968894+0000","modified":"2024-03-28T18:10:49.565429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24580,"mds_1":24569,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6837/4034607523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":4034607523},{"type":"v1","addr":"172.21.15.59:6837","nonce":4034607523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6839/1575518279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":1575518279},{"type":"v1","addr":"172.21.15.170:6839","nonce":1575518279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6839/2091629078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":2091629078},{"type":"v1","addr":"172.21.15.59:6839","nonce":2091629078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24580":{"gid":24580,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.37:6841/1923263622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6840","nonce":1923263622},{"type":"v1","addr":"172.21.15.37:6841","nonce":1923263622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24580,"qdb_cluster":[24580,24550,24569]},"id":1}]} 2024-03-28T18:11:30.508 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 23 2024-03-28T18:11:30.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.511+0000 7fac7a800640 1 -- 172.21.15.37:0/1346941108 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fac5c0a2a00 msgr2=0x7fac5c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:30.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.511+0000 7fac7a800640 1 --2- 172.21.15.37:0/1346941108 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fac5c0a2a00 0x7fac5c0a4ec0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fac60005d00 tx=0x7fac60009550 comp rx=0 tx=0).stop 2024-03-28T18:11:30.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.511+0000 7fac7a800640 1 -- 172.21.15.37:0/1346941108 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac74068d40 msgr2=0x7fac74199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:30.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.511+0000 7fac7a800640 1 --2- 172.21.15.37:0/1346941108 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac74068d40 0x7fac74199f60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fac68004af0 tx=0x7fac68032f40 comp rx=0 tx=0).stop 2024-03-28T18:11:30.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.511+0000 7fac7a800640 1 -- 172.21.15.37:0/1346941108 shutdown_connections 2024-03-28T18:11:30.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.511+0000 7fac7a800640 1 --2- 172.21.15.37:0/1346941108 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fac5c0a2a00 0x7fac5c0a4ec0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.511+0000 7fac7a800640 1 --2- 172.21.15.37:0/1346941108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac7410dd50 0x7fac7419e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.511+0000 7fac7a800640 1 --2- 172.21.15.37:0/1346941108 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac741023b0 0x7fac7419a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.511+0000 7fac7a800640 1 --2- 172.21.15.37:0/1346941108 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac74068d40 0x7fac74199f60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.511+0000 7fac7a800640 1 -- 172.21.15.37:0/1346941108 >> 172.21.15.37:0/1346941108 conn(0x7fac74075fb0 msgr2=0x7fac740ff640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:30.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.511+0000 7fac7a800640 1 -- 172.21.15.37:0/1346941108 shutdown_connections 2024-03-28T18:11:30.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.512+0000 7fac7a800640 1 -- 172.21.15.37:0/1346941108 wait complete. 2024-03-28T18:11:30.534 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:11:30.534 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:11:30.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.763+0000 7f5a1da00640 1 -- 172.21.15.37:0/374685675 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a18074360 msgr2=0x7f5a18111fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:30.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.763+0000 7f5a1da00640 1 --2- 172.21.15.37:0/374685675 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a18074360 0x7f5a18111fd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5a00009900 tx=0x7f5a0002bcc0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.763+0000 7f5a1da00640 1 -- 172.21.15.37:0/374685675 shutdown_connections 2024-03-28T18:11:30.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.763+0000 7f5a1da00640 1 --2- 172.21.15.37:0/374685675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a18112510 0x7f5a18114900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.763+0000 7f5a1da00640 1 --2- 172.21.15.37:0/374685675 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a18074360 0x7f5a18111fd0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.763+0000 7f5a1da00640 1 --2- 172.21.15.37:0/374685675 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a18073a40 0x7f5a18073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.763+0000 7f5a1da00640 1 -- 172.21.15.37:0/374685675 >> 172.21.15.37:0/374685675 conn(0x7f5a180fd010 msgr2=0x7f5a180ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:30.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.764+0000 7f5a1da00640 1 -- 172.21.15.37:0/374685675 shutdown_connections 2024-03-28T18:11:30.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.764+0000 7f5a1da00640 1 -- 172.21.15.37:0/374685675 wait complete. 2024-03-28T18:11:30.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.765+0000 7f5a1da00640 1 Processor -- start 2024-03-28T18:11:30.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.765+0000 7f5a1da00640 1 -- start start 2024-03-28T18:11:30.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.765+0000 7f5a1da00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a18073a40 0x7f5a1819e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.765+0000 7f5a1da00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a18074360 0x7f5a1819e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.766+0000 7f5a1da00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a18112510 0x7f5a181a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.766+0000 7f5a1da00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a18117f60 con 0x7f5a18074360 2024-03-28T18:11:30.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.766+0000 7f5a1da00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a18117de0 con 0x7f5a18073a40 2024-03-28T18:11:30.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.766+0000 7f5a1da00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a181180e0 con 0x7f5a18112510 2024-03-28T18:11:30.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.766+0000 7f5a17e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a18073a40 0x7f5a1819e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.766+0000 7f5a17e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a18073a40 0x7f5a1819e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:56914/0 (socket says 172.21.15.37:56914) 2024-03-28T18:11:30.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.766+0000 7f5a17e00640 1 -- 172.21.15.37:0/767444780 learned_addr learned my addr 172.21.15.37:0/767444780 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:30.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.766+0000 7f5a1d000640 1 --2- 172.21.15.37:0/767444780 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a18112510 0x7f5a181a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.766+0000 7f5a0fe00640 1 --2- 172.21.15.37:0/767444780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a18074360 0x7f5a1819e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.766+0000 7f5a17e00640 1 -- 172.21.15.37:0/767444780 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a18112510 msgr2=0x7f5a181a2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:30.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.766+0000 7f5a17e00640 1 --2- 172.21.15.37:0/767444780 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a18112510 0x7f5a181a2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.766+0000 7f5a17e00640 1 -- 172.21.15.37:0/767444780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a18074360 msgr2=0x7f5a1819e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:30.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.766+0000 7f5a17e00640 1 --2- 172.21.15.37:0/767444780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a18074360 0x7f5a1819e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.766+0000 7f5a17e00640 1 -- 172.21.15.37:0/767444780 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a181a33f0 con 0x7f5a18073a40 2024-03-28T18:11:30.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.766+0000 7f5a0fe00640 1 --2- 172.21.15.37:0/767444780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a18074360 0x7f5a1819e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:11:30.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.767+0000 7f5a17e00640 1 --2- 172.21.15.37:0/767444780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a18073a40 0x7f5a1819e400 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5a0800d950 tx=0x7f5a0800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:30.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.767+0000 7f5a15600640 1 -- 172.21.15.37:0/767444780 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a08017030 con 0x7f5a18073a40 2024-03-28T18:11:30.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.767+0000 7f5a15600640 1 -- 172.21.15.37:0/767444780 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a08004020 con 0x7f5a18073a40 2024-03-28T18:11:30.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.767+0000 7f5a1da00640 1 -- 172.21.15.37:0/767444780 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a181a36e0 con 0x7f5a18073a40 2024-03-28T18:11:30.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.767+0000 7f5a15600640 1 -- 172.21.15.37:0/767444780 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a080047a0 con 0x7f5a18073a40 2024-03-28T18:11:30.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.767+0000 7f5a1da00640 1 -- 172.21.15.37:0/767444780 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a181aaf70 con 0x7f5a18073a40 2024-03-28T18:11:30.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.770+0000 7f5a21dab740 1 Processor -- start 2024-03-28T18:11:30.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.770+0000 7f5a15600640 1 -- 172.21.15.37:0/767444780 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 25) v1 ==== 137466+0+0 (secure 0 0 0) 0x7f5a080041d0 con 0x7f5a18073a40 2024-03-28T18:11:30.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.770+0000 7f5a21dab740 1 -- start start 2024-03-28T18:11:30.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.771+0000 7f5a21dab740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a5b8b0 0x5613b8a5bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.771+0000 7f5a21dab740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a5c260 0x5613b8a64810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.771+0000 7f5a21dab740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a64de0 0x5613b8a67230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.771+0000 7f5a21dab740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613b89bf2c0 con 0x5613b8a5c260 2024-03-28T18:11:30.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.771+0000 7f5a21dab740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613b8967c90 con 0x5613b8a64de0 2024-03-28T18:11:30.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.771+0000 7f5a21dab740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613b89bb360 con 0x5613b8a5b8b0 2024-03-28T18:11:30.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.771+0000 7f5a1d000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a5b8b0 0x5613b8a5bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.771+0000 7f5a1d000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a5b8b0 0x5613b8a5bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:54918/0 (socket says 172.21.15.37:54918) 2024-03-28T18:11:30.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.771+0000 7f5a1d000640 1 -- 172.21.15.37:0/885164226 learned_addr learned my addr 172.21.15.37:0/885164226 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:30.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.771+0000 7f5a0fe00640 1 --2- 172.21.15.37:0/885164226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a64de0 0x5613b8a67230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.771+0000 7f5a17e00640 1 --2- 172.21.15.37:0/885164226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a5c260 0x5613b8a64810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.771+0000 7f5a1d000640 1 -- 172.21.15.37:0/885164226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a64de0 msgr2=0x5613b8a67230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:30.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.771+0000 7f5a1d000640 1 --2- 172.21.15.37:0/885164226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a64de0 0x5613b8a67230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.771+0000 7f5a1d000640 1 -- 172.21.15.37:0/885164226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a5c260 msgr2=0x5613b8a64810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:30.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.771+0000 7f5a1d000640 1 --2- 172.21.15.37:0/885164226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a5c260 0x5613b8a64810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.771+0000 7f5a1d000640 1 -- 172.21.15.37:0/885164226 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5613b898c080 con 0x5613b8a5b8b0 2024-03-28T18:11:30.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.772+0000 7f5a1d000640 1 --2- 172.21.15.37:0/885164226 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a5b8b0 0x5613b8a5bc90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5a10009f40 tx=0x7f5a10010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63eb7291ddb1414b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:30.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.772+0000 7f5a0d600640 1 -- 172.21.15.37:0/885164226 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a10016070 con 0x5613b8a5b8b0 2024-03-28T18:11:30.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.772+0000 7f5a0d600640 1 -- 172.21.15.37:0/885164226 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a10004d20 con 0x5613b8a5b8b0 2024-03-28T18:11:30.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.772+0000 7f5a0d600640 1 -- 172.21.15.37:0/885164226 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a10012850 con 0x5613b8a5b8b0 2024-03-28T18:11:30.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.772+0000 7f5a15600640 1 --2- 172.21.15.37:0/767444780 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f59fc0a2a80 0x7f59fc0a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.772+0000 7f5a15600640 1 -- 172.21.15.37:0/767444780 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5a080ce980 con 0x7f5a18073a40 2024-03-28T18:11:30.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.772+0000 7f5a17e00640 1 --2- 172.21.15.37:0/767444780 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f59fc0a2a80 0x7f59fc0a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.773+0000 7f5a21dab740 1 -- 172.21.15.37:0/885164226 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a5b8b0 msgr2=0x5613b8a5bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:30.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.773+0000 7f5a21dab740 1 --2- 172.21.15.37:0/885164226 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a5b8b0 0x5613b8a5bc90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5a10009f40 tx=0x7f5a10010800 comp rx=0 tx=0).stop 2024-03-28T18:11:30.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.773+0000 7f5a21dab740 1 -- 172.21.15.37:0/885164226 shutdown_connections 2024-03-28T18:11:30.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.773+0000 7f5a21dab740 1 --2- 172.21.15.37:0/885164226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a64de0 0x5613b8a67230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.773+0000 7f5a21dab740 1 --2- 172.21.15.37:0/885164226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a5c260 0x5613b8a64810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.773+0000 7f5a21dab740 1 --2- 172.21.15.37:0/885164226 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a5b8b0 0x5613b8a5bc90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.773+0000 7f5a21dab740 1 -- 172.21.15.37:0/885164226 >> 172.21.15.37:0/885164226 conn(0x5613b8a55f80 msgr2=0x5613b8a56360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:30.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.773+0000 7f5a21dab740 1 -- 172.21.15.37:0/885164226 shutdown_connections 2024-03-28T18:11:30.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.773+0000 7f5a21dab740 1 -- 172.21.15.37:0/885164226 wait complete. 2024-03-28T18:11:30.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.773+0000 7f5a17e00640 1 --2- 172.21.15.37:0/767444780 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f59fc0a2a80 0x7f59fc0a4f40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5a080cc040 tx=0x7f5a08000bc0 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:30.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.774+0000 7f5a21dab740 1 -- start start 2024-03-28T18:11:30.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.774+0000 7f5a21dab740 1 Processor -- start 2024-03-28T18:11:30.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.774+0000 7f5a21dab740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5b8b0 0x5613b8a64750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.774+0000 7f5a21dab740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a5c260 0x5613b8a61ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.774+0000 7f5a21dab740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a64de0 0x5613b8a620f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.774+0000 7f5a21dab740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613b8a67d50 con 0x5613b8a5c260 2024-03-28T18:11:30.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.774+0000 7f5a21dab740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613b897b0d0 con 0x5613b8a5b8b0 2024-03-28T18:11:30.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.774+0000 7f5a21dab740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613b8a67ed0 con 0x5613b8a64de0 2024-03-28T18:11:30.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.774+0000 7f5a21dab740 10 client.0 _ll_register_callbacks cb 0x5613b897fe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:11:30.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.775+0000 7f5a1d000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5b8b0 0x5613b8a64750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.775+0000 7f5a0fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a5c260 0x5613b8a61ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.775+0000 7f5a1d000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5b8b0 0x5613b8a64750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:56926/0 (socket says 172.21.15.37:56926) 2024-03-28T18:11:30.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.775+0000 7f5a1d000640 1 -- 172.21.15.37:0/3540374348 learned_addr learned my addr 172.21.15.37:0/3540374348 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:30.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.775+0000 7f5a1d000640 1 --2- 172.21.15.37:0/3540374348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a64de0 0x5613b8a620f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.775+0000 7f5a1d000640 1 -- 172.21.15.37:0/3540374348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a64de0 msgr2=0x5613b8a620f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:30.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.775+0000 7f5a1d000640 1 --2- 172.21.15.37:0/3540374348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a64de0 0x5613b8a620f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.775+0000 7f5a1d000640 1 -- 172.21.15.37:0/3540374348 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a5c260 msgr2=0x5613b8a61ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:30.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.775+0000 7f5a1d000640 1 --2- 172.21.15.37:0/3540374348 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a5c260 0x5613b8a61ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.775+0000 7f5a1d000640 1 -- 172.21.15.37:0/3540374348 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5613b86d33f0 con 0x5613b8a5b8b0 2024-03-28T18:11:30.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.775+0000 7f5a1d000640 1 --2- 172.21.15.37:0/3540374348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5b8b0 0x5613b8a64750 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5a10012550 tx=0x7f5a10012450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32ddad51a3c58588 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:30.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.775+0000 7f5a06000640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:11:30.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.776+0000 7f5a06000640 1 -- 172.21.15.37:0/3540374348 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1001e020 con 0x5613b8a5b8b0 2024-03-28T18:11:30.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.776+0000 7f5a06000640 1 -- 172.21.15.37:0/3540374348 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5a10012c80 con 0x5613b8a5b8b0 2024-03-28T18:11:30.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.776+0000 7f5a21dab740 1 -- 172.21.15.37:0/3540374348 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5a10004d20 con 0x5613b8a5b8b0 2024-03-28T18:11:30.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.776+0000 7f5a06000640 1 -- 172.21.15.37:0/3540374348 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a10031050 con 0x5613b8a5b8b0 2024-03-28T18:11:30.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.776+0000 7f5a21dab740 10 client.25469 fetch_fsmap learned FSMap version 23 2024-03-28T18:11:30.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.776+0000 7f5a21dab740 1 -- 172.21.15.37:0/3540374348 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5613b879d7d0 con 0x5613b8a5b8b0 2024-03-28T18:11:30.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.776+0000 7f5a06000640 1 -- 172.21.15.37:0/3540374348 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a10004d20 con 0x5613b8a5b8b0 2024-03-28T18:11:30.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.776+0000 7f5a06000640 1 -- 172.21.15.37:0/3540374348 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f5a10029020 con 0x5613b8a5b8b0 2024-03-28T18:11:30.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.776+0000 7f5a21dab740 10 client.25469 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:11:30.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.776+0000 7f5a21dab740 10 client.25469 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-28T18:11:30.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.776+0000 7f5a21dab740 20 client.25469 populate_metadata read hostname 'smithi037' 2024-03-28T18:11:30.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.776+0000 7f5a21dab740 1 --2- 172.21.15.37:0/3540374348 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5613b8a74c40 0x5613b8a77030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.776+0000 7f5a21dab740 4 client.25469 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:11:30.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.776+0000 7f5a21dab740 1 -- 172.21.15.37:0/3540374348 --> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5613b89cc800 con 0x5613b8a74c40 2024-03-28T18:11:30.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.777+0000 7f5a0fe00640 1 --2- 172.21.15.37:0/3540374348 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5613b8a74c40 0x5613b8a77030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.777+0000 7f5a0fe00640 1 --2- 172.21.15.37:0/3540374348 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5613b8a74c40 0x5613b8a77030 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6faedd6843da3483 server_cookie=d5815f296b1b29fd in_seq=0 out_seq=0 2024-03-28T18:11:30.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.777+0000 7f5a06000640 10 client.25469 ms_handle_connect on v2:172.21.15.170:6838/1575518279 2024-03-28T18:11:30.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.778+0000 7f5a06000640 1 -- 172.21.15.37:0/3540374348 <== mds.2 v2:172.21.15.170:6838/1575518279 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5613b89cc800 con 0x5613b8a74c40 2024-03-28T18:11:30.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.778+0000 7f5a06000640 10 client.25469 handle_command_reply: tid=0 2024-03-28T18:11:30.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.778+0000 7f5a21dab740 1 client.25469 shutdown 2024-03-28T18:11:30.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.778+0000 7f5a21dab740 1 -- 172.21.15.37:0/3540374348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5b8b0 msgr2=0x5613b8a64750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:30.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.778+0000 7f5a21dab740 1 --2- 172.21.15.37:0/3540374348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5b8b0 0x5613b8a64750 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5a10012550 tx=0x7f5a10012450 comp rx=0 tx=0).stop 2024-03-28T18:11:30.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.778+0000 7f5a21dab740 1 -- 172.21.15.37:0/3540374348 shutdown_connections 2024-03-28T18:11:30.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.778+0000 7f5a21dab740 1 --2- 172.21.15.37:0/3540374348 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5613b8a74c40 0x5613b8a77030 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.778+0000 7f5a21dab740 1 --2- 172.21.15.37:0/3540374348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a64de0 0x5613b8a620f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.778+0000 7f5a21dab740 1 --2- 172.21.15.37:0/3540374348 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a5c260 0x5613b8a61ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.778+0000 7f5a21dab740 1 --2- 172.21.15.37:0/3540374348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5b8b0 0x5613b8a64750 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.778+0000 7f5a21dab740 1 -- 172.21.15.37:0/3540374348 >> 172.21.15.37:0/3540374348 conn(0x5613b8a55f80 msgr2=0x5613b8a57cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:30.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.778+0000 7f5a06000640 0 client.25469 ms_handle_reset on v2:172.21.15.170:6838/1575518279 2024-03-28T18:11:30.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.778+0000 7f5a21dab740 1 -- 172.21.15.37:0/3540374348 shutdown_connections 2024-03-28T18:11:30.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.779+0000 7f5a21dab740 1 -- 172.21.15.37:0/3540374348 wait complete. 2024-03-28T18:11:30.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.779+0000 7f5a21dab740 20 client.25469 trim_cache size 0 max 16384 2024-03-28T18:11:30.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.793+0000 7f5a21dab740 1 Processor -- start 2024-03-28T18:11:30.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.793+0000 7f5a21dab740 1 -- start start 2024-03-28T18:11:30.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.793+0000 7f5a21dab740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a5b8b0 0x5613b8a55160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.793+0000 7f5a21dab740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5c260 0x5613b8a64210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.793+0000 7f5a21dab740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a64de0 0x5613b8a64750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.793+0000 7f5a21dab740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a181b5bd0 con 0x5613b8a64de0 2024-03-28T18:11:30.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.793+0000 7f5a21dab740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a181b36d0 con 0x5613b8a5c260 2024-03-28T18:11:30.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.793+0000 7f5a21dab740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a181b1770 con 0x5613b8a5b8b0 2024-03-28T18:11:30.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.793+0000 7f5a0fe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5c260 0x5613b8a64210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.793+0000 7f5a1d000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a5b8b0 0x5613b8a55160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.793+0000 7f5a0fe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5c260 0x5613b8a64210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:56938/0 (socket says 172.21.15.37:56938) 2024-03-28T18:11:30.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.793+0000 7f5a0fe00640 1 -- 172.21.15.37:0/811617307 learned_addr learned my addr 172.21.15.37:0/811617307 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:30.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.793+0000 7f5a1d000640 1 --2- 172.21.15.37:0/811617307 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a64de0 0x5613b8a64750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.794+0000 7f5a0fe00640 1 -- 172.21.15.37:0/811617307 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a5b8b0 msgr2=0x5613b8a55160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:30.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.794+0000 7f5a0fe00640 1 --2- 172.21.15.37:0/811617307 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a5b8b0 0x5613b8a55160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.794+0000 7f5a0fe00640 1 -- 172.21.15.37:0/811617307 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a64de0 msgr2=0x5613b8a64750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:30.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.794+0000 7f5a0fe00640 1 --2- 172.21.15.37:0/811617307 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a64de0 0x5613b8a64750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.794+0000 7f5a0fe00640 1 -- 172.21.15.37:0/811617307 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5613b8a62460 con 0x5613b8a5c260 2024-03-28T18:11:30.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.794+0000 7f5a0fe00640 1 --2- 172.21.15.37:0/811617307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5c260 0x5613b8a64210 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5a0002c2a0 tx=0x7f5a0002c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2620551d10e71bca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:30.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.794+0000 7f5a0cc00640 1 -- 172.21.15.37:0/811617307 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a000042a0 con 0x5613b8a5c260 2024-03-28T18:11:30.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.794+0000 7f5a0cc00640 1 -- 172.21.15.37:0/811617307 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a0002cdf0 con 0x5613b8a5c260 2024-03-28T18:11:30.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.794+0000 7f5a0cc00640 1 -- 172.21.15.37:0/811617307 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a000046c0 con 0x5613b8a5c260 2024-03-28T18:11:30.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.795+0000 7f5a21dab740 1 -- 172.21.15.37:0/811617307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5c260 msgr2=0x5613b8a64210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:30.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.795+0000 7f5a21dab740 1 --2- 172.21.15.37:0/811617307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5c260 0x5613b8a64210 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5a0002c2a0 tx=0x7f5a0002c860 comp rx=0 tx=0).stop 2024-03-28T18:11:30.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.795+0000 7f5a21dab740 1 -- 172.21.15.37:0/811617307 shutdown_connections 2024-03-28T18:11:30.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.795+0000 7f5a21dab740 1 --2- 172.21.15.37:0/811617307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5c260 0x5613b8a64210 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.795+0000 7f5a21dab740 1 --2- 172.21.15.37:0/811617307 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a64de0 0x5613b8a64750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.795+0000 7f5a21dab740 1 --2- 172.21.15.37:0/811617307 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a5b8b0 0x5613b8a55160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.795+0000 7f5a21dab740 1 -- 172.21.15.37:0/811617307 >> 172.21.15.37:0/811617307 conn(0x5613b8a55f80 msgr2=0x5613b8a5c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:30.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.795+0000 7f5a21dab740 1 -- 172.21.15.37:0/811617307 shutdown_connections 2024-03-28T18:11:30.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.795+0000 7f5a21dab740 1 -- 172.21.15.37:0/811617307 wait complete. 2024-03-28T18:11:30.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.796+0000 7f5a21dab740 1 -- start start 2024-03-28T18:11:30.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.796+0000 7f5a21dab740 1 Processor -- start 2024-03-28T18:11:30.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.797+0000 7f5a21dab740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5b8b0 0x5613b8a6cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.797+0000 7f5a21dab740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a5c260 0x5613b8a6b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.797+0000 7f5a21dab740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a64de0 0x5613b8a6b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.797+0000 7f5a21dab740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a181b1d70 con 0x5613b8a5c260 2024-03-28T18:11:30.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.797+0000 7f5a21dab740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a181b3070 con 0x5613b8a5b8b0 2024-03-28T18:11:30.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.797+0000 7f5a21dab740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a181b23b0 con 0x5613b8a64de0 2024-03-28T18:11:30.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.797+0000 7f5a21dab740 10 client.0 _ll_register_callbacks cb 0x7f5a00009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:11:30.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.797+0000 7f5a1d000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5b8b0 0x5613b8a6cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.797+0000 7f5a1d000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5b8b0 0x5613b8a6cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:56944/0 (socket says 172.21.15.37:56944) 2024-03-28T18:11:30.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.797+0000 7f5a1d000640 1 -- 172.21.15.37:0/474955239 learned_addr learned my addr 172.21.15.37:0/474955239 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:30.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.797+0000 7f5a0fe00640 1 --2- 172.21.15.37:0/474955239 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a5c260 0x5613b8a6b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.797+0000 7f5a1d000640 1 --2- 172.21.15.37:0/474955239 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a64de0 0x5613b8a6b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.797+0000 7f5a1d000640 1 -- 172.21.15.37:0/474955239 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a64de0 msgr2=0x5613b8a6b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:30.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.797+0000 7f5a1d000640 1 --2- 172.21.15.37:0/474955239 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a64de0 0x5613b8a6b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.797+0000 7f5a1d000640 1 -- 172.21.15.37:0/474955239 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a5c260 msgr2=0x5613b8a6b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:30.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.797+0000 7f5a1d000640 1 --2- 172.21.15.37:0/474955239 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a5c260 0x5613b8a6b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.797+0000 7f5a1d000640 1 -- 172.21.15.37:0/474955239 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5613b8a6c000 con 0x5613b8a5b8b0 2024-03-28T18:11:30.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.797+0000 7f5a0fe00640 1 --2- 172.21.15.37:0/474955239 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a5c260 0x5613b8a6b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:11:30.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.798+0000 7f5a1d000640 1 --2- 172.21.15.37:0/474955239 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5b8b0 0x5613b8a6cea0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5a1001a6a0 tx=0x7f5a10005a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f939e11228eb58a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:30.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.798+0000 7f5a06000640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:11:30.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.798+0000 7f5a06000640 1 -- 172.21.15.37:0/474955239 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1001e040 con 0x5613b8a5b8b0 2024-03-28T18:11:30.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.798+0000 7f5a06000640 1 -- 172.21.15.37:0/474955239 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5a10012820 con 0x5613b8a5b8b0 2024-03-28T18:11:30.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.798+0000 7f5a21dab740 1 -- 172.21.15.37:0/474955239 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5a0002cdf0 con 0x5613b8a5b8b0 2024-03-28T18:11:30.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.798+0000 7f5a06000640 1 -- 172.21.15.37:0/474955239 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a10044050 con 0x5613b8a5b8b0 2024-03-28T18:11:30.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.798+0000 7f5a21dab740 10 client.25475 fetch_fsmap learned FSMap version 23 2024-03-28T18:11:30.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.798+0000 7f5a21dab740 1 -- 172.21.15.37:0/474955239 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5613b8ad12a0 con 0x5613b8a5b8b0 2024-03-28T18:11:30.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.799+0000 7f5a06000640 1 -- 172.21.15.37:0/474955239 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a0002cdf0 con 0x5613b8a5b8b0 2024-03-28T18:11:30.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.799+0000 7f5a06000640 1 -- 172.21.15.37:0/474955239 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f5a10029040 con 0x5613b8a5b8b0 2024-03-28T18:11:30.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.799+0000 7f5a21dab740 10 client.25475 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:11:30.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.799+0000 7f5a21dab740 10 client.25475 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-28T18:11:30.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.799+0000 7f5a21dab740 20 client.25475 populate_metadata read hostname 'smithi037' 2024-03-28T18:11:30.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.799+0000 7f5a21dab740 1 --2- 172.21.15.37:0/474955239 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5613b8ad17e0 0x5613b8ad3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:30.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.799+0000 7f5a21dab740 4 client.25475 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:11:30.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.799+0000 7f5a21dab740 1 -- 172.21.15.37:0/474955239 --> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5613b89c6b50 con 0x5613b8ad17e0 2024-03-28T18:11:30.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.799+0000 7f5a0fe00640 1 --2- 172.21.15.37:0/474955239 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5613b8ad17e0 0x5613b8ad3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:30.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.800+0000 7f5a0fe00640 1 --2- 172.21.15.37:0/474955239 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5613b8ad17e0 0x5613b8ad3c00 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=60cb28ba70929f4b server_cookie=822dd310144283b3 in_seq=0 out_seq=0 2024-03-28T18:11:30.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.800+0000 7f5a06000640 10 client.25475 ms_handle_connect on v2:172.21.15.170:6838/1575518279 2024-03-28T18:11:30.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.800+0000 7f5a06000640 1 -- 172.21.15.37:0/474955239 <== mds.2 v2:172.21.15.170:6838/1575518279 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5613b89c6b50 con 0x5613b8ad17e0 2024-03-28T18:11:30.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.800+0000 7f5a06000640 10 client.25475 handle_command_reply: tid=0 2024-03-28T18:11:30.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.800+0000 7f5a21dab740 1 client.25475 shutdown 2024-03-28T18:11:30.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.800+0000 7f5a21dab740 1 -- 172.21.15.37:0/474955239 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5b8b0 msgr2=0x5613b8a6cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:30.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.800+0000 7f5a21dab740 1 --2- 172.21.15.37:0/474955239 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5b8b0 0x5613b8a6cea0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5a1001a6a0 tx=0x7f5a10005a90 comp rx=0 tx=0).stop 2024-03-28T18:11:30.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.800+0000 7f5a21dab740 1 -- 172.21.15.37:0/474955239 shutdown_connections 2024-03-28T18:11:30.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.800+0000 7f5a21dab740 1 --2- 172.21.15.37:0/474955239 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5613b8ad17e0 0x5613b8ad3c00 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.800+0000 7f5a21dab740 1 --2- 172.21.15.37:0/474955239 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5613b8a64de0 0x5613b8a6b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.800+0000 7f5a21dab740 1 --2- 172.21.15.37:0/474955239 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5613b8a5c260 0x5613b8a6b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.800+0000 7f5a21dab740 1 --2- 172.21.15.37:0/474955239 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5613b8a5b8b0 0x5613b8a6cea0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.800+0000 7f5a21dab740 1 -- 172.21.15.37:0/474955239 >> 172.21.15.37:0/474955239 conn(0x5613b8a55f80 msgr2=0x5613b8a54fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:30.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.800+0000 7f5a06000640 0 client.25475 ms_handle_reset on v2:172.21.15.170:6838/1575518279 2024-03-28T18:11:30.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.800+0000 7f5a21dab740 1 -- 172.21.15.37:0/474955239 shutdown_connections 2024-03-28T18:11:30.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.800+0000 7f5a21dab740 1 -- 172.21.15.37:0/474955239 wait complete. 2024-03-28T18:11:30.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.801+0000 7f5a21dab740 20 client.25475 trim_cache size 0 max 16384 2024-03-28T18:11:30.804 INFO:teuthology.orchestra.run.smithi037.stdout:[] 2024-03-28T18:11:30.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.801+0000 7f5a1da00640 1 -- 172.21.15.37:0/767444780 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f59fc0a2a80 msgr2=0x7f59fc0a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:30.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.801+0000 7f5a1da00640 1 --2- 172.21.15.37:0/767444780 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f59fc0a2a80 0x7f59fc0a4f40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5a080cc040 tx=0x7f5a08000bc0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.801+0000 7f5a1da00640 1 -- 172.21.15.37:0/767444780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a18073a40 msgr2=0x7f5a1819e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:30.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.801+0000 7f5a1da00640 1 --2- 172.21.15.37:0/767444780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a18073a40 0x7f5a1819e400 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5a0800d950 tx=0x7f5a0800de10 comp rx=0 tx=0).stop 2024-03-28T18:11:30.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.801+0000 7f5a1da00640 1 -- 172.21.15.37:0/767444780 shutdown_connections 2024-03-28T18:11:30.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.801+0000 7f5a1da00640 1 --2- 172.21.15.37:0/767444780 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f59fc0a2a80 0x7f59fc0a4f40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.801+0000 7f5a1da00640 1 --2- 172.21.15.37:0/767444780 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a18112510 0x7f5a181a2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.801+0000 7f5a1da00640 1 --2- 172.21.15.37:0/767444780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a18074360 0x7f5a1819e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.801+0000 7f5a1da00640 1 --2- 172.21.15.37:0/767444780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a18073a40 0x7f5a1819e400 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:30.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.801+0000 7f5a1da00640 1 -- 172.21.15.37:0/767444780 >> 172.21.15.37:0/767444780 conn(0x7f5a180fd010 msgr2=0x7f5a180fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:30.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.801+0000 7f5a1da00640 1 -- 172.21.15.37:0/767444780 shutdown_connections 2024-03-28T18:11:30.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:30.801+0000 7f5a1da00640 1 -- 172.21.15.37:0/767444780 wait complete. 2024-03-28T18:11:30.817 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:30 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1346941108' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:11:30.819 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:11:30.820 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:11:31.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.034+0000 7f75bbe00640 1 -- 172.21.15.37:0/2450235706 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75bc101530 msgr2=0x7f75bc1019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:31.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.034+0000 7f75bbe00640 1 --2- 172.21.15.37:0/2450235706 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75bc101530 0x7f75bc1019b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f75a8009930 tx=0x7f75a802bc30 comp rx=0 tx=0).stop 2024-03-28T18:11:31.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.034+0000 7f75bbe00640 1 -- 172.21.15.37:0/2450235706 shutdown_connections 2024-03-28T18:11:31.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.034+0000 7f75bbe00640 1 --2- 172.21.15.37:0/2450235706 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75bc101ef0 0x7f75bc10e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.034+0000 7f75bbe00640 1 --2- 172.21.15.37:0/2450235706 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75bc101530 0x7f75bc1019b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.034+0000 7f75bbe00640 1 --2- 172.21.15.37:0/2450235706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75bc1075f0 0x7f75bc1079d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.034+0000 7f75bbe00640 1 -- 172.21.15.37:0/2450235706 >> 172.21.15.37:0/2450235706 conn(0x7f75bc0fd040 msgr2=0x7f75bc0ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:31.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.034+0000 7f75bbe00640 1 -- 172.21.15.37:0/2450235706 shutdown_connections 2024-03-28T18:11:31.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.034+0000 7f75bbe00640 1 -- 172.21.15.37:0/2450235706 wait complete. 2024-03-28T18:11:31.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:30 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1346941108' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:11:31.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:30 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1346941108' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:11:31.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.036+0000 7f75bbe00640 1 Processor -- start 2024-03-28T18:11:31.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.036+0000 7f75bbe00640 1 -- start start 2024-03-28T18:11:31.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.037+0000 7f75bbe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75bc101530 0x7f75bc0739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.037+0000 7f75bbe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75bc101ef0 0x7f75bc073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.037+0000 7f75bbe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75bc1075f0 0x7f75bc074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.037+0000 7f75bbe00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc111900 con 0x7f75bc101ef0 2024-03-28T18:11:31.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.037+0000 7f75bbe00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc111780 con 0x7f75bc101530 2024-03-28T18:11:31.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.037+0000 7f75bbe00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc111a80 con 0x7f75bc1075f0 2024-03-28T18:11:31.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.037+0000 7f75baa00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75bc101530 0x7f75bc0739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.037+0000 7f75baa00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75bc101530 0x7f75bc0739a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:56952/0 (socket says 172.21.15.37:56952) 2024-03-28T18:11:31.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.037+0000 7f75baa00640 1 -- 172.21.15.37:0/1662680992 learned_addr learned my addr 172.21.15.37:0/1662680992 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:31.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.037+0000 7f75bb400640 1 --2- 172.21.15.37:0/1662680992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75bc1075f0 0x7f75bc074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.037+0000 7f75ba000640 1 --2- 172.21.15.37:0/1662680992 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75bc101ef0 0x7f75bc073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.037+0000 7f75baa00640 1 -- 172.21.15.37:0/1662680992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75bc1075f0 msgr2=0x7f75bc074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:31.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.037+0000 7f75baa00640 1 --2- 172.21.15.37:0/1662680992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75bc1075f0 0x7f75bc074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.037+0000 7f75baa00640 1 -- 172.21.15.37:0/1662680992 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75bc101ef0 msgr2=0x7f75bc073ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:31.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.037+0000 7f75baa00640 1 --2- 172.21.15.37:0/1662680992 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75bc101ef0 0x7f75bc073ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.037+0000 7f75baa00640 1 -- 172.21.15.37:0/1662680992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc1a6260 con 0x7f75bc101530 2024-03-28T18:11:31.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.038+0000 7f75ba000640 1 --2- 172.21.15.37:0/1662680992 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75bc101ef0 0x7f75bc073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:11:31.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.038+0000 7f75baa00640 1 --2- 172.21.15.37:0/1662680992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75bc101530 0x7f75bc0739a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f75ac00e9f0 tx=0x7f75ac00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:31.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.038+0000 7f75b3400640 1 -- 172.21.15.37:0/1662680992 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac00cd80 con 0x7f75bc101530 2024-03-28T18:11:31.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.038+0000 7f75b3400640 1 -- 172.21.15.37:0/1662680992 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75ac010910 con 0x7f75bc101530 2024-03-28T18:11:31.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.038+0000 7f75b3400640 1 -- 172.21.15.37:0/1662680992 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac018990 con 0x7f75bc101530 2024-03-28T18:11:31.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.038+0000 7f75bbe00640 1 -- 172.21.15.37:0/1662680992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75bc1a6550 con 0x7f75bc101530 2024-03-28T18:11:31.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.038+0000 7f75bbe00640 1 -- 172.21.15.37:0/1662680992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75bc1a6a20 con 0x7f75bc101530 2024-03-28T18:11:31.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.041+0000 7f75b3400640 1 -- 172.21.15.37:0/1662680992 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 25) v1 ==== 137466+0+0 (secure 0 0 0) 0x7f75ac010ac0 con 0x7f75bc101530 2024-03-28T18:11:31.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.042+0000 7f75c4a09740 1 Processor -- start 2024-03-28T18:11:31.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.042+0000 7f75c4a09740 1 -- start start 2024-03-28T18:11:31.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.042+0000 7f75c4a09740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d25f8b0 0x55b56d25fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.042+0000 7f75c4a09740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d260260 0x55b56d268810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.042+0000 7f75c4a09740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55b56d268de0 0x55b56d26b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.042+0000 7f75c4a09740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b56d1c32c0 con 0x55b56d268de0 2024-03-28T18:11:31.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.042+0000 7f75c4a09740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b56d16bc90 con 0x55b56d25f8b0 2024-03-28T18:11:31.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.042+0000 7f75c4a09740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b56d1bf360 con 0x55b56d260260 2024-03-28T18:11:31.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.042+0000 7f75bb400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d25f8b0 0x55b56d25fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.042+0000 7f75baa00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d260260 0x55b56d268810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.042+0000 7f75bb400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d25f8b0 0x55b56d25fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:56962/0 (socket says 172.21.15.37:56962) 2024-03-28T18:11:31.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.042+0000 7f75bb400640 1 -- 172.21.15.37:0/991583950 learned_addr learned my addr 172.21.15.37:0/991583950 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:31.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.043+0000 7f75ba000640 1 --2- 172.21.15.37:0/991583950 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55b56d268de0 0x55b56d26b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.043+0000 7f75bb400640 1 -- 172.21.15.37:0/991583950 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d260260 msgr2=0x55b56d268810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:31.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.043+0000 7f75bb400640 1 --2- 172.21.15.37:0/991583950 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d260260 0x55b56d268810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.043+0000 7f75bb400640 1 -- 172.21.15.37:0/991583950 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55b56d268de0 msgr2=0x55b56d26b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:31.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.043+0000 7f75bb400640 1 --2- 172.21.15.37:0/991583950 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55b56d268de0 0x55b56d26b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.043+0000 7f75bb400640 1 -- 172.21.15.37:0/991583950 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b56d190080 con 0x55b56d25f8b0 2024-03-28T18:11:31.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.043+0000 7f75b3400640 1 --2- 172.21.15.37:0/1662680992 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f75a00a2a80 0x7f75a00a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.043+0000 7f75bb400640 1 --2- 172.21.15.37:0/991583950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d25f8b0 0x55b56d25fc90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009fd0 tx=0x7f75b40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fc70dbaeb930814 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:31.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.043+0000 7f75b3400640 1 -- 172.21.15.37:0/1662680992 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f75ac0cf310 con 0x7f75bc101530 2024-03-28T18:11:31.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.043+0000 7f75baa00640 1 --2- 172.21.15.37:0/1662680992 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f75a00a2a80 0x7f75a00a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.043+0000 7f75a7e00640 1 -- 172.21.15.37:0/991583950 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b4004d20 con 0x55b56d25f8b0 2024-03-28T18:11:31.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.044+0000 7f75a7e00640 1 -- 172.21.15.37:0/991583950 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75b4004050 con 0x55b56d25f8b0 2024-03-28T18:11:31.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.044+0000 7f75a7e00640 1 -- 172.21.15.37:0/991583950 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b4005260 con 0x55b56d25f8b0 2024-03-28T18:11:31.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.044+0000 7f75baa00640 1 --2- 172.21.15.37:0/1662680992 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f75a00a2a80 0x7f75a00a4f40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f75ac0cc9d0 tx=0x7f75ac0d2d10 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:31.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.044+0000 7f75c4a09740 1 -- 172.21.15.37:0/991583950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d25f8b0 msgr2=0x55b56d25fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:31.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.044+0000 7f75c4a09740 1 --2- 172.21.15.37:0/991583950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d25f8b0 0x55b56d25fc90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009fd0 tx=0x7f75b40118d0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.044+0000 7f75c4a09740 1 -- 172.21.15.37:0/991583950 shutdown_connections 2024-03-28T18:11:31.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.044+0000 7f75c4a09740 1 --2- 172.21.15.37:0/991583950 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55b56d268de0 0x55b56d26b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.044+0000 7f75c4a09740 1 --2- 172.21.15.37:0/991583950 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d260260 0x55b56d268810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.044+0000 7f75c4a09740 1 --2- 172.21.15.37:0/991583950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d25f8b0 0x55b56d25fc90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.044+0000 7f75c4a09740 1 -- 172.21.15.37:0/991583950 >> 172.21.15.37:0/991583950 conn(0x55b56d259f80 msgr2=0x55b56d25a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:31.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.044+0000 7f75c4a09740 1 -- 172.21.15.37:0/991583950 shutdown_connections 2024-03-28T18:11:31.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.045+0000 7f75c4a09740 1 -- 172.21.15.37:0/991583950 wait complete. 2024-03-28T18:11:31.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.045+0000 7f75c4a09740 1 -- start start 2024-03-28T18:11:31.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.045+0000 7f75c4a09740 1 Processor -- start 2024-03-28T18:11:31.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.046+0000 7f75c4a09740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d25f8b0 0x55b56d2686a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.046+0000 7f75c4a09740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d260260 0x55b56d265a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.046+0000 7f75c4a09740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55b56d268de0 0x55b56d265f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.046+0000 7f75c4a09740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b56d26bd50 con 0x55b56d268de0 2024-03-28T18:11:31.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.046+0000 7f75c4a09740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b56d17f0d0 con 0x55b56d25f8b0 2024-03-28T18:11:31.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.046+0000 7f75c4a09740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b56d26bed0 con 0x55b56d260260 2024-03-28T18:11:31.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.046+0000 7f75c4a09740 10 client.0 _ll_register_callbacks cb 0x55b56d183e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:11:31.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.046+0000 7f75bb400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d25f8b0 0x55b56d2686a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.046+0000 7f75ba000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d260260 0x55b56d265a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.046+0000 7f75bb400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d25f8b0 0x55b56d2686a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:56966/0 (socket says 172.21.15.37:56966) 2024-03-28T18:11:31.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.046+0000 7f75bb400640 1 -- 172.21.15.37:0/3518033621 learned_addr learned my addr 172.21.15.37:0/3518033621 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:31.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.046+0000 7f75bb400640 1 --2- 172.21.15.37:0/3518033621 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55b56d268de0 0x55b56d265f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.047+0000 7f75ba000640 1 -- 172.21.15.37:0/3518033621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d25f8b0 msgr2=0x55b56d2686a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:31.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.047+0000 7f75ba000640 1 --2- 172.21.15.37:0/3518033621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d25f8b0 0x55b56d2686a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.047+0000 7f75ba000640 1 -- 172.21.15.37:0/3518033621 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55b56d268de0 msgr2=0x55b56d265f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:31.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.047+0000 7f75ba000640 1 --2- 172.21.15.37:0/3518033621 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55b56d268de0 0x55b56d265f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.047+0000 7f75ba000640 1 -- 172.21.15.37:0/3518033621 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b56ced73f0 con 0x55b56d260260 2024-03-28T18:11:31.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.047+0000 7f75ba000640 1 --2- 172.21.15.37:0/3518033621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d260260 0x55b56d265a30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f75a8036040 tx=0x7f75a8004560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1835e86c1584cdb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:31.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.047+0000 7f75a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:11:31.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.047+0000 7f75a4c00640 1 -- 172.21.15.37:0/3518033621 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a802cc90 con 0x55b56d260260 2024-03-28T18:11:31.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.047+0000 7f75a4c00640 1 -- 172.21.15.37:0/3518033621 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f75a8005030 con 0x55b56d260260 2024-03-28T18:11:31.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.047+0000 7f75c4a09740 1 -- 172.21.15.37:0/3518033621 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75b4004050 con 0x55b56d260260 2024-03-28T18:11:31.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.048+0000 7f75a4c00640 1 -- 172.21.15.37:0/3518033621 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75a8005230 con 0x55b56d260260 2024-03-28T18:11:31.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.048+0000 7f75c4a09740 10 client.25513 fetch_fsmap learned FSMap version 23 2024-03-28T18:11:31.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.048+0000 7f75c4a09740 1 -- 172.21.15.37:0/3518033621 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b56cfa17d0 con 0x55b56d260260 2024-03-28T18:11:31.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.048+0000 7f75a4c00640 1 -- 172.21.15.37:0/3518033621 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75b4004050 con 0x55b56d260260 2024-03-28T18:11:31.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.048+0000 7f75a4c00640 1 -- 172.21.15.37:0/3518033621 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f75a80035f0 con 0x55b56d260260 2024-03-28T18:11:31.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.048+0000 7f75c4a09740 10 client.25513 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:11:31.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.048+0000 7f75c4a09740 10 client.25513 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-28T18:11:31.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.048+0000 7f75c4a09740 20 client.25513 populate_metadata read hostname 'smithi037' 2024-03-28T18:11:31.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.048+0000 7f75c4a09740 1 --2- 172.21.15.37:0/3518033621 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55b56d278b70 0x55b56d27af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.048+0000 7f75c4a09740 4 client.25513 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:11:31.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.048+0000 7f75c4a09740 1 -- 172.21.15.37:0/3518033621 --> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b56d1d0800 con 0x55b56d278b70 2024-03-28T18:11:31.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.049+0000 7f75bb400640 1 --2- 172.21.15.37:0/3518033621 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55b56d278b70 0x55b56d27af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.049+0000 7f75bb400640 1 --2- 172.21.15.37:0/3518033621 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55b56d278b70 0x55b56d27af60 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=591cf8901592eba1 server_cookie=e8a4e13f1ef019fb in_seq=0 out_seq=0 2024-03-28T18:11:31.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.049+0000 7f75a4c00640 10 client.25513 ms_handle_connect on v2:172.21.15.59:6838/2091629078 2024-03-28T18:11:31.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.050+0000 7f75a4c00640 1 -- 172.21.15.37:0/3518033621 <== mds.1 v2:172.21.15.59:6838/2091629078 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b56d1d0800 con 0x55b56d278b70 2024-03-28T18:11:31.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.050+0000 7f75a4c00640 10 client.25513 handle_command_reply: tid=0 2024-03-28T18:11:31.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.050+0000 7f75c4a09740 1 client.25513 shutdown 2024-03-28T18:11:31.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.050+0000 7f75c4a09740 1 -- 172.21.15.37:0/3518033621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d260260 msgr2=0x55b56d265a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:31.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.050+0000 7f75c4a09740 1 --2- 172.21.15.37:0/3518033621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d260260 0x55b56d265a30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f75a8036040 tx=0x7f75a8004560 comp rx=0 tx=0).stop 2024-03-28T18:11:31.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.050+0000 7f75c4a09740 1 -- 172.21.15.37:0/3518033621 shutdown_connections 2024-03-28T18:11:31.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.050+0000 7f75c4a09740 1 --2- 172.21.15.37:0/3518033621 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55b56d278b70 0x55b56d27af60 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.050+0000 7f75c4a09740 1 --2- 172.21.15.37:0/3518033621 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55b56d268de0 0x55b56d265f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.050+0000 7f75c4a09740 1 --2- 172.21.15.37:0/3518033621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d260260 0x55b56d265a30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.050+0000 7f75a4c00640 0 client.25513 ms_handle_reset on v2:172.21.15.59:6838/2091629078 2024-03-28T18:11:31.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.050+0000 7f75c4a09740 1 --2- 172.21.15.37:0/3518033621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d25f8b0 0x55b56d2686a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.050+0000 7f75c4a09740 1 -- 172.21.15.37:0/3518033621 >> 172.21.15.37:0/3518033621 conn(0x55b56d259f80 msgr2=0x55b56d25bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:31.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.050+0000 7f75c4a09740 1 -- 172.21.15.37:0/3518033621 shutdown_connections 2024-03-28T18:11:31.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.050+0000 7f75c4a09740 1 -- 172.21.15.37:0/3518033621 wait complete. 2024-03-28T18:11:31.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.050+0000 7f75c4a09740 20 client.25513 trim_cache size 0 max 16384 2024-03-28T18:11:31.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.065+0000 7f75c4a09740 1 Processor -- start 2024-03-28T18:11:31.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.065+0000 7f75c4a09740 1 -- start start 2024-03-28T18:11:31.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.065+0000 7f75c4a09740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55b56d25f8b0 0x55b56d268160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.065+0000 7f75c4a09740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d260260 0x55b56d2686a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.065+0000 7f75c4a09740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d268de0 0x55b56d262c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.065+0000 7f75c4a09740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc1b1770 con 0x55b56d25f8b0 2024-03-28T18:11:31.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.065+0000 7f75c4a09740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc1af270 con 0x55b56d260260 2024-03-28T18:11:31.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.065+0000 7f75c4a09740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc1ad310 con 0x55b56d268de0 2024-03-28T18:11:31.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.066+0000 7f75ba000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d260260 0x55b56d2686a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.066+0000 7f75ba000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d260260 0x55b56d2686a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:56968/0 (socket says 172.21.15.37:56968) 2024-03-28T18:11:31.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.066+0000 7f75ba000640 1 -- 172.21.15.37:0/236631018 learned_addr learned my addr 172.21.15.37:0/236631018 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:31.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.066+0000 7f75bb400640 1 --2- 172.21.15.37:0/236631018 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55b56d25f8b0 0x55b56d268160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.066+0000 7f75bb400640 1 --2- 172.21.15.37:0/236631018 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d268de0 0x55b56d262c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.066+0000 7f75ba000640 1 -- 172.21.15.37:0/236631018 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d268de0 msgr2=0x55b56d262c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:31.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.066+0000 7f75ba000640 1 --2- 172.21.15.37:0/236631018 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d268de0 0x55b56d262c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.066+0000 7f75ba000640 1 -- 172.21.15.37:0/236631018 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55b56d25f8b0 msgr2=0x55b56d268160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:31.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.066+0000 7f75ba000640 1 --2- 172.21.15.37:0/236631018 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55b56d25f8b0 0x55b56d268160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.066+0000 7f75ba000640 1 -- 172.21.15.37:0/236631018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b56d266300 con 0x55b56d260260 2024-03-28T18:11:31.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.066+0000 7f75bb400640 1 --2- 172.21.15.37:0/236631018 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d268de0 0x55b56d262c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:11:31.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.066+0000 7f75ba000640 1 --2- 172.21.15.37:0/236631018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d260260 0x55b56d2686a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f75a8005ab0 tx=0x7f75a8009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7e2107d35cc7b1f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:31.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.067+0000 7f75a7400640 1 -- 172.21.15.37:0/236631018 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a8005d50 con 0x55b56d260260 2024-03-28T18:11:31.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.067+0000 7f75a7400640 1 -- 172.21.15.37:0/236631018 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75a8056070 con 0x55b56d260260 2024-03-28T18:11:31.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.067+0000 7f75a7400640 1 -- 172.21.15.37:0/236631018 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a8004e00 con 0x55b56d260260 2024-03-28T18:11:31.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.067+0000 7f75c4a09740 1 -- 172.21.15.37:0/236631018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d260260 msgr2=0x55b56d2686a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:31.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.067+0000 7f75c4a09740 1 --2- 172.21.15.37:0/236631018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d260260 0x55b56d2686a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f75a8005ab0 tx=0x7f75a8009680 comp rx=0 tx=0).stop 2024-03-28T18:11:31.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.067+0000 7f75c4a09740 1 -- 172.21.15.37:0/236631018 shutdown_connections 2024-03-28T18:11:31.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.067+0000 7f75c4a09740 1 --2- 172.21.15.37:0/236631018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d260260 0x55b56d2686a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.067+0000 7f75c4a09740 1 --2- 172.21.15.37:0/236631018 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d268de0 0x55b56d262c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.067+0000 7f75c4a09740 1 --2- 172.21.15.37:0/236631018 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55b56d25f8b0 0x55b56d268160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.068+0000 7f75c4a09740 1 -- 172.21.15.37:0/236631018 >> 172.21.15.37:0/236631018 conn(0x55b56d259f80 msgr2=0x55b56d259160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:31.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.068+0000 7f75c4a09740 1 -- 172.21.15.37:0/236631018 shutdown_connections 2024-03-28T18:11:31.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.068+0000 7f75c4a09740 1 -- 172.21.15.37:0/236631018 wait complete. 2024-03-28T18:11:31.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.068+0000 7f75c4a09740 1 -- start start 2024-03-28T18:11:31.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.068+0000 7f75c4a09740 1 Processor -- start 2024-03-28T18:11:31.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.069+0000 7f75c4a09740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55b56d25f8b0 0x55b56d2d4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.069+0000 7f75c4a09740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d260260 0x55b56d2d5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.069+0000 7f75c4a09740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d268de0 0x55b56d26ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.069+0000 7f75c4a09740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc1ad910 con 0x55b56d25f8b0 2024-03-28T18:11:31.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.069+0000 7f75c4a09740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc1aec10 con 0x55b56d268de0 2024-03-28T18:11:31.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.069+0000 7f75c4a09740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc1adf50 con 0x55b56d260260 2024-03-28T18:11:31.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.069+0000 7f75c4a09740 10 client.0 _ll_register_callbacks cb 0x7f75b40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:11:31.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.069+0000 7f75ba000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d260260 0x55b56d2d5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.069+0000 7f75bb400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55b56d25f8b0 0x55b56d2d4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.069+0000 7f75ba000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d260260 0x55b56d2d5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:54988/0 (socket says 172.21.15.37:54988) 2024-03-28T18:11:31.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.069+0000 7f75ba000640 1 -- 172.21.15.37:0/2284056681 learned_addr learned my addr 172.21.15.37:0/2284056681 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:31.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.070+0000 7f75bb400640 1 --2- 172.21.15.37:0/2284056681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d268de0 0x55b56d26ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.070+0000 7f75ba000640 1 -- 172.21.15.37:0/2284056681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d268de0 msgr2=0x55b56d26ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:31.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.070+0000 7f75ba000640 1 --2- 172.21.15.37:0/2284056681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d268de0 0x55b56d26ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.070+0000 7f75ba000640 1 -- 172.21.15.37:0/2284056681 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55b56d25f8b0 msgr2=0x55b56d2d4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:31.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.070+0000 7f75ba000640 1 --2- 172.21.15.37:0/2284056681 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55b56d25f8b0 0x55b56d2d4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.070+0000 7f75ba000640 1 -- 172.21.15.37:0/2284056681 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b56d26d620 con 0x55b56d260260 2024-03-28T18:11:31.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.070+0000 7f75ba000640 1 --2- 172.21.15.37:0/2284056681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d260260 0x55b56d2d5460 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f75a8004650 tx=0x7f75a8005990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c680e57ad6b36103 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:31.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.070+0000 7f75a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:11:31.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.070+0000 7f75a4c00640 1 -- 172.21.15.37:0/2284056681 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a8003ac0 con 0x55b56d260260 2024-03-28T18:11:31.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.071+0000 7f75a4c00640 1 -- 172.21.15.37:0/2284056681 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f75a8003c60 con 0x55b56d260260 2024-03-28T18:11:31.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.071+0000 7f75c4a09740 1 -- 172.21.15.37:0/2284056681 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75a8056070 con 0x55b56d260260 2024-03-28T18:11:31.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.071+0000 7f75a4c00640 1 -- 172.21.15.37:0/2284056681 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75a802ca90 con 0x55b56d260260 2024-03-28T18:11:31.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.071+0000 7f75c4a09740 10 client.25516 fetch_fsmap learned FSMap version 23 2024-03-28T18:11:31.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.071+0000 7f75c4a09740 1 -- 172.21.15.37:0/2284056681 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b56d2dd720 con 0x55b56d260260 2024-03-28T18:11:31.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.071+0000 7f75a4c00640 1 -- 172.21.15.37:0/2284056681 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75a8056070 con 0x55b56d260260 2024-03-28T18:11:31.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.071+0000 7f75a4c00640 1 -- 172.21.15.37:0/2284056681 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f75a8004ba0 con 0x55b56d260260 2024-03-28T18:11:31.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.071+0000 7f75c4a09740 10 client.25516 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:11:31.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.071+0000 7f75c4a09740 10 client.25516 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-28T18:11:31.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.071+0000 7f75c4a09740 20 client.25516 populate_metadata read hostname 'smithi037' 2024-03-28T18:11:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.071+0000 7f75c4a09740 1 --2- 172.21.15.37:0/2284056681 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55b56d2ddda0 0x55b56d2e01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.071+0000 7f75c4a09740 4 client.25516 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:11:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.071+0000 7f75c4a09740 1 -- 172.21.15.37:0/2284056681 --> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b56d1cab50 con 0x55b56d2ddda0 2024-03-28T18:11:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.072+0000 7f75bb400640 1 --2- 172.21.15.37:0/2284056681 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55b56d2ddda0 0x55b56d2e01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.072+0000 7f75bb400640 1 --2- 172.21.15.37:0/2284056681 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55b56d2ddda0 0x55b56d2e01c0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5f56a11ce37dd1c6 server_cookie=62ddf7e42a050190 in_seq=0 out_seq=0 2024-03-28T18:11:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.072+0000 7f75a4c00640 10 client.25516 ms_handle_connect on v2:172.21.15.59:6838/2091629078 2024-03-28T18:11:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.072+0000 7f75a4c00640 1 -- 172.21.15.37:0/2284056681 <== mds.1 v2:172.21.15.59:6838/2091629078 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b56d1cab50 con 0x55b56d2ddda0 2024-03-28T18:11:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.072+0000 7f75a4c00640 10 client.25516 handle_command_reply: tid=0 2024-03-28T18:11:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.072+0000 7f75c4a09740 1 client.25516 shutdown 2024-03-28T18:11:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.072+0000 7f75c4a09740 1 -- 172.21.15.37:0/2284056681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d260260 msgr2=0x55b56d2d5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.072+0000 7f75c4a09740 1 --2- 172.21.15.37:0/2284056681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d260260 0x55b56d2d5460 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f75a8004650 tx=0x7f75a8005990 comp rx=0 tx=0).stop 2024-03-28T18:11:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.072+0000 7f75c4a09740 1 -- 172.21.15.37:0/2284056681 shutdown_connections 2024-03-28T18:11:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.072+0000 7f75c4a09740 1 --2- 172.21.15.37:0/2284056681 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55b56d2ddda0 0x55b56d2e01c0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.072+0000 7f75c4a09740 1 --2- 172.21.15.37:0/2284056681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b56d268de0 0x55b56d26ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.072+0000 7f75c4a09740 1 --2- 172.21.15.37:0/2284056681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b56d260260 0x55b56d2d5460 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.072+0000 7f75a4c00640 0 client.25516 ms_handle_reset on v2:172.21.15.59:6838/2091629078 2024-03-28T18:11:31.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.072+0000 7f75c4a09740 1 --2- 172.21.15.37:0/2284056681 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55b56d25f8b0 0x55b56d2d4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.072+0000 7f75c4a09740 1 -- 172.21.15.37:0/2284056681 >> 172.21.15.37:0/2284056681 conn(0x55b56d259f80 msgr2=0x55b56d260640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:31.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.073+0000 7f75c4a09740 1 -- 172.21.15.37:0/2284056681 shutdown_connections 2024-03-28T18:11:31.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.073+0000 7f75c4a09740 1 -- 172.21.15.37:0/2284056681 wait complete. 2024-03-28T18:11:31.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.073+0000 7f75c4a09740 20 client.25516 trim_cache size 0 max 16384 2024-03-28T18:11:31.080 INFO:teuthology.orchestra.run.smithi037.stdout:[] 2024-03-28T18:11:31.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.073+0000 7f75bbe00640 1 -- 172.21.15.37:0/1662680992 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f75a00a2a80 msgr2=0x7f75a00a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:31.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.073+0000 7f75bbe00640 1 --2- 172.21.15.37:0/1662680992 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f75a00a2a80 0x7f75a00a4f40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f75ac0cc9d0 tx=0x7f75ac0d2d10 comp rx=0 tx=0).stop 2024-03-28T18:11:31.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.073+0000 7f75bbe00640 1 -- 172.21.15.37:0/1662680992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75bc101530 msgr2=0x7f75bc0739a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:31.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.073+0000 7f75bbe00640 1 --2- 172.21.15.37:0/1662680992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75bc101530 0x7f75bc0739a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f75ac00e9f0 tx=0x7f75ac00eeb0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.074+0000 7f75bbe00640 1 -- 172.21.15.37:0/1662680992 shutdown_connections 2024-03-28T18:11:31.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.074+0000 7f75bbe00640 1 --2- 172.21.15.37:0/1662680992 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f75a00a2a80 0x7f75a00a4f40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.074+0000 7f75bbe00640 1 --2- 172.21.15.37:0/1662680992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75bc1075f0 0x7f75bc074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.074+0000 7f75bbe00640 1 --2- 172.21.15.37:0/1662680992 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75bc101ef0 0x7f75bc073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.074+0000 7f75bbe00640 1 --2- 172.21.15.37:0/1662680992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75bc101530 0x7f75bc0739a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.074+0000 7f75bbe00640 1 -- 172.21.15.37:0/1662680992 >> 172.21.15.37:0/1662680992 conn(0x7f75bc0fd040 msgr2=0x7f75bc0fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:31.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.074+0000 7f75bbe00640 1 -- 172.21.15.37:0/1662680992 shutdown_connections 2024-03-28T18:11:31.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.074+0000 7f75bbe00640 1 -- 172.21.15.37:0/1662680992 wait complete. 2024-03-28T18:11:31.091 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:11:31.091 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:11:31.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.306+0000 7fbf8ec00640 1 -- 172.21.15.37:0/3950844718 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf88103840 msgr2=0x7fbf88103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:31.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.306+0000 7fbf8ec00640 1 --2- 172.21.15.37:0/3950844718 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf88103840 0x7fbf88103cc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbf70009930 tx=0x7fbf7002bc30 comp rx=0 tx=0).stop 2024-03-28T18:11:31.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.306+0000 7fbf8ec00640 1 -- 172.21.15.37:0/3950844718 shutdown_connections 2024-03-28T18:11:31.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.306+0000 7fbf8ec00640 1 --2- 172.21.15.37:0/3950844718 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbf88104200 0x7fbf88110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.306+0000 7fbf8ec00640 1 --2- 172.21.15.37:0/3950844718 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf88103840 0x7fbf88103cc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.306+0000 7fbf8ec00640 1 --2- 172.21.15.37:0/3950844718 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf88109850 0x7fbf88109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.306+0000 7fbf8ec00640 1 -- 172.21.15.37:0/3950844718 >> 172.21.15.37:0/3950844718 conn(0x7fbf880ff2f0 msgr2=0x7fbf88101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:31.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.307+0000 7fbf8ec00640 1 -- 172.21.15.37:0/3950844718 shutdown_connections 2024-03-28T18:11:31.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.307+0000 7fbf8ec00640 1 -- 172.21.15.37:0/3950844718 wait complete. 2024-03-28T18:11:31.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.308+0000 7fbf8ec00640 1 Processor -- start 2024-03-28T18:11:31.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.308+0000 7fbf8ec00640 1 -- start start 2024-03-28T18:11:31.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.308+0000 7fbf8ec00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf88103840 0x7fbf88075bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.309+0000 7fbf8ec00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbf88104200 0x7fbf880760f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.309+0000 7fbf8ec00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf88109850 0x7fbf88076630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.309+0000 7fbf8ec00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf88113cb0 con 0x7fbf88104200 2024-03-28T18:11:31.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.309+0000 7fbf8ec00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf88113b30 con 0x7fbf88109850 2024-03-28T18:11:31.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.309+0000 7fbf8ec00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf88113e30 con 0x7fbf88103840 2024-03-28T18:11:31.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.309+0000 7fbf87e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf88103840 0x7fbf88075bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.309+0000 7fbf87e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf88103840 0x7fbf88075bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55018/0 (socket says 172.21.15.37:55018) 2024-03-28T18:11:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.309+0000 7fbf87e00640 1 -- 172.21.15.37:0/2141578152 learned_addr learned my addr 172.21.15.37:0/2141578152 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.309+0000 7fbf8cc00640 1 --2- 172.21.15.37:0/2141578152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf88109850 0x7fbf88076630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.309+0000 7fbf7f400640 1 --2- 172.21.15.37:0/2141578152 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbf88104200 0x7fbf880760f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.309+0000 7fbf87e00640 1 -- 172.21.15.37:0/2141578152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf88109850 msgr2=0x7fbf88076630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.309+0000 7fbf87e00640 1 --2- 172.21.15.37:0/2141578152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf88109850 0x7fbf88076630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.309+0000 7fbf87e00640 1 -- 172.21.15.37:0/2141578152 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbf88104200 msgr2=0x7fbf880760f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.309+0000 7fbf87e00640 1 --2- 172.21.15.37:0/2141578152 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbf88104200 0x7fbf880760f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.309+0000 7fbf87e00640 1 -- 172.21.15.37:0/2141578152 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf881aa890 con 0x7fbf88103840 2024-03-28T18:11:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.310+0000 7fbf87e00640 1 --2- 172.21.15.37:0/2141578152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf88103840 0x7fbf88075bb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbf7800ebc0 tx=0x7fbf7800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.310+0000 7fbf85600640 1 -- 172.21.15.37:0/2141578152 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf78004d20 con 0x7fbf88103840 2024-03-28T18:11:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.310+0000 7fbf85600640 1 -- 172.21.15.37:0/2141578152 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf78005500 con 0x7fbf88103840 2024-03-28T18:11:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.310+0000 7fbf8ec00640 1 -- 172.21.15.37:0/2141578152 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf881aab80 con 0x7fbf88103840 2024-03-28T18:11:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.310+0000 7fbf85600640 1 -- 172.21.15.37:0/2141578152 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf780107f0 con 0x7fbf88103840 2024-03-28T18:11:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.310+0000 7fbf8ec00640 1 -- 172.21.15.37:0/2141578152 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf881ab0a0 con 0x7fbf88103840 2024-03-28T18:11:31.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.315+0000 7fbf85600640 1 -- 172.21.15.37:0/2141578152 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 25) v1 ==== 137466+0+0 (secure 0 0 0) 0x7fbf780056b0 con 0x7fbf88103840 2024-03-28T18:11:31.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.315+0000 7fbf915fd740 1 Processor -- start 2024-03-28T18:11:31.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.315+0000 7fbf915fd740 1 -- start start 2024-03-28T18:11:31.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.315+0000 7fbf915fd740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa486d0 0x56543aa48ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.315+0000 7fbf915fd740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa49080 0x56543aa51630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.315+0000 7fbf915fd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa51c00 0x56543aa54050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.315+0000 7fbf915fd740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x56543a9ac2c0 con 0x56543aa49080 2024-03-28T18:11:31.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.315+0000 7fbf915fd740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x56543a954c90 con 0x56543aa486d0 2024-03-28T18:11:31.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.315+0000 7fbf915fd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56543a9a8360 con 0x56543aa51c00 2024-03-28T18:11:31.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.316+0000 7fbf8cc00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa486d0 0x56543aa48ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.316+0000 7fbf8cc00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa486d0 0x56543aa48ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:57010/0 (socket says 172.21.15.37:57010) 2024-03-28T18:11:31.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.316+0000 7fbf8cc00640 1 -- 172.21.15.37:0/520144509 learned_addr learned my addr 172.21.15.37:0/520144509 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:31.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.316+0000 7fbf7f400640 1 --2- 172.21.15.37:0/520144509 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa51c00 0x56543aa54050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.316+0000 7fbf87e00640 1 --2- 172.21.15.37:0/520144509 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa49080 0x56543aa51630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.316+0000 7fbf8cc00640 1 -- 172.21.15.37:0/520144509 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa51c00 msgr2=0x56543aa54050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:31.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.316+0000 7fbf8cc00640 1 --2- 172.21.15.37:0/520144509 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa51c00 0x56543aa54050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.316+0000 7fbf8cc00640 1 -- 172.21.15.37:0/520144509 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa49080 msgr2=0x56543aa51630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:31.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.316+0000 7fbf8cc00640 1 --2- 172.21.15.37:0/520144509 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa49080 0x56543aa51630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.316+0000 7fbf8cc00640 1 -- 172.21.15.37:0/520144509 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56543a979080 con 0x56543aa486d0 2024-03-28T18:11:31.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.316+0000 7fbf85600640 1 --2- 172.21.15.37:0/2141578152 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fbf6c0a2a80 0x7fbf6c0a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.316+0000 7fbf8cc00640 1 --2- 172.21.15.37:0/520144509 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa486d0 0x56543aa48ab0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbf8000b700 tx=0x7fbf8000bbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e0df838761062f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:31.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.317+0000 7fbf85600640 1 -- 172.21.15.37:0/2141578152 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbf780cf550 con 0x7fbf88103840 2024-03-28T18:11:31.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.317+0000 7fbf87e00640 1 --2- 172.21.15.37:0/2141578152 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fbf6c0a2a80 0x7fbf6c0a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.317+0000 7fbf7d200640 1 -- 172.21.15.37:0/520144509 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf80003dd0 con 0x56543aa486d0 2024-03-28T18:11:31.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.317+0000 7fbf7d200640 1 -- 172.21.15.37:0/520144509 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf80015070 con 0x56543aa486d0 2024-03-28T18:11:31.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.317+0000 7fbf7d200640 1 -- 172.21.15.37:0/520144509 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf800106a0 con 0x56543aa486d0 2024-03-28T18:11:31.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.318+0000 7fbf915fd740 1 -- 172.21.15.37:0/520144509 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa486d0 msgr2=0x56543aa48ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:31.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.318+0000 7fbf915fd740 1 --2- 172.21.15.37:0/520144509 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa486d0 0x56543aa48ab0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbf8000b700 tx=0x7fbf8000bbc0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.318+0000 7fbf87e00640 1 --2- 172.21.15.37:0/2141578152 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fbf6c0a2a80 0x7fbf6c0a4f40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbf780ccc10 tx=0x7fbf78009a70 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:31.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.318+0000 7fbf915fd740 1 -- 172.21.15.37:0/520144509 shutdown_connections 2024-03-28T18:11:31.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.318+0000 7fbf915fd740 1 --2- 172.21.15.37:0/520144509 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa51c00 0x56543aa54050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.318+0000 7fbf915fd740 1 --2- 172.21.15.37:0/520144509 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa49080 0x56543aa51630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.318+0000 7fbf915fd740 1 --2- 172.21.15.37:0/520144509 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa486d0 0x56543aa48ab0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.318+0000 7fbf915fd740 1 -- 172.21.15.37:0/520144509 >> 172.21.15.37:0/520144509 conn(0x56543aa42da0 msgr2=0x56543aa43180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:31.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.318+0000 7fbf915fd740 1 -- 172.21.15.37:0/520144509 shutdown_connections 2024-03-28T18:11:31.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.318+0000 7fbf915fd740 1 -- 172.21.15.37:0/520144509 wait complete. 2024-03-28T18:11:31.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.318+0000 7fbf915fd740 1 -- start start 2024-03-28T18:11:31.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.318+0000 7fbf915fd740 1 Processor -- start 2024-03-28T18:11:31.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.319+0000 7fbf915fd740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa486d0 0x56543aa51520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.319+0000 7fbf915fd740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa49080 0x56543aa4e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.319+0000 7fbf915fd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa51c00 0x56543aa4ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.319+0000 7fbf915fd740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x56543aa54b70 con 0x56543aa49080 2024-03-28T18:11:31.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.319+0000 7fbf915fd740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x56543a9680d0 con 0x56543aa486d0 2024-03-28T18:11:31.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.319+0000 7fbf915fd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56543aa54cf0 con 0x56543aa51c00 2024-03-28T18:11:31.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.319+0000 7fbf8cc00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa486d0 0x56543aa51520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.319+0000 7fbf8cc00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa486d0 0x56543aa51520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:57014/0 (socket says 172.21.15.37:57014) 2024-03-28T18:11:31.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.319+0000 7fbf8cc00640 1 -- 172.21.15.37:0/1717367846 learned_addr learned my addr 172.21.15.37:0/1717367846 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:31.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.319+0000 7fbf7f400640 1 --2- 172.21.15.37:0/1717367846 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa49080 0x56543aa4e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.319+0000 7fbf915fd740 10 client.0 _ll_register_callbacks cb 0x56543a96ce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:11:31.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.319+0000 7fbf8cc00640 1 --2- 172.21.15.37:0/1717367846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa51c00 0x56543aa4ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.320+0000 7fbf8cc00640 1 -- 172.21.15.37:0/1717367846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa51c00 msgr2=0x56543aa4ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:31.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.320+0000 7fbf8cc00640 1 --2- 172.21.15.37:0/1717367846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa51c00 0x56543aa4ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.320+0000 7fbf8cc00640 1 -- 172.21.15.37:0/1717367846 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa49080 msgr2=0x56543aa4e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:31.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.320+0000 7fbf8cc00640 1 --2- 172.21.15.37:0/1717367846 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa49080 0x56543aa4e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.320+0000 7fbf8cc00640 1 -- 172.21.15.37:0/1717367846 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56543a6c03f0 con 0x56543aa486d0 2024-03-28T18:11:31.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.320+0000 7fbf8cc00640 1 --2- 172.21.15.37:0/1717367846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa486d0 0x56543aa51520 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbf80011f60 tx=0x7fbf80011e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5afe465e6b8416c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:31.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.320+0000 7fbf75600640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:11:31.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.320+0000 7fbf75600640 1 -- 172.21.15.37:0/1717367846 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf80023040 con 0x56543aa486d0 2024-03-28T18:11:31.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.320+0000 7fbf75600640 1 -- 172.21.15.37:0/1717367846 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbf80010840 con 0x56543aa486d0 2024-03-28T18:11:31.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.320+0000 7fbf915fd740 1 -- 172.21.15.37:0/1717367846 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbf80015070 con 0x56543aa486d0 2024-03-28T18:11:31.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.321+0000 7fbf75600640 1 -- 172.21.15.37:0/1717367846 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf80010a40 con 0x56543aa486d0 2024-03-28T18:11:31.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.321+0000 7fbf915fd740 10 client.25505 fetch_fsmap learned FSMap version 23 2024-03-28T18:11:31.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.321+0000 7fbf915fd740 1 -- 172.21.15.37:0/1717367846 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56543a78a7d0 con 0x56543aa486d0 2024-03-28T18:11:31.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.321+0000 7fbf75600640 1 -- 172.21.15.37:0/1717367846 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf80015070 con 0x56543aa486d0 2024-03-28T18:11:31.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.321+0000 7fbf75600640 1 -- 172.21.15.37:0/1717367846 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fbf80011a40 con 0x56543aa486d0 2024-03-28T18:11:31.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.321+0000 7fbf915fd740 10 client.25505 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:11:31.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.321+0000 7fbf915fd740 10 client.25505 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:11:31.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.321+0000 7fbf915fd740 20 client.25505 populate_metadata read hostname 'smithi037' 2024-03-28T18:11:31.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.321+0000 7fbf915fd740 1 --2- 172.21.15.37:0/1717367846 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x56543aa619b0 0x56543aa63da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.321+0000 7fbf915fd740 4 client.25505 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:11:31.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.321+0000 7fbf915fd740 1 -- 172.21.15.37:0/1717367846 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56543a9b9800 con 0x56543aa619b0 2024-03-28T18:11:31.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.322+0000 7fbf7f400640 1 --2- 172.21.15.37:0/1717367846 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x56543aa619b0 0x56543aa63da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.322+0000 7fbf7f400640 1 --2- 172.21.15.37:0/1717367846 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x56543aa619b0 0x56543aa63da0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ba3956fb2c1e061 server_cookie=e1f321d73a9c870d in_seq=0 out_seq=0 2024-03-28T18:11:31.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.322+0000 7fbf75600640 10 client.25505 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:11:31.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.323+0000 7fbf75600640 1 -- 172.21.15.37:0/1717367846 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56543a9b9800 con 0x56543aa619b0 2024-03-28T18:11:31.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.323+0000 7fbf75600640 10 client.25505 handle_command_reply: tid=0 2024-03-28T18:11:31.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.323+0000 7fbf915fd740 1 client.25505 shutdown 2024-03-28T18:11:31.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.323+0000 7fbf915fd740 1 -- 172.21.15.37:0/1717367846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa486d0 msgr2=0x56543aa51520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:31.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.323+0000 7fbf915fd740 1 --2- 172.21.15.37:0/1717367846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa486d0 0x56543aa51520 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbf80011f60 tx=0x7fbf80011e60 comp rx=0 tx=0).stop 2024-03-28T18:11:31.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.323+0000 7fbf915fd740 1 -- 172.21.15.37:0/1717367846 shutdown_connections 2024-03-28T18:11:31.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.323+0000 7fbf915fd740 1 --2- 172.21.15.37:0/1717367846 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x56543aa619b0 0x56543aa63da0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.323+0000 7fbf915fd740 1 --2- 172.21.15.37:0/1717367846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa51c00 0x56543aa4ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.323+0000 7fbf915fd740 1 --2- 172.21.15.37:0/1717367846 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa49080 0x56543aa4e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.323+0000 7fbf915fd740 1 --2- 172.21.15.37:0/1717367846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa486d0 0x56543aa51520 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.323+0000 7fbf75600640 0 client.25505 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:11:31.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.323+0000 7fbf915fd740 1 -- 172.21.15.37:0/1717367846 >> 172.21.15.37:0/1717367846 conn(0x56543aa42da0 msgr2=0x56543aa44af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:31.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.324+0000 7fbf915fd740 1 -- 172.21.15.37:0/1717367846 shutdown_connections 2024-03-28T18:11:31.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.324+0000 7fbf915fd740 1 -- 172.21.15.37:0/1717367846 wait complete. 2024-03-28T18:11:31.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.324+0000 7fbf915fd740 20 client.25505 trim_cache size 0 max 16384 2024-03-28T18:11:31.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.339+0000 7fbf915fd740 1 Processor -- start 2024-03-28T18:11:31.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.339+0000 7fbf915fd740 1 -- start start 2024-03-28T18:11:31.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.339+0000 7fbf915fd740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa486d0 0x56543aa41620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.339+0000 7fbf915fd740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa49080 0x56543aa4e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.339+0000 7fbf915fd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa51c00 0x56543aa4e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.339+0000 7fbf915fd740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf881b5e30 con 0x56543aa486d0 2024-03-28T18:11:31.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.339+0000 7fbf915fd740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf881b3930 con 0x56543aa49080 2024-03-28T18:11:31.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.339+0000 7fbf915fd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf881b19d0 con 0x56543aa51c00 2024-03-28T18:11:31.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.340+0000 7fbf8cc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa486d0 0x56543aa41620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.340+0000 7fbf8cc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa486d0 0x56543aa41620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:42092/0 (socket says 172.21.15.37:42092) 2024-03-28T18:11:31.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.340+0000 7fbf8cc00640 1 -- 172.21.15.37:0/55288464 learned_addr learned my addr 172.21.15.37:0/55288464 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:31.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.340+0000 7fbf7f400640 1 --2- 172.21.15.37:0/55288464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa49080 0x56543aa4e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.340+0000 7fbf8cc00640 1 --2- 172.21.15.37:0/55288464 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa51c00 0x56543aa4e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.340+0000 7fbf7f400640 1 -- 172.21.15.37:0/55288464 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa51c00 msgr2=0x56543aa4e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:31.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.340+0000 7fbf7f400640 1 --2- 172.21.15.37:0/55288464 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa51c00 0x56543aa4e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.340+0000 7fbf7f400640 1 -- 172.21.15.37:0/55288464 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa486d0 msgr2=0x56543aa41620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:31.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.340+0000 7fbf7f400640 1 --2- 172.21.15.37:0/55288464 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa486d0 0x56543aa41620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.340+0000 7fbf7f400640 1 -- 172.21.15.37:0/55288464 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56543aa544b0 con 0x56543aa49080 2024-03-28T18:11:31.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.340+0000 7fbf8cc00640 1 --2- 172.21.15.37:0/55288464 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa486d0 0x56543aa41620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:11:31.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.341+0000 7fbf7f400640 1 --2- 172.21.15.37:0/55288464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa49080 0x56543aa4e0d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbf70036040 tx=0x7fbf7002df90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39bed07bd5d87660 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:31.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.341+0000 7fbf77e00640 1 -- 172.21.15.37:0/55288464 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf70004040 con 0x56543aa49080 2024-03-28T18:11:31.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.341+0000 7fbf77e00640 1 -- 172.21.15.37:0/55288464 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf70051030 con 0x56543aa49080 2024-03-28T18:11:31.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.341+0000 7fbf77e00640 1 -- 172.21.15.37:0/55288464 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf70004aa0 con 0x56543aa49080 2024-03-28T18:11:31.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.342+0000 7fbf915fd740 1 -- 172.21.15.37:0/55288464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa49080 msgr2=0x56543aa4e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:31.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.342+0000 7fbf915fd740 1 --2- 172.21.15.37:0/55288464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa49080 0x56543aa4e0d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbf70036040 tx=0x7fbf7002df90 comp rx=0 tx=0).stop 2024-03-28T18:11:31.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.342+0000 7fbf915fd740 1 -- 172.21.15.37:0/55288464 shutdown_connections 2024-03-28T18:11:31.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.342+0000 7fbf915fd740 1 --2- 172.21.15.37:0/55288464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa49080 0x56543aa4e0d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.342+0000 7fbf915fd740 1 --2- 172.21.15.37:0/55288464 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa51c00 0x56543aa4e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.342+0000 7fbf915fd740 1 --2- 172.21.15.37:0/55288464 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa486d0 0x56543aa41620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.342+0000 7fbf915fd740 1 -- 172.21.15.37:0/55288464 >> 172.21.15.37:0/55288464 conn(0x56543aa42da0 msgr2=0x56543aa49cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:31.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.342+0000 7fbf915fd740 1 -- 172.21.15.37:0/55288464 shutdown_connections 2024-03-28T18:11:31.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.342+0000 7fbf915fd740 1 -- 172.21.15.37:0/55288464 wait complete. 2024-03-28T18:11:31.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.342+0000 7fbf915fd740 1 -- start start 2024-03-28T18:11:31.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.342+0000 7fbf915fd740 1 Processor -- start 2024-03-28T18:11:31.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.343+0000 7fbf915fd740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa486d0 0x56543aa59ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.343+0000 7fbf915fd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa49080 0x56543aa57f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.343+0000 7fbf915fd740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa51c00 0x56543aa58480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.343+0000 7fbf915fd740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf881b1fd0 con 0x56543aa51c00 2024-03-28T18:11:31.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.343+0000 7fbf915fd740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf881b32d0 con 0x56543aa486d0 2024-03-28T18:11:31.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.343+0000 7fbf915fd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf881b2610 con 0x56543aa49080 2024-03-28T18:11:31.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.343+0000 7fbf915fd740 10 client.0 _ll_register_callbacks cb 0x7fbf70009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:11:31.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.343+0000 7fbf7f400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa49080 0x56543aa57f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.343+0000 7fbf7f400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa49080 0x56543aa57f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55058/0 (socket says 172.21.15.37:55058) 2024-03-28T18:11:31.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.343+0000 7fbf7f400640 1 -- 172.21.15.37:0/259322757 learned_addr learned my addr 172.21.15.37:0/259322757 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:31.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.343+0000 7fbf8cc00640 1 --2- 172.21.15.37:0/259322757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa486d0 0x56543aa59ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.343+0000 7fbf8cc00640 1 --2- 172.21.15.37:0/259322757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa51c00 0x56543aa58480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.344+0000 7fbf7f400640 1 -- 172.21.15.37:0/259322757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa486d0 msgr2=0x56543aa59ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:31.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.344+0000 7fbf7f400640 1 --2- 172.21.15.37:0/259322757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa486d0 0x56543aa59ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.344+0000 7fbf7f400640 1 -- 172.21.15.37:0/259322757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa51c00 msgr2=0x56543aa58480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:31.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.344+0000 7fbf7f400640 1 --2- 172.21.15.37:0/259322757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa51c00 0x56543aa58480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.344+0000 7fbf7f400640 1 -- 172.21.15.37:0/259322757 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56543aa58b40 con 0x56543aa49080 2024-03-28T18:11:31.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.344+0000 7fbf8cc00640 1 --2- 172.21.15.37:0/259322757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa51c00 0x56543aa58480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:11:31.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.344+0000 7fbf7f400640 1 --2- 172.21.15.37:0/259322757 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa49080 0x56543aa57f40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbf70005ec0 tx=0x7fbf70000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39c2f511f5c3cbc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:31.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.344+0000 7fbf75600640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:11:31.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.344+0000 7fbf75600640 1 -- 172.21.15.37:0/259322757 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf70004030 con 0x56543aa49080 2024-03-28T18:11:31.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.344+0000 7fbf915fd740 1 -- 172.21.15.37:0/259322757 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbf70051030 con 0x56543aa49080 2024-03-28T18:11:31.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.344+0000 7fbf75600640 1 -- 172.21.15.37:0/259322757 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbf700041d0 con 0x56543aa49080 2024-03-28T18:11:31.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.344+0000 7fbf75600640 1 -- 172.21.15.37:0/259322757 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf70065050 con 0x56543aa49080 2024-03-28T18:11:31.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.344+0000 7fbf915fd740 10 client.25534 fetch_fsmap learned FSMap version 23 2024-03-28T18:11:31.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.344+0000 7fbf915fd740 1 -- 172.21.15.37:0/259322757 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56543aabdf20 con 0x56543aa49080 2024-03-28T18:11:31.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.345+0000 7fbf75600640 1 -- 172.21.15.37:0/259322757 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf70051030 con 0x56543aa49080 2024-03-28T18:11:31.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.345+0000 7fbf75600640 1 -- 172.21.15.37:0/259322757 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fbf700051e0 con 0x56543aa49080 2024-03-28T18:11:31.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.345+0000 7fbf915fd740 10 client.25534 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:11:31.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.345+0000 7fbf915fd740 10 client.25534 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:11:31.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.345+0000 7fbf915fd740 20 client.25534 populate_metadata read hostname 'smithi037' 2024-03-28T18:11:31.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.345+0000 7fbf915fd740 1 --2- 172.21.15.37:0/259322757 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x56543aabe680 0x56543aac0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:31.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.345+0000 7fbf915fd740 4 client.25534 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:11:31.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.345+0000 7fbf915fd740 1 -- 172.21.15.37:0/259322757 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56543a9b3b50 con 0x56543aabe680 2024-03-28T18:11:31.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.345+0000 7fbf8cc00640 1 --2- 172.21.15.37:0/259322757 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x56543aabe680 0x56543aac0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:31.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.346+0000 7fbf8cc00640 1 --2- 172.21.15.37:0/259322757 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x56543aabe680 0x56543aac0aa0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5ed698513a35dd7 server_cookie=10413e7895d40aa8 in_seq=0 out_seq=0 2024-03-28T18:11:31.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.346+0000 7fbf75600640 10 client.25534 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:11:31.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.346+0000 7fbf75600640 1 -- 172.21.15.37:0/259322757 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56543a9b3b50 con 0x56543aabe680 2024-03-28T18:11:31.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.346+0000 7fbf75600640 10 client.25534 handle_command_reply: tid=0 2024-03-28T18:11:31.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.346+0000 7fbf915fd740 1 client.25534 shutdown 2024-03-28T18:11:31.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.346+0000 7fbf915fd740 1 -- 172.21.15.37:0/259322757 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa49080 msgr2=0x56543aa57f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:31.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.346+0000 7fbf915fd740 1 --2- 172.21.15.37:0/259322757 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa49080 0x56543aa57f40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbf70005ec0 tx=0x7fbf70000bc0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.346+0000 7fbf915fd740 1 -- 172.21.15.37:0/259322757 shutdown_connections 2024-03-28T18:11:31.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.346+0000 7fbf915fd740 1 --2- 172.21.15.37:0/259322757 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x56543aabe680 0x56543aac0aa0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.346+0000 7fbf915fd740 1 --2- 172.21.15.37:0/259322757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56543aa51c00 0x56543aa58480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.346+0000 7fbf915fd740 1 --2- 172.21.15.37:0/259322757 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56543aa49080 0x56543aa57f40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.346+0000 7fbf75600640 0 client.25534 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:11:31.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.346+0000 7fbf915fd740 1 --2- 172.21.15.37:0/259322757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56543aa486d0 0x56543aa59ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.346+0000 7fbf915fd740 1 -- 172.21.15.37:0/259322757 >> 172.21.15.37:0/259322757 conn(0x56543aa42da0 msgr2=0x56543aa68450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:31.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.346+0000 7fbf915fd740 1 -- 172.21.15.37:0/259322757 shutdown_connections 2024-03-28T18:11:31.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.346+0000 7fbf915fd740 1 -- 172.21.15.37:0/259322757 wait complete. 2024-03-28T18:11:31.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.347+0000 7fbf915fd740 20 client.25534 trim_cache size 0 max 16384 2024-03-28T18:11:31.351 INFO:teuthology.orchestra.run.smithi037.stdout:[] 2024-03-28T18:11:31.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.347+0000 7fbf8ec00640 1 -- 172.21.15.37:0/2141578152 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fbf6c0a2a80 msgr2=0x7fbf6c0a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:31.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.347+0000 7fbf8ec00640 1 --2- 172.21.15.37:0/2141578152 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fbf6c0a2a80 0x7fbf6c0a4f40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbf780ccc10 tx=0x7fbf78009a70 comp rx=0 tx=0).stop 2024-03-28T18:11:31.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.347+0000 7fbf8ec00640 1 -- 172.21.15.37:0/2141578152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf88103840 msgr2=0x7fbf88075bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:31.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.347+0000 7fbf8ec00640 1 --2- 172.21.15.37:0/2141578152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf88103840 0x7fbf88075bb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbf7800ebc0 tx=0x7fbf7800c6a0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.347+0000 7fbf8ec00640 1 -- 172.21.15.37:0/2141578152 shutdown_connections 2024-03-28T18:11:31.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.347+0000 7fbf8ec00640 1 --2- 172.21.15.37:0/2141578152 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fbf6c0a2a80 0x7fbf6c0a4f40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.347+0000 7fbf8ec00640 1 --2- 172.21.15.37:0/2141578152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf88109850 0x7fbf88076630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.347+0000 7fbf8ec00640 1 --2- 172.21.15.37:0/2141578152 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbf88104200 0x7fbf880760f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.347+0000 7fbf8ec00640 1 --2- 172.21.15.37:0/2141578152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf88103840 0x7fbf88075bb0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:31.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.347+0000 7fbf8ec00640 1 -- 172.21.15.37:0/2141578152 >> 172.21.15.37:0/2141578152 conn(0x7fbf880ff2f0 msgr2=0x7fbf880ffb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:31.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.347+0000 7fbf8ec00640 1 -- 172.21.15.37:0/2141578152 shutdown_connections 2024-03-28T18:11:31.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:31.347+0000 7fbf8ec00640 1 -- 172.21.15.37:0/2141578152 wait complete. 2024-03-28T18:11:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:31 smithi037 ceph-mon[31460]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:31.935 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:31 smithi059 ceph-mon[36190]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:32.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:31 smithi170 ceph-mon[36280]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:32.365 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:11:32.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:11:32.366 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:11:32.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.605+0000 7fb741c00640 1 -- 172.21.15.37:0/4164982052 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb73c10de80 msgr2=0x7fb73c110270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:32.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.605+0000 7fb741c00640 1 --2- 172.21.15.37:0/4164982052 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb73c10de80 0x7fb73c110270 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb734009be0 tx=0x7fb73402bef0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.605+0000 7fb741c00640 1 -- 172.21.15.37:0/4164982052 shutdown_connections 2024-03-28T18:11:32.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.605+0000 7fb741c00640 1 --2- 172.21.15.37:0/4164982052 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb73c10de80 0x7fb73c110270 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.605+0000 7fb741c00640 1 --2- 172.21.15.37:0/4164982052 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb73c100c80 0x7fb73c10d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.605+0000 7fb741c00640 1 --2- 172.21.15.37:0/4164982052 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb73c1002d0 0x7fb73c1006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.605+0000 7fb741c00640 1 -- 172.21.15.37:0/4164982052 >> 172.21.15.37:0/4164982052 conn(0x7fb73c0fbef0 msgr2=0x7fb73c0fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:32.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.605+0000 7fb741c00640 1 -- 172.21.15.37:0/4164982052 shutdown_connections 2024-03-28T18:11:32.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.605+0000 7fb741c00640 1 -- 172.21.15.37:0/4164982052 wait complete. 2024-03-28T18:11:32.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.607+0000 7fb741c00640 1 Processor -- start 2024-03-28T18:11:32.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.607+0000 7fb741c00640 1 -- start start 2024-03-28T18:11:32.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.607+0000 7fb741c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb73c1002d0 0x7fb73c199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.607+0000 7fb741c00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb73c100c80 0x7fb73c19a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.607+0000 7fb741c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb73c10de80 0x7fb73c19e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.607+0000 7fb741c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c113b70 con 0x7fb73c1002d0 2024-03-28T18:11:32.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.607+0000 7fb741c00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c1139f0 con 0x7fb73c10de80 2024-03-28T18:11:32.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.607+0000 7fb741c00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c113cf0 con 0x7fb73c100c80 2024-03-28T18:11:32.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.608+0000 7fb73be00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb73c1002d0 0x7fb73c199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.608+0000 7fb73b400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb73c100c80 0x7fb73c19a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.608+0000 7fb73b400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb73c100c80 0x7fb73c19a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55072/0 (socket says 172.21.15.37:55072) 2024-03-28T18:11:32.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.608+0000 7fb73b400640 1 -- 172.21.15.37:0/2300330090 learned_addr learned my addr 172.21.15.37:0/2300330090 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:32.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.608+0000 7fb73be00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb73c1002d0 0x7fb73c199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:42124/0 (socket says 172.21.15.37:42124) 2024-03-28T18:11:32.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.608+0000 7fb741200640 1 --2- 172.21.15.37:0/2300330090 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb73c10de80 0x7fb73c19e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.608+0000 7fb73b400640 1 -- 172.21.15.37:0/2300330090 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb73c10de80 msgr2=0x7fb73c19e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:32.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.608+0000 7fb73b400640 1 --2- 172.21.15.37:0/2300330090 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb73c10de80 0x7fb73c19e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.608+0000 7fb73b400640 1 -- 172.21.15.37:0/2300330090 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb73c1002d0 msgr2=0x7fb73c199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:32.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.608+0000 7fb73b400640 1 --2- 172.21.15.37:0/2300330090 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb73c1002d0 0x7fb73c199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.608+0000 7fb73b400640 1 -- 172.21.15.37:0/2300330090 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73c19efe0 con 0x7fb73c100c80 2024-03-28T18:11:32.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.608+0000 7fb73b400640 1 --2- 172.21.15.37:0/2300330090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb73c100c80 0x7fb73c19a530 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb72800e9c0 tx=0x7fb72800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:32.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.609+0000 7fb738c00640 1 -- 172.21.15.37:0/2300330090 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72800cde0 con 0x7fb73c100c80 2024-03-28T18:11:32.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.609+0000 7fb738c00640 1 -- 172.21.15.37:0/2300330090 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7280035f0 con 0x7fb73c100c80 2024-03-28T18:11:32.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.609+0000 7fb738c00640 1 -- 172.21.15.37:0/2300330090 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb728010690 con 0x7fb73c100c80 2024-03-28T18:11:32.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.609+0000 7fb741c00640 1 -- 172.21.15.37:0/2300330090 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73c19f2d0 con 0x7fb73c100c80 2024-03-28T18:11:32.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.609+0000 7fb741c00640 1 -- 172.21.15.37:0/2300330090 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb73c1a6b60 con 0x7fb73c100c80 2024-03-28T18:11:32.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.612+0000 7fb746033740 1 Processor -- start 2024-03-28T18:11:32.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.612+0000 7fb746033740 1 -- start start 2024-03-28T18:11:32.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.613+0000 7fb746033740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562c07772290 0x562c07772670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.613+0000 7fb738c00640 1 -- 172.21.15.37:0/2300330090 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 25) v1 ==== 137466+0+0 (secure 0 0 0) 0x7fb7280037a0 con 0x7fb73c100c80 2024-03-28T18:11:32.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.613+0000 7fb746033740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c07772bb0 0x562c0777b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.613+0000 7fb746033740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c0777b790 0x562c0777db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.613+0000 7fb746033740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c076d6eb0 con 0x562c07772290 2024-03-28T18:11:32.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.613+0000 7fb746033740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c0767fa60 con 0x562c07772bb0 2024-03-28T18:11:32.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.613+0000 7fb746033740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c076d2f50 con 0x562c0777b790 2024-03-28T18:11:32.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.613+0000 7fb73be00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c07772bb0 0x562c0777b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.613+0000 7fb741200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562c07772290 0x562c07772670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.613+0000 7fb73be00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c07772bb0 0x562c0777b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:57060/0 (socket says 172.21.15.37:57060) 2024-03-28T18:11:32.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.613+0000 7fb73be00640 1 -- 172.21.15.37:0/119873358 learned_addr learned my addr 172.21.15.37:0/119873358 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:32.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.613+0000 7fb73b400640 1 --2- 172.21.15.37:0/119873358 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c0777b790 0x562c0777db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.614+0000 7fb73be00640 1 -- 172.21.15.37:0/119873358 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c0777b790 msgr2=0x562c0777db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:32.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.614+0000 7fb73be00640 1 --2- 172.21.15.37:0/119873358 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c0777b790 0x562c0777db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.614+0000 7fb73be00640 1 -- 172.21.15.37:0/119873358 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562c07772290 msgr2=0x562c07772670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:32.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.614+0000 7fb73be00640 1 --2- 172.21.15.37:0/119873358 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562c07772290 0x562c07772670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.614+0000 7fb73be00640 1 -- 172.21.15.37:0/119873358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c074b37e0 con 0x562c07772bb0 2024-03-28T18:11:32.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.614+0000 7fb73be00640 1 --2- 172.21.15.37:0/119873358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c07772bb0 0x562c0777b250 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009cf0 tx=0x7fb72c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=298f35784bc28a9d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:32.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.614+0000 7fb731600640 1 -- 172.21.15.37:0/119873358 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c004230 con 0x562c07772bb0 2024-03-28T18:11:32.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.614+0000 7fb731600640 1 -- 172.21.15.37:0/119873358 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb72c0043d0 con 0x562c07772bb0 2024-03-28T18:11:32.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.614+0000 7fb731600640 1 -- 172.21.15.37:0/119873358 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c019960 con 0x562c07772bb0 2024-03-28T18:11:32.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.614+0000 7fb738c00640 1 --2- 172.21.15.37:0/2300330090 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fb7200a2a80 0x7fb7200a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.615+0000 7fb738c00640 1 -- 172.21.15.37:0/2300330090 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb7280cf220 con 0x7fb73c100c80 2024-03-28T18:11:32.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.615+0000 7fb741200640 1 --2- 172.21.15.37:0/2300330090 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fb7200a2a80 0x7fb7200a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.615+0000 7fb746033740 1 -- 172.21.15.37:0/119873358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c07772bb0 msgr2=0x562c0777b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:32.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.615+0000 7fb746033740 1 --2- 172.21.15.37:0/119873358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c07772bb0 0x562c0777b250 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009cf0 tx=0x7fb72c011810 comp rx=0 tx=0).stop 2024-03-28T18:11:32.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.615+0000 7fb746033740 1 -- 172.21.15.37:0/119873358 shutdown_connections 2024-03-28T18:11:32.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.615+0000 7fb746033740 1 --2- 172.21.15.37:0/119873358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c07772bb0 0x562c0777b250 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.615+0000 7fb746033740 1 --2- 172.21.15.37:0/119873358 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c0777b790 0x562c0777db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.615+0000 7fb746033740 1 --2- 172.21.15.37:0/119873358 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562c07772290 0x562c07772670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.615+0000 7fb746033740 1 -- 172.21.15.37:0/119873358 >> 172.21.15.37:0/119873358 conn(0x562c0776c9c0 msgr2=0x562c0776cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:32.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.615+0000 7fb746033740 1 -- 172.21.15.37:0/119873358 shutdown_connections 2024-03-28T18:11:32.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.615+0000 7fb746033740 1 -- 172.21.15.37:0/119873358 wait complete. 2024-03-28T18:11:32.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.616+0000 7fb741200640 1 --2- 172.21.15.37:0/2300330090 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fb7200a2a80 0x7fb7200a4f40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb73402c430 tx=0x7fb734005fd0 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:32.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.616+0000 7fb746033740 1 -- start start 2024-03-28T18:11:32.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.616+0000 7fb746033740 1 Processor -- start 2024-03-28T18:11:32.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.617+0000 7fb746033740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562c07772290 0x562c0777b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.617+0000 7fb746033740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772bb0 0x562c07778430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.617+0000 7fb746033740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c0777b790 0x562c07778970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.617+0000 7fb746033740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c0777e580 con 0x562c07772290 2024-03-28T18:11:32.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.617+0000 7fb746033740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c07692eb0 con 0x562c0777b790 2024-03-28T18:11:32.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.617+0000 7fb746033740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c0777e700 con 0x562c07772bb0 2024-03-28T18:11:32.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.617+0000 7fb746033740 10 client.0 _ll_register_callbacks cb 0x562c07687150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:11:32.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.617+0000 7fb73b400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772bb0 0x562c07778430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.617+0000 7fb73be00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562c07772290 0x562c0777b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.617+0000 7fb73b400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772bb0 0x562c07778430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55100/0 (socket says 172.21.15.37:55100) 2024-03-28T18:11:32.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.617+0000 7fb73b400640 1 -- 172.21.15.37:0/3255937299 learned_addr learned my addr 172.21.15.37:0/3255937299 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:32.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.617+0000 7fb741200640 1 --2- 172.21.15.37:0/3255937299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c0777b790 0x562c07778970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.618+0000 7fb73b400640 1 -- 172.21.15.37:0/3255937299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c0777b790 msgr2=0x562c07778970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:32.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.618+0000 7fb73b400640 1 --2- 172.21.15.37:0/3255937299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c0777b790 0x562c07778970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.618+0000 7fb73b400640 1 -- 172.21.15.37:0/3255937299 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562c07772290 msgr2=0x562c0777b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:32.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.618+0000 7fb73b400640 1 --2- 172.21.15.37:0/3255937299 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562c07772290 0x562c0777b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.618+0000 7fb73b400640 1 -- 172.21.15.37:0/3255937299 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562c07513f70 con 0x562c07772bb0 2024-03-28T18:11:32.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.618+0000 7fb73b400640 1 --2- 172.21.15.37:0/3255937299 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772bb0 0x562c07778430 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb72800c6a0 tx=0x7fb728004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67f01aaee910f672 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:32.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.618+0000 7fb726000640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:11:32.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.618+0000 7fb726000640 1 -- 172.21.15.37:0/3255937299 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72809b510 con 0x562c07772bb0 2024-03-28T18:11:32.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.618+0000 7fb726000640 1 -- 172.21.15.37:0/3255937299 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb7280ccb90 con 0x562c07772bb0 2024-03-28T18:11:32.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.618+0000 7fb746033740 1 -- 172.21.15.37:0/3255937299 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb72c0043d0 con 0x562c07772bb0 2024-03-28T18:11:32.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.618+0000 7fb726000640 1 -- 172.21.15.37:0/3255937299 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb72809b0a0 con 0x562c07772bb0 2024-03-28T18:11:32.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.618+0000 7fb746033740 10 client.25546 fetch_fsmap learned FSMap version 23 2024-03-28T18:11:32.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.618+0000 7fb746033740 1 -- 172.21.15.37:0/3255937299 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562c073ab170 con 0x562c07772bb0 2024-03-28T18:11:32.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.619+0000 7fb726000640 1 -- 172.21.15.37:0/3255937299 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb72c0043d0 con 0x562c07772bb0 2024-03-28T18:11:32.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.619+0000 7fb726000640 1 -- 172.21.15.37:0/3255937299 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fb72809ab90 con 0x562c07772bb0 2024-03-28T18:11:32.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.619+0000 7fb746033740 10 client.25546 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:11:32.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.619+0000 7fb746033740 10 client.25546 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:11:32.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.619+0000 7fb746033740 20 client.25546 populate_metadata read hostname 'smithi037' 2024-03-28T18:11:32.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.619+0000 7fb746033740 1 --2- 172.21.15.37:0/3255937299 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x562c0778b790 0x562c0778db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.619+0000 7fb746033740 4 client.25546 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:11:32.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.619+0000 7fb746033740 1 -- 172.21.15.37:0/3255937299 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562c076e43f0 con 0x562c0778b790 2024-03-28T18:11:32.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.619+0000 7fb73be00640 1 --2- 172.21.15.37:0/3255937299 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x562c0778b790 0x562c0778db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.620+0000 7fb73be00640 1 --2- 172.21.15.37:0/3255937299 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x562c0778b790 0x562c0778db80 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f15a5df28236094d server_cookie=b20e82a0e50ce95f in_seq=0 out_seq=0 2024-03-28T18:11:32.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.620+0000 7fb726000640 10 client.25546 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:11:32.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.621+0000 7fb726000640 1 -- 172.21.15.37:0/3255937299 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562c076e43f0 con 0x562c0778b790 2024-03-28T18:11:32.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.621+0000 7fb726000640 10 client.25546 handle_command_reply: tid=0 2024-03-28T18:11:32.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.621+0000 7fb746033740 1 client.25546 shutdown 2024-03-28T18:11:32.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.621+0000 7fb746033740 1 -- 172.21.15.37:0/3255937299 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772bb0 msgr2=0x562c07778430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:32.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.621+0000 7fb746033740 1 --2- 172.21.15.37:0/3255937299 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772bb0 0x562c07778430 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb72800c6a0 tx=0x7fb728004730 comp rx=0 tx=0).stop 2024-03-28T18:11:32.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.621+0000 7fb746033740 1 -- 172.21.15.37:0/3255937299 shutdown_connections 2024-03-28T18:11:32.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.621+0000 7fb746033740 1 --2- 172.21.15.37:0/3255937299 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x562c0778b790 0x562c0778db80 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.621+0000 7fb746033740 1 --2- 172.21.15.37:0/3255937299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c0777b790 0x562c07778970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.621+0000 7fb746033740 1 --2- 172.21.15.37:0/3255937299 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772bb0 0x562c07778430 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.621+0000 7fb746033740 1 --2- 172.21.15.37:0/3255937299 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562c07772290 0x562c0777b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.621+0000 7fb726000640 0 client.25546 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:11:32.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.621+0000 7fb746033740 1 -- 172.21.15.37:0/3255937299 >> 172.21.15.37:0/3255937299 conn(0x562c0776c9c0 msgr2=0x562c0776e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:32.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.621+0000 7fb746033740 1 -- 172.21.15.37:0/3255937299 shutdown_connections 2024-03-28T18:11:32.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.621+0000 7fb746033740 1 -- 172.21.15.37:0/3255937299 wait complete. 2024-03-28T18:11:32.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.621+0000 7fb746033740 20 client.25546 trim_cache size 0 max 16384 2024-03-28T18:11:32.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.639+0000 7fb746033740 1 Processor -- start 2024-03-28T18:11:32.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.639+0000 7fb746033740 1 -- start start 2024-03-28T18:11:32.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.640+0000 7fb746033740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562c07772290 0x562c0777ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.640+0000 7fb746033740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772bb0 0x562c0777b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.640+0000 7fb746033740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c0777b790 0x562c077736a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.640+0000 7fb746033740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c1b16f0 con 0x562c07772290 2024-03-28T18:11:32.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.640+0000 7fb746033740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c1af1f0 con 0x562c0777b790 2024-03-28T18:11:32.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.640+0000 7fb746033740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c1ad290 con 0x562c07772bb0 2024-03-28T18:11:32.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.640+0000 7fb73b400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772bb0 0x562c0777b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.640+0000 7fb73b400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772bb0 0x562c0777b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55108/0 (socket says 172.21.15.37:55108) 2024-03-28T18:11:32.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.640+0000 7fb73b400640 1 -- 172.21.15.37:0/1793792993 learned_addr learned my addr 172.21.15.37:0/1793792993 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:32.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.640+0000 7fb741200640 1 --2- 172.21.15.37:0/1793792993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c0777b790 0x562c077736a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.640+0000 7fb73be00640 1 --2- 172.21.15.37:0/1793792993 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562c07772290 0x562c0777ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.640+0000 7fb73b400640 1 -- 172.21.15.37:0/1793792993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c0777b790 msgr2=0x562c077736a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:32.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.640+0000 7fb73b400640 1 --2- 172.21.15.37:0/1793792993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c0777b790 0x562c077736a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.640+0000 7fb73b400640 1 -- 172.21.15.37:0/1793792993 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562c07772290 msgr2=0x562c0777ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:32.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.640+0000 7fb73b400640 1 --2- 172.21.15.37:0/1793792993 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562c07772290 0x562c0777ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.640+0000 7fb73b400640 1 -- 172.21.15.37:0/1793792993 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c0778e1b0 con 0x562c07772bb0 2024-03-28T18:11:32.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.641+0000 7fb73b400640 1 --2- 172.21.15.37:0/1793792993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772bb0 0x562c0777b0a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb728097ba0 tx=0x7fb728000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59818736a7a39d4f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:32.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.641+0000 7fb730c00640 1 -- 172.21.15.37:0/1793792993 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7280d6030 con 0x562c07772bb0 2024-03-28T18:11:32.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.641+0000 7fb730c00640 1 -- 172.21.15.37:0/1793792993 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7280d2880 con 0x562c07772bb0 2024-03-28T18:11:32.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.641+0000 7fb730c00640 1 -- 172.21.15.37:0/1793792993 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72809b280 con 0x562c07772bb0 2024-03-28T18:11:32.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.641+0000 7fb746033740 1 -- 172.21.15.37:0/1793792993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772bb0 msgr2=0x562c0777b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:32.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.641+0000 7fb746033740 1 --2- 172.21.15.37:0/1793792993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772bb0 0x562c0777b0a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb728097ba0 tx=0x7fb728000bc0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.642+0000 7fb746033740 1 -- 172.21.15.37:0/1793792993 shutdown_connections 2024-03-28T18:11:32.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.642+0000 7fb746033740 1 --2- 172.21.15.37:0/1793792993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c0777b790 0x562c077736a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.642+0000 7fb746033740 1 --2- 172.21.15.37:0/1793792993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772bb0 0x562c0777b0a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.642+0000 7fb746033740 1 --2- 172.21.15.37:0/1793792993 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562c07772290 0x562c0777ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.642+0000 7fb746033740 1 -- 172.21.15.37:0/1793792993 >> 172.21.15.37:0/1793792993 conn(0x562c0776c9c0 msgr2=0x562c07772f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:32.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.642+0000 7fb746033740 1 -- 172.21.15.37:0/1793792993 shutdown_connections 2024-03-28T18:11:32.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.642+0000 7fb746033740 1 -- 172.21.15.37:0/1793792993 wait complete. 2024-03-28T18:11:32.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.642+0000 7fb746033740 1 -- start start 2024-03-28T18:11:32.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.642+0000 7fb746033740 1 Processor -- start 2024-03-28T18:11:32.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.643+0000 7fb746033740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772290 0x562c07776040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.643+0000 7fb746033740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562c07772bb0 0x562c07780920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.643+0000 7fb746033740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c0777b790 0x562c07780e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.643+0000 7fb746033740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c1ad890 con 0x562c07772bb0 2024-03-28T18:11:32.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.643+0000 7fb746033740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c1aeb90 con 0x562c0777b790 2024-03-28T18:11:32.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.643+0000 7fb746033740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c1aded0 con 0x562c07772290 2024-03-28T18:11:32.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.643+0000 7fb746033740 10 client.0 _ll_register_callbacks cb 0x7fb7280d2a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:11:32.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.643+0000 7fb73be00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772290 0x562c07776040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.643+0000 7fb73be00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772290 0x562c07776040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55118/0 (socket says 172.21.15.37:55118) 2024-03-28T18:11:32.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.643+0000 7fb73be00640 1 -- 172.21.15.37:0/3969588042 learned_addr learned my addr 172.21.15.37:0/3969588042 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:32.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.643+0000 7fb73b400640 1 --2- 172.21.15.37:0/3969588042 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562c07772bb0 0x562c07780920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.643+0000 7fb741200640 1 --2- 172.21.15.37:0/3969588042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c0777b790 0x562c07780e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.643+0000 7fb73be00640 1 -- 172.21.15.37:0/3969588042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c0777b790 msgr2=0x562c07780e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:32.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.643+0000 7fb73be00640 1 --2- 172.21.15.37:0/3969588042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c0777b790 0x562c07780e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.643+0000 7fb73be00640 1 -- 172.21.15.37:0/3969588042 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562c07772bb0 msgr2=0x562c07780920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:32.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.643+0000 7fb73be00640 1 --2- 172.21.15.37:0/3969588042 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562c07772bb0 0x562c07780920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.643+0000 7fb73be00640 1 -- 172.21.15.37:0/3969588042 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562c07781550 con 0x562c07772290 2024-03-28T18:11:32.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.644+0000 7fb73be00640 1 --2- 172.21.15.37:0/3969588042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772290 0x562c07776040 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb72c00a010 tx=0x7fb72c009320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e84261b09b314ee5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:32.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.644+0000 7fb726000640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:11:32.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.644+0000 7fb726000640 1 -- 172.21.15.37:0/3969588042 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c02b310 con 0x562c07772290 2024-03-28T18:11:32.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.644+0000 7fb746033740 1 -- 172.21.15.37:0/3969588042 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562c076de740 con 0x562c07772290 2024-03-28T18:11:32.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.644+0000 7fb726000640 1 -- 172.21.15.37:0/3969588042 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb72c02a070 con 0x562c07772290 2024-03-28T18:11:32.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.644+0000 7fb726000640 1 -- 172.21.15.37:0/3969588042 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb72c005250 con 0x562c07772290 2024-03-28T18:11:32.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.644+0000 7fb746033740 10 client.25552 fetch_fsmap learned FSMap version 23 2024-03-28T18:11:32.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.644+0000 7fb746033740 1 -- 172.21.15.37:0/3969588042 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562c077e7ef0 con 0x562c07772290 2024-03-28T18:11:32.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.645+0000 7fb726000640 1 -- 172.21.15.37:0/3969588042 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x562c076de740 con 0x562c07772290 2024-03-28T18:11:32.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.645+0000 7fb726000640 1 -- 172.21.15.37:0/3969588042 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fb72c019530 con 0x562c07772290 2024-03-28T18:11:32.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.645+0000 7fb746033740 10 client.25552 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:11:32.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.645+0000 7fb746033740 10 client.25552 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:11:32.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.645+0000 7fb746033740 20 client.25552 populate_metadata read hostname 'smithi037' 2024-03-28T18:11:32.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.645+0000 7fb746033740 1 --2- 172.21.15.37:0/3969588042 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x562c077e83d0 0x562c077ea7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.645+0000 7fb746033740 4 client.25552 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:11:32.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.645+0000 7fb746033740 1 -- 172.21.15.37:0/3969588042 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x562c0765ca40 con 0x562c077e83d0 2024-03-28T18:11:32.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.645+0000 7fb73b400640 1 --2- 172.21.15.37:0/3969588042 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x562c077e83d0 0x562c077ea7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.646+0000 7fb73b400640 1 --2- 172.21.15.37:0/3969588042 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x562c077e83d0 0x562c077ea7c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35474f29e4201a17 server_cookie=3890fd088b653b47 in_seq=0 out_seq=0 2024-03-28T18:11:32.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.646+0000 7fb726000640 10 client.25552 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:11:32.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.647+0000 7fb726000640 1 -- 172.21.15.37:0/3969588042 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x562c0765ca40 con 0x562c077e83d0 2024-03-28T18:11:32.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.647+0000 7fb726000640 10 client.25552 handle_command_reply: tid=0 2024-03-28T18:11:32.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.647+0000 7fb746033740 1 client.25552 shutdown 2024-03-28T18:11:32.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.647+0000 7fb746033740 1 -- 172.21.15.37:0/3969588042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772290 msgr2=0x562c07776040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:32.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.647+0000 7fb746033740 1 --2- 172.21.15.37:0/3969588042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772290 0x562c07776040 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb72c00a010 tx=0x7fb72c009320 comp rx=0 tx=0).stop 2024-03-28T18:11:32.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.647+0000 7fb746033740 1 -- 172.21.15.37:0/3969588042 shutdown_connections 2024-03-28T18:11:32.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.647+0000 7fb746033740 1 --2- 172.21.15.37:0/3969588042 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x562c077e83d0 0x562c077ea7c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.647+0000 7fb746033740 1 --2- 172.21.15.37:0/3969588042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562c0777b790 0x562c07780e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.647+0000 7fb746033740 1 --2- 172.21.15.37:0/3969588042 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562c07772bb0 0x562c07780920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.647+0000 7fb746033740 1 --2- 172.21.15.37:0/3969588042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c07772290 0x562c07776040 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.647+0000 7fb726000640 0 client.25552 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:11:32.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.647+0000 7fb746033740 1 -- 172.21.15.37:0/3969588042 >> 172.21.15.37:0/3969588042 conn(0x562c0776c9c0 msgr2=0x562c07773820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:32.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.648+0000 7fb746033740 1 -- 172.21.15.37:0/3969588042 shutdown_connections 2024-03-28T18:11:32.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.648+0000 7fb746033740 1 -- 172.21.15.37:0/3969588042 wait complete. 2024-03-28T18:11:32.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.648+0000 7fb746033740 20 client.25552 trim_cache size 0 max 16384 2024-03-28T18:11:32.651 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:11:32.651 INFO:teuthology.orchestra.run.smithi037.stdout: "return_code": 0, 2024-03-28T18:11:32.652 INFO:teuthology.orchestra.run.smithi037.stdout: "scrub_tag": "59d357e0-5aaf-4517-bd4c-d439d845171d", 2024-03-28T18:11:32.652 INFO:teuthology.orchestra.run.smithi037.stdout: "mode": "asynchronous" 2024-03-28T18:11:32.652 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:11:32.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.649+0000 7fb741c00640 1 -- 172.21.15.37:0/2300330090 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fb7200a2a80 msgr2=0x7fb7200a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:32.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.649+0000 7fb741c00640 1 --2- 172.21.15.37:0/2300330090 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fb7200a2a80 0x7fb7200a4f40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb73402c430 tx=0x7fb734005fd0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.649+0000 7fb741c00640 1 -- 172.21.15.37:0/2300330090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb73c100c80 msgr2=0x7fb73c19a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:32.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.649+0000 7fb741c00640 1 --2- 172.21.15.37:0/2300330090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb73c100c80 0x7fb73c19a530 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb72800e9c0 tx=0x7fb72800ee80 comp rx=0 tx=0).stop 2024-03-28T18:11:32.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.649+0000 7fb741c00640 1 -- 172.21.15.37:0/2300330090 shutdown_connections 2024-03-28T18:11:32.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.649+0000 7fb741c00640 1 --2- 172.21.15.37:0/2300330090 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fb7200a2a80 0x7fb7200a4f40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.649+0000 7fb741c00640 1 --2- 172.21.15.37:0/2300330090 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb73c10de80 0x7fb73c19e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.649+0000 7fb741c00640 1 --2- 172.21.15.37:0/2300330090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb73c100c80 0x7fb73c19a530 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.649+0000 7fb741c00640 1 --2- 172.21.15.37:0/2300330090 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb73c1002d0 0x7fb73c199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.649+0000 7fb741c00640 1 -- 172.21.15.37:0/2300330090 >> 172.21.15.37:0/2300330090 conn(0x7fb73c0fbef0 msgr2=0x7fb73c10f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:32.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.650+0000 7fb741c00640 1 -- 172.21.15.37:0/2300330090 shutdown_connections 2024-03-28T18:11:32.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.650+0000 7fb741c00640 1 -- 172.21.15.37:0/2300330090 wait complete. 2024-03-28T18:11:32.667 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:11:32.668 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:11:32.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.889+0000 7f83aa400640 1 -- 172.21.15.37:0/1016407540 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83a4103050 msgr2=0x7f83a410b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:32.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.889+0000 7f83aa400640 1 --2- 172.21.15.37:0/1016407540 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83a4103050 0x7f83a410b530 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8390009900 tx=0x7f839002bcd0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.890+0000 7f83aa400640 1 -- 172.21.15.37:0/1016407540 shutdown_connections 2024-03-28T18:11:32.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.890+0000 7f83aa400640 1 --2- 172.21.15.37:0/1016407540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83a410bb00 0x7f83a410df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.890+0000 7f83aa400640 1 --2- 172.21.15.37:0/1016407540 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83a4103050 0x7f83a410b530 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.890+0000 7f83aa400640 1 --2- 172.21.15.37:0/1016407540 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83a4102730 0x7f83a4102b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.890+0000 7f83aa400640 1 -- 172.21.15.37:0/1016407540 >> 172.21.15.37:0/1016407540 conn(0x7f83a40fbef0 msgr2=0x7f83a40fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:32.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.890+0000 7f83aa400640 1 -- 172.21.15.37:0/1016407540 shutdown_connections 2024-03-28T18:11:32.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.890+0000 7f83aa400640 1 -- 172.21.15.37:0/1016407540 wait complete. 2024-03-28T18:11:32.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.891+0000 7f83aa400640 1 Processor -- start 2024-03-28T18:11:32.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.891+0000 7f83aa400640 1 -- start start 2024-03-28T18:11:32.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.892+0000 7f83aa400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83a4102730 0x7f83a4197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.892+0000 7f83aa400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83a4103050 0x7f83a4198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.892+0000 7f83aa400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83a410bb00 0x7f83a419c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.892+0000 7f83aa400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a4111920 con 0x7f83a4103050 2024-03-28T18:11:32.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.892+0000 7f83aa400640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a41117a0 con 0x7f83a410bb00 2024-03-28T18:11:32.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.892+0000 7f83aa400640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a4111aa0 con 0x7f83a4102730 2024-03-28T18:11:32.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.892+0000 7f83a9000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83a4102730 0x7f83a4197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.892+0000 7f83a9000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83a4102730 0x7f83a4197e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:55140/0 (socket says 172.21.15.37:55140) 2024-03-28T18:11:32.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.892+0000 7f83a9000640 1 -- 172.21.15.37:0/2597444166 learned_addr learned my addr 172.21.15.37:0/2597444166 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:32.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.892+0000 7f83a9a00640 1 --2- 172.21.15.37:0/2597444166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83a410bb00 0x7f83a419c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.892+0000 7f83a3e00640 1 --2- 172.21.15.37:0/2597444166 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83a4103050 0x7f83a4198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.892+0000 7f83a9000640 1 -- 172.21.15.37:0/2597444166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83a410bb00 msgr2=0x7f83a419c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:32.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.892+0000 7f83a9000640 1 --2- 172.21.15.37:0/2597444166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83a410bb00 0x7f83a419c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.892+0000 7f83a9000640 1 -- 172.21.15.37:0/2597444166 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83a4103050 msgr2=0x7f83a4198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:32.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.892+0000 7f83a9000640 1 --2- 172.21.15.37:0/2597444166 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83a4103050 0x7f83a4198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.892+0000 7f83a9000640 1 -- 172.21.15.37:0/2597444166 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a419cde0 con 0x7f83a4102730 2024-03-28T18:11:32.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.892+0000 7f83a3e00640 1 --2- 172.21.15.37:0/2597444166 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83a4103050 0x7f83a4198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:11:32.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.893+0000 7f83a9000640 1 --2- 172.21.15.37:0/2597444166 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83a4102730 0x7f83a4197e40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f839800e9f0 tx=0x7f839800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:32.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.893+0000 7f83a1600640 1 -- 172.21.15.37:0/2597444166 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839800cd80 con 0x7f83a4102730 2024-03-28T18:11:32.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.893+0000 7f83a1600640 1 -- 172.21.15.37:0/2597444166 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8398010910 con 0x7f83a4102730 2024-03-28T18:11:32.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.893+0000 7f83a1600640 1 -- 172.21.15.37:0/2597444166 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8398018990 con 0x7f83a4102730 2024-03-28T18:11:32.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.893+0000 7f83aa400640 1 -- 172.21.15.37:0/2597444166 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a419d0d0 con 0x7f83a4102730 2024-03-28T18:11:32.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.893+0000 7f83aa400640 1 -- 172.21.15.37:0/2597444166 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a41a4960 con 0x7f83a4102730 2024-03-28T18:11:32.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.897+0000 7f83a1600640 1 -- 172.21.15.37:0/2597444166 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 25) v1 ==== 137466+0+0 (secure 0 0 0) 0x7f8398010ac0 con 0x7f83a4102730 2024-03-28T18:11:32.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.897+0000 7f83ae7c0740 1 Processor -- start 2024-03-28T18:11:32.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.897+0000 7f83ae7c0740 1 -- start start 2024-03-28T18:11:32.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.897+0000 7f83ae7c0740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 0x55c2ab994f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.897+0000 7f83ae7c0740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab995500 0x55c2ab995980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.897+0000 7f83ae7c0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c2ab99e870 0x55c2ab9a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.897+0000 7f83ae7c0740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ab8f5250 con 0x55c2ab995500 2024-03-28T18:11:32.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.897+0000 7f83ae7c0740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ab89da30 con 0x55c2ab9961e0 2024-03-28T18:11:32.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.897+0000 7f83ae7c0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ab8f12f0 con 0x55c2ab99e870 2024-03-28T18:11:32.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.898+0000 7f83a9a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 0x55c2ab994f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.898+0000 7f83a9a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 0x55c2ab994f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:57112/0 (socket says 172.21.15.37:57112) 2024-03-28T18:11:32.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.898+0000 7f83a9a00640 1 -- 172.21.15.37:0/814060633 learned_addr learned my addr 172.21.15.37:0/814060633 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:32.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.898+0000 7f83a9000640 1 --2- 172.21.15.37:0/814060633 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab995500 0x55c2ab995980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.898+0000 7f83a3e00640 1 --2- 172.21.15.37:0/814060633 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c2ab99e870 0x55c2ab9a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.898+0000 7f83a9a00640 1 -- 172.21.15.37:0/814060633 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c2ab99e870 msgr2=0x55c2ab9a0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:32.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.898+0000 7f83a9a00640 1 --2- 172.21.15.37:0/814060633 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c2ab99e870 0x55c2ab9a0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.898+0000 7f83a9a00640 1 -- 172.21.15.37:0/814060633 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab995500 msgr2=0x55c2ab995980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:32.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.898+0000 7f83a9a00640 1 --2- 172.21.15.37:0/814060633 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab995500 0x55c2ab995980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.898+0000 7f83a9a00640 1 -- 172.21.15.37:0/814060633 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2ab888690 con 0x55c2ab9961e0 2024-03-28T18:11:32.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.898+0000 7f83a1600640 1 --2- 172.21.15.37:0/2597444166 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f838c0a2a80 0x7f838c0a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.899+0000 7f83a9a00640 1 --2- 172.21.15.37:0/814060633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 0x55c2ab994f30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f839c009f50 tx=0x7f839c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e8ad260fdcc7b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:32.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.899+0000 7f83a1600640 1 -- 172.21.15.37:0/2597444166 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f83980cf380 con 0x7f83a4102730 2024-03-28T18:11:32.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.899+0000 7f83a9000640 1 --2- 172.21.15.37:0/2597444166 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f838c0a2a80 0x7f838c0a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.899+0000 7f8396000640 1 -- 172.21.15.37:0/814060633 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c004d20 con 0x55c2ab9961e0 2024-03-28T18:11:32.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.899+0000 7f8396000640 1 -- 172.21.15.37:0/814060633 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f839c004050 con 0x55c2ab9961e0 2024-03-28T18:11:32.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.899+0000 7f8396000640 1 -- 172.21.15.37:0/814060633 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c005280 con 0x55c2ab9961e0 2024-03-28T18:11:32.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.899+0000 7f83ae7c0740 1 -- 172.21.15.37:0/814060633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 msgr2=0x55c2ab994f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:32.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.899+0000 7f83a9000640 1 --2- 172.21.15.37:0/2597444166 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f838c0a2a80 0x7f838c0a4f40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f83980cca40 tx=0x7f83980d2d80 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:32.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.899+0000 7f83ae7c0740 1 --2- 172.21.15.37:0/814060633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 0x55c2ab994f30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f839c009f50 tx=0x7f839c0118d0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.900+0000 7f83ae7c0740 1 -- 172.21.15.37:0/814060633 shutdown_connections 2024-03-28T18:11:32.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.900+0000 7f83ae7c0740 1 --2- 172.21.15.37:0/814060633 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c2ab99e870 0x55c2ab9a0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.900+0000 7f83ae7c0740 1 --2- 172.21.15.37:0/814060633 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab995500 0x55c2ab995980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.900+0000 7f83ae7c0740 1 --2- 172.21.15.37:0/814060633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 0x55c2ab994f30 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.900+0000 7f83ae7c0740 1 -- 172.21.15.37:0/814060633 >> 172.21.15.37:0/814060633 conn(0x55c2ab958200 msgr2=0x55c2ab98ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:32.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.900+0000 7f83ae7c0740 1 -- 172.21.15.37:0/814060633 shutdown_connections 2024-03-28T18:11:32.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.900+0000 7f83ae7c0740 1 -- 172.21.15.37:0/814060633 wait complete. 2024-03-28T18:11:32.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.900+0000 7f83ae7c0740 1 -- start start 2024-03-28T18:11:32.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.900+0000 7f83ae7c0740 1 Processor -- start 2024-03-28T18:11:32.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.901+0000 7f83ae7c0740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab995500 0x55c2ab99df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.901+0000 7f83ae7c0740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 0x55c2ab99b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.901+0000 7f83ae7c0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c2ab99e870 0x55c2ab99b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.901+0000 7f83ae7c0740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ab9a16e0 con 0x55c2ab995500 2024-03-28T18:11:32.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.901+0000 7f83ae7c0740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ab8b0e80 con 0x55c2ab9961e0 2024-03-28T18:11:32.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.901+0000 7f83ae7c0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ab9a1860 con 0x55c2ab99e870 2024-03-28T18:11:32.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.901+0000 7f83ae7c0740 10 client.0 _ll_register_callbacks cb 0x55c2ab8c3130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:11:32.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.901+0000 7f83a3e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 0x55c2ab99b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.901+0000 7f83a3e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 0x55c2ab99b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:57128/0 (socket says 172.21.15.37:57128) 2024-03-28T18:11:32.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.901+0000 7f83a3e00640 1 -- 172.21.15.37:0/2375613181 learned_addr learned my addr 172.21.15.37:0/2375613181 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:32.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.901+0000 7f83a9a00640 1 --2- 172.21.15.37:0/2375613181 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab995500 0x55c2ab99df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.902+0000 7f83a3e00640 1 -- 172.21.15.37:0/2375613181 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c2ab99e870 msgr2=0x55c2ab99b840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:11:32.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.902+0000 7f83a3e00640 1 --2- 172.21.15.37:0/2375613181 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c2ab99e870 0x55c2ab99b840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.902+0000 7f83a3e00640 1 -- 172.21.15.37:0/2375613181 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab995500 msgr2=0x55c2ab99df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:32.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.902+0000 7f83a3e00640 1 --2- 172.21.15.37:0/2375613181 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab995500 0x55c2ab99df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.902+0000 7f83a3e00640 1 -- 172.21.15.37:0/2375613181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2ab6d37d0 con 0x55c2ab9961e0 2024-03-28T18:11:32.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.902+0000 7f83a9a00640 1 --2- 172.21.15.37:0/2375613181 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab995500 0x55c2ab99df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:11:32.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.902+0000 7f83a3e00640 1 --2- 172.21.15.37:0/2375613181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 0x55c2ab99b300 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8390005c40 tx=0x7f8390005e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9317d281b59dc201 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:32.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.902+0000 7f838aa00640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:11:32.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.902+0000 7f838aa00640 1 -- 172.21.15.37:0/2375613181 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8390043070 con 0x55c2ab9961e0 2024-03-28T18:11:32.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.902+0000 7f838aa00640 1 -- 172.21.15.37:0/2375613181 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f8390005030 con 0x55c2ab9961e0 2024-03-28T18:11:32.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.902+0000 7f83ae7c0740 1 -- 172.21.15.37:0/2375613181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f839c004050 con 0x55c2ab9961e0 2024-03-28T18:11:32.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.903+0000 7f838aa00640 1 -- 172.21.15.37:0/2375613181 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f839004f050 con 0x55c2ab9961e0 2024-03-28T18:11:32.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.903+0000 7f83ae7c0740 10 client.25541 fetch_fsmap learned FSMap version 23 2024-03-28T18:11:32.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.903+0000 7f83ae7c0740 1 -- 172.21.15.37:0/2375613181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c2ab6093f0 con 0x55c2ab9961e0 2024-03-28T18:11:32.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.903+0000 7f838aa00640 1 -- 172.21.15.37:0/2375613181 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f839c004050 con 0x55c2ab9961e0 2024-03-28T18:11:32.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.903+0000 7f838aa00640 1 -- 172.21.15.37:0/2375613181 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f83900035f0 con 0x55c2ab9961e0 2024-03-28T18:11:32.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.903+0000 7f83ae7c0740 10 client.25541 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:11:32.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.903+0000 7f83ae7c0740 10 client.25541 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:11:32.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.903+0000 7f83ae7c0740 20 client.25541 populate_metadata read hostname 'smithi037' 2024-03-28T18:11:32.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.903+0000 7f83ae7c0740 1 --2- 172.21.15.37:0/2375613181 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55c2ab9ae6e0 0x55c2ab9b0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.903+0000 7f83ae7c0740 4 client.25541 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:11:32.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.903+0000 7f83ae7c0740 1 -- 172.21.15.37:0/2375613181 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c2ab902790 con 0x55c2ab9ae6e0 2024-03-28T18:11:32.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.904+0000 7f83a9a00640 1 --2- 172.21.15.37:0/2375613181 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55c2ab9ae6e0 0x55c2ab9b0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.904+0000 7f83a9a00640 1 --2- 172.21.15.37:0/2375613181 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55c2ab9ae6e0 0x55c2ab9b0b60 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab4e8f7638219704 server_cookie=d38985a40526f6e1 in_seq=0 out_seq=0 2024-03-28T18:11:32.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.904+0000 7f838aa00640 10 client.25541 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:11:32.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.905+0000 7f838aa00640 1 -- 172.21.15.37:0/2375613181 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c2ab902790 con 0x55c2ab9ae6e0 2024-03-28T18:11:32.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.905+0000 7f838aa00640 10 client.25541 handle_command_reply: tid=0 2024-03-28T18:11:32.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.905+0000 7f83ae7c0740 1 client.25541 shutdown 2024-03-28T18:11:32.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.905+0000 7f83ae7c0740 1 -- 172.21.15.37:0/2375613181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 msgr2=0x55c2ab99b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:32.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.905+0000 7f83ae7c0740 1 --2- 172.21.15.37:0/2375613181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 0x55c2ab99b300 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8390005c40 tx=0x7f8390005e30 comp rx=0 tx=0).stop 2024-03-28T18:11:32.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.906+0000 7f83ae7c0740 1 -- 172.21.15.37:0/2375613181 shutdown_connections 2024-03-28T18:11:32.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.906+0000 7f83ae7c0740 1 --2- 172.21.15.37:0/2375613181 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55c2ab9ae6e0 0x55c2ab9b0b60 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.906+0000 7f83ae7c0740 1 --2- 172.21.15.37:0/2375613181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 0x55c2ab99b300 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.906+0000 7f83ae7c0740 1 --2- 172.21.15.37:0/2375613181 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c2ab99e870 0x55c2ab99b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.906+0000 7f83ae7c0740 1 --2- 172.21.15.37:0/2375613181 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab995500 0x55c2ab99df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.906+0000 7f83ae7c0740 1 -- 172.21.15.37:0/2375613181 >> 172.21.15.37:0/2375613181 conn(0x55c2ab958200 msgr2=0x55c2ab991ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:32.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.906+0000 7f838aa00640 0 client.25541 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:11:32.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.906+0000 7f83ae7c0740 1 -- 172.21.15.37:0/2375613181 shutdown_connections 2024-03-28T18:11:32.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.906+0000 7f83ae7c0740 1 -- 172.21.15.37:0/2375613181 wait complete. 2024-03-28T18:11:32.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.906+0000 7f83ae7c0740 20 client.25541 trim_cache size 0 max 16384 2024-03-28T18:11:32.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.924+0000 7f83ae7c0740 1 Processor -- start 2024-03-28T18:11:32.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.924+0000 7f83ae7c0740 1 -- start start 2024-03-28T18:11:32.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.924+0000 7f83ae7c0740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab995500 0x55c2ab99da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.924+0000 7f83ae7c0740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab9961e0 0x55c2ab99df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.924+0000 7f83ae7c0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c2ab99e870 0x55c2ab99fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.924+0000 7f83ae7c0740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a41ad340 con 0x55c2ab9961e0 2024-03-28T18:11:32.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.924+0000 7f83ae7c0740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a41aae40 con 0x55c2ab995500 2024-03-28T18:11:32.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.924+0000 7f83ae7c0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a41a8ee0 con 0x55c2ab99e870 2024-03-28T18:11:32.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.924+0000 7f83a9a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab995500 0x55c2ab99da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.924+0000 7f83a9a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab995500 0x55c2ab99da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:57144/0 (socket says 172.21.15.37:57144) 2024-03-28T18:11:32.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.924+0000 7f83a9a00640 1 -- 172.21.15.37:0/3712170831 learned_addr learned my addr 172.21.15.37:0/3712170831 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:32.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.925+0000 7f83a9a00640 1 --2- 172.21.15.37:0/3712170831 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c2ab99e870 0x55c2ab99fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.925+0000 7f83a3e00640 1 --2- 172.21.15.37:0/3712170831 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab9961e0 0x55c2ab99df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.925+0000 7f83a9a00640 1 -- 172.21.15.37:0/3712170831 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c2ab99e870 msgr2=0x55c2ab99fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:32.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.925+0000 7f83a9a00640 1 --2- 172.21.15.37:0/3712170831 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c2ab99e870 0x55c2ab99fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.925+0000 7f83a9a00640 1 -- 172.21.15.37:0/3712170831 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab9961e0 msgr2=0x55c2ab99df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:32.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.925+0000 7f83a9a00640 1 --2- 172.21.15.37:0/3712170831 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab9961e0 0x55c2ab99df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.925+0000 7f83a9a00640 1 -- 172.21.15.37:0/3712170831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2ab5c90f0 con 0x55c2ab995500 2024-03-28T18:11:32.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.925+0000 7f83a9a00640 1 --2- 172.21.15.37:0/3712170831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab995500 0x55c2ab99da30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f839c009d60 tx=0x7f839c009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c03f115fd824056 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:32.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.925+0000 7f8395600640 1 -- 172.21.15.37:0/3712170831 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c003d00 con 0x55c2ab995500 2024-03-28T18:11:32.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.926+0000 7f8395600640 1 -- 172.21.15.37:0/3712170831 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f839c004530 con 0x55c2ab995500 2024-03-28T18:11:32.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.926+0000 7f8395600640 1 -- 172.21.15.37:0/3712170831 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c00fba0 con 0x55c2ab995500 2024-03-28T18:11:32.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.926+0000 7f83ae7c0740 1 -- 172.21.15.37:0/3712170831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab995500 msgr2=0x55c2ab99da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:32.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.926+0000 7f83ae7c0740 1 --2- 172.21.15.37:0/3712170831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab995500 0x55c2ab99da30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f839c009d60 tx=0x7f839c009b30 comp rx=0 tx=0).stop 2024-03-28T18:11:32.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.926+0000 7f83ae7c0740 1 -- 172.21.15.37:0/3712170831 shutdown_connections 2024-03-28T18:11:32.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.926+0000 7f83ae7c0740 1 --2- 172.21.15.37:0/3712170831 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c2ab99e870 0x55c2ab99fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.926+0000 7f83ae7c0740 1 --2- 172.21.15.37:0/3712170831 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab9961e0 0x55c2ab99df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.926+0000 7f83ae7c0740 1 --2- 172.21.15.37:0/3712170831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab995500 0x55c2ab99da30 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.926+0000 7f83ae7c0740 1 -- 172.21.15.37:0/3712170831 >> 172.21.15.37:0/3712170831 conn(0x55c2ab958200 msgr2=0x55c2ab98fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:32.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.926+0000 7f83ae7c0740 1 -- 172.21.15.37:0/3712170831 shutdown_connections 2024-03-28T18:11:32.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.926+0000 7f83ae7c0740 1 -- 172.21.15.37:0/3712170831 wait complete. 2024-03-28T18:11:32.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.927+0000 7f83ae7c0740 1 -- start start 2024-03-28T18:11:32.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.927+0000 7f83ae7c0740 1 Processor -- start 2024-03-28T18:11:32.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.927+0000 7f83ae7c0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c2ab995500 0x55c2ab9985a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.927+0000 7f83ae7c0740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 0x55c2ab998ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.927+0000 7f83ae7c0740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab99e870 0x55c2ab9a95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.927+0000 7f83ae7c0740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a41a94e0 con 0x55c2ab99e870 2024-03-28T18:11:32.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.927+0000 7f83ae7c0740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a41aa7e0 con 0x55c2ab9961e0 2024-03-28T18:11:32.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.927+0000 7f83ae7c0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a41a9b20 con 0x55c2ab995500 2024-03-28T18:11:32.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.928+0000 7f83ae7c0740 10 client.0 _ll_register_callbacks cb 0x7f839c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:11:32.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.928+0000 7f83a3e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 0x55c2ab998ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.928+0000 7f83a3e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 0x55c2ab998ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:57150/0 (socket says 172.21.15.37:57150) 2024-03-28T18:11:32.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.928+0000 7f83a9a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c2ab995500 0x55c2ab9985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.928+0000 7f83a3e00640 1 -- 172.21.15.37:0/87401855 learned_addr learned my addr 172.21.15.37:0/87401855 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:11:32.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.928+0000 7f83a9a00640 1 --2- 172.21.15.37:0/87401855 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab99e870 0x55c2ab9a95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.928+0000 7f83a3e00640 1 -- 172.21.15.37:0/87401855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c2ab995500 msgr2=0x55c2ab9985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:32.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.928+0000 7f83a3e00640 1 --2- 172.21.15.37:0/87401855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c2ab995500 0x55c2ab9985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.928+0000 7f83a3e00640 1 -- 172.21.15.37:0/87401855 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab99e870 msgr2=0x55c2ab9a95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:11:32.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.928+0000 7f83a3e00640 1 --2- 172.21.15.37:0/87401855 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab99e870 0x55c2ab9a95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.928+0000 7f83a3e00640 1 -- 172.21.15.37:0/87401855 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2ab9a9b00 con 0x55c2ab9961e0 2024-03-28T18:11:32.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.928+0000 7f83a9a00640 1 --2- 172.21.15.37:0/87401855 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab99e870 0x55c2ab9a95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:11:32.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.929+0000 7f83a3e00640 1 --2- 172.21.15.37:0/87401855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 0x55c2ab998ae0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55c2ab99c8d0 tx=0x7f8390004450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70869268e3d3c905 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:11:32.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.929+0000 7f838aa00640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:11:32.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.929+0000 7f838aa00640 1 -- 172.21.15.37:0/87401855 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8390043040 con 0x55c2ab9961e0 2024-03-28T18:11:32.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.929+0000 7f838aa00640 1 -- 172.21.15.37:0/87401855 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f8390005240 con 0x55c2ab9961e0 2024-03-28T18:11:32.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.929+0000 7f83ae7c0740 1 -- 172.21.15.37:0/87401855 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f839c004530 con 0x55c2ab9961e0 2024-03-28T18:11:32.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.929+0000 7f838aa00640 1 -- 172.21.15.37:0/87401855 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8390003800 con 0x55c2ab9961e0 2024-03-28T18:11:32.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.929+0000 7f83ae7c0740 10 client.25547 fetch_fsmap learned FSMap version 23 2024-03-28T18:11:32.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.929+0000 7f83ae7c0740 1 -- 172.21.15.37:0/87401855 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c2aba0a730 con 0x55c2ab9961e0 2024-03-28T18:11:32.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.929+0000 7f838aa00640 1 -- 172.21.15.37:0/87401855 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f839c004530 con 0x55c2ab9961e0 2024-03-28T18:11:32.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.930+0000 7f838aa00640 1 -- 172.21.15.37:0/87401855 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8390034440 con 0x55c2ab9961e0 2024-03-28T18:11:32.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.930+0000 7f83ae7c0740 10 client.25547 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:11:32.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.930+0000 7f83ae7c0740 10 client.25547 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:11:32.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.930+0000 7f83ae7c0740 20 client.25547 populate_metadata read hostname 'smithi037' 2024-03-28T18:11:32.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.930+0000 7f83ae7c0740 1 --2- 172.21.15.37:0/87401855 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55c2aba0ae50 0x55c2aba0d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:11:32.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.930+0000 7f83ae7c0740 4 client.25547 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:11:32.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.930+0000 7f83ae7c0740 1 -- 172.21.15.37:0/87401855 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c2ab8fcae0 con 0x55c2aba0ae50 2024-03-28T18:11:32.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.930+0000 7f83a9a00640 1 --2- 172.21.15.37:0/87401855 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55c2aba0ae50 0x55c2aba0d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:11:32.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.931+0000 7f838aa00640 10 client.25547 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:11:32.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.931+0000 7f83a9a00640 1 --2- 172.21.15.37:0/87401855 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55c2aba0ae50 0x55c2aba0d270 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d8a0406a9f62592 server_cookie=646de7bc455bdf9a in_seq=0 out_seq=0 2024-03-28T18:11:32.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.931+0000 7f838aa00640 1 -- 172.21.15.37:0/87401855 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55c2ab8fcae0 con 0x55c2aba0ae50 2024-03-28T18:11:32.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.931+0000 7f838aa00640 10 client.25547 handle_command_reply: tid=0 2024-03-28T18:11:32.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.931+0000 7f83ae7c0740 1 client.25547 shutdown 2024-03-28T18:11:32.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.931+0000 7f83ae7c0740 1 -- 172.21.15.37:0/87401855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 msgr2=0x55c2ab998ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:32.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.931+0000 7f83ae7c0740 1 --2- 172.21.15.37:0/87401855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 0x55c2ab998ae0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55c2ab99c8d0 tx=0x7f8390004450 comp rx=0 tx=0).stop 2024-03-28T18:11:32.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.931+0000 7f83ae7c0740 1 -- 172.21.15.37:0/87401855 shutdown_connections 2024-03-28T18:11:32.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.931+0000 7f83ae7c0740 1 --2- 172.21.15.37:0/87401855 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55c2aba0ae50 0x55c2aba0d270 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.931+0000 7f83ae7c0740 1 --2- 172.21.15.37:0/87401855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c2ab9961e0 0x55c2ab998ae0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.931+0000 7f83ae7c0740 1 --2- 172.21.15.37:0/87401855 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c2ab99e870 0x55c2ab9a95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.931+0000 7f838aa00640 0 client.25547 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:11:32.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.931+0000 7f83ae7c0740 1 --2- 172.21.15.37:0/87401855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c2ab995500 0x55c2ab9985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.931+0000 7f83ae7c0740 1 -- 172.21.15.37:0/87401855 >> 172.21.15.37:0/87401855 conn(0x55c2ab958200 msgr2=0x55c2ab98a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:32.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.931+0000 7f83ae7c0740 1 -- 172.21.15.37:0/87401855 shutdown_connections 2024-03-28T18:11:32.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.931+0000 7f83ae7c0740 1 -- 172.21.15.37:0/87401855 wait complete. 2024-03-28T18:11:32.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.932+0000 7f83ae7c0740 20 client.25547 trim_cache size 0 max 16384 2024-03-28T18:11:32.935 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:11:32.936 INFO:teuthology.orchestra.run.smithi037.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:11:32.936 INFO:teuthology.orchestra.run.smithi037.stdout: "scrubs": { 2024-03-28T18:11:32.936 INFO:teuthology.orchestra.run.smithi037.stdout: "59d357e0-5aaf-4517-bd4c-d439d845171d": { 2024-03-28T18:11:32.936 INFO:teuthology.orchestra.run.smithi037.stdout: "path": "/", 2024-03-28T18:11:32.936 INFO:teuthology.orchestra.run.smithi037.stdout: "tag": "59d357e0-5aaf-4517-bd4c-d439d845171d", 2024-03-28T18:11:32.936 INFO:teuthology.orchestra.run.smithi037.stdout: "options": "recursive,force" 2024-03-28T18:11:32.936 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-03-28T18:11:32.936 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-03-28T18:11:32.936 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:11:32.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.932+0000 7f83aa400640 1 -- 172.21.15.37:0/2597444166 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f838c0a2a80 msgr2=0x7f838c0a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:32.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.932+0000 7f83aa400640 1 --2- 172.21.15.37:0/2597444166 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f838c0a2a80 0x7f838c0a4f40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f83980cca40 tx=0x7f83980d2d80 comp rx=0 tx=0).stop 2024-03-28T18:11:32.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.932+0000 7f83aa400640 1 -- 172.21.15.37:0/2597444166 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83a4102730 msgr2=0x7f83a4197e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:11:32.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.932+0000 7f83aa400640 1 --2- 172.21.15.37:0/2597444166 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83a4102730 0x7f83a4197e40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f839800e9f0 tx=0x7f839800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.933+0000 7f83aa400640 1 -- 172.21.15.37:0/2597444166 shutdown_connections 2024-03-28T18:11:32.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.933+0000 7f83aa400640 1 --2- 172.21.15.37:0/2597444166 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f838c0a2a80 0x7f838c0a4f40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.933+0000 7f83aa400640 1 --2- 172.21.15.37:0/2597444166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83a410bb00 0x7f83a419c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.933+0000 7f83aa400640 1 --2- 172.21.15.37:0/2597444166 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83a4103050 0x7f83a4198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.933+0000 7f83aa400640 1 --2- 172.21.15.37:0/2597444166 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83a4102730 0x7f83a4197e40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:11:32.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.933+0000 7f83aa400640 1 -- 172.21.15.37:0/2597444166 >> 172.21.15.37:0/2597444166 conn(0x7f83a40fbef0 msgr2=0x7f83a40fcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:11:32.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.933+0000 7f83aa400640 1 -- 172.21.15.37:0/2597444166 shutdown_connections 2024-03-28T18:11:32.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:11:32.933+0000 7f83aa400640 1 -- 172.21.15.37:0/2597444166 wait complete. 2024-03-28T18:11:32.954 INFO:tasks.cephfs.filesystem:scrub status for tag:59d357e0-5aaf-4517-bd4c-d439d845171d - {'path': '/', 'tag': '59d357e0-5aaf-4517-bd4c-d439d845171d', 'options': 'recursive,force'} 2024-03-28T18:11:33.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:33 smithi037 ceph-mon[31460]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:11:34.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:33 smithi059 ceph-mon[36190]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:11:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:33 smithi170 ceph-mon[36280]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:11:34.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:34 smithi037 ceph-mon[31460]: scrub queued for path: / 2024-03-28T18:11:34.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:34 smithi037 ceph-mon[31460]: scrub summary: idle+waiting paths [/] 2024-03-28T18:11:34.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:34 smithi037 ceph-mon[31460]: scrub summary: active paths [/] 2024-03-28T18:11:34.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:34 smithi037 ceph-mon[31460]: scrub summary: idle+waiting paths [/] 2024-03-28T18:11:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:34 smithi059 ceph-mon[36190]: scrub queued for path: / 2024-03-28T18:11:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:34 smithi059 ceph-mon[36190]: scrub summary: idle+waiting paths [/] 2024-03-28T18:11:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:34 smithi059 ceph-mon[36190]: scrub summary: active paths [/] 2024-03-28T18:11:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:34 smithi059 ceph-mon[36190]: scrub summary: idle+waiting paths [/] 2024-03-28T18:11:35.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:34 smithi170 ceph-mon[36280]: scrub queued for path: / 2024-03-28T18:11:35.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:34 smithi170 ceph-mon[36280]: scrub summary: idle+waiting paths [/] 2024-03-28T18:11:35.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:34 smithi170 ceph-mon[36280]: scrub summary: active paths [/] 2024-03-28T18:11:35.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:34 smithi170 ceph-mon[36280]: scrub summary: idle+waiting paths [/] 2024-03-28T18:11:35.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:35 smithi037 ceph-mon[31460]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:35.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:35 smithi037 ceph-mon[31460]: scrub summary: idle 2024-03-28T18:11:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:35 smithi059 ceph-mon[36190]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:35 smithi059 ceph-mon[36190]: scrub summary: idle 2024-03-28T18:11:36.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:35 smithi170 ceph-mon[36280]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:36.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:35 smithi170 ceph-mon[36280]: scrub summary: idle 2024-03-28T18:11:37.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:37 smithi037 ceph-mon[31460]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:38.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:37 smithi059 ceph-mon[36190]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:38.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:37 smithi170 ceph-mon[36280]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:39.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:39 smithi037 ceph-mon[31460]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T18:11:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:39 smithi059 ceph-mon[36190]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T18:11:40.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:39 smithi170 ceph-mon[36280]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T18:11:42.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:41 smithi059 ceph-mon[36190]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T18:11:42.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:41 smithi170 ceph-mon[36280]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T18:11:42.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:41 smithi037 ceph-mon[31460]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T18:11:44.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:43 smithi059 ceph-mon[36190]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T18:11:44.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:43 smithi170 ceph-mon[36280]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T18:11:44.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:43 smithi037 ceph-mon[31460]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T18:11:46.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:45 smithi059 ceph-mon[36190]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T18:11:46.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:45 smithi170 ceph-mon[36280]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T18:11:46.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:45 smithi037 ceph-mon[31460]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T18:11:48.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:47 smithi059 ceph-mon[36190]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T18:11:48.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:47 smithi170 ceph-mon[36280]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T18:11:48.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:47 smithi037 ceph-mon[31460]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T18:11:50.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:49 smithi059 ceph-mon[36190]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T18:11:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:49 smithi170 ceph-mon[36280]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T18:11:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:49 smithi037 ceph-mon[31460]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T18:11:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:51 smithi059 ceph-mon[36190]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:51 smithi170 ceph-mon[36280]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:51 smithi037 ceph-mon[31460]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:54.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:53 smithi059 ceph-mon[36190]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-28T18:11:54.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:53 smithi170 ceph-mon[36280]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-28T18:11:54.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:53 smithi037 ceph-mon[31460]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-28T18:11:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:55 smithi059 ceph-mon[36190]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-28T18:11:56.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:55 smithi170 ceph-mon[36280]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-28T18:11:56.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:55 smithi037 ceph-mon[31460]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-28T18:11:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:57 smithi059 ceph-mon[36190]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-28T18:11:58.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:57 smithi170 ceph-mon[36280]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-28T18:11:58.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:57 smithi037 ceph-mon[31460]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-28T18:12:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:11:59 smithi059 ceph-mon[36190]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-28T18:12:00.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:11:59 smithi170 ceph-mon[36280]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-28T18:12:00.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:11:59 smithi037 ceph-mon[31460]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-28T18:12:01.375 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:12:01 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:12:01.009+0000 7f0f92a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:12:01.375 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:12:01 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:12:01.009+0000 7f0f92a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:12:01.375 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:12:01 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:12:01.009+0000 7f0f92a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:12:01.375 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:12:01 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:12:01.009+0000 7f0f92a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:12:01.375 INFO:journalctl@ceph.mgr.x.smithi037.stdout:Mar 28 18:12:01 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mgr-x[31690]: 2024-03-28T18:12:01.009+0000 7f0f92a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:12:02.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:01 smithi059 ceph-mon[36190]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-28T18:12:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:01 smithi170 ceph-mon[36280]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-28T18:12:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:01 smithi037 ceph-mon[31460]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-28T18:12:02.954 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:12:02.955 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:12:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.160+0000 7f639ac00640 1 -- 172.21.15.37:0/4058591460 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6394101530 msgr2=0x7f63941019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.160+0000 7f639ac00640 1 --2- 172.21.15.37:0/4058591460 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6394101530 0x7f63941019b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6384009930 tx=0x7f638402bc30 comp rx=0 tx=0).stop 2024-03-28T18:12:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.160+0000 7f639ac00640 1 -- 172.21.15.37:0/4058591460 shutdown_connections 2024-03-28T18:12:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.160+0000 7f639ac00640 1 --2- 172.21.15.37:0/4058591460 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6394101530 0x7f63941019b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.160+0000 7f639ac00640 1 --2- 172.21.15.37:0/4058591460 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6394101ef0 0x7f639410e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.160+0000 7f639ac00640 1 --2- 172.21.15.37:0/4058591460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63941075f0 0x7f63941079d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.160+0000 7f639ac00640 1 -- 172.21.15.37:0/4058591460 >> 172.21.15.37:0/4058591460 conn(0x7f63940fd040 msgr2=0x7f63940ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.160+0000 7f639ac00640 1 -- 172.21.15.37:0/4058591460 shutdown_connections 2024-03-28T18:12:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.160+0000 7f639ac00640 1 -- 172.21.15.37:0/4058591460 wait complete. 2024-03-28T18:12:03.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.161+0000 7f639ac00640 1 Processor -- start 2024-03-28T18:12:03.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.162+0000 7f639ac00640 1 -- start start 2024-03-28T18:12:03.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.162+0000 7f639ac00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6394101530 0x7f6394199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.162+0000 7f639ac00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6394101ef0 0x7f639419a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.162+0000 7f639ac00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63941075f0 0x7f639419e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.162+0000 7f639ac00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6394111920 con 0x7f6394101ef0 2024-03-28T18:12:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.162+0000 7f639ac00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63941117a0 con 0x7f63941075f0 2024-03-28T18:12:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.162+0000 7f639ac00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6394111aa0 con 0x7f6394101530 2024-03-28T18:12:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.162+0000 7f6399800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6394101530 0x7f6394199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.162+0000 7f639a200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63941075f0 0x7f639419e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.162+0000 7f6398e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6394101ef0 0x7f639419a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.162+0000 7f6399800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6394101530 0x7f6394199fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:47490/0 (socket says 172.21.15.37:47490) 2024-03-28T18:12:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.162+0000 7f6399800640 1 -- 172.21.15.37:0/1651176610 learned_addr learned my addr 172.21.15.37:0/1651176610 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.163+0000 7f6399800640 1 -- 172.21.15.37:0/1651176610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63941075f0 msgr2=0x7f639419e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.163+0000 7f6399800640 1 --2- 172.21.15.37:0/1651176610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63941075f0 0x7f639419e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.163+0000 7f6399800640 1 -- 172.21.15.37:0/1651176610 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6394101ef0 msgr2=0x7f639419a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.163+0000 7f6399800640 1 --2- 172.21.15.37:0/1651176610 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6394101ef0 0x7f639419a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.163+0000 7f6399800640 1 -- 172.21.15.37:0/1651176610 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639419efa0 con 0x7f6394101530 2024-03-28T18:12:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.163+0000 7f6399800640 1 --2- 172.21.15.37:0/1651176610 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6394101530 0x7f6394199fb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f638800e9f0 tx=0x7f638800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:03.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.163+0000 7f6392000640 1 -- 172.21.15.37:0/1651176610 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638800cd80 con 0x7f6394101530 2024-03-28T18:12:03.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.163+0000 7f6392000640 1 -- 172.21.15.37:0/1651176610 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6388005500 con 0x7f6394101530 2024-03-28T18:12:03.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.163+0000 7f6392000640 1 -- 172.21.15.37:0/1651176610 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6388010690 con 0x7f6394101530 2024-03-28T18:12:03.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.163+0000 7f639ac00640 1 -- 172.21.15.37:0/1651176610 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f639419f290 con 0x7f6394101530 2024-03-28T18:12:03.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.163+0000 7f639ac00640 1 -- 172.21.15.37:0/1651176610 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63941a6b20 con 0x7f6394101530 2024-03-28T18:12:03.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.168+0000 7f639f003740 1 Processor -- start 2024-03-28T18:12:03.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.168+0000 7f639f003740 1 -- start start 2024-03-28T18:12:03.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.168+0000 7f6392000640 1 -- 172.21.15.37:0/1651176610 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f63880046c0 con 0x7f6394101530 2024-03-28T18:12:03.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.168+0000 7f639f003740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee6dd0 0x55dafdee5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.168+0000 7f639f003740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdee60f0 0x55dafdee6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.168+0000 7f639f003740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdeef460 0x55dafdef1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.168+0000 7f639f003740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dafde4a250 con 0x55dafdee60f0 2024-03-28T18:12:03.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.168+0000 7f639f003740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dafddf2a30 con 0x55dafdee6dd0 2024-03-28T18:12:03.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.168+0000 7f639f003740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dafde462f0 con 0x55dafdeef460 2024-03-28T18:12:03.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.169+0000 7f639a200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee6dd0 0x55dafdee5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.169+0000 7f6399800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdee60f0 0x55dafdee6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.169+0000 7f639a200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee6dd0 0x55dafdee5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:58448/0 (socket says 172.21.15.37:58448) 2024-03-28T18:12:03.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.169+0000 7f639a200640 1 -- 172.21.15.37:0/2891022724 learned_addr learned my addr 172.21.15.37:0/2891022724 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:03.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.169+0000 7f6398e00640 1 --2- 172.21.15.37:0/2891022724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdeef460 0x55dafdef1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.169+0000 7f639a200640 1 -- 172.21.15.37:0/2891022724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdeef460 msgr2=0x55dafdef1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:03.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.169+0000 7f639a200640 1 --2- 172.21.15.37:0/2891022724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdeef460 0x55dafdef1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.169+0000 7f639a200640 1 -- 172.21.15.37:0/2891022724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdee60f0 msgr2=0x55dafdee6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:03.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.169+0000 7f639a200640 1 --2- 172.21.15.37:0/2891022724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdee60f0 0x55dafdee6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.169+0000 7f639a200640 1 -- 172.21.15.37:0/2891022724 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dafdddd690 con 0x55dafdee6dd0 2024-03-28T18:12:03.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.169+0000 7f6398e00640 1 --2- 172.21.15.37:0/2891022724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdeef460 0x55dafdef1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:12:03.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.170+0000 7f639a200640 1 --2- 172.21.15.37:0/2891022724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee6dd0 0x55dafdee5b20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f638c009fd0 tx=0x7f638c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9debc048c98d5da2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.170+0000 7f6392000640 1 --2- 172.21.15.37:0/1651176610 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f637c0a2b10 0x7f637c0a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.170+0000 7f6382a00640 1 -- 172.21.15.37:0/2891022724 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638c004d20 con 0x55dafdee6dd0 2024-03-28T18:12:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.170+0000 7f6382a00640 1 -- 172.21.15.37:0/2891022724 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f638c004050 con 0x55dafdee6dd0 2024-03-28T18:12:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.170+0000 7f6382a00640 1 -- 172.21.15.37:0/2891022724 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638c005260 con 0x55dafdee6dd0 2024-03-28T18:12:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.170+0000 7f6392000640 1 -- 172.21.15.37:0/1651176610 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f63880cf460 con 0x7f6394101530 2024-03-28T18:12:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.170+0000 7f6399800640 1 --2- 172.21.15.37:0/1651176610 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f637c0a2b10 0x7f637c0a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.171+0000 7f639f003740 1 -- 172.21.15.37:0/2891022724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee6dd0 msgr2=0x55dafdee5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.171+0000 7f639f003740 1 --2- 172.21.15.37:0/2891022724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee6dd0 0x55dafdee5b20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f638c009fd0 tx=0x7f638c0118d0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.171+0000 7f639f003740 1 -- 172.21.15.37:0/2891022724 shutdown_connections 2024-03-28T18:12:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.171+0000 7f639f003740 1 --2- 172.21.15.37:0/2891022724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdeef460 0x55dafdef1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.171+0000 7f639f003740 1 --2- 172.21.15.37:0/2891022724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdee60f0 0x55dafdee6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.171+0000 7f639f003740 1 --2- 172.21.15.37:0/2891022724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee6dd0 0x55dafdee5b20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.171+0000 7f639f003740 1 -- 172.21.15.37:0/2891022724 >> 172.21.15.37:0/2891022724 conn(0x55dafdead200 msgr2=0x55dafdee0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.171+0000 7f6399800640 1 --2- 172.21.15.37:0/1651176610 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f637c0a2b10 0x7f637c0a4fd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f63880ccac0 tx=0x7f6388004a40 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.171+0000 7f639f003740 1 -- 172.21.15.37:0/2891022724 shutdown_connections 2024-03-28T18:12:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.171+0000 7f639f003740 1 -- 172.21.15.37:0/2891022724 wait complete. 2024-03-28T18:12:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.172+0000 7f639f003740 1 -- start start 2024-03-28T18:12:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.172+0000 7f639f003740 1 Processor -- start 2024-03-28T18:12:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.172+0000 7f639f003740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdee60f0 0x55dafdeeeb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.172+0000 7f639f003740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee6dd0 0x55dafdeebf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.172+0000 7f639f003740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdeef460 0x55dafdeec460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.172+0000 7f639f003740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dafdef22d0 con 0x55dafdeef460 2024-03-28T18:12:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.172+0000 7f639f003740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dafde05e80 con 0x55dafdee6dd0 2024-03-28T18:12:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.172+0000 7f639f003740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dafdef2450 con 0x55dafdee60f0 2024-03-28T18:12:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.172+0000 7f639f003740 10 client.0 _ll_register_callbacks cb 0x55dafde18130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.173+0000 7f639a200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdee60f0 0x55dafdeeeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.173+0000 7f639a200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdee60f0 0x55dafdeeeb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:47496/0 (socket says 172.21.15.37:47496) 2024-03-28T18:12:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.173+0000 7f639a200640 1 -- 172.21.15.37:0/2711645019 learned_addr learned my addr 172.21.15.37:0/2711645019 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.173+0000 7f639a200640 1 --2- 172.21.15.37:0/2711645019 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdeef460 0x55dafdeec460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.173+0000 7f6398e00640 1 --2- 172.21.15.37:0/2711645019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee6dd0 0x55dafdeebf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.173+0000 7f6398e00640 1 -- 172.21.15.37:0/2711645019 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdee60f0 msgr2=0x55dafdeeeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.173+0000 7f6398e00640 1 --2- 172.21.15.37:0/2711645019 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdee60f0 0x55dafdeeeb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.173+0000 7f6398e00640 1 -- 172.21.15.37:0/2711645019 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdeef460 msgr2=0x55dafdeec460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.173+0000 7f6398e00640 1 --2- 172.21.15.37:0/2711645019 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdeef460 0x55dafdeec460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.173+0000 7f6398e00640 1 -- 172.21.15.37:0/2711645019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dafdc287d0 con 0x55dafdee6dd0 2024-03-28T18:12:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.173+0000 7f639a200640 1 --2- 172.21.15.37:0/2711645019 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdee60f0 0x55dafdeeeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.173+0000 7f6398e00640 1 --2- 172.21.15.37:0/2711645019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee6dd0 0x55dafdeebf20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6384036040 tx=0x7f6384005db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=417b21f2eff8315c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.173+0000 7f6377400640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:12:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.174+0000 7f6377400640 1 -- 172.21.15.37:0/2711645019 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638402cb50 con 0x55dafdee6dd0 2024-03-28T18:12:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.174+0000 7f6377400640 1 -- 172.21.15.37:0/2711645019 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f63840046c0 con 0x55dafdee6dd0 2024-03-28T18:12:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.174+0000 7f639f003740 1 -- 172.21.15.37:0/2711645019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f638c004050 con 0x55dafdee6dd0 2024-03-28T18:12:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.174+0000 7f6377400640 1 -- 172.21.15.37:0/2711645019 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f638402ccf0 con 0x55dafdee6dd0 2024-03-28T18:12:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.174+0000 7f639f003740 10 client.25565 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.174+0000 7f639f003740 1 -- 172.21.15.37:0/2711645019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55dafdb5e3f0 con 0x55dafdee6dd0 2024-03-28T18:12:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.174+0000 7f6377400640 1 -- 172.21.15.37:0/2711645019 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f638c004050 con 0x55dafdee6dd0 2024-03-28T18:12:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.175+0000 7f6377400640 1 -- 172.21.15.37:0/2711645019 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f63840035f0 con 0x55dafdee6dd0 2024-03-28T18:12:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.175+0000 7f639f003740 10 client.25565 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.175+0000 7f639f003740 10 client.25565 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:12:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.175+0000 7f639f003740 20 client.25565 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.175+0000 7f639f003740 1 --2- 172.21.15.37:0/2711645019 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55dafdeff3d0 0x55dafdf017c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.175+0000 7f639f003740 4 client.25565 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:12:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.175+0000 7f639f003740 1 -- 172.21.15.37:0/2711645019 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dafde57790 con 0x55dafdeff3d0 2024-03-28T18:12:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.175+0000 7f639a200640 1 --2- 172.21.15.37:0/2711645019 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55dafdeff3d0 0x55dafdf017c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.176+0000 7f639a200640 1 --2- 172.21.15.37:0/2711645019 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55dafdeff3d0 0x55dafdf017c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83e54c937679eff9 server_cookie=e520f9149b013a97 in_seq=0 out_seq=0 2024-03-28T18:12:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.176+0000 7f6377400640 10 client.25565 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.176+0000 7f6377400640 1 -- 172.21.15.37:0/2711645019 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55dafde57790 con 0x55dafdeff3d0 2024-03-28T18:12:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.176+0000 7f6377400640 10 client.25565 handle_command_reply: tid=0 2024-03-28T18:12:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.177+0000 7f639f003740 1 client.25565 shutdown 2024-03-28T18:12:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.177+0000 7f639f003740 1 -- 172.21.15.37:0/2711645019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee6dd0 msgr2=0x55dafdeebf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.177+0000 7f639f003740 1 --2- 172.21.15.37:0/2711645019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee6dd0 0x55dafdeebf20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6384036040 tx=0x7f6384005db0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.177+0000 7f639f003740 1 -- 172.21.15.37:0/2711645019 shutdown_connections 2024-03-28T18:12:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.177+0000 7f639f003740 1 --2- 172.21.15.37:0/2711645019 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55dafdeff3d0 0x55dafdf017c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.177+0000 7f639f003740 1 --2- 172.21.15.37:0/2711645019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee6dd0 0x55dafdeebf20 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.177+0000 7f639f003740 1 --2- 172.21.15.37:0/2711645019 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdeef460 0x55dafdeec460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.177+0000 7f639f003740 1 --2- 172.21.15.37:0/2711645019 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdee60f0 0x55dafdeeeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.177+0000 7f6377400640 0 client.25565 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.177+0000 7f639f003740 1 -- 172.21.15.37:0/2711645019 >> 172.21.15.37:0/2711645019 conn(0x55dafdead200 msgr2=0x55dafdee2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.177+0000 7f639f003740 1 -- 172.21.15.37:0/2711645019 shutdown_connections 2024-03-28T18:12:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.177+0000 7f639f003740 1 -- 172.21.15.37:0/2711645019 wait complete. 2024-03-28T18:12:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.177+0000 7f639f003740 20 client.25565 trim_cache size 0 max 16384 2024-03-28T18:12:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.212+0000 7f639f003740 1 Processor -- start 2024-03-28T18:12:03.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.212+0000 7f639f003740 1 -- start start 2024-03-28T18:12:03.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.212+0000 7f639f003740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdee60f0 0x55dafdeee650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.212+0000 7f639f003740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee6dd0 0x55dafdeeeb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.212+0000 7f639f003740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdeef460 0x55dafdefaba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.212+0000 7f639f003740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63941b15a0 con 0x55dafdeef460 2024-03-28T18:12:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.212+0000 7f639f003740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63941af0a0 con 0x55dafdee6dd0 2024-03-28T18:12:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.212+0000 7f639f003740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63941ad140 con 0x55dafdee60f0 2024-03-28T18:12:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.213+0000 7f6398e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee6dd0 0x55dafdeeeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.213+0000 7f639a200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdee60f0 0x55dafdeee650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.213+0000 7f6398e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee6dd0 0x55dafdeeeb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:58466/0 (socket says 172.21.15.37:58466) 2024-03-28T18:12:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.213+0000 7f6398e00640 1 -- 172.21.15.37:0/510308075 learned_addr learned my addr 172.21.15.37:0/510308075 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.213+0000 7f639a200640 1 --2- 172.21.15.37:0/510308075 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdeef460 0x55dafdefaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.213+0000 7f6398e00640 1 -- 172.21.15.37:0/510308075 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdee60f0 msgr2=0x55dafdeee650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:03.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.213+0000 7f6398e00640 1 --2- 172.21.15.37:0/510308075 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdee60f0 0x55dafdeee650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.213+0000 7f6398e00640 1 -- 172.21.15.37:0/510308075 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdeef460 msgr2=0x55dafdefaba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:03.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.213+0000 7f6398e00640 1 --2- 172.21.15.37:0/510308075 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdeef460 0x55dafdefaba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.213+0000 7f6398e00640 1 -- 172.21.15.37:0/510308075 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dafdb1e0f0 con 0x55dafdee6dd0 2024-03-28T18:12:03.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.213+0000 7f639a200640 1 --2- 172.21.15.37:0/510308075 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdee60f0 0x55dafdeee650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:03.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.213+0000 7f6398e00640 1 --2- 172.21.15.37:0/510308075 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee6dd0 0x55dafdeeeb90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f638402c170 tx=0x7f6384005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e76a128c2c6681 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:03.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.214+0000 7f6382000640 1 -- 172.21.15.37:0/510308075 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6384057070 con 0x55dafdee6dd0 2024-03-28T18:12:03.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.214+0000 7f6382000640 1 -- 172.21.15.37:0/510308075 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6384009520 con 0x55dafdee6dd0 2024-03-28T18:12:03.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.214+0000 7f6382000640 1 -- 172.21.15.37:0/510308075 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6384005140 con 0x55dafdee6dd0 2024-03-28T18:12:03.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.214+0000 7f639f003740 1 -- 172.21.15.37:0/510308075 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee6dd0 msgr2=0x55dafdeeeb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:03.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.214+0000 7f639f003740 1 --2- 172.21.15.37:0/510308075 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee6dd0 0x55dafdeeeb90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f638402c170 tx=0x7f6384005fd0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.215+0000 7f639f003740 1 -- 172.21.15.37:0/510308075 shutdown_connections 2024-03-28T18:12:03.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.215+0000 7f639f003740 1 --2- 172.21.15.37:0/510308075 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee6dd0 0x55dafdeeeb90 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.215+0000 7f639f003740 1 --2- 172.21.15.37:0/510308075 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdeef460 0x55dafdefaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.215+0000 7f639f003740 1 --2- 172.21.15.37:0/510308075 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdee60f0 0x55dafdeee650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.215+0000 7f639f003740 1 -- 172.21.15.37:0/510308075 >> 172.21.15.37:0/510308075 conn(0x55dafdead200 msgr2=0x55dafdee0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:03.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.215+0000 7f639f003740 1 -- 172.21.15.37:0/510308075 shutdown_connections 2024-03-28T18:12:03.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.215+0000 7f639f003740 1 -- 172.21.15.37:0/510308075 wait complete. 2024-03-28T18:12:03.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.215+0000 7f639f003740 1 -- start start 2024-03-28T18:12:03.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.215+0000 7f639f003740 1 Processor -- start 2024-03-28T18:12:03.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.216+0000 7f639f003740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee60f0 0x55dafdee91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.216+0000 7f639f003740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdee6dd0 0x55dafdee9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.216+0000 7f639f003740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdeef460 0x55dafdee9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.216+0000 7f639f003740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63941ad740 con 0x55dafdeef460 2024-03-28T18:12:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.216+0000 7f639f003740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63941aea40 con 0x55dafdee60f0 2024-03-28T18:12:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.216+0000 7f639f003740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63941add80 con 0x55dafdee6dd0 2024-03-28T18:12:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.216+0000 7f639f003740 10 client.0 _ll_register_callbacks cb 0x7f6388005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.216+0000 7f6398e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdee6dd0 0x55dafdee9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.216+0000 7f6398e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdee6dd0 0x55dafdee9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:47512/0 (socket says 172.21.15.37:47512) 2024-03-28T18:12:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.216+0000 7f6398e00640 1 -- 172.21.15.37:0/2351542588 learned_addr learned my addr 172.21.15.37:0/2351542588 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.216+0000 7f639a200640 1 --2- 172.21.15.37:0/2351542588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee60f0 0x55dafdee91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.216+0000 7f639a200640 1 --2- 172.21.15.37:0/2351542588 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdeef460 0x55dafdee9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.216+0000 7f6398e00640 1 -- 172.21.15.37:0/2351542588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee60f0 msgr2=0x55dafdee91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.216+0000 7f6398e00640 1 --2- 172.21.15.37:0/2351542588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee60f0 0x55dafdee91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.216+0000 7f6398e00640 1 -- 172.21.15.37:0/2351542588 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdeef460 msgr2=0x55dafdee9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.216+0000 7f6398e00640 1 --2- 172.21.15.37:0/2351542588 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdeef460 0x55dafdee9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.216+0000 7f6398e00640 1 -- 172.21.15.37:0/2351542588 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dafdeefb50 con 0x55dafdee6dd0 2024-03-28T18:12:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.216+0000 7f639a200640 1 --2- 172.21.15.37:0/2351542588 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdeef460 0x55dafdee9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:12:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.217+0000 7f6398e00640 1 --2- 172.21.15.37:0/2351542588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdee6dd0 0x55dafdee9720 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f63840045c0 tx=0x7f6384004c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9fb616a0586c650 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.217+0000 7f6377400640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:12:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.217+0000 7f6377400640 1 -- 172.21.15.37:0/2351542588 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6384057040 con 0x55dafdee6dd0 2024-03-28T18:12:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.217+0000 7f6377400640 1 -- 172.21.15.37:0/2351542588 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f63840050e0 con 0x55dafdee6dd0 2024-03-28T18:12:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.217+0000 7f639f003740 1 -- 172.21.15.37:0/2351542588 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6384009520 con 0x55dafdee6dd0 2024-03-28T18:12:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.217+0000 7f6377400640 1 -- 172.21.15.37:0/2351542588 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f638402cb50 con 0x55dafdee6dd0 2024-03-28T18:12:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.217+0000 7f639f003740 10 client.25591 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.217+0000 7f639f003740 1 -- 172.21.15.37:0/2351542588 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55dafdf5b040 con 0x55dafdee6dd0 2024-03-28T18:12:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.218+0000 7f6377400640 1 -- 172.21.15.37:0/2351542588 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6384009520 con 0x55dafdee6dd0 2024-03-28T18:12:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.218+0000 7f6377400640 1 -- 172.21.15.37:0/2351542588 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f638404a510 con 0x55dafdee6dd0 2024-03-28T18:12:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.218+0000 7f639f003740 10 client.25591 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.218+0000 7f639f003740 10 client.25591 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:12:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.218+0000 7f639f003740 20 client.25591 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:03.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.218+0000 7f639f003740 1 --2- 172.21.15.37:0/2351542588 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55dafdf5b670 0x55dafdf5da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.218+0000 7f639f003740 4 client.25591 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:12:03.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.218+0000 7f639f003740 1 -- 172.21.15.37:0/2351542588 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55dafde51ae0 con 0x55dafdf5b670 2024-03-28T18:12:03.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.218+0000 7f639a200640 1 --2- 172.21.15.37:0/2351542588 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55dafdf5b670 0x55dafdf5da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.219+0000 7f639a200640 1 --2- 172.21.15.37:0/2351542588 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55dafdf5b670 0x55dafdf5da90 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=250069fce85b81bd server_cookie=9f082466a2fda972 in_seq=0 out_seq=0 2024-03-28T18:12:03.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.219+0000 7f6377400640 10 client.25591 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.219+0000 7f6377400640 1 -- 172.21.15.37:0/2351542588 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55dafde51ae0 con 0x55dafdf5b670 2024-03-28T18:12:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.219+0000 7f6377400640 10 client.25591 handle_command_reply: tid=0 2024-03-28T18:12:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.219+0000 7f639f003740 1 client.25591 shutdown 2024-03-28T18:12:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.219+0000 7f639f003740 1 -- 172.21.15.37:0/2351542588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdee6dd0 msgr2=0x55dafdee9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.219+0000 7f639f003740 1 --2- 172.21.15.37:0/2351542588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdee6dd0 0x55dafdee9720 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f63840045c0 tx=0x7f6384004c10 comp rx=0 tx=0).stop 2024-03-28T18:12:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.220+0000 7f639f003740 1 -- 172.21.15.37:0/2351542588 shutdown_connections 2024-03-28T18:12:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.220+0000 7f639f003740 1 --2- 172.21.15.37:0/2351542588 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55dafdf5b670 0x55dafdf5da90 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.220+0000 7f639f003740 1 --2- 172.21.15.37:0/2351542588 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55dafdeef460 0x55dafdee9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.220+0000 7f639f003740 1 --2- 172.21.15.37:0/2351542588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dafdee6dd0 0x55dafdee9720 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.220+0000 7f639f003740 1 --2- 172.21.15.37:0/2351542588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55dafdee60f0 0x55dafdee91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.220+0000 7f639f003740 1 -- 172.21.15.37:0/2351542588 >> 172.21.15.37:0/2351542588 conn(0x55dafdead200 msgr2=0x55dafdeea830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:03.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.220+0000 7f6377400640 0 client.25591 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:03.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.220+0000 7f639f003740 1 -- 172.21.15.37:0/2351542588 shutdown_connections 2024-03-28T18:12:03.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.220+0000 7f639f003740 1 -- 172.21.15.37:0/2351542588 wait complete. 2024-03-28T18:12:03.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.220+0000 7f639f003740 20 client.25591 trim_cache size 0 max 16384 2024-03-28T18:12:03.223 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:12:03.223 INFO:teuthology.orchestra.run.smithi037.stdout: "status": "no active scrubs running", 2024-03-28T18:12:03.223 INFO:teuthology.orchestra.run.smithi037.stdout: "scrubs": {} 2024-03-28T18:12:03.223 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:12:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.221+0000 7f639ac00640 1 -- 172.21.15.37:0/1651176610 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f637c0a2b10 msgr2=0x7f637c0a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.221+0000 7f639ac00640 1 --2- 172.21.15.37:0/1651176610 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f637c0a2b10 0x7f637c0a4fd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f63880ccac0 tx=0x7f6388004a40 comp rx=0 tx=0).stop 2024-03-28T18:12:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.221+0000 7f639ac00640 1 -- 172.21.15.37:0/1651176610 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6394101530 msgr2=0x7f6394199fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.221+0000 7f639ac00640 1 --2- 172.21.15.37:0/1651176610 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6394101530 0x7f6394199fb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f638800e9f0 tx=0x7f638800edf0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.221+0000 7f639ac00640 1 -- 172.21.15.37:0/1651176610 shutdown_connections 2024-03-28T18:12:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.221+0000 7f639ac00640 1 --2- 172.21.15.37:0/1651176610 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f637c0a2b10 0x7f637c0a4fd0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.221+0000 7f639ac00640 1 --2- 172.21.15.37:0/1651176610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63941075f0 0x7f639419e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.221+0000 7f639ac00640 1 --2- 172.21.15.37:0/1651176610 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6394101ef0 0x7f639419a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.221+0000 7f639ac00640 1 --2- 172.21.15.37:0/1651176610 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6394101530 0x7f6394199fb0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.221+0000 7f639ac00640 1 -- 172.21.15.37:0/1651176610 >> 172.21.15.37:0/1651176610 conn(0x7f63940fd040 msgr2=0x7f63940fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:03.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.221+0000 7f639ac00640 1 -- 172.21.15.37:0/1651176610 shutdown_connections 2024-03-28T18:12:03.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.221+0000 7f639ac00640 1 -- 172.21.15.37:0/1651176610 wait complete. 2024-03-28T18:12:03.236 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:12:03.237 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:12:03.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.444+0000 7f1a15400640 1 -- 172.21.15.37:0/2160277602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a10068800 msgr2=0x7f1a10068c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:03.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.444+0000 7f1a15400640 1 --2- 172.21.15.37:0/2160277602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a10068800 0x7f1a10068c80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f19f8009930 tx=0x7f19f802bc20 comp rx=0 tx=0).stop 2024-03-28T18:12:03.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.444+0000 7f1a15400640 1 -- 172.21.15.37:0/2160277602 shutdown_connections 2024-03-28T18:12:03.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.444+0000 7f1a15400640 1 --2- 172.21.15.37:0/2160277602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a10068800 0x7f1a10068c80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.444+0000 7f1a15400640 1 --2- 172.21.15.37:0/2160277602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a100691c0 0x7f1a1010e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.444+0000 7f1a15400640 1 --2- 172.21.15.37:0/2160277602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a10107590 0x7f1a10107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.444+0000 7f1a15400640 1 -- 172.21.15.37:0/2160277602 >> 172.21.15.37:0/2160277602 conn(0x7f1a10075fb0 msgr2=0x7f1a100763c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:03.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.444+0000 7f1a15400640 1 -- 172.21.15.37:0/2160277602 shutdown_connections 2024-03-28T18:12:03.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.444+0000 7f1a15400640 1 -- 172.21.15.37:0/2160277602 wait complete. 2024-03-28T18:12:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.445+0000 7f1a15400640 1 Processor -- start 2024-03-28T18:12:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.445+0000 7f1a15400640 1 -- start start 2024-03-28T18:12:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.446+0000 7f1a15400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a10068800 0x7f1a10108390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.446+0000 7f1a15400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a100691c0 0x7f1a101088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.446+0000 7f1a15400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a10107590 0x7f1a10108e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.446+0000 7f1a15400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a101118a0 con 0x7f1a10107590 2024-03-28T18:12:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.446+0000 7f1a15400640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a10111720 con 0x7f1a100691c0 2024-03-28T18:12:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.446+0000 7f1a15400640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a10111a20 con 0x7f1a10068800 2024-03-28T18:12:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.446+0000 7f1a0fe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a10068800 0x7f1a10108390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.446+0000 7f1a0f400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a100691c0 0x7f1a101088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.446+0000 7f1a0fe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a10068800 0x7f1a10108390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:53564/0 (socket says 172.21.15.37:53564) 2024-03-28T18:12:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.446+0000 7f1a0fe00640 1 -- 172.21.15.37:0/1957322356 learned_addr learned my addr 172.21.15.37:0/1957322356 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.446+0000 7f1a14a00640 1 --2- 172.21.15.37:0/1957322356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a10107590 0x7f1a10108e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.446+0000 7f1a0f400640 1 -- 172.21.15.37:0/1957322356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a10068800 msgr2=0x7f1a10108390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.446+0000 7f1a0f400640 1 --2- 172.21.15.37:0/1957322356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a10068800 0x7f1a10108390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.446+0000 7f1a0f400640 1 -- 172.21.15.37:0/1957322356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a10107590 msgr2=0x7f1a10108e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.446+0000 7f1a0f400640 1 --2- 172.21.15.37:0/1957322356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a10107590 0x7f1a10108e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.446+0000 7f1a0f400640 1 -- 172.21.15.37:0/1957322356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a101aa5c0 con 0x7f1a100691c0 2024-03-28T18:12:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.446+0000 7f1a14a00640 1 --2- 172.21.15.37:0/1957322356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a10107590 0x7f1a10108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:12:03.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.447+0000 7f1a0f400640 1 --2- 172.21.15.37:0/1957322356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a100691c0 0x7f1a101088d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f19f8036040 tx=0x7f19f8004250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:03.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.447+0000 7f1a0cc00640 1 -- 172.21.15.37:0/1957322356 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f8043020 con 0x7f1a100691c0 2024-03-28T18:12:03.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.447+0000 7f1a0cc00640 1 -- 172.21.15.37:0/1957322356 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19f8005000 con 0x7f1a100691c0 2024-03-28T18:12:03.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.447+0000 7f1a0cc00640 1 -- 172.21.15.37:0/1957322356 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f8003a80 con 0x7f1a100691c0 2024-03-28T18:12:03.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.447+0000 7f1a15400640 1 -- 172.21.15.37:0/1957322356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a101aa850 con 0x7f1a100691c0 2024-03-28T18:12:03.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.447+0000 7f1a15400640 1 -- 172.21.15.37:0/1957322356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a101aad50 con 0x7f1a100691c0 2024-03-28T18:12:03.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.449+0000 7f1a15400640 1 -- 172.21.15.37:0/1957322356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a10102bb0 con 0x7f1a100691c0 2024-03-28T18:12:03.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.450+0000 7f1a0cc00640 1 -- 172.21.15.37:0/1957322356 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f19f8046020 con 0x7f1a100691c0 2024-03-28T18:12:03.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.452+0000 7f1a0cc00640 1 --2- 172.21.15.37:0/1957322356 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f19f40a2b10 0x7f19f40a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:03.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.452+0000 7f1a0cc00640 1 -- 172.21.15.37:0/1957322356 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f19f80f0c00 con 0x7f1a100691c0 2024-03-28T18:12:03.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.452+0000 7f1a0fe00640 1 --2- 172.21.15.37:0/1957322356 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f19f40a2b10 0x7f19f40a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:03.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.453+0000 7f1a0fe00640 1 --2- 172.21.15.37:0/1957322356 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f19f40a2b10 0x7f19f40a4fd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1a00009fd0 tx=0x7f1a00009500 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:03.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.456+0000 7f1a0cc00640 1 -- 172.21.15.37:0/1957322356 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f19f80bd7f0 con 0x7f1a100691c0 2024-03-28T18:12:03.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.741+0000 7f1a15400640 1 -- 172.21.15.37:0/1957322356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a101ab070 con 0x7f1a100691c0 2024-03-28T18:12:03.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.741+0000 7f1a0cc00640 1 -- 172.21.15.37:0/1957322356 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11383 (secure 0 0 0) 0x7f19f80bd0f0 con 0x7f1a100691c0 2024-03-28T18:12:03.742 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:12:03.742 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1926270371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6838","nonce":1926270371},{"type":"v1","addr":"172.21.15.37:6839","nonce":1926270371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:08:19.968894+0000","modified":"2024-03-28T18:10:49.565429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24580,"mds_1":24569,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6837/4034607523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":4034607523},{"type":"v1","addr":"172.21.15.59:6837","nonce":4034607523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6839/1575518279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":1575518279},{"type":"v1","addr":"172.21.15.170:6839","nonce":1575518279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6839/2091629078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":2091629078},{"type":"v1","addr":"172.21.15.59:6839","nonce":2091629078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24580":{"gid":24580,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.37:6841/1923263622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6840","nonce":1923263622},{"type":"v1","addr":"172.21.15.37:6841","nonce":1923263622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24580,"qdb_cluster":[24580,24550,24569]},"id":1}]} 2024-03-28T18:12:03.742 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 23 2024-03-28T18:12:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.744+0000 7f1a15400640 1 -- 172.21.15.37:0/1957322356 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f19f40a2b10 msgr2=0x7f19f40a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.744+0000 7f1a15400640 1 --2- 172.21.15.37:0/1957322356 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f19f40a2b10 0x7f19f40a4fd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1a00009fd0 tx=0x7f1a00009500 comp rx=0 tx=0).stop 2024-03-28T18:12:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.744+0000 7f1a15400640 1 -- 172.21.15.37:0/1957322356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a100691c0 msgr2=0x7f1a101088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.744+0000 7f1a15400640 1 --2- 172.21.15.37:0/1957322356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a100691c0 0x7f1a101088d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f19f8036040 tx=0x7f19f8004250 comp rx=0 tx=0).stop 2024-03-28T18:12:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.744+0000 7f1a15400640 1 -- 172.21.15.37:0/1957322356 shutdown_connections 2024-03-28T18:12:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.744+0000 7f1a15400640 1 --2- 172.21.15.37:0/1957322356 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f19f40a2b10 0x7f19f40a4fd0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.744+0000 7f1a15400640 1 --2- 172.21.15.37:0/1957322356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a100691c0 0x7f1a101088d0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.744+0000 7f1a15400640 1 --2- 172.21.15.37:0/1957322356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a10107590 0x7f1a10108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.744+0000 7f1a15400640 1 --2- 172.21.15.37:0/1957322356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a10068800 0x7f1a10108390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.744+0000 7f1a15400640 1 -- 172.21.15.37:0/1957322356 >> 172.21.15.37:0/1957322356 conn(0x7f1a10075fb0 msgr2=0x7f1a1010c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.744+0000 7f1a15400640 1 -- 172.21.15.37:0/1957322356 shutdown_connections 2024-03-28T18:12:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:03.745+0000 7f1a15400640 1 -- 172.21.15.37:0/1957322356 wait complete. 2024-03-28T18:12:03.762 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:12:03.762 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:12:04.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.002+0000 7f5999400640 1 -- 172.21.15.37:0/20373611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5994109730 msgr2=0x7f5994109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.002+0000 7f5999400640 1 --2- 172.21.15.37:0/20373611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5994109730 0x7f5994109b10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5984009890 tx=0x7f598402bb70 comp rx=0 tx=0).stop 2024-03-28T18:12:04.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.002+0000 7f5999400640 1 -- 172.21.15.37:0/20373611 shutdown_connections 2024-03-28T18:12:04.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.002+0000 7f5999400640 1 --2- 172.21.15.37:0/20373611 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59941040e0 0x7f59941105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.002+0000 7f5999400640 1 --2- 172.21.15.37:0/20373611 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5994103720 0x7f5994103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.002+0000 7f5999400640 1 --2- 172.21.15.37:0/20373611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5994109730 0x7f5994109b10 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.002+0000 7f5999400640 1 -- 172.21.15.37:0/20373611 >> 172.21.15.37:0/20373611 conn(0x7f59940ff1b0 msgr2=0x7f59941015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:04.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.002+0000 7f5999400640 1 -- 172.21.15.37:0/20373611 shutdown_connections 2024-03-28T18:12:04.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.002+0000 7f5999400640 1 -- 172.21.15.37:0/20373611 wait complete. 2024-03-28T18:12:04.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.004+0000 7f5999400640 1 Processor -- start 2024-03-28T18:12:04.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.004+0000 7f5999400640 1 -- start start 2024-03-28T18:12:04.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.005+0000 7f5999400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5994103720 0x7f599419a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.005+0000 7f5999400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59941040e0 0x7f599419a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.005+0000 7f5999400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5994109730 0x7f599419e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.005+0000 7f5999400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5994113b00 con 0x7f59941040e0 2024-03-28T18:12:04.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.005+0000 7f5999400640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5994113980 con 0x7f5994109730 2024-03-28T18:12:04.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.005+0000 7f5999400640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5994113c80 con 0x7f5994103720 2024-03-28T18:12:04.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.005+0000 7f5993e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5994103720 0x7f599419a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.005+0000 7f5993e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5994103720 0x7f599419a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:53582/0 (socket says 172.21.15.37:53582) 2024-03-28T18:12:04.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.005+0000 7f5993e00640 1 -- 172.21.15.37:0/2369712510 learned_addr learned my addr 172.21.15.37:0/2369712510 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:04.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.005+0000 7f5993400640 1 --2- 172.21.15.37:0/2369712510 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59941040e0 0x7f599419a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.005+0000 7f5998a00640 1 --2- 172.21.15.37:0/2369712510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5994109730 0x7f599419e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.005+0000 7f5993e00640 1 -- 172.21.15.37:0/2369712510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5994109730 msgr2=0x7f599419e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.005+0000 7f5993e00640 1 --2- 172.21.15.37:0/2369712510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5994109730 0x7f599419e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.005+0000 7f5993e00640 1 -- 172.21.15.37:0/2369712510 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59941040e0 msgr2=0x7f599419a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.005+0000 7f5993e00640 1 --2- 172.21.15.37:0/2369712510 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59941040e0 0x7f599419a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.005+0000 7f5993e00640 1 -- 172.21.15.37:0/2369712510 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f599419f050 con 0x7f5994103720 2024-03-28T18:12:04.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.005+0000 7f5998a00640 1 --2- 172.21.15.37:0/2369712510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5994109730 0x7f599419e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:04.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.006+0000 7f5993e00640 1 --2- 172.21.15.37:0/2369712510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5994103720 0x7f599419a060 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f59840057c0 tx=0x7f59840045d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:04.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.006+0000 7f5990c00640 1 -- 172.21.15.37:0/2369712510 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984041020 con 0x7f5994103720 2024-03-28T18:12:04.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.006+0000 7f5990c00640 1 -- 172.21.15.37:0/2369712510 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f598402c9f0 con 0x7f5994103720 2024-03-28T18:12:04.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.006+0000 7f5990c00640 1 -- 172.21.15.37:0/2369712510 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984039040 con 0x7f5994103720 2024-03-28T18:12:04.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.006+0000 7f5999400640 1 -- 172.21.15.37:0/2369712510 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f599419f2e0 con 0x7f5994103720 2024-03-28T18:12:04.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.006+0000 7f5999400640 1 -- 172.21.15.37:0/2369712510 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59941a6bd0 con 0x7f5994103720 2024-03-28T18:12:04.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.009+0000 7f599d80f740 1 Processor -- start 2024-03-28T18:12:04.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.010+0000 7f599d80f740 1 -- start start 2024-03-28T18:12:04.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.010+0000 7f5990c00640 1 -- 172.21.15.37:0/2369712510 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f5984044020 con 0x7f5994103720 2024-03-28T18:12:04.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.010+0000 7f599d80f740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a458148b0 0x560a45814c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.010+0000 7f599d80f740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a45815260 0x560a4581d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.010+0000 7f599d80f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a4581dde0 0x560a45820230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.010+0000 7f599d80f740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a457782c0 con 0x560a458148b0 2024-03-28T18:12:04.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.010+0000 7f599d80f740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a45720c90 con 0x560a45815260 2024-03-28T18:12:04.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.010+0000 7f599d80f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a45774360 con 0x560a4581dde0 2024-03-28T18:12:04.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.010+0000 7f5993e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a45815260 0x560a4581d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.010+0000 7f5998a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a458148b0 0x560a45814c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.010+0000 7f5993400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a4581dde0 0x560a45820230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.010+0000 7f5993400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a4581dde0 0x560a45820230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:53588/0 (socket says 172.21.15.37:53588) 2024-03-28T18:12:04.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.010+0000 7f5993400640 1 -- 172.21.15.37:0/538252762 learned_addr learned my addr 172.21.15.37:0/538252762 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:04.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.011+0000 7f5993400640 1 -- 172.21.15.37:0/538252762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a45815260 msgr2=0x560a4581d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.011+0000 7f5993400640 1 --2- 172.21.15.37:0/538252762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a45815260 0x560a4581d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.011+0000 7f5993400640 1 -- 172.21.15.37:0/538252762 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a458148b0 msgr2=0x560a45814c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.011+0000 7f5993400640 1 --2- 172.21.15.37:0/538252762 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a458148b0 0x560a45814c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.011+0000 7f5993400640 1 -- 172.21.15.37:0/538252762 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a45745080 con 0x560a4581dde0 2024-03-28T18:12:04.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.011+0000 7f5998a00640 1 --2- 172.21.15.37:0/538252762 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a458148b0 0x560a45814c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:12:04.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.011+0000 7f5993400640 1 --2- 172.21.15.37:0/538252762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a4581dde0 0x560a45820230 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5980009c90 tx=0x7f5980011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d036a44525de3102 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:04.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.012+0000 7f5990c00640 1 --2- 172.21.15.37:0/2369712510 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f59780a2b10 0x7f59780a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.012+0000 7f5989200640 1 -- 172.21.15.37:0/538252762 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598001e070 con 0x560a4581dde0 2024-03-28T18:12:04.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.012+0000 7f5989200640 1 -- 172.21.15.37:0/538252762 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5980019b10 con 0x560a4581dde0 2024-03-28T18:12:04.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.012+0000 7f5989200640 1 -- 172.21.15.37:0/538252762 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598000eac0 con 0x560a4581dde0 2024-03-28T18:12:04.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.012+0000 7f5990c00640 1 -- 172.21.15.37:0/2369712510 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f59840f0810 con 0x7f5994103720 2024-03-28T18:12:04.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.012+0000 7f5998a00640 1 --2- 172.21.15.37:0/2369712510 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f59780a2b10 0x7f59780a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.013+0000 7f599d80f740 1 -- 172.21.15.37:0/538252762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a4581dde0 msgr2=0x560a45820230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.013+0000 7f599d80f740 1 --2- 172.21.15.37:0/538252762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a4581dde0 0x560a45820230 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5980009c90 tx=0x7f5980011810 comp rx=0 tx=0).stop 2024-03-28T18:12:04.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.013+0000 7f599d80f740 1 -- 172.21.15.37:0/538252762 shutdown_connections 2024-03-28T18:12:04.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.013+0000 7f599d80f740 1 --2- 172.21.15.37:0/538252762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a45815260 0x560a4581d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.013+0000 7f599d80f740 1 --2- 172.21.15.37:0/538252762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a4581dde0 0x560a45820230 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.013+0000 7f599d80f740 1 --2- 172.21.15.37:0/538252762 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a458148b0 0x560a45814c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.013+0000 7f599d80f740 1 -- 172.21.15.37:0/538252762 >> 172.21.15.37:0/538252762 conn(0x560a4580ef80 msgr2=0x560a4580f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:04.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.013+0000 7f5998a00640 1 --2- 172.21.15.37:0/2369712510 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f59780a2b10 0x7f59780a4fd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f598c005fd0 tx=0x7f598c0095c0 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:04.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.013+0000 7f599d80f740 1 -- 172.21.15.37:0/538252762 shutdown_connections 2024-03-28T18:12:04.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.013+0000 7f599d80f740 1 -- 172.21.15.37:0/538252762 wait complete. 2024-03-28T18:12:04.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.013+0000 7f599d80f740 1 -- start start 2024-03-28T18:12:04.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.014+0000 7f599d80f740 1 Processor -- start 2024-03-28T18:12:04.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.014+0000 7f599d80f740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a458148b0 0x560a4581d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.014+0000 7f599d80f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a45815260 0x560a4581aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.014+0000 7f599d80f740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4581dde0 0x560a4581aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.014+0000 7f599d80f740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a45829560 con 0x560a458148b0 2024-03-28T18:12:04.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.014+0000 7f599d80f740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a457340d0 con 0x560a4581dde0 2024-03-28T18:12:04.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.014+0000 7f599d80f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a458296e0 con 0x560a45815260 2024-03-28T18:12:04.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.015+0000 7f599d80f740 10 client.0 _ll_register_callbacks cb 0x560a45738e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:04.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.015+0000 7f5993400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a45815260 0x560a4581aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.015+0000 7f5993e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a458148b0 0x560a4581d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.015+0000 7f5993400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a45815260 0x560a4581aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:53596/0 (socket says 172.21.15.37:53596) 2024-03-28T18:12:04.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.015+0000 7f5993400640 1 -- 172.21.15.37:0/305298940 learned_addr learned my addr 172.21.15.37:0/305298940 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:04.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.015+0000 7f5998a00640 1 --2- 172.21.15.37:0/305298940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4581dde0 0x560a4581aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.015+0000 7f5993400640 1 -- 172.21.15.37:0/305298940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4581dde0 msgr2=0x560a4581aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.015+0000 7f5993400640 1 --2- 172.21.15.37:0/305298940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4581dde0 0x560a4581aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.015+0000 7f5993400640 1 -- 172.21.15.37:0/305298940 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a458148b0 msgr2=0x560a4581d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.015+0000 7f5993400640 1 --2- 172.21.15.37:0/305298940 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a458148b0 0x560a4581d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.015+0000 7f5993400640 1 -- 172.21.15.37:0/305298940 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a4548c3f0 con 0x560a45815260 2024-03-28T18:12:04.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.016+0000 7f5993400640 1 --2- 172.21.15.37:0/305298940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a45815260 0x560a4581aa70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5980019a20 tx=0x7f5980019a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3062bfcc84da0e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:04.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.016+0000 7f597d600640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:12:04.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.016+0000 7f597d600640 1 -- 172.21.15.37:0/305298940 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598001e070 con 0x560a45815260 2024-03-28T18:12:04.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.016+0000 7f597d600640 1 -- 172.21.15.37:0/305298940 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5980004030 con 0x560a45815260 2024-03-28T18:12:04.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.016+0000 7f599d80f740 1 -- 172.21.15.37:0/305298940 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5980019b10 con 0x560a45815260 2024-03-28T18:12:04.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.016+0000 7f597d600640 1 -- 172.21.15.37:0/305298940 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5980033050 con 0x560a45815260 2024-03-28T18:12:04.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.016+0000 7f599d80f740 10 client.25609 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:04.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.016+0000 7f599d80f740 1 -- 172.21.15.37:0/305298940 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560a455567d0 con 0x560a45815260 2024-03-28T18:12:04.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.016+0000 7f597d600640 1 -- 172.21.15.37:0/305298940 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5980019b10 con 0x560a45815260 2024-03-28T18:12:04.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.017+0000 7f597d600640 1 -- 172.21.15.37:0/305298940 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f598000e7c0 con 0x560a45815260 2024-03-28T18:12:04.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.017+0000 7f599d80f740 10 client.25609 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:04.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.017+0000 7f599d80f740 10 client.25609 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-28T18:12:04.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.017+0000 7f599d80f740 20 client.25609 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:04.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.017+0000 7f599d80f740 1 --2- 172.21.15.37:0/305298940 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x560a45836390 0x560a45838780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.017+0000 7f599d80f740 4 client.25609 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:12:04.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.017+0000 7f599d80f740 1 -- 172.21.15.37:0/305298940 --> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560a45785800 con 0x560a45836390 2024-03-28T18:12:04.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.017+0000 7f5993e00640 1 --2- 172.21.15.37:0/305298940 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x560a45836390 0x560a45838780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.018+0000 7f597d600640 10 client.25609 ms_handle_connect on v2:172.21.15.170:6838/1575518279 2024-03-28T18:12:04.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.018+0000 7f5993e00640 1 --2- 172.21.15.37:0/305298940 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x560a45836390 0x560a45838780 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c6e8e909ae9695c4 server_cookie=b71023962ce315f9 in_seq=0 out_seq=0 2024-03-28T18:12:04.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.019+0000 7f597d600640 1 -- 172.21.15.37:0/305298940 <== mds.2 v2:172.21.15.170:6838/1575518279 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560a45785800 con 0x560a45836390 2024-03-28T18:12:04.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.019+0000 7f597d600640 10 client.25609 handle_command_reply: tid=0 2024-03-28T18:12:04.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.019+0000 7f599d80f740 1 client.25609 shutdown 2024-03-28T18:12:04.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.019+0000 7f599d80f740 1 -- 172.21.15.37:0/305298940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a45815260 msgr2=0x560a4581aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.019+0000 7f599d80f740 1 --2- 172.21.15.37:0/305298940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a45815260 0x560a4581aa70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5980019a20 tx=0x7f5980019a50 comp rx=0 tx=0).stop 2024-03-28T18:12:04.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.019+0000 7f599d80f740 1 -- 172.21.15.37:0/305298940 shutdown_connections 2024-03-28T18:12:04.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.019+0000 7f599d80f740 1 --2- 172.21.15.37:0/305298940 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x560a45836390 0x560a45838780 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.019+0000 7f599d80f740 1 --2- 172.21.15.37:0/305298940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4581dde0 0x560a4581aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.019+0000 7f599d80f740 1 --2- 172.21.15.37:0/305298940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a45815260 0x560a4581aa70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.019+0000 7f599d80f740 1 --2- 172.21.15.37:0/305298940 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a458148b0 0x560a4581d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.019+0000 7f599d80f740 1 -- 172.21.15.37:0/305298940 >> 172.21.15.37:0/305298940 conn(0x560a4580ef80 msgr2=0x560a45810cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:04.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.019+0000 7f597d600640 0 client.25609 ms_handle_reset on v2:172.21.15.170:6838/1575518279 2024-03-28T18:12:04.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.019+0000 7f599d80f740 1 -- 172.21.15.37:0/305298940 shutdown_connections 2024-03-28T18:12:04.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.019+0000 7f599d80f740 1 -- 172.21.15.37:0/305298940 wait complete. 2024-03-28T18:12:04.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.020+0000 7f599d80f740 20 client.25609 trim_cache size 0 max 16384 2024-03-28T18:12:04.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:03 smithi059 ceph-mon[36190]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T18:12:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:03 smithi059 ceph-mon[36190]: mgrmap e26: x(active, since 52s), standbys: y, z 2024-03-28T18:12:04.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:03 smithi170 ceph-mon[36280]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T18:12:04.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:03 smithi170 ceph-mon[36280]: mgrmap e26: x(active, since 52s), standbys: y, z 2024-03-28T18:12:04.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.042+0000 7f599d80f740 1 Processor -- start 2024-03-28T18:12:04.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.042+0000 7f599d80f740 1 -- start start 2024-03-28T18:12:04.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.043+0000 7f599d80f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a458148b0 0x560a4581a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.043+0000 7f599d80f740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a45815260 0x560a4581ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.043+0000 7f599d80f740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4581dde0 0x560a45817cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.043+0000 7f599d80f740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59941ad330 con 0x560a45815260 2024-03-28T18:12:04.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.043+0000 7f599d80f740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59941aae30 con 0x560a4581dde0 2024-03-28T18:12:04.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.043+0000 7f599d80f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59941a8ed0 con 0x560a458148b0 2024-03-28T18:12:04.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.043+0000 7f5993e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a458148b0 0x560a4581a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.043+0000 7f5993400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a45815260 0x560a4581ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.043+0000 7f5993e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a458148b0 0x560a4581a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:53604/0 (socket says 172.21.15.37:53604) 2024-03-28T18:12:04.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.043+0000 7f5998a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4581dde0 0x560a45817cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.043+0000 7f5993e00640 1 -- 172.21.15.37:0/2938721627 learned_addr learned my addr 172.21.15.37:0/2938721627 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:04.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.043+0000 7f5993e00640 1 -- 172.21.15.37:0/2938721627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4581dde0 msgr2=0x560a45817cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.043+0000 7f5993e00640 1 --2- 172.21.15.37:0/2938721627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4581dde0 0x560a45817cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.043+0000 7f5993e00640 1 -- 172.21.15.37:0/2938721627 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a45815260 msgr2=0x560a4581ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.043+0000 7f5993e00640 1 --2- 172.21.15.37:0/2938721627 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a45815260 0x560a4581ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.043+0000 7f5993e00640 1 -- 172.21.15.37:0/2938721627 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a4581b360 con 0x560a458148b0 2024-03-28T18:12:04.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.044+0000 7f5993e00640 1 --2- 172.21.15.37:0/2938721627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a458148b0 0x560a4581a220 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f59840099b0 tx=0x7f59840f2c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4510630959961ab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:04.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.044+0000 7f597fe00640 1 -- 172.21.15.37:0/2938721627 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59840bc840 con 0x560a458148b0 2024-03-28T18:12:04.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.044+0000 7f597fe00640 1 -- 172.21.15.37:0/2938721627 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59840d3020 con 0x560a458148b0 2024-03-28T18:12:04.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.044+0000 7f597fe00640 1 -- 172.21.15.37:0/2938721627 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59840bc660 con 0x560a458148b0 2024-03-28T18:12:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.045+0000 7f599d80f740 1 -- 172.21.15.37:0/2938721627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a458148b0 msgr2=0x560a4581a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.045+0000 7f599d80f740 1 --2- 172.21.15.37:0/2938721627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a458148b0 0x560a4581a220 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f59840099b0 tx=0x7f59840f2c60 comp rx=0 tx=0).stop 2024-03-28T18:12:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.045+0000 7f599d80f740 1 -- 172.21.15.37:0/2938721627 shutdown_connections 2024-03-28T18:12:04.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.045+0000 7f599d80f740 1 --2- 172.21.15.37:0/2938721627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4581dde0 0x560a45817cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.045+0000 7f599d80f740 1 --2- 172.21.15.37:0/2938721627 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a45815260 0x560a4581ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.045+0000 7f599d80f740 1 --2- 172.21.15.37:0/2938721627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a458148b0 0x560a4581a220 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.045+0000 7f599d80f740 1 -- 172.21.15.37:0/2938721627 >> 172.21.15.37:0/2938721627 conn(0x560a4580ef80 msgr2=0x560a4580e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:04.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.045+0000 7f599d80f740 1 -- 172.21.15.37:0/2938721627 shutdown_connections 2024-03-28T18:12:04.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.045+0000 7f599d80f740 1 -- 172.21.15.37:0/2938721627 wait complete. 2024-03-28T18:12:04.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.045+0000 7f599d80f740 1 -- start start 2024-03-28T18:12:04.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.046+0000 7f599d80f740 1 Processor -- start 2024-03-28T18:12:04.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.046+0000 7f599d80f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a458148b0 0x560a4582c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.046+0000 7f599d80f740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a45815260 0x560a4582ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.046+0000 7f599d80f740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4581dde0 0x560a45890ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.046+0000 7f599d80f740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59941a94d0 con 0x560a45815260 2024-03-28T18:12:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.046+0000 7f599d80f740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59941aa7d0 con 0x560a4581dde0 2024-03-28T18:12:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.046+0000 7f599d80f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59941a9b10 con 0x560a458148b0 2024-03-28T18:12:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.046+0000 7f599d80f740 10 client.0 _ll_register_callbacks cb 0x7f598001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.046+0000 7f5993e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a458148b0 0x560a4582c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.046+0000 7f5993400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a45815260 0x560a4582ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.046+0000 7f5993e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a458148b0 0x560a4582c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:53608/0 (socket says 172.21.15.37:53608) 2024-03-28T18:12:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.046+0000 7f5998a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4581dde0 0x560a45890ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.046+0000 7f5993e00640 1 -- 172.21.15.37:0/1448801417 learned_addr learned my addr 172.21.15.37:0/1448801417 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.047+0000 7f5993e00640 1 -- 172.21.15.37:0/1448801417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4581dde0 msgr2=0x560a45890ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.047+0000 7f5993e00640 1 --2- 172.21.15.37:0/1448801417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4581dde0 0x560a45890ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.047+0000 7f5993e00640 1 -- 172.21.15.37:0/1448801417 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a45815260 msgr2=0x560a4582ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.047+0000 7f5993e00640 1 --2- 172.21.15.37:0/1448801417 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a45815260 0x560a4582ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.047+0000 7f5993e00640 1 -- 172.21.15.37:0/1448801417 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a458910d0 con 0x560a458148b0 2024-03-28T18:12:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.047+0000 7f5993e00640 1 --2- 172.21.15.37:0/1448801417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a458148b0 0x560a4582c520 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f59840b9160 tx=0x7f59840f0a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a24116ce3592328 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.047+0000 7f597d600640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:12:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.047+0000 7f597d600640 1 -- 172.21.15.37:0/1448801417 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59840d9020 con 0x560a458148b0 2024-03-28T18:12:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.047+0000 7f597d600640 1 -- 172.21.15.37:0/1448801417 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5984034640 con 0x560a458148b0 2024-03-28T18:12:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.047+0000 7f599d80f740 1 -- 172.21.15.37:0/1448801417 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f59840d3020 con 0x560a458148b0 2024-03-28T18:12:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.048+0000 7f597d600640 1 -- 172.21.15.37:0/1448801417 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59840bc660 con 0x560a458148b0 2024-03-28T18:12:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.048+0000 7f599d80f740 10 client.25615 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.048+0000 7f599d80f740 1 -- 172.21.15.37:0/1448801417 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560a458923b0 con 0x560a458148b0 2024-03-28T18:12:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.048+0000 7f597d600640 1 -- 172.21.15.37:0/1448801417 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59840d3020 con 0x560a458148b0 2024-03-28T18:12:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.048+0000 7f597d600640 1 -- 172.21.15.37:0/1448801417 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f598403c030 con 0x560a458148b0 2024-03-28T18:12:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.048+0000 7f599d80f740 10 client.25615 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.048+0000 7f599d80f740 10 client.25615 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-28T18:12:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.048+0000 7f599d80f740 20 client.25615 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.048+0000 7f599d80f740 1 --2- 172.21.15.37:0/1448801417 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x560a45892a20 0x560a45894e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.048+0000 7f599d80f740 4 client.25615 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:12:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.048+0000 7f599d80f740 1 -- 172.21.15.37:0/1448801417 --> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560a4577fb50 con 0x560a45892a20 2024-03-28T18:12:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.048+0000 7f5993400640 1 --2- 172.21.15.37:0/1448801417 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x560a45892a20 0x560a45894e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.049+0000 7f5993400640 1 --2- 172.21.15.37:0/1448801417 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x560a45892a20 0x560a45894e10 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=eae8fb149fd7d485 server_cookie=325e46f0cd8e8ba1 in_seq=0 out_seq=0 2024-03-28T18:12:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.049+0000 7f597d600640 10 client.25615 ms_handle_connect on v2:172.21.15.170:6838/1575518279 2024-03-28T18:12:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.049+0000 7f597d600640 1 -- 172.21.15.37:0/1448801417 <== mds.2 v2:172.21.15.170:6838/1575518279 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560a4577fb50 con 0x560a45892a20 2024-03-28T18:12:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.049+0000 7f597d600640 10 client.25615 handle_command_reply: tid=0 2024-03-28T18:12:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.049+0000 7f599d80f740 1 client.25615 shutdown 2024-03-28T18:12:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.049+0000 7f599d80f740 1 -- 172.21.15.37:0/1448801417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a458148b0 msgr2=0x560a4582c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.049+0000 7f599d80f740 1 --2- 172.21.15.37:0/1448801417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a458148b0 0x560a4582c520 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f59840b9160 tx=0x7f59840f0a90 comp rx=0 tx=0).stop 2024-03-28T18:12:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.049+0000 7f599d80f740 1 -- 172.21.15.37:0/1448801417 shutdown_connections 2024-03-28T18:12:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.049+0000 7f599d80f740 1 --2- 172.21.15.37:0/1448801417 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x560a45892a20 0x560a45894e10 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.049+0000 7f599d80f740 1 --2- 172.21.15.37:0/1448801417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4581dde0 0x560a45890ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.049+0000 7f599d80f740 1 --2- 172.21.15.37:0/1448801417 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560a45815260 0x560a4582ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.049+0000 7f599d80f740 1 --2- 172.21.15.37:0/1448801417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560a458148b0 0x560a4582c520 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.049+0000 7f597d600640 0 client.25615 ms_handle_reset on v2:172.21.15.170:6838/1575518279 2024-03-28T18:12:04.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.049+0000 7f599d80f740 1 -- 172.21.15.37:0/1448801417 >> 172.21.15.37:0/1448801417 conn(0x560a4580ef80 msgr2=0x560a45815ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:04.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.049+0000 7f599d80f740 1 -- 172.21.15.37:0/1448801417 shutdown_connections 2024-03-28T18:12:04.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.050+0000 7f599d80f740 1 -- 172.21.15.37:0/1448801417 wait complete. 2024-03-28T18:12:04.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.050+0000 7f599d80f740 20 client.25615 trim_cache size 0 max 16384 2024-03-28T18:12:04.064 INFO:teuthology.orchestra.run.smithi037.stdout:[] 2024-03-28T18:12:04.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.050+0000 7f5999400640 1 -- 172.21.15.37:0/2369712510 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f59780a2b10 msgr2=0x7f59780a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.050+0000 7f5999400640 1 --2- 172.21.15.37:0/2369712510 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f59780a2b10 0x7f59780a4fd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f598c005fd0 tx=0x7f598c0095c0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.050+0000 7f5999400640 1 -- 172.21.15.37:0/2369712510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5994103720 msgr2=0x7f599419a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.050+0000 7f5999400640 1 --2- 172.21.15.37:0/2369712510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5994103720 0x7f599419a060 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f59840057c0 tx=0x7f59840045d0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.050+0000 7f5999400640 1 -- 172.21.15.37:0/2369712510 shutdown_connections 2024-03-28T18:12:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.050+0000 7f5999400640 1 --2- 172.21.15.37:0/2369712510 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f59780a2b10 0x7f59780a4fd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.050+0000 7f5999400640 1 --2- 172.21.15.37:0/2369712510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5994109730 0x7f599419e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.050+0000 7f5999400640 1 --2- 172.21.15.37:0/2369712510 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59941040e0 0x7f599419a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.050+0000 7f5999400640 1 --2- 172.21.15.37:0/2369712510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5994103720 0x7f599419a060 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.050+0000 7f5999400640 1 -- 172.21.15.37:0/2369712510 >> 172.21.15.37:0/2369712510 conn(0x7f59940ff1b0 msgr2=0x7f59940ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.051+0000 7f5999400640 1 -- 172.21.15.37:0/2369712510 shutdown_connections 2024-03-28T18:12:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.051+0000 7f5999400640 1 -- 172.21.15.37:0/2369712510 wait complete. 2024-03-28T18:12:04.067 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:03 smithi037 ceph-mon[31460]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T18:12:04.067 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:03 smithi037 ceph-mon[31460]: mgrmap e26: x(active, since 52s), standbys: y, z 2024-03-28T18:12:04.068 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:12:04.068 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:12:04.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.282+0000 7f4f67200640 1 -- 172.21.15.37:0/1635061640 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f68103720 msgr2=0x7f4f68103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.282+0000 7f4f67200640 1 --2- 172.21.15.37:0/1635061640 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f68103720 0x7f4f68103ba0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4f54009900 tx=0x7f4f5402bc30 comp rx=0 tx=0).stop 2024-03-28T18:12:04.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.282+0000 7f4f67200640 1 -- 172.21.15.37:0/1635061640 shutdown_connections 2024-03-28T18:12:04.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.282+0000 7f4f67200640 1 --2- 172.21.15.37:0/1635061640 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f68103720 0x7f4f68103ba0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.282+0000 7f4f67200640 1 --2- 172.21.15.37:0/1635061640 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f681040e0 0x7f4f681105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.282+0000 7f4f67200640 1 --2- 172.21.15.37:0/1635061640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f68109730 0x7f4f68109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.282+0000 7f4f67200640 1 -- 172.21.15.37:0/1635061640 >> 172.21.15.37:0/1635061640 conn(0x7f4f680ff1b0 msgr2=0x7f4f681015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:04.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.282+0000 7f4f67200640 1 -- 172.21.15.37:0/1635061640 shutdown_connections 2024-03-28T18:12:04.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.282+0000 7f4f67200640 1 -- 172.21.15.37:0/1635061640 wait complete. 2024-03-28T18:12:04.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.283+0000 7f4f67200640 1 Processor -- start 2024-03-28T18:12:04.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.284+0000 7f4f67200640 1 -- start start 2024-03-28T18:12:04.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.284+0000 7f4f67200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f68103720 0x7f4f6819a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.284+0000 7f4f67200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f681040e0 0x7f4f6819a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.284+0000 7f4f67200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f68109730 0x7f4f6819e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.284+0000 7f4f67200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f68113b90 con 0x7f4f681040e0 2024-03-28T18:12:04.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.284+0000 7f4f67200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f68113a10 con 0x7f4f68109730 2024-03-28T18:12:04.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.284+0000 7f4f67200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f68113d10 con 0x7f4f68103720 2024-03-28T18:12:04.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.284+0000 7f4f65e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f68103720 0x7f4f6819a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.284+0000 7f4f66800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f68109730 0x7f4f6819e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.284+0000 7f4f65e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f68103720 0x7f4f6819a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:53624/0 (socket says 172.21.15.37:53624) 2024-03-28T18:12:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.284+0000 7f4f65e00640 1 -- 172.21.15.37:0/3795263948 learned_addr learned my addr 172.21.15.37:0/3795263948 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.284+0000 7f4f65400640 1 --2- 172.21.15.37:0/3795263948 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f681040e0 0x7f4f6819a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.285+0000 7f4f65e00640 1 -- 172.21.15.37:0/3795263948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f68109730 msgr2=0x7f4f6819e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.285+0000 7f4f65e00640 1 --2- 172.21.15.37:0/3795263948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f68109730 0x7f4f6819e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.285+0000 7f4f65e00640 1 -- 172.21.15.37:0/3795263948 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f681040e0 msgr2=0x7f4f6819a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.285+0000 7f4f65e00640 1 --2- 172.21.15.37:0/3795263948 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f681040e0 0x7f4f6819a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.285+0000 7f4f65e00640 1 -- 172.21.15.37:0/3795263948 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6819f060 con 0x7f4f68103720 2024-03-28T18:12:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.285+0000 7f4f65e00640 1 --2- 172.21.15.37:0/3795263948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f68103720 0x7f4f6819a070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4f5800e9f0 tx=0x7f4f5800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.285+0000 7f4f5ea00640 1 -- 172.21.15.37:0/3795263948 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5800ccc0 con 0x7f4f68103720 2024-03-28T18:12:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.285+0000 7f4f5ea00640 1 -- 172.21.15.37:0/3795263948 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f58010910 con 0x7f4f68103720 2024-03-28T18:12:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.285+0000 7f4f67200640 1 -- 172.21.15.37:0/3795263948 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f6819f350 con 0x7f4f68103720 2024-03-28T18:12:04.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.285+0000 7f4f5ea00640 1 -- 172.21.15.37:0/3795263948 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f580188a0 con 0x7f4f68103720 2024-03-28T18:12:04.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.286+0000 7f4f67200640 1 -- 172.21.15.37:0/3795263948 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f681a6be0 con 0x7f4f68103720 2024-03-28T18:12:04.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.289+0000 7f4f6fbfc740 1 Processor -- start 2024-03-28T18:12:04.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.289+0000 7f4f6fbfc740 1 -- start start 2024-03-28T18:12:04.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.289+0000 7f4f6fbfc740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff568b0 0x55d9dff56c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.289+0000 7f4f6fbfc740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff57260 0x55d9dff5f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.289+0000 7f4f6fbfc740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff5fde0 0x55d9dff62230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.289+0000 7f4f6fbfc740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9dfeba2c0 con 0x55d9dff5fde0 2024-03-28T18:12:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.289+0000 7f4f6fbfc740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9dfe62c90 con 0x55d9dff57260 2024-03-28T18:12:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.289+0000 7f4f6fbfc740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9dfeb6360 con 0x55d9dff568b0 2024-03-28T18:12:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.290+0000 7f4f66800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff568b0 0x55d9dff56c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.290+0000 7f4f66800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff568b0 0x55d9dff56c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:53628/0 (socket says 172.21.15.37:53628) 2024-03-28T18:12:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.290+0000 7f4f66800640 1 -- 172.21.15.37:0/1214120777 learned_addr learned my addr 172.21.15.37:0/1214120777 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.290+0000 7f4f65400640 1 --2- 172.21.15.37:0/1214120777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff5fde0 0x55d9dff62230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.290+0000 7f4f66800640 1 -- 172.21.15.37:0/1214120777 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff57260 msgr2=0x55d9dff5f810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:12:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.290+0000 7f4f66800640 1 --2- 172.21.15.37:0/1214120777 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff57260 0x55d9dff5f810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.290+0000 7f4f66800640 1 -- 172.21.15.37:0/1214120777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff5fde0 msgr2=0x55d9dff62230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.290+0000 7f4f66800640 1 --2- 172.21.15.37:0/1214120777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff5fde0 0x55d9dff62230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.290+0000 7f4f66800640 1 -- 172.21.15.37:0/1214120777 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9dfe87080 con 0x55d9dff568b0 2024-03-28T18:12:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.290+0000 7f4f66800640 1 --2- 172.21.15.37:0/1214120777 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff568b0 0x55d9dff56c90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4f60011440 tx=0x7f4f60011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b525868290b07039 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.290+0000 7f4f5ea00640 1 -- 172.21.15.37:0/3795263948 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f4f58010ac0 con 0x7f4f68103720 2024-03-28T18:12:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.291+0000 7f4f53400640 1 -- 172.21.15.37:0/1214120777 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60003c30 con 0x55d9dff568b0 2024-03-28T18:12:04.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.291+0000 7f4f53400640 1 -- 172.21.15.37:0/1214120777 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f60003dd0 con 0x55d9dff568b0 2024-03-28T18:12:04.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.291+0000 7f4f53400640 1 -- 172.21.15.37:0/1214120777 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6000fbb0 con 0x55d9dff568b0 2024-03-28T18:12:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.291+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/1214120777 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff568b0 msgr2=0x55d9dff56c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.291+0000 7f4f6fbfc740 1 --2- 172.21.15.37:0/1214120777 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff568b0 0x55d9dff56c90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4f60011440 tx=0x7f4f60011900 comp rx=0 tx=0).stop 2024-03-28T18:12:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.292+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/1214120777 shutdown_connections 2024-03-28T18:12:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.292+0000 7f4f6fbfc740 1 --2- 172.21.15.37:0/1214120777 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff57260 0x55d9dff5f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.292+0000 7f4f6fbfc740 1 --2- 172.21.15.37:0/1214120777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff5fde0 0x55d9dff62230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.292+0000 7f4f6fbfc740 1 --2- 172.21.15.37:0/1214120777 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff568b0 0x55d9dff56c90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.292+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/1214120777 >> 172.21.15.37:0/1214120777 conn(0x55d9dff50f80 msgr2=0x55d9dff51360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.292+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/1214120777 shutdown_connections 2024-03-28T18:12:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.292+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/1214120777 wait complete. 2024-03-28T18:12:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.292+0000 7f4f6fbfc740 1 -- start start 2024-03-28T18:12:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.293+0000 7f4f5ea00640 1 --2- 172.21.15.37:0/3795263948 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f4f4c0a2a20 0x7f4f4c0a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.293+0000 7f4f6fbfc740 1 Processor -- start 2024-03-28T18:12:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.293+0000 7f4f5ea00640 1 -- 172.21.15.37:0/3795263948 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4f580cf3e0 con 0x7f4f68103720 2024-03-28T18:12:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.293+0000 7f4f66800640 1 --2- 172.21.15.37:0/3795263948 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f4f4c0a2a20 0x7f4f4c0a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.293+0000 7f4f6fbfc740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff568b0 0x55d9dff5f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.293+0000 7f4f6fbfc740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff57260 0x55d9dff5cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.293+0000 7f4f6fbfc740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff5fde0 0x55d9dff5d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.293+0000 7f4f6fbfc740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9dff62d50 con 0x55d9dff5fde0 2024-03-28T18:12:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.294+0000 7f4f6fbfc740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9dfe760d0 con 0x55d9dff57260 2024-03-28T18:12:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.294+0000 7f4f6fbfc740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9dff62ed0 con 0x55d9dff568b0 2024-03-28T18:12:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.294+0000 7f4f6fbfc740 10 client.0 _ll_register_callbacks cb 0x55d9dfe7ae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.294+0000 7f4f65e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff568b0 0x55d9dff5f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.294+0000 7f4f65e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff568b0 0x55d9dff5f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:53644/0 (socket says 172.21.15.37:53644) 2024-03-28T18:12:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.294+0000 7f4f65e00640 1 -- 172.21.15.37:0/189668315 learned_addr learned my addr 172.21.15.37:0/189668315 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.294+0000 7f4f66800640 1 --2- 172.21.15.37:0/3795263948 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f4f4c0a2a20 0x7f4f4c0a4ee0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4f6001b040 tx=0x7f4f60005fd0 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.294+0000 7f4f66800640 1 --2- 172.21.15.37:0/189668315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff5fde0 0x55d9dff5d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.294+0000 7f4f65400640 1 --2- 172.21.15.37:0/189668315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff57260 0x55d9dff5cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.294+0000 7f4f65e00640 1 -- 172.21.15.37:0/189668315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff57260 msgr2=0x55d9dff5cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.294+0000 7f4f65e00640 1 --2- 172.21.15.37:0/189668315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff57260 0x55d9dff5cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.294+0000 7f4f65e00640 1 -- 172.21.15.37:0/189668315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff5fde0 msgr2=0x55d9dff5d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.294+0000 7f4f65e00640 1 --2- 172.21.15.37:0/189668315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff5fde0 0x55d9dff5d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.294+0000 7f4f65e00640 1 -- 172.21.15.37:0/189668315 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d9dfbce3f0 con 0x55d9dff568b0 2024-03-28T18:12:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.294+0000 7f4f66800640 1 --2- 172.21.15.37:0/189668315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff5fde0 0x55d9dff5d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:12:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.295+0000 7f4f65e00640 1 --2- 172.21.15.37:0/189668315 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff568b0 0x55d9dff5f700 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4f580ccaf0 tx=0x7f4f5809b450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45c5bcd05ac8524a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.295+0000 7f4f47e00640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:12:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.295+0000 7f4f47e00640 1 -- 172.21.15.37:0/189668315 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f580ccec0 con 0x55d9dff568b0 2024-03-28T18:12:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.295+0000 7f4f47e00640 1 -- 172.21.15.37:0/189668315 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4f5809cb50 con 0x55d9dff568b0 2024-03-28T18:12:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.295+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/189668315 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f60003dd0 con 0x55d9dff568b0 2024-03-28T18:12:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.296+0000 7f4f47e00640 1 -- 172.21.15.37:0/189668315 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f5809f970 con 0x55d9dff568b0 2024-03-28T18:12:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.296+0000 7f4f6fbfc740 10 client.25627 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.296+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/189668315 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d9dfc987d0 con 0x55d9dff568b0 2024-03-28T18:12:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.296+0000 7f4f47e00640 1 -- 172.21.15.37:0/189668315 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f60003dd0 con 0x55d9dff568b0 2024-03-28T18:12:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.296+0000 7f4f47e00640 1 -- 172.21.15.37:0/189668315 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f4f58097570 con 0x55d9dff568b0 2024-03-28T18:12:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.296+0000 7f4f6fbfc740 10 client.25627 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.296+0000 7f4f6fbfc740 10 client.25627 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-28T18:12:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.296+0000 7f4f6fbfc740 20 client.25627 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.296+0000 7f4f6fbfc740 1 --2- 172.21.15.37:0/189668315 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55d9dff70030 0x55d9dff72420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.296+0000 7f4f6fbfc740 4 client.25627 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:12:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.296+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/189668315 --> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d9dfec7800 con 0x55d9dff70030 2024-03-28T18:12:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.297+0000 7f4f65400640 1 --2- 172.21.15.37:0/189668315 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55d9dff70030 0x55d9dff72420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.297+0000 7f4f65400640 1 --2- 172.21.15.37:0/189668315 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55d9dff70030 0x55d9dff72420 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=14895d243a5ef0c5 server_cookie=f29c77b8b281cfd3 in_seq=0 out_seq=0 2024-03-28T18:12:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.297+0000 7f4f47e00640 10 client.25627 ms_handle_connect on v2:172.21.15.59:6838/2091629078 2024-03-28T18:12:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.298+0000 7f4f47e00640 1 -- 172.21.15.37:0/189668315 <== mds.1 v2:172.21.15.59:6838/2091629078 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d9dfec7800 con 0x55d9dff70030 2024-03-28T18:12:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.298+0000 7f4f47e00640 10 client.25627 handle_command_reply: tid=0 2024-03-28T18:12:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.298+0000 7f4f6fbfc740 1 client.25627 shutdown 2024-03-28T18:12:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.298+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/189668315 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff568b0 msgr2=0x55d9dff5f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.298+0000 7f4f6fbfc740 1 --2- 172.21.15.37:0/189668315 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff568b0 0x55d9dff5f700 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4f580ccaf0 tx=0x7f4f5809b450 comp rx=0 tx=0).stop 2024-03-28T18:12:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.298+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/189668315 shutdown_connections 2024-03-28T18:12:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.298+0000 7f4f6fbfc740 1 --2- 172.21.15.37:0/189668315 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55d9dff70030 0x55d9dff72420 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.298+0000 7f4f47e00640 0 client.25627 ms_handle_reset on v2:172.21.15.59:6838/2091629078 2024-03-28T18:12:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.298+0000 7f4f6fbfc740 1 --2- 172.21.15.37:0/189668315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff57260 0x55d9dff5cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.298+0000 7f4f6fbfc740 1 --2- 172.21.15.37:0/189668315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff5fde0 0x55d9dff5d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.298+0000 7f4f6fbfc740 1 --2- 172.21.15.37:0/189668315 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff568b0 0x55d9dff5f700 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.298+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/189668315 >> 172.21.15.37:0/189668315 conn(0x55d9dff50f80 msgr2=0x55d9dff52cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.298+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/189668315 shutdown_connections 2024-03-28T18:12:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.298+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/189668315 wait complete. 2024-03-28T18:12:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.299+0000 7f4f6fbfc740 20 client.25627 trim_cache size 0 max 16384 2024-03-28T18:12:04.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.312+0000 7f4f6fbfc740 1 Processor -- start 2024-03-28T18:12:04.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.312+0000 7f4f6fbfc740 1 -- start start 2024-03-28T18:12:04.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.313+0000 7f4f6fbfc740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff568b0 0x55d9dff5f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.313+0000 7f4f6fbfc740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff57260 0x55d9dff59d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.313+0000 7f4f6fbfc740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff5fde0 0x55d9dff5a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.313+0000 7f4f6fbfc740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f681b16f0 con 0x55d9dff57260 2024-03-28T18:12:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.313+0000 7f4f6fbfc740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f681af1f0 con 0x55d9dff568b0 2024-03-28T18:12:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.313+0000 7f4f6fbfc740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f681ad290 con 0x55d9dff5fde0 2024-03-28T18:12:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.313+0000 7f4f65e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff568b0 0x55d9dff5f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.313+0000 7f4f65e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff568b0 0x55d9dff5f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:46602/0 (socket says 172.21.15.37:46602) 2024-03-28T18:12:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.313+0000 7f4f65e00640 1 -- 172.21.15.37:0/3146742481 learned_addr learned my addr 172.21.15.37:0/3146742481 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.313+0000 7f4f66800640 1 --2- 172.21.15.37:0/3146742481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff5fde0 0x55d9dff5a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.313+0000 7f4f65400640 1 --2- 172.21.15.37:0/3146742481 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff57260 0x55d9dff59d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.313+0000 7f4f65e00640 1 -- 172.21.15.37:0/3146742481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff5fde0 msgr2=0x55d9dff5a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.313+0000 7f4f65e00640 1 --2- 172.21.15.37:0/3146742481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff5fde0 0x55d9dff5a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.313+0000 7f4f65e00640 1 -- 172.21.15.37:0/3146742481 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff57260 msgr2=0x55d9dff59d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.313+0000 7f4f65e00640 1 --2- 172.21.15.37:0/3146742481 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff57260 0x55d9dff59d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.313+0000 7f4f65e00640 1 -- 172.21.15.37:0/3146742481 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9dff72790 con 0x55d9dff568b0 2024-03-28T18:12:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.313+0000 7f4f65400640 1 --2- 172.21.15.37:0/3146742481 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff57260 0x55d9dff59d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:12:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.314+0000 7f4f65e00640 1 --2- 172.21.15.37:0/3146742481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff568b0 0x55d9dff5f1e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4f5800e9c0 tx=0x7f4f58009700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d10c6e68ca1b0c46 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.314+0000 7f4f52a00640 1 -- 172.21.15.37:0/3146742481 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5809bf20 con 0x55d9dff568b0 2024-03-28T18:12:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.314+0000 7f4f52a00640 1 -- 172.21.15.37:0/3146742481 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f580a3fc0 con 0x55d9dff568b0 2024-03-28T18:12:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.314+0000 7f4f52a00640 1 -- 172.21.15.37:0/3146742481 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f580ccf20 con 0x55d9dff568b0 2024-03-28T18:12:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.315+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/3146742481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff568b0 msgr2=0x55d9dff5f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.315+0000 7f4f6fbfc740 1 --2- 172.21.15.37:0/3146742481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff568b0 0x55d9dff5f1e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4f5800e9c0 tx=0x7f4f58009700 comp rx=0 tx=0).stop 2024-03-28T18:12:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.315+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/3146742481 shutdown_connections 2024-03-28T18:12:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.315+0000 7f4f6fbfc740 1 --2- 172.21.15.37:0/3146742481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff5fde0 0x55d9dff5a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.315+0000 7f4f6fbfc740 1 --2- 172.21.15.37:0/3146742481 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff57260 0x55d9dff59d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.315+0000 7f4f6fbfc740 1 --2- 172.21.15.37:0/3146742481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff568b0 0x55d9dff5f1e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.315+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/3146742481 >> 172.21.15.37:0/3146742481 conn(0x55d9dff50f80 msgr2=0x55d9dff50160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.315+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/3146742481 shutdown_connections 2024-03-28T18:12:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.315+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/3146742481 wait complete. 2024-03-28T18:12:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.315+0000 7f4f6fbfc740 1 -- start start 2024-03-28T18:12:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.316+0000 7f4f6fbfc740 1 Processor -- start 2024-03-28T18:12:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.316+0000 7f4f6fbfc740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff568b0 0x55d9dff67be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.316+0000 7f4f6fbfc740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff57260 0x55d9dff65f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.316+0000 7f4f6fbfc740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff5fde0 0x55d9dff664c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.316+0000 7f4f6fbfc740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f681ad890 con 0x55d9dff57260 2024-03-28T18:12:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.316+0000 7f4f6fbfc740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f681aeb90 con 0x55d9dff568b0 2024-03-28T18:12:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.316+0000 7f4f6fbfc740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f681aded0 con 0x55d9dff5fde0 2024-03-28T18:12:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.316+0000 7f4f6fbfc740 10 client.0 _ll_register_callbacks cb 0x7f4f58010e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.317+0000 7f4f65e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff568b0 0x55d9dff67be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.317+0000 7f4f65400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff57260 0x55d9dff65f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.317+0000 7f4f65e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff568b0 0x55d9dff67be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:46610/0 (socket says 172.21.15.37:46610) 2024-03-28T18:12:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.317+0000 7f4f65400640 1 --2- 172.21.15.37:0/2048672040 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff57260 0x55d9dff65f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57768/0 (socket says 172.21.15.37:57768) 2024-03-28T18:12:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.317+0000 7f4f65e00640 1 -- 172.21.15.37:0/2048672040 learned_addr learned my addr 172.21.15.37:0/2048672040 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.317+0000 7f4f66800640 1 --2- 172.21.15.37:0/2048672040 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff5fde0 0x55d9dff664c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.317+0000 7f4f65e00640 1 -- 172.21.15.37:0/2048672040 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff5fde0 msgr2=0x55d9dff664c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.317+0000 7f4f65e00640 1 --2- 172.21.15.37:0/2048672040 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff5fde0 0x55d9dff664c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.317+0000 7f4f65e00640 1 -- 172.21.15.37:0/2048672040 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff57260 msgr2=0x55d9dff65f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.317+0000 7f4f65e00640 1 --2- 172.21.15.37:0/2048672040 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff57260 0x55d9dff65f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.317+0000 7f4f65e00640 1 -- 172.21.15.37:0/2048672040 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d9dff66a00 con 0x55d9dff568b0 2024-03-28T18:12:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.317+0000 7f4f65e00640 1 --2- 172.21.15.37:0/2048672040 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff568b0 0x55d9dff67be0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4f5809b990 tx=0x7f4f5809ab90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7419dbf9cb0c439a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.317+0000 7f4f47e00640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:12:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.318+0000 7f4f47e00640 1 -- 172.21.15.37:0/2048672040 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f580ccec0 con 0x55d9dff568b0 2024-03-28T18:12:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.318+0000 7f4f47e00640 1 -- 172.21.15.37:0/2048672040 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4f5809cb50 con 0x55d9dff568b0 2024-03-28T18:12:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.318+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/2048672040 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f580a3fc0 con 0x55d9dff568b0 2024-03-28T18:12:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.318+0000 7f4f47e00640 1 -- 172.21.15.37:0/2048672040 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f5809ae80 con 0x55d9dff568b0 2024-03-28T18:12:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.318+0000 7f4f6fbfc740 10 client.25610 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.318+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/2048672040 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d9dff66a00 con 0x55d9dff568b0 2024-03-28T18:12:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.318+0000 7f4f47e00640 1 -- 172.21.15.37:0/2048672040 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f580a3fc0 con 0x55d9dff568b0 2024-03-28T18:12:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.318+0000 7f4f47e00640 1 -- 172.21.15.37:0/2048672040 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f4f58018a40 con 0x55d9dff568b0 2024-03-28T18:12:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.318+0000 7f4f6fbfc740 10 client.25610 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:04.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.318+0000 7f4f6fbfc740 10 client.25610 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-28T18:12:04.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.318+0000 7f4f6fbfc740 20 client.25610 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:04.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.318+0000 7f4f6fbfc740 1 --2- 172.21.15.37:0/2048672040 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55d9dffcc8a0 0x55d9dffcec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.318+0000 7f4f6fbfc740 4 client.25610 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:12:04.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.318+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/2048672040 --> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d9dfec1b50 con 0x55d9dffcc8a0 2024-03-28T18:12:04.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.319+0000 7f4f65400640 1 --2- 172.21.15.37:0/2048672040 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55d9dffcc8a0 0x55d9dffcec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.320+0000 7f4f47e00640 10 client.25610 ms_handle_connect on v2:172.21.15.59:6838/2091629078 2024-03-28T18:12:04.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.320+0000 7f4f65400640 1 --2- 172.21.15.37:0/2048672040 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55d9dffcc8a0 0x55d9dffcec90 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=43ff6cd05c853078 server_cookie=d738a527b2e6a2d1 in_seq=0 out_seq=0 2024-03-28T18:12:04.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.320+0000 7f4f47e00640 1 -- 172.21.15.37:0/2048672040 <== mds.1 v2:172.21.15.59:6838/2091629078 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d9dfec1b50 con 0x55d9dffcc8a0 2024-03-28T18:12:04.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.320+0000 7f4f47e00640 10 client.25610 handle_command_reply: tid=0 2024-03-28T18:12:04.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.320+0000 7f4f6fbfc740 1 client.25610 shutdown 2024-03-28T18:12:04.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.320+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/2048672040 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff568b0 msgr2=0x55d9dff67be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.320+0000 7f4f6fbfc740 1 --2- 172.21.15.37:0/2048672040 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff568b0 0x55d9dff67be0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4f5809b990 tx=0x7f4f5809ab90 comp rx=0 tx=0).stop 2024-03-28T18:12:04.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.320+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/2048672040 shutdown_connections 2024-03-28T18:12:04.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.320+0000 7f4f6fbfc740 1 --2- 172.21.15.37:0/2048672040 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55d9dffcc8a0 0x55d9dffcec90 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.320+0000 7f4f6fbfc740 1 --2- 172.21.15.37:0/2048672040 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d9dff5fde0 0x55d9dff664c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.320+0000 7f4f6fbfc740 1 --2- 172.21.15.37:0/2048672040 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55d9dff57260 0x55d9dff65f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.320+0000 7f4f47e00640 0 client.25610 ms_handle_reset on v2:172.21.15.59:6838/2091629078 2024-03-28T18:12:04.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.320+0000 7f4f6fbfc740 1 --2- 172.21.15.37:0/2048672040 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d9dff568b0 0x55d9dff67be0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.320+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/2048672040 >> 172.21.15.37:0/2048672040 conn(0x55d9dff50f80 msgr2=0x55d9dff52210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:04.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.320+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/2048672040 shutdown_connections 2024-03-28T18:12:04.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.320+0000 7f4f6fbfc740 1 -- 172.21.15.37:0/2048672040 wait complete. 2024-03-28T18:12:04.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.321+0000 7f4f6fbfc740 20 client.25610 trim_cache size 0 max 16384 2024-03-28T18:12:04.325 INFO:teuthology.orchestra.run.smithi037.stdout:[] 2024-03-28T18:12:04.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.321+0000 7f4f67200640 1 -- 172.21.15.37:0/3795263948 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f4f4c0a2a20 msgr2=0x7f4f4c0a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.321+0000 7f4f67200640 1 --2- 172.21.15.37:0/3795263948 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f4f4c0a2a20 0x7f4f4c0a4ee0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4f6001b040 tx=0x7f4f60005fd0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.321+0000 7f4f67200640 1 -- 172.21.15.37:0/3795263948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f68103720 msgr2=0x7f4f6819a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.321+0000 7f4f67200640 1 --2- 172.21.15.37:0/3795263948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f68103720 0x7f4f6819a070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4f5800e9f0 tx=0x7f4f5800edf0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.321+0000 7f4f67200640 1 -- 172.21.15.37:0/3795263948 shutdown_connections 2024-03-28T18:12:04.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.321+0000 7f4f67200640 1 --2- 172.21.15.37:0/3795263948 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f4f4c0a2a20 0x7f4f4c0a4ee0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.321+0000 7f4f67200640 1 --2- 172.21.15.37:0/3795263948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f68109730 0x7f4f6819e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.321+0000 7f4f67200640 1 --2- 172.21.15.37:0/3795263948 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f681040e0 0x7f4f6819a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.321+0000 7f4f67200640 1 --2- 172.21.15.37:0/3795263948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f68103720 0x7f4f6819a070 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.321+0000 7f4f67200640 1 -- 172.21.15.37:0/3795263948 >> 172.21.15.37:0/3795263948 conn(0x7f4f680ff1b0 msgr2=0x7f4f680ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:04.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.322+0000 7f4f67200640 1 -- 172.21.15.37:0/3795263948 shutdown_connections 2024-03-28T18:12:04.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.322+0000 7f4f67200640 1 -- 172.21.15.37:0/3795263948 wait complete. 2024-03-28T18:12:04.339 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:12:04.340 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:12:04.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.555+0000 7fad3e200640 1 -- 172.21.15.37:0/3415426369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad38100c60 msgr2=0x7fad3810d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.555+0000 7fad3e200640 1 --2- 172.21.15.37:0/3415426369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad38100c60 0x7fad3810d920 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fad24009900 tx=0x7fad2402bc30 comp rx=0 tx=0).stop 2024-03-28T18:12:04.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.556+0000 7fad3e200640 1 -- 172.21.15.37:0/3415426369 shutdown_connections 2024-03-28T18:12:04.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.556+0000 7fad3e200640 1 --2- 172.21.15.37:0/3415426369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad38100c60 0x7fad3810d920 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.556+0000 7fad3e200640 1 --2- 172.21.15.37:0/3415426369 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad3810de60 0x7fad38110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.556+0000 7fad3e200640 1 --2- 172.21.15.37:0/3415426369 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad381002b0 0x7fad38100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.556+0000 7fad3e200640 1 -- 172.21.15.37:0/3415426369 >> 172.21.15.37:0/3415426369 conn(0x7fad380fbef0 msgr2=0x7fad380fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:04.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.556+0000 7fad3e200640 1 -- 172.21.15.37:0/3415426369 shutdown_connections 2024-03-28T18:12:04.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.556+0000 7fad3e200640 1 -- 172.21.15.37:0/3415426369 wait complete. 2024-03-28T18:12:04.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.557+0000 7fad3e200640 1 Processor -- start 2024-03-28T18:12:04.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.557+0000 7fad3e200640 1 -- start start 2024-03-28T18:12:04.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.557+0000 7fad3e200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad381002b0 0x7fad3819a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.557+0000 7fad3e200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad38100c60 0x7fad3819a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.557+0000 7fad3e200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad3810de60 0x7fad3819e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.557+0000 7fad3e200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad38113b60 con 0x7fad38100c60 2024-03-28T18:12:04.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.557+0000 7fad3e200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad381139e0 con 0x7fad3810de60 2024-03-28T18:12:04.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.557+0000 7fad3e200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad38113ce0 con 0x7fad381002b0 2024-03-28T18:12:04.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.558+0000 7fad3ce00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad381002b0 0x7fad3819a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.558+0000 7fad3ce00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad381002b0 0x7fad3819a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:53686/0 (socket says 172.21.15.37:53686) 2024-03-28T18:12:04.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.558+0000 7fad3ce00640 1 -- 172.21.15.37:0/1963645260 learned_addr learned my addr 172.21.15.37:0/1963645260 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:04.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.558+0000 7fad37e00640 1 --2- 172.21.15.37:0/1963645260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad38100c60 0x7fad3819a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.558+0000 7fad3d800640 1 --2- 172.21.15.37:0/1963645260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad3810de60 0x7fad3819e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.558+0000 7fad3ce00640 1 -- 172.21.15.37:0/1963645260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad3810de60 msgr2=0x7fad3819e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.558+0000 7fad3ce00640 1 --2- 172.21.15.37:0/1963645260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad3810de60 0x7fad3819e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.558+0000 7fad3ce00640 1 -- 172.21.15.37:0/1963645260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad38100c60 msgr2=0x7fad3819a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.558+0000 7fad3ce00640 1 --2- 172.21.15.37:0/1963645260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad38100c60 0x7fad3819a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.558+0000 7fad3ce00640 1 -- 172.21.15.37:0/1963645260 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3819efc0 con 0x7fad381002b0 2024-03-28T18:12:04.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.559+0000 7fad3ce00640 1 --2- 172.21.15.37:0/1963645260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad381002b0 0x7fad3819a020 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fad2c00e9f0 tx=0x7fad2c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:04.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.559+0000 7fad35600640 1 -- 172.21.15.37:0/1963645260 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c00cd80 con 0x7fad381002b0 2024-03-28T18:12:04.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.559+0000 7fad35600640 1 -- 172.21.15.37:0/1963645260 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad2c010910 con 0x7fad381002b0 2024-03-28T18:12:04.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.559+0000 7fad3e200640 1 -- 172.21.15.37:0/1963645260 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3819f2b0 con 0x7fad381002b0 2024-03-28T18:12:04.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.559+0000 7fad35600640 1 -- 172.21.15.37:0/1963645260 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c018990 con 0x7fad381002b0 2024-03-28T18:12:04.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.559+0000 7fad3e200640 1 -- 172.21.15.37:0/1963645260 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad381a6b40 con 0x7fad381002b0 2024-03-28T18:12:04.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.562+0000 7fad35600640 1 -- 172.21.15.37:0/1963645260 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7fad2c010ac0 con 0x7fad381002b0 2024-03-28T18:12:04.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.563+0000 7fad42447740 1 Processor -- start 2024-03-28T18:12:04.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.563+0000 7fad42447740 1 -- start start 2024-03-28T18:12:04.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.564+0000 7fad42447740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5598421508b0 0x559842150c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.564+0000 7fad42447740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559842151260 0x559842159810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.564+0000 7fad35600640 1 --2- 172.21.15.37:0/1963645260 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fad200a2a90 0x7fad200a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.564+0000 7fad42447740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559842159de0 0x55984215c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.564+0000 7fad42447740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598420b42c0 con 0x5598421508b0 2024-03-28T18:12:04.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.564+0000 7fad42447740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55984205cc90 con 0x559842159de0 2024-03-28T18:12:04.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.564+0000 7fad42447740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598420b0360 con 0x559842151260 2024-03-28T18:12:04.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.564+0000 7fad35600640 1 -- 172.21.15.37:0/1963645260 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fad2c0cf490 con 0x7fad381002b0 2024-03-28T18:12:04.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.564+0000 7fad3ce00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559842151260 0x559842159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.564+0000 7fad3d800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5598421508b0 0x559842150c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.564+0000 7fad3ce00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559842151260 0x559842159810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:53688/0 (socket says 172.21.15.37:53688) 2024-03-28T18:12:04.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.564+0000 7fad3ce00640 1 -- 172.21.15.37:0/2840049182 learned_addr learned my addr 172.21.15.37:0/2840049182 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:04.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.564+0000 7fad3d800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5598421508b0 0x559842150c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57800/0 (socket says 172.21.15.37:57800) 2024-03-28T18:12:04.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.564+0000 7fad3d800640 1 --2- 172.21.15.37:0/2840049182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559842159de0 0x55984215c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.565+0000 7fad3ce00640 1 -- 172.21.15.37:0/2840049182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559842159de0 msgr2=0x55984215c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.565+0000 7fad3ce00640 1 --2- 172.21.15.37:0/2840049182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559842159de0 0x55984215c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.565+0000 7fad3ce00640 1 -- 172.21.15.37:0/2840049182 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5598421508b0 msgr2=0x559842150c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.565+0000 7fad3ce00640 1 --2- 172.21.15.37:0/2840049182 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5598421508b0 0x559842150c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.565+0000 7fad3ce00640 1 -- 172.21.15.37:0/2840049182 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559842081080 con 0x559842151260 2024-03-28T18:12:04.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.565+0000 7fad3d800640 1 --2- 172.21.15.37:0/2840049182 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5598421508b0 0x559842150c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:12:04.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.565+0000 7fad37e00640 1 --2- 172.21.15.37:0/1963645260 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fad200a2a90 0x7fad200a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.565+0000 7fad3ce00640 1 --2- 172.21.15.37:0/2840049182 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559842151260 0x559842159810 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fad2c09b620 tx=0x7fad2c09b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67e3047cb2ecb44b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:04.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.565+0000 7fad2a000640 1 -- 172.21.15.37:0/2840049182 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c084e60 con 0x559842151260 2024-03-28T18:12:04.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.565+0000 7fad2a000640 1 -- 172.21.15.37:0/2840049182 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad2c097f90 con 0x559842151260 2024-03-28T18:12:04.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.565+0000 7fad2a000640 1 -- 172.21.15.37:0/2840049182 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c018b30 con 0x559842151260 2024-03-28T18:12:04.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.566+0000 7fad42447740 1 -- 172.21.15.37:0/2840049182 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559842151260 msgr2=0x559842159810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.566+0000 7fad37e00640 1 --2- 172.21.15.37:0/1963645260 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fad200a2a90 0x7fad200a4f50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fad24000c00 tx=0x7fad2402df90 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:04.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.566+0000 7fad42447740 1 --2- 172.21.15.37:0/2840049182 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559842151260 0x559842159810 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fad2c09b620 tx=0x7fad2c09b920 comp rx=0 tx=0).stop 2024-03-28T18:12:04.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.566+0000 7fad42447740 1 -- 172.21.15.37:0/2840049182 shutdown_connections 2024-03-28T18:12:04.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.566+0000 7fad42447740 1 --2- 172.21.15.37:0/2840049182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559842159de0 0x55984215c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.566+0000 7fad42447740 1 --2- 172.21.15.37:0/2840049182 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559842151260 0x559842159810 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.566+0000 7fad42447740 1 --2- 172.21.15.37:0/2840049182 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5598421508b0 0x559842150c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.566+0000 7fad42447740 1 -- 172.21.15.37:0/2840049182 >> 172.21.15.37:0/2840049182 conn(0x55984214af80 msgr2=0x55984214b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:04.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.566+0000 7fad42447740 1 -- 172.21.15.37:0/2840049182 shutdown_connections 2024-03-28T18:12:04.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.566+0000 7fad42447740 1 -- 172.21.15.37:0/2840049182 wait complete. 2024-03-28T18:12:04.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.567+0000 7fad42447740 1 -- start start 2024-03-28T18:12:04.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.567+0000 7fad42447740 1 Processor -- start 2024-03-28T18:12:04.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.567+0000 7fad42447740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5598421508b0 0x559842159720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.567+0000 7fad42447740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x559842151260 0x559842156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.567+0000 7fad42447740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559842159de0 0x5598421570c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.567+0000 7fad42447740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55984215cd50 con 0x559842151260 2024-03-28T18:12:04.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.567+0000 7fad42447740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598420700d0 con 0x559842159de0 2024-03-28T18:12:04.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.567+0000 7fad42447740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55984215ced0 con 0x5598421508b0 2024-03-28T18:12:04.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.567+0000 7fad42447740 10 client.0 _ll_register_callbacks cb 0x559842074e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:04.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.568+0000 7fad3ce00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5598421508b0 0x559842159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.568+0000 7fad3ce00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5598421508b0 0x559842159720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:53704/0 (socket says 172.21.15.37:53704) 2024-03-28T18:12:04.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.568+0000 7fad3ce00640 1 -- 172.21.15.37:0/4038924387 learned_addr learned my addr 172.21.15.37:0/4038924387 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:04.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.568+0000 7fad3d800640 1 --2- 172.21.15.37:0/4038924387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559842159de0 0x5598421570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.568+0000 7fad37e00640 1 --2- 172.21.15.37:0/4038924387 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x559842151260 0x559842156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.568+0000 7fad3ce00640 1 -- 172.21.15.37:0/4038924387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559842159de0 msgr2=0x5598421570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.568+0000 7fad3ce00640 1 --2- 172.21.15.37:0/4038924387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559842159de0 0x5598421570c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.568+0000 7fad3ce00640 1 -- 172.21.15.37:0/4038924387 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x559842151260 msgr2=0x559842156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.568+0000 7fad3ce00640 1 --2- 172.21.15.37:0/4038924387 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x559842151260 0x559842156ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.568+0000 7fad3ce00640 1 -- 172.21.15.37:0/4038924387 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559841dc83f0 con 0x5598421508b0 2024-03-28T18:12:04.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.568+0000 7fad3d800640 1 --2- 172.21.15.37:0/4038924387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559842159de0 0x5598421570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:04.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.568+0000 7fad3ce00640 1 --2- 172.21.15.37:0/4038924387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5598421508b0 0x559842159720 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fad2c09b8b0 tx=0x7fad2c009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c488584394976818 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:04.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.568+0000 7fad1ea00640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:12:04.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.569+0000 7fad1ea00640 1 -- 172.21.15.37:0/4038924387 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c0b4070 con 0x5598421508b0 2024-03-28T18:12:04.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.569+0000 7fad1ea00640 1 -- 172.21.15.37:0/4038924387 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fad2c0862c0 con 0x5598421508b0 2024-03-28T18:12:04.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.569+0000 7fad42447740 1 -- 172.21.15.37:0/4038924387 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fad2c097f90 con 0x5598421508b0 2024-03-28T18:12:04.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.569+0000 7fad1ea00640 1 -- 172.21.15.37:0/4038924387 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad2c084e60 con 0x5598421508b0 2024-03-28T18:12:04.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.569+0000 7fad42447740 10 client.25645 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:04.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.569+0000 7fad42447740 1 -- 172.21.15.37:0/4038924387 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559841e927d0 con 0x5598421508b0 2024-03-28T18:12:04.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.569+0000 7fad1ea00640 1 -- 172.21.15.37:0/4038924387 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad2c097f90 con 0x5598421508b0 2024-03-28T18:12:04.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.569+0000 7fad1ea00640 1 -- 172.21.15.37:0/4038924387 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fad2c066020 con 0x5598421508b0 2024-03-28T18:12:04.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.569+0000 7fad42447740 10 client.25645 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:04.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.569+0000 7fad42447740 10 client.25645 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:12:04.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.569+0000 7fad42447740 20 client.25645 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:04.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.569+0000 7fad42447740 1 --2- 172.21.15.37:0/4038924387 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x559842169b10 0x55984216bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.569+0000 7fad42447740 4 client.25645 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:12:04.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.569+0000 7fad42447740 1 -- 172.21.15.37:0/4038924387 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5598420c1800 con 0x559842169b10 2024-03-28T18:12:04.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.570+0000 7fad37e00640 1 --2- 172.21.15.37:0/4038924387 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x559842169b10 0x55984216bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.571+0000 7fad1ea00640 10 client.25645 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:04.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.571+0000 7fad37e00640 1 --2- 172.21.15.37:0/4038924387 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x559842169b10 0x55984216bf00 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58b252fd6405175b server_cookie=ab473805f444009d in_seq=0 out_seq=0 2024-03-28T18:12:04.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.571+0000 7fad1ea00640 1 -- 172.21.15.37:0/4038924387 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5598420c1800 con 0x559842169b10 2024-03-28T18:12:04.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.571+0000 7fad1ea00640 10 client.25645 handle_command_reply: tid=0 2024-03-28T18:12:04.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.571+0000 7fad42447740 1 client.25645 shutdown 2024-03-28T18:12:04.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.572+0000 7fad42447740 1 -- 172.21.15.37:0/4038924387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5598421508b0 msgr2=0x559842159720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.572+0000 7fad42447740 1 --2- 172.21.15.37:0/4038924387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5598421508b0 0x559842159720 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fad2c09b8b0 tx=0x7fad2c009700 comp rx=0 tx=0).stop 2024-03-28T18:12:04.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.572+0000 7fad42447740 1 -- 172.21.15.37:0/4038924387 shutdown_connections 2024-03-28T18:12:04.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.572+0000 7fad42447740 1 --2- 172.21.15.37:0/4038924387 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x559842169b10 0x55984216bf00 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.572+0000 7fad42447740 1 --2- 172.21.15.37:0/4038924387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559842159de0 0x5598421570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.572+0000 7fad42447740 1 --2- 172.21.15.37:0/4038924387 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x559842151260 0x559842156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.572+0000 7fad42447740 1 --2- 172.21.15.37:0/4038924387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5598421508b0 0x559842159720 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.572+0000 7fad1ea00640 0 client.25645 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:04.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.572+0000 7fad42447740 1 -- 172.21.15.37:0/4038924387 >> 172.21.15.37:0/4038924387 conn(0x55984214af80 msgr2=0x55984214ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:04.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.572+0000 7fad42447740 1 -- 172.21.15.37:0/4038924387 shutdown_connections 2024-03-28T18:12:04.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.572+0000 7fad42447740 1 -- 172.21.15.37:0/4038924387 wait complete. 2024-03-28T18:12:04.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.572+0000 7fad42447740 20 client.25645 trim_cache size 0 max 16384 2024-03-28T18:12:04.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.585+0000 7fad42447740 1 Processor -- start 2024-03-28T18:12:04.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.586+0000 7fad42447740 1 -- start start 2024-03-28T18:12:04.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.586+0000 7fad42447740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5598421508b0 0x559842156390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.586+0000 7fad42447740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559842151260 0x5598421591e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.586+0000 7fad42447740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559842159de0 0x559842159720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.586+0000 7fad42447740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad381b1770 con 0x5598421508b0 2024-03-28T18:12:04.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.586+0000 7fad42447740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad381af270 con 0x559842151260 2024-03-28T18:12:04.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.586+0000 7fad42447740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad381ad310 con 0x559842159de0 2024-03-28T18:12:04.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.586+0000 7fad37e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559842151260 0x5598421591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.586+0000 7fad3ce00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5598421508b0 0x559842156390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.586+0000 7fad37e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559842151260 0x5598421591e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:46670/0 (socket says 172.21.15.37:46670) 2024-03-28T18:12:04.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.586+0000 7fad37e00640 1 -- 172.21.15.37:0/1390624702 learned_addr learned my addr 172.21.15.37:0/1390624702 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:04.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.586+0000 7fad3d800640 1 --2- 172.21.15.37:0/1390624702 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559842159de0 0x559842159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.587+0000 7fad37e00640 1 -- 172.21.15.37:0/1390624702 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559842159de0 msgr2=0x559842159720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.587+0000 7fad37e00640 1 --2- 172.21.15.37:0/1390624702 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559842159de0 0x559842159720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.587+0000 7fad37e00640 1 -- 172.21.15.37:0/1390624702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5598421508b0 msgr2=0x559842156390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.587+0000 7fad37e00640 1 --2- 172.21.15.37:0/1390624702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5598421508b0 0x559842156390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.587+0000 7fad37e00640 1 -- 172.21.15.37:0/1390624702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559842157430 con 0x559842151260 2024-03-28T18:12:04.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.587+0000 7fad37e00640 1 --2- 172.21.15.37:0/1390624702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559842151260 0x5598421591e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fad24009230 tx=0x7fad24043000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=959d5ab544e97dfa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:04.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.587+0000 7fad29600640 1 -- 172.21.15.37:0/1390624702 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2402c880 con 0x559842151260 2024-03-28T18:12:04.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.587+0000 7fad29600640 1 -- 172.21.15.37:0/1390624702 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad24053da0 con 0x559842151260 2024-03-28T18:12:04.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.587+0000 7fad29600640 1 -- 172.21.15.37:0/1390624702 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad240047d0 con 0x559842151260 2024-03-28T18:12:04.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.588+0000 7fad42447740 1 -- 172.21.15.37:0/1390624702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559842151260 msgr2=0x5598421591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.588+0000 7fad42447740 1 --2- 172.21.15.37:0/1390624702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559842151260 0x5598421591e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fad24009230 tx=0x7fad24043000 comp rx=0 tx=0).stop 2024-03-28T18:12:04.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.588+0000 7fad42447740 1 -- 172.21.15.37:0/1390624702 shutdown_connections 2024-03-28T18:12:04.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.588+0000 7fad42447740 1 --2- 172.21.15.37:0/1390624702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559842151260 0x5598421591e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.588+0000 7fad42447740 1 --2- 172.21.15.37:0/1390624702 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559842159de0 0x559842159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.588+0000 7fad42447740 1 --2- 172.21.15.37:0/1390624702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5598421508b0 0x559842156390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.588+0000 7fad42447740 1 -- 172.21.15.37:0/1390624702 >> 172.21.15.37:0/1390624702 conn(0x55984214af80 msgr2=0x55984214a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:04.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.588+0000 7fad42447740 1 -- 172.21.15.37:0/1390624702 shutdown_connections 2024-03-28T18:12:04.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.588+0000 7fad42447740 1 -- 172.21.15.37:0/1390624702 wait complete. 2024-03-28T18:12:04.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.588+0000 7fad42447740 1 -- start start 2024-03-28T18:12:04.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.589+0000 7fad42447740 1 Processor -- start 2024-03-28T18:12:04.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.589+0000 7fad42447740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5598421508b0 0x55984216c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.589+0000 7fad42447740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559842151260 0x55984216cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.589+0000 7fad42447740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x559842159de0 0x5598421c4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.589+0000 7fad42447740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad381ad910 con 0x559842159de0 2024-03-28T18:12:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.589+0000 7fad42447740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad381aec10 con 0x5598421508b0 2024-03-28T18:12:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.589+0000 7fad42447740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad381adf50 con 0x559842151260 2024-03-28T18:12:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.589+0000 7fad42447740 10 client.0 _ll_register_callbacks cb 0x7fad30009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.589+0000 7fad3ce00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5598421508b0 0x55984216c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.589+0000 7fad37e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559842151260 0x55984216cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.589+0000 7fad3ce00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5598421508b0 0x55984216c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:46682/0 (socket says 172.21.15.37:46682) 2024-03-28T18:12:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.589+0000 7fad3ce00640 1 -- 172.21.15.37:0/2216826780 learned_addr learned my addr 172.21.15.37:0/2216826780 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.590+0000 7fad3d800640 1 --2- 172.21.15.37:0/2216826780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x559842159de0 0x5598421c4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.590+0000 7fad3ce00640 1 -- 172.21.15.37:0/2216826780 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559842151260 msgr2=0x55984216cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.590+0000 7fad3ce00640 1 --2- 172.21.15.37:0/2216826780 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559842151260 0x55984216cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.590+0000 7fad3ce00640 1 -- 172.21.15.37:0/2216826780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x559842159de0 msgr2=0x5598421c4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:04.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.590+0000 7fad3ce00640 1 --2- 172.21.15.37:0/2216826780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x559842159de0 0x5598421c4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.590+0000 7fad3ce00640 1 -- 172.21.15.37:0/2216826780 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55984216d420 con 0x5598421508b0 2024-03-28T18:12:04.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.590+0000 7fad3d800640 1 --2- 172.21.15.37:0/2216826780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x559842159de0 0x5598421c4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:12:04.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.590+0000 7fad37e00640 1 --2- 172.21.15.37:0/2216826780 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559842151260 0x55984216cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:04.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.590+0000 7fad3ce00640 1 --2- 172.21.15.37:0/2216826780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5598421508b0 0x55984216c650 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fad2c09ae80 tx=0x7fad2c010870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a536e92edb34e999 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:04.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.590+0000 7fad1ea00640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:12:04.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.590+0000 7fad1ea00640 1 -- 172.21.15.37:0/2216826780 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c0b4070 con 0x5598421508b0 2024-03-28T18:12:04.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.590+0000 7fad1ea00640 1 -- 172.21.15.37:0/2216826780 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fad2c0862c0 con 0x5598421508b0 2024-03-28T18:12:04.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.590+0000 7fad42447740 1 -- 172.21.15.37:0/2216826780 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fad24053da0 con 0x5598421508b0 2024-03-28T18:12:04.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.590+0000 7fad1ea00640 1 -- 172.21.15.37:0/2216826780 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad2c0cc970 con 0x5598421508b0 2024-03-28T18:12:04.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.591+0000 7fad42447740 10 client.25628 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:04.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.591+0000 7fad42447740 1 -- 172.21.15.37:0/2216826780 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5598421c5650 con 0x5598421508b0 2024-03-28T18:12:04.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.591+0000 7fad1ea00640 1 -- 172.21.15.37:0/2216826780 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad24053da0 con 0x5598421508b0 2024-03-28T18:12:04.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.591+0000 7fad1ea00640 1 -- 172.21.15.37:0/2216826780 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fad2c066070 con 0x5598421508b0 2024-03-28T18:12:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.591+0000 7fad42447740 10 client.25628 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.591+0000 7fad42447740 10 client.25628 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:12:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.591+0000 7fad42447740 20 client.25628 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.591+0000 7fad42447740 1 --2- 172.21.15.37:0/2216826780 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x5598421c59e0 0x5598421c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.591+0000 7fad42447740 4 client.25628 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:12:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.591+0000 7fad42447740 1 -- 172.21.15.37:0/2216826780 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5598420bbb50 con 0x5598421c59e0 2024-03-28T18:12:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.591+0000 7fad37e00640 1 --2- 172.21.15.37:0/2216826780 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x5598421c59e0 0x5598421c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.592+0000 7fad37e00640 1 --2- 172.21.15.37:0/2216826780 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x5598421c59e0 0x5598421c7e00 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffd4d5225c9a1c07 server_cookie=d9966997dd86bd9e in_seq=0 out_seq=0 2024-03-28T18:12:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.592+0000 7fad1ea00640 10 client.25628 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.592+0000 7fad1ea00640 1 -- 172.21.15.37:0/2216826780 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5598420bbb50 con 0x5598421c59e0 2024-03-28T18:12:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.592+0000 7fad1ea00640 10 client.25628 handle_command_reply: tid=0 2024-03-28T18:12:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.592+0000 7fad42447740 1 client.25628 shutdown 2024-03-28T18:12:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.592+0000 7fad42447740 1 -- 172.21.15.37:0/2216826780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5598421508b0 msgr2=0x55984216c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.592+0000 7fad42447740 1 --2- 172.21.15.37:0/2216826780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5598421508b0 0x55984216c650 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fad2c09ae80 tx=0x7fad2c010870 comp rx=0 tx=0).stop 2024-03-28T18:12:04.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.592+0000 7fad42447740 1 -- 172.21.15.37:0/2216826780 shutdown_connections 2024-03-28T18:12:04.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.592+0000 7fad42447740 1 --2- 172.21.15.37:0/2216826780 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x5598421c59e0 0x5598421c7e00 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.592+0000 7fad42447740 1 --2- 172.21.15.37:0/2216826780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x559842159de0 0x5598421c4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.592+0000 7fad42447740 1 --2- 172.21.15.37:0/2216826780 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559842151260 0x55984216cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.592+0000 7fad1ea00640 0 client.25628 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:04.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.592+0000 7fad42447740 1 --2- 172.21.15.37:0/2216826780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5598421508b0 0x55984216c650 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.593+0000 7fad42447740 1 -- 172.21.15.37:0/2216826780 >> 172.21.15.37:0/2216826780 conn(0x55984214af80 msgr2=0x559842151640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:04.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.593+0000 7fad42447740 1 -- 172.21.15.37:0/2216826780 shutdown_connections 2024-03-28T18:12:04.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.593+0000 7fad42447740 1 -- 172.21.15.37:0/2216826780 wait complete. 2024-03-28T18:12:04.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.593+0000 7fad42447740 20 client.25628 trim_cache size 0 max 16384 2024-03-28T18:12:04.597 INFO:teuthology.orchestra.run.smithi037.stdout:[] 2024-03-28T18:12:04.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.594+0000 7fad3e200640 1 -- 172.21.15.37:0/1963645260 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fad200a2a90 msgr2=0x7fad200a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.594+0000 7fad3e200640 1 --2- 172.21.15.37:0/1963645260 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fad200a2a90 0x7fad200a4f50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fad24000c00 tx=0x7fad2402df90 comp rx=0 tx=0).stop 2024-03-28T18:12:04.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.594+0000 7fad3e200640 1 -- 172.21.15.37:0/1963645260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad381002b0 msgr2=0x7fad3819a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:04.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.594+0000 7fad3e200640 1 --2- 172.21.15.37:0/1963645260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad381002b0 0x7fad3819a020 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fad2c00e9f0 tx=0x7fad2c00eeb0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.594+0000 7fad3e200640 1 -- 172.21.15.37:0/1963645260 shutdown_connections 2024-03-28T18:12:04.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.594+0000 7fad3e200640 1 --2- 172.21.15.37:0/1963645260 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fad200a2a90 0x7fad200a4f50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.594+0000 7fad3e200640 1 --2- 172.21.15.37:0/1963645260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad3810de60 0x7fad3819e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.594+0000 7fad3e200640 1 --2- 172.21.15.37:0/1963645260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad38100c60 0x7fad3819a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.594+0000 7fad3e200640 1 --2- 172.21.15.37:0/1963645260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad381002b0 0x7fad3819a020 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:04.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.594+0000 7fad3e200640 1 -- 172.21.15.37:0/1963645260 >> 172.21.15.37:0/1963645260 conn(0x7fad380fbef0 msgr2=0x7fad3810f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:04.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.594+0000 7fad3e200640 1 -- 172.21.15.37:0/1963645260 shutdown_connections 2024-03-28T18:12:04.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:04.594+0000 7fad3e200640 1 -- 172.21.15.37:0/1963645260 wait complete. 2024-03-28T18:12:04.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:04 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1957322356' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:12:05.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:04 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1957322356' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:12:05.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:04 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1957322356' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:12:05.611 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:12:05.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:12:05.612 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:12:05.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.857+0000 7fe4ba800640 1 -- 172.21.15.37:0/4081924984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4b4109730 msgr2=0x7fe4b4109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:05.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.857+0000 7fe4ba800640 1 --2- 172.21.15.37:0/4081924984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4b4109730 0x7fe4b4109b10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8009930 tx=0x7fe4a802bc20 comp rx=0 tx=0).stop 2024-03-28T18:12:05.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.857+0000 7fe4ba800640 1 -- 172.21.15.37:0/4081924984 shutdown_connections 2024-03-28T18:12:05.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.857+0000 7fe4ba800640 1 --2- 172.21.15.37:0/4081924984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4b4103720 0x7fe4b4103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.857+0000 7fe4ba800640 1 --2- 172.21.15.37:0/4081924984 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4b41040e0 0x7fe4b41105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.857+0000 7fe4ba800640 1 --2- 172.21.15.37:0/4081924984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4b4109730 0x7fe4b4109b10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.857+0000 7fe4ba800640 1 -- 172.21.15.37:0/4081924984 >> 172.21.15.37:0/4081924984 conn(0x7fe4b40ff1b0 msgr2=0x7fe4b41015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:05.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.857+0000 7fe4ba800640 1 -- 172.21.15.37:0/4081924984 shutdown_connections 2024-03-28T18:12:05.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.857+0000 7fe4ba800640 1 -- 172.21.15.37:0/4081924984 wait complete. 2024-03-28T18:12:05.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.858+0000 7fe4ba800640 1 Processor -- start 2024-03-28T18:12:05.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.858+0000 7fe4ba800640 1 -- start start 2024-03-28T18:12:05.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.859+0000 7fe4ba800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4b4103720 0x7fe4b4199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:05.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.859+0000 7fe4ba800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4b41040e0 0x7fe4b419a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:05.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.859+0000 7fe4ba800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4b4109730 0x7fe4b419e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:05.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.859+0000 7fe4ba800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b4113b90 con 0x7fe4b4109730 2024-03-28T18:12:05.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.859+0000 7fe4ba800640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b4113a10 con 0x7fe4b41040e0 2024-03-28T18:12:05.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.859+0000 7fe4ba800640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b4113d10 con 0x7fe4b4103720 2024-03-28T18:12:05.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.859+0000 7fe4b8a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4b41040e0 0x7fe4b419a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:05.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.859+0000 7fe4b9400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4b4103720 0x7fe4b4199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:05.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.859+0000 7fe4b8a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4b41040e0 0x7fe4b419a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:46698/0 (socket says 172.21.15.37:46698) 2024-03-28T18:12:05.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.859+0000 7fe4b8a00640 1 -- 172.21.15.37:0/2957505934 learned_addr learned my addr 172.21.15.37:0/2957505934 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:05.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.859+0000 7fe4b9e00640 1 --2- 172.21.15.37:0/2957505934 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4b4109730 0x7fe4b419e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:05.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.860+0000 7fe4b9400640 1 -- 172.21.15.37:0/2957505934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4b41040e0 msgr2=0x7fe4b419a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:05.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.860+0000 7fe4b9400640 1 --2- 172.21.15.37:0/2957505934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4b41040e0 0x7fe4b419a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.860+0000 7fe4b9400640 1 -- 172.21.15.37:0/2957505934 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4b4109730 msgr2=0x7fe4b419e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:05.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.860+0000 7fe4b9400640 1 --2- 172.21.15.37:0/2957505934 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4b4109730 0x7fe4b419e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.860+0000 7fe4b9400640 1 -- 172.21.15.37:0/2957505934 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b419efe0 con 0x7fe4b4103720 2024-03-28T18:12:05.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.860+0000 7fe4b9e00640 1 --2- 172.21.15.37:0/2957505934 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4b4109730 0x7fe4b419e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:12:05.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.860+0000 7fe4b8a00640 1 --2- 172.21.15.37:0/2957505934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4b41040e0 0x7fe4b419a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:05.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.860+0000 7fe4b9400640 1 --2- 172.21.15.37:0/2957505934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4b4103720 0x7fe4b4199ff0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8036040 tx=0x7fe4a8004340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:05.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.860+0000 7fe4b2000640 1 -- 172.21.15.37:0/2957505934 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a8043020 con 0x7fe4b4103720 2024-03-28T18:12:05.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.860+0000 7fe4ba800640 1 -- 172.21.15.37:0/2957505934 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b419f270 con 0x7fe4b4103720 2024-03-28T18:12:05.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.860+0000 7fe4b2000640 1 -- 172.21.15.37:0/2957505934 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4a80054e0 con 0x7fe4b4103720 2024-03-28T18:12:05.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.860+0000 7fe4b2000640 1 -- 172.21.15.37:0/2957505934 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a8003ab0 con 0x7fe4b4103720 2024-03-28T18:12:05.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.860+0000 7fe4ba800640 1 -- 172.21.15.37:0/2957505934 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b41a6b60 con 0x7fe4b4103720 2024-03-28T18:12:05.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.863+0000 7fe4b2000640 1 -- 172.21.15.37:0/2957505934 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7fe4a8046020 con 0x7fe4b4103720 2024-03-28T18:12:05.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.864+0000 7fe4beb86740 1 Processor -- start 2024-03-28T18:12:05.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.864+0000 7fe4beb86740 1 -- start start 2024-03-28T18:12:05.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.864+0000 7fe4beb86740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35387290 0x55cc35387670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:05.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.864+0000 7fe4beb86740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35387bb0 0x55cc35390250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:05.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.864+0000 7fe4beb86740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35390790 0x55cc35392b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:05.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.864+0000 7fe4beb86740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc352ebeb0 con 0x55cc35390790 2024-03-28T18:12:05.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.864+0000 7fe4beb86740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc35294a60 con 0x55cc35387290 2024-03-28T18:12:05.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.864+0000 7fe4beb86740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc352e7f50 con 0x55cc35387bb0 2024-03-28T18:12:05.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.865+0000 7fe4b9e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35387290 0x55cc35387670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:05.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.865+0000 7fe4b9e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35387290 0x55cc35387670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:46708/0 (socket says 172.21.15.37:46708) 2024-03-28T18:12:05.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.865+0000 7fe4b9e00640 1 -- 172.21.15.37:0/900146875 learned_addr learned my addr 172.21.15.37:0/900146875 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:05.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.865+0000 7fe4b9400640 1 --2- 172.21.15.37:0/900146875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35387bb0 0x55cc35390250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:05.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.865+0000 7fe4b8a00640 1 --2- 172.21.15.37:0/900146875 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35390790 0x55cc35392b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:05.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.865+0000 7fe4b9e00640 1 -- 172.21.15.37:0/900146875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35387bb0 msgr2=0x55cc35390250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:05.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.865+0000 7fe4b9e00640 1 --2- 172.21.15.37:0/900146875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35387bb0 0x55cc35390250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.865+0000 7fe4b9e00640 1 -- 172.21.15.37:0/900146875 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35390790 msgr2=0x55cc35392b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:05.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.865+0000 7fe4b9e00640 1 --2- 172.21.15.37:0/900146875 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35390790 0x55cc35392b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.865+0000 7fe4b9e00640 1 -- 172.21.15.37:0/900146875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc350c87e0 con 0x55cc35387290 2024-03-28T18:12:05.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.865+0000 7fe4b9400640 1 --2- 172.21.15.37:0/900146875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35387bb0 0x55cc35390250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:12:05.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.865+0000 7fe4b2000640 1 --2- 172.21.15.37:0/2957505934 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fe49c0a2a90 0x7fe49c0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:05.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.865+0000 7fe4b9e00640 1 --2- 172.21.15.37:0/900146875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35387290 0x55cc35387670 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac007e60 tx=0x7fe4ac00e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29060fe1f5e1eab5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:05.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.866+0000 7fe4b2000640 1 -- 172.21.15.37:0/2957505934 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe4a80f0fc0 con 0x7fe4b4103720 2024-03-28T18:12:05.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.866+0000 7fe4b9400640 1 --2- 172.21.15.37:0/2957505934 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fe49c0a2a90 0x7fe49c0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:05.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.866+0000 7fe4a6600640 1 -- 172.21.15.37:0/900146875 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac014070 con 0x55cc35387290 2024-03-28T18:12:05.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.866+0000 7fe4a6600640 1 -- 172.21.15.37:0/900146875 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4ac003e50 con 0x55cc35387290 2024-03-28T18:12:05.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.866+0000 7fe4a6600640 1 -- 172.21.15.37:0/900146875 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac004810 con 0x55cc35387290 2024-03-28T18:12:05.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.867+0000 7fe4beb86740 1 -- 172.21.15.37:0/900146875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35387290 msgr2=0x55cc35387670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:05.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.867+0000 7fe4b9400640 1 --2- 172.21.15.37:0/2957505934 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fe49c0a2a90 0x7fe49c0a4f50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe4a80ee6d0 tx=0x7fe4a8000bc0 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:05.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.867+0000 7fe4beb86740 1 --2- 172.21.15.37:0/900146875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35387290 0x55cc35387670 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac007e60 tx=0x7fe4ac00e720 comp rx=0 tx=0).stop 2024-03-28T18:12:05.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.867+0000 7fe4beb86740 1 -- 172.21.15.37:0/900146875 shutdown_connections 2024-03-28T18:12:05.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.867+0000 7fe4beb86740 1 --2- 172.21.15.37:0/900146875 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35390790 0x55cc35392b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.867+0000 7fe4beb86740 1 --2- 172.21.15.37:0/900146875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35387bb0 0x55cc35390250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.867+0000 7fe4beb86740 1 --2- 172.21.15.37:0/900146875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35387290 0x55cc35387670 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.867+0000 7fe4beb86740 1 -- 172.21.15.37:0/900146875 >> 172.21.15.37:0/900146875 conn(0x55cc353819c0 msgr2=0x55cc35381da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:05.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.867+0000 7fe4beb86740 1 -- 172.21.15.37:0/900146875 shutdown_connections 2024-03-28T18:12:05.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.867+0000 7fe4beb86740 1 -- 172.21.15.37:0/900146875 wait complete. 2024-03-28T18:12:05.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.867+0000 7fe4beb86740 1 -- start start 2024-03-28T18:12:05.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.868+0000 7fe4beb86740 1 Processor -- start 2024-03-28T18:12:05.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.868+0000 7fe4beb86740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35387290 0x55cc35390180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:05.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.868+0000 7fe4beb86740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35387bb0 0x55cc3538d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:05.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.868+0000 7fe4beb86740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35390790 0x55cc3538db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:05.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.868+0000 7fe4beb86740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc35393580 con 0x55cc35387290 2024-03-28T18:12:05.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.868+0000 7fe4beb86740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc352a7eb0 con 0x55cc35390790 2024-03-28T18:12:05.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.868+0000 7fe4beb86740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc35393700 con 0x55cc35387bb0 2024-03-28T18:12:05.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.869+0000 7fe4b9e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35387290 0x55cc35390180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:05.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.869+0000 7fe4b8a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35387bb0 0x55cc3538d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:05.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.869+0000 7fe4beb86740 10 client.0 _ll_register_callbacks cb 0x55cc3529c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:05.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.869+0000 7fe4b9e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35387290 0x55cc35390180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57862/0 (socket says 172.21.15.37:57862) 2024-03-28T18:12:05.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.869+0000 7fe4b8a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35387bb0 0x55cc3538d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:53768/0 (socket says 172.21.15.37:53768) 2024-03-28T18:12:05.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.869+0000 7fe4b9e00640 1 -- 172.21.15.37:0/4141965082 learned_addr learned my addr 172.21.15.37:0/4141965082 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:05.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.869+0000 7fe4b9e00640 1 --2- 172.21.15.37:0/4141965082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35390790 0x55cc3538db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:05.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.869+0000 7fe4b8a00640 1 -- 172.21.15.37:0/4141965082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35390790 msgr2=0x55cc3538db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:05.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.869+0000 7fe4b8a00640 1 --2- 172.21.15.37:0/4141965082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35390790 0x55cc3538db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.869+0000 7fe4b8a00640 1 -- 172.21.15.37:0/4141965082 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35387290 msgr2=0x55cc35390180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:05.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.869+0000 7fe4b8a00640 1 --2- 172.21.15.37:0/4141965082 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35387290 0x55cc35390180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.869+0000 7fe4b8a00640 1 -- 172.21.15.37:0/4141965082 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc35128f70 con 0x55cc35387bb0 2024-03-28T18:12:05.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.869+0000 7fe4b8a00640 1 --2- 172.21.15.37:0/4141965082 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35387bb0 0x55cc3538d510 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe4b41017e0 tx=0x7fe4a000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e65c3bb1c944a01 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:05.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.870+0000 7fe49aa00640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:12:05.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.870+0000 7fe49aa00640 1 -- 172.21.15.37:0/4141965082 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a00120d0 con 0x55cc35387bb0 2024-03-28T18:12:05.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.870+0000 7fe49aa00640 1 -- 172.21.15.37:0/4141965082 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe4a0010430 con 0x55cc35387bb0 2024-03-28T18:12:05.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.870+0000 7fe4beb86740 1 -- 172.21.15.37:0/4141965082 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe4ac003e50 con 0x55cc35387bb0 2024-03-28T18:12:05.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.870+0000 7fe49aa00640 1 -- 172.21.15.37:0/4141965082 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4a0012270 con 0x55cc35387bb0 2024-03-28T18:12:05.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.870+0000 7fe4beb86740 10 client.25663 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:05.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.870+0000 7fe4beb86740 1 -- 172.21.15.37:0/4141965082 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55cc34fc0170 con 0x55cc35387bb0 2024-03-28T18:12:05.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.870+0000 7fe49aa00640 1 -- 172.21.15.37:0/4141965082 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4ac003e50 con 0x55cc35387bb0 2024-03-28T18:12:05.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.871+0000 7fe49aa00640 1 -- 172.21.15.37:0/4141965082 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe4a001c790 con 0x55cc35387bb0 2024-03-28T18:12:05.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.871+0000 7fe4beb86740 10 client.25663 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:05.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.871+0000 7fe4beb86740 10 client.25663 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:12:05.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.871+0000 7fe4beb86740 20 client.25663 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:05.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.871+0000 7fe4beb86740 1 --2- 172.21.15.37:0/4141965082 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55cc353a0910 0x55cc353a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:05.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.871+0000 7fe4beb86740 4 client.25663 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:12:05.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.871+0000 7fe4beb86740 1 -- 172.21.15.37:0/4141965082 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cc352f93f0 con 0x55cc353a0910 2024-03-28T18:12:05.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.871+0000 7fe4b9e00640 1 --2- 172.21.15.37:0/4141965082 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55cc353a0910 0x55cc353a2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:05.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.872+0000 7fe4b9e00640 1 --2- 172.21.15.37:0/4141965082 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55cc353a0910 0x55cc353a2d00 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1545c6cedae7bc4f server_cookie=83190c25b677055b in_seq=0 out_seq=0 2024-03-28T18:12:05.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.872+0000 7fe49aa00640 10 client.25663 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:05.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.873+0000 7fe49aa00640 1 -- 172.21.15.37:0/4141965082 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cc352f93f0 con 0x55cc353a0910 2024-03-28T18:12:05.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.873+0000 7fe49aa00640 10 client.25663 handle_command_reply: tid=0 2024-03-28T18:12:05.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.873+0000 7fe4beb86740 1 client.25663 shutdown 2024-03-28T18:12:05.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.873+0000 7fe4beb86740 1 -- 172.21.15.37:0/4141965082 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35387bb0 msgr2=0x55cc3538d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:05.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.873+0000 7fe4beb86740 1 --2- 172.21.15.37:0/4141965082 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35387bb0 0x55cc3538d510 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe4b41017e0 tx=0x7fe4a000f790 comp rx=0 tx=0).stop 2024-03-28T18:12:05.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.873+0000 7fe4beb86740 1 -- 172.21.15.37:0/4141965082 shutdown_connections 2024-03-28T18:12:05.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.873+0000 7fe4beb86740 1 --2- 172.21.15.37:0/4141965082 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55cc353a0910 0x55cc353a2d00 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.873+0000 7fe4beb86740 1 --2- 172.21.15.37:0/4141965082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35390790 0x55cc3538db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.873+0000 7fe4beb86740 1 --2- 172.21.15.37:0/4141965082 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35387bb0 0x55cc3538d510 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.873+0000 7fe4beb86740 1 --2- 172.21.15.37:0/4141965082 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35387290 0x55cc35390180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.873+0000 7fe49aa00640 0 client.25663 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:05.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.873+0000 7fe4beb86740 1 -- 172.21.15.37:0/4141965082 >> 172.21.15.37:0/4141965082 conn(0x55cc353819c0 msgr2=0x55cc35383710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:05.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.873+0000 7fe4beb86740 1 -- 172.21.15.37:0/4141965082 shutdown_connections 2024-03-28T18:12:05.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.873+0000 7fe4beb86740 1 -- 172.21.15.37:0/4141965082 wait complete. 2024-03-28T18:12:05.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.873+0000 7fe4beb86740 20 client.25663 trim_cache size 0 max 16384 2024-03-28T18:12:05.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.893+0000 7fe4beb86740 1 Processor -- start 2024-03-28T18:12:05.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.893+0000 7fe4beb86740 1 -- start start 2024-03-28T18:12:05.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.893+0000 7fe4beb86740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35387290 0x55cc353834f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:05.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.893+0000 7fe4beb86740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35387bb0 0x55cc3538fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:05.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.894+0000 7fe4beb86740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35390790 0x55cc35390180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:05.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.894+0000 7fe4beb86740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b41b17a0 con 0x55cc35387bb0 2024-03-28T18:12:05.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.894+0000 7fe4beb86740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b41af2a0 con 0x55cc35390790 2024-03-28T18:12:05.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.894+0000 7fe4beb86740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b41ad340 con 0x55cc35387290 2024-03-28T18:12:05.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.894+0000 7fe4b8a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35387bb0 0x55cc3538fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:05.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.894+0000 7fe4b8a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35387bb0 0x55cc3538fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57878/0 (socket says 172.21.15.37:57878) 2024-03-28T18:12:05.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.894+0000 7fe4b8a00640 1 -- 172.21.15.37:0/559085879 learned_addr learned my addr 172.21.15.37:0/559085879 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:05.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.894+0000 7fe4b9e00640 1 --2- 172.21.15.37:0/559085879 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35387290 0x55cc353834f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:05.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.894+0000 7fe4b9e00640 1 --2- 172.21.15.37:0/559085879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35390790 0x55cc35390180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:05.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.894+0000 7fe4b8a00640 1 -- 172.21.15.37:0/559085879 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35387290 msgr2=0x55cc353834f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:05.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.894+0000 7fe4b8a00640 1 --2- 172.21.15.37:0/559085879 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35387290 0x55cc353834f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.894+0000 7fe4b8a00640 1 -- 172.21.15.37:0/559085879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35390790 msgr2=0x55cc35390180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:05.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.894+0000 7fe4b8a00640 1 --2- 172.21.15.37:0/559085879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35390790 0x55cc35390180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.894+0000 7fe4b8a00640 1 -- 172.21.15.37:0/559085879 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc35392ef0 con 0x55cc35387bb0 2024-03-28T18:12:05.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.894+0000 7fe4b9e00640 1 --2- 172.21.15.37:0/559085879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35390790 0x55cc35390180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:05.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.895+0000 7fe4b8a00640 1 --2- 172.21.15.37:0/559085879 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35387bb0 0x55cc3538fc40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0000c00 tx=0x7fe4a0009530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3f6225058cb0dd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:05.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.895+0000 7fe4a5c00640 1 -- 172.21.15.37:0/559085879 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0029030 con 0x55cc35387bb0 2024-03-28T18:12:05.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.895+0000 7fe4a5c00640 1 -- 172.21.15.37:0/559085879 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4a001c900 con 0x55cc35387bb0 2024-03-28T18:12:05.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.895+0000 7fe4a5c00640 1 -- 172.21.15.37:0/559085879 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0004680 con 0x55cc35387bb0 2024-03-28T18:12:05.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.896+0000 7fe4beb86740 1 -- 172.21.15.37:0/559085879 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35387bb0 msgr2=0x55cc3538fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:05.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.896+0000 7fe4beb86740 1 --2- 172.21.15.37:0/559085879 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35387bb0 0x55cc3538fc40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0000c00 tx=0x7fe4a0009530 comp rx=0 tx=0).stop 2024-03-28T18:12:05.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.896+0000 7fe4beb86740 1 -- 172.21.15.37:0/559085879 shutdown_connections 2024-03-28T18:12:05.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.896+0000 7fe4beb86740 1 --2- 172.21.15.37:0/559085879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35390790 0x55cc35390180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.896+0000 7fe4beb86740 1 --2- 172.21.15.37:0/559085879 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35387bb0 0x55cc3538fc40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.896+0000 7fe4beb86740 1 --2- 172.21.15.37:0/559085879 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35387290 0x55cc353834f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.896+0000 7fe4beb86740 1 -- 172.21.15.37:0/559085879 >> 172.21.15.37:0/559085879 conn(0x55cc353819c0 msgr2=0x55cc35382de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:05.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.896+0000 7fe4beb86740 1 -- 172.21.15.37:0/559085879 shutdown_connections 2024-03-28T18:12:05.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.896+0000 7fe4beb86740 1 -- 172.21.15.37:0/559085879 wait complete. 2024-03-28T18:12:05.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.896+0000 7fe4beb86740 1 -- start start 2024-03-28T18:12:05.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.897+0000 7fe4beb86740 1 Processor -- start 2024-03-28T18:12:05.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.897+0000 7fe4beb86740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35387290 0x55cc35395970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:05.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.897+0000 7fe4beb86740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35387bb0 0x55cc35395eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:05.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.897+0000 7fe4beb86740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35390790 0x55cc353963f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:05.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.897+0000 7fe4beb86740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b41ad940 con 0x55cc35387290 2024-03-28T18:12:05.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.897+0000 7fe4beb86740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b41aec40 con 0x55cc35387bb0 2024-03-28T18:12:05.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.897+0000 7fe4beb86740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b41adf80 con 0x55cc35390790 2024-03-28T18:12:05.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.897+0000 7fe4beb86740 10 client.0 _ll_register_callbacks cb 0x7fe4a0009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:05.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.898+0000 7fe4b8a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35387bb0 0x55cc35395eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:05.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.898+0000 7fe4b9e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35387290 0x55cc35395970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:05.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.898+0000 7fe4b8a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35387bb0 0x55cc35395eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:46728/0 (socket says 172.21.15.37:46728) 2024-03-28T18:12:05.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.898+0000 7fe4b8a00640 1 -- 172.21.15.37:0/4086284265 learned_addr learned my addr 172.21.15.37:0/4086284265 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:05.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.898+0000 7fe4b9e00640 1 --2- 172.21.15.37:0/4086284265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35390790 0x55cc353963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:05.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.898+0000 7fe4b8a00640 1 -- 172.21.15.37:0/4086284265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35390790 msgr2=0x55cc353963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:05.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.898+0000 7fe4b8a00640 1 --2- 172.21.15.37:0/4086284265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35390790 0x55cc353963f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.898+0000 7fe4b8a00640 1 -- 172.21.15.37:0/4086284265 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35387290 msgr2=0x55cc35395970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:05.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.898+0000 7fe4b8a00640 1 --2- 172.21.15.37:0/4086284265 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35387290 0x55cc35395970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.898+0000 7fe4b8a00640 1 -- 172.21.15.37:0/4086284265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc353fc350 con 0x55cc35387bb0 2024-03-28T18:12:05.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.898+0000 7fe4b9e00640 1 --2- 172.21.15.37:0/4086284265 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35387290 0x55cc35395970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:05.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.899+0000 7fe4b8a00640 1 --2- 172.21.15.37:0/4086284265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35387bb0 0x55cc35395eb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000a010 tx=0x7fe4a001cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e97dd40e61e67955 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:05.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.899+0000 7fe49aa00640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:12:05.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.899+0000 7fe49aa00640 1 -- 172.21.15.37:0/4086284265 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0025450 con 0x55cc35387bb0 2024-03-28T18:12:05.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.899+0000 7fe4beb86740 1 -- 172.21.15.37:0/4086284265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe4a001c900 con 0x55cc35387bb0 2024-03-28T18:12:05.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.899+0000 7fe49aa00640 1 -- 172.21.15.37:0/4086284265 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe4a0005bb0 con 0x55cc35387bb0 2024-03-28T18:12:05.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.899+0000 7fe49aa00640 1 -- 172.21.15.37:0/4086284265 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4a00255f0 con 0x55cc35387bb0 2024-03-28T18:12:05.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.899+0000 7fe4beb86740 10 client.25646 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:05.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.899+0000 7fe4beb86740 1 -- 172.21.15.37:0/4086284265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55cc353fd650 con 0x55cc35387bb0 2024-03-28T18:12:05.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.899+0000 7fe49aa00640 1 -- 172.21.15.37:0/4086284265 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4a001c900 con 0x55cc35387bb0 2024-03-28T18:12:05.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.900+0000 7fe49aa00640 1 -- 172.21.15.37:0/4086284265 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe4a002e740 con 0x55cc35387bb0 2024-03-28T18:12:05.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.900+0000 7fe4beb86740 10 client.25646 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:05.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.900+0000 7fe4beb86740 10 client.25646 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:12:05.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.900+0000 7fe4beb86740 20 client.25646 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:05.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.900+0000 7fe4beb86740 1 --2- 172.21.15.37:0/4086284265 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55cc353fdbe0 0x55cc35400000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:05.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.900+0000 7fe4beb86740 4 client.25646 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:12:05.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.900+0000 7fe4beb86740 1 -- 172.21.15.37:0/4086284265 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55cc352f3740 con 0x55cc353fdbe0 2024-03-28T18:12:05.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.900+0000 7fe4b9e00640 1 --2- 172.21.15.37:0/4086284265 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55cc353fdbe0 0x55cc35400000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:05.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.901+0000 7fe4b9e00640 1 --2- 172.21.15.37:0/4086284265 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55cc353fdbe0 0x55cc35400000 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=815245037b1b0db5 server_cookie=a8c4c2b278845967 in_seq=0 out_seq=0 2024-03-28T18:12:05.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.901+0000 7fe49aa00640 10 client.25646 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:05.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.902+0000 7fe49aa00640 1 -- 172.21.15.37:0/4086284265 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55cc352f3740 con 0x55cc353fdbe0 2024-03-28T18:12:05.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.902+0000 7fe49aa00640 10 client.25646 handle_command_reply: tid=0 2024-03-28T18:12:05.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.902+0000 7fe4beb86740 1 client.25646 shutdown 2024-03-28T18:12:05.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.903+0000 7fe4beb86740 1 -- 172.21.15.37:0/4086284265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35387bb0 msgr2=0x55cc35395eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:05.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.903+0000 7fe4beb86740 1 --2- 172.21.15.37:0/4086284265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35387bb0 0x55cc35395eb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000a010 tx=0x7fe4a001cc80 comp rx=0 tx=0).stop 2024-03-28T18:12:05.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.903+0000 7fe4beb86740 1 -- 172.21.15.37:0/4086284265 shutdown_connections 2024-03-28T18:12:05.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.903+0000 7fe4beb86740 1 --2- 172.21.15.37:0/4086284265 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55cc353fdbe0 0x55cc35400000 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.903+0000 7fe4beb86740 1 --2- 172.21.15.37:0/4086284265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55cc35387bb0 0x55cc35395eb0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.903+0000 7fe4beb86740 1 --2- 172.21.15.37:0/4086284265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc35390790 0x55cc353963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.903+0000 7fe4beb86740 1 --2- 172.21.15.37:0/4086284265 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55cc35387290 0x55cc35395970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.903+0000 7fe4beb86740 1 -- 172.21.15.37:0/4086284265 >> 172.21.15.37:0/4086284265 conn(0x55cc353819c0 msgr2=0x55cc35388960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:05.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.903+0000 7fe49aa00640 0 client.25646 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:05.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.903+0000 7fe4beb86740 1 -- 172.21.15.37:0/4086284265 shutdown_connections 2024-03-28T18:12:05.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.903+0000 7fe4beb86740 1 -- 172.21.15.37:0/4086284265 wait complete. 2024-03-28T18:12:05.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.903+0000 7fe4beb86740 20 client.25646 trim_cache size 0 max 16384 2024-03-28T18:12:05.905 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:12:05.905 INFO:teuthology.orchestra.run.smithi037.stdout: "return_code": 0, 2024-03-28T18:12:05.905 INFO:teuthology.orchestra.run.smithi037.stdout: "scrub_tag": "de771fd3-8dc5-4e66-a43b-8207502d730b", 2024-03-28T18:12:05.905 INFO:teuthology.orchestra.run.smithi037.stdout: "mode": "asynchronous" 2024-03-28T18:12:05.905 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:12:05.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.904+0000 7fe4ba800640 1 -- 172.21.15.37:0/2957505934 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fe49c0a2a90 msgr2=0x7fe49c0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:05.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.904+0000 7fe4ba800640 1 --2- 172.21.15.37:0/2957505934 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fe49c0a2a90 0x7fe49c0a4f50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe4a80ee6d0 tx=0x7fe4a8000bc0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.904+0000 7fe4ba800640 1 -- 172.21.15.37:0/2957505934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4b4103720 msgr2=0x7fe4b4199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:05.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.904+0000 7fe4ba800640 1 --2- 172.21.15.37:0/2957505934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4b4103720 0x7fe4b4199ff0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8036040 tx=0x7fe4a8004340 comp rx=0 tx=0).stop 2024-03-28T18:12:05.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.905+0000 7fe4ba800640 1 -- 172.21.15.37:0/2957505934 shutdown_connections 2024-03-28T18:12:05.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.905+0000 7fe4ba800640 1 --2- 172.21.15.37:0/2957505934 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fe49c0a2a90 0x7fe49c0a4f50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.905+0000 7fe4ba800640 1 --2- 172.21.15.37:0/2957505934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4b41040e0 0x7fe4b419a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.905+0000 7fe4ba800640 1 --2- 172.21.15.37:0/2957505934 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4b4109730 0x7fe4b419e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.905+0000 7fe4ba800640 1 --2- 172.21.15.37:0/2957505934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4b4103720 0x7fe4b4199ff0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:05.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.905+0000 7fe4ba800640 1 -- 172.21.15.37:0/2957505934 >> 172.21.15.37:0/2957505934 conn(0x7fe4b40ff1b0 msgr2=0x7fe4b40ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:05.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.905+0000 7fe4ba800640 1 -- 172.21.15.37:0/2957505934 shutdown_connections 2024-03-28T18:12:05.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:05.905+0000 7fe4ba800640 1 -- 172.21.15.37:0/2957505934 wait complete. 2024-03-28T18:12:05.925 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:12:05.925 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:12:05.989 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:05 smithi037 ceph-mon[31460]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T18:12:06.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:05 smithi059 ceph-mon[36190]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T18:12:06.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:05 smithi170 ceph-mon[36280]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T18:12:06.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.126+0000 7f0f0cc00640 1 -- 172.21.15.37:0/390738109 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f08101520 msgr2=0x7f0f081019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:06.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.126+0000 7f0f0cc00640 1 --2- 172.21.15.37:0/390738109 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f08101520 0x7f0f081019a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009930 tx=0x7f0ef402bc30 comp rx=0 tx=0).stop 2024-03-28T18:12:06.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.126+0000 7f0f0cc00640 1 -- 172.21.15.37:0/390738109 shutdown_connections 2024-03-28T18:12:06.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.126+0000 7f0f0cc00640 1 --2- 172.21.15.37:0/390738109 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f08101ee0 0x7f0f0810e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.126+0000 7f0f0cc00640 1 --2- 172.21.15.37:0/390738109 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f08101520 0x7f0f081019a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.126+0000 7f0f0cc00640 1 --2- 172.21.15.37:0/390738109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f081075e0 0x7f0f081079c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.126+0000 7f0f0cc00640 1 -- 172.21.15.37:0/390738109 >> 172.21.15.37:0/390738109 conn(0x7f0f080fd010 msgr2=0x7f0f080ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:06.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.126+0000 7f0f0cc00640 1 -- 172.21.15.37:0/390738109 shutdown_connections 2024-03-28T18:12:06.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.126+0000 7f0f0cc00640 1 -- 172.21.15.37:0/390738109 wait complete. 2024-03-28T18:12:06.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.127+0000 7f0f0cc00640 1 Processor -- start 2024-03-28T18:12:06.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.127+0000 7f0f0cc00640 1 -- start start 2024-03-28T18:12:06.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.128+0000 7f0f0cc00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f08101520 0x7f0f08199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:06.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.128+0000 7f0f0cc00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f08101ee0 0x7f0f0819a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:06.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.128+0000 7f0f0cc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f081075e0 0x7f0f0819e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:06.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.128+0000 7f0f0cc00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f081118f0 con 0x7f0f081075e0 2024-03-28T18:12:06.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.128+0000 7f0f0cc00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f08111770 con 0x7f0f08101ee0 2024-03-28T18:12:06.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.128+0000 7f0f0cc00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f08111a70 con 0x7f0f08101520 2024-03-28T18:12:06.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.128+0000 7f0f07400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f08101520 0x7f0f08199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:06.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.128+0000 7f0f06a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f08101ee0 0x7f0f0819a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:06.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.128+0000 7f0f07400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f08101520 0x7f0f08199f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:53810/0 (socket says 172.21.15.37:53810) 2024-03-28T18:12:06.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.128+0000 7f0f07400640 1 -- 172.21.15.37:0/4175513774 learned_addr learned my addr 172.21.15.37:0/4175513774 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:06.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.128+0000 7f0f07e00640 1 --2- 172.21.15.37:0/4175513774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f081075e0 0x7f0f0819e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:06.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.129+0000 7f0f06a00640 1 -- 172.21.15.37:0/4175513774 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f08101520 msgr2=0x7f0f08199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:06.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.129+0000 7f0f06a00640 1 --2- 172.21.15.37:0/4175513774 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f08101520 0x7f0f08199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.129+0000 7f0f06a00640 1 -- 172.21.15.37:0/4175513774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f081075e0 msgr2=0x7f0f0819e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:06.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.129+0000 7f0f06a00640 1 --2- 172.21.15.37:0/4175513774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f081075e0 0x7f0f0819e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.129+0000 7f0f06a00640 1 -- 172.21.15.37:0/4175513774 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0819ef70 con 0x7f0f08101ee0 2024-03-28T18:12:06.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.129+0000 7f0f07400640 1 --2- 172.21.15.37:0/4175513774 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f08101520 0x7f0f08199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:06.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.129+0000 7f0f06a00640 1 --2- 172.21.15.37:0/4175513774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f08101ee0 0x7f0f0819a4c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4036040 tx=0x7f0ef4004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:06.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.129+0000 7f0effe00640 1 -- 172.21.15.37:0/4175513774 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4043020 con 0x7f0f08101ee0 2024-03-28T18:12:06.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.129+0000 7f0effe00640 1 -- 172.21.15.37:0/4175513774 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ef40054e0 con 0x7f0f08101ee0 2024-03-28T18:12:06.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.129+0000 7f0effe00640 1 -- 172.21.15.37:0/4175513774 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4003b80 con 0x7f0f08101ee0 2024-03-28T18:12:06.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.129+0000 7f0f0cc00640 1 -- 172.21.15.37:0/4175513774 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f0819f200 con 0x7f0f08101ee0 2024-03-28T18:12:06.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.130+0000 7f0f0cc00640 1 -- 172.21.15.37:0/4175513774 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f081a6af0 con 0x7f0f08101ee0 2024-03-28T18:12:06.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.133+0000 7f0f10e6f740 1 Processor -- start 2024-03-28T18:12:06.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.133+0000 7f0effe00640 1 -- 172.21.15.37:0/4175513774 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f0ef4046020 con 0x7f0f08101ee0 2024-03-28T18:12:06.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.133+0000 7f0f10e6f740 1 -- start start 2024-03-28T18:12:06.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.133+0000 7f0f10e6f740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b92643dd0 0x555b92642b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:06.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.134+0000 7f0f10e6f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b926430f0 0x555b92643570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:06.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.134+0000 7f0f10e6f740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b9264c460 0x555b9264e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:06.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.134+0000 7f0f10e6f740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b925a7250 con 0x555b92643dd0 2024-03-28T18:12:06.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.134+0000 7f0f10e6f740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b9254fa30 con 0x555b9264c460 2024-03-28T18:12:06.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.134+0000 7f0f10e6f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b925a32f0 con 0x555b926430f0 2024-03-28T18:12:06.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.134+0000 7f0f07e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b92643dd0 0x555b92642b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:06.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.134+0000 7f0f06a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b9264c460 0x555b9264e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:06.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.134+0000 7f0f06a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b9264c460 0x555b9264e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:46756/0 (socket says 172.21.15.37:46756) 2024-03-28T18:12:06.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.134+0000 7f0f07e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b92643dd0 0x555b92642b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57918/0 (socket says 172.21.15.37:57918) 2024-03-28T18:12:06.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.134+0000 7f0f06a00640 1 -- 172.21.15.37:0/718610765 learned_addr learned my addr 172.21.15.37:0/718610765 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:06.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.134+0000 7f0f07400640 1 --2- 172.21.15.37:0/718610765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b926430f0 0x555b92643570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:06.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.135+0000 7f0f06a00640 1 -- 172.21.15.37:0/718610765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b926430f0 msgr2=0x555b92643570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:06.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.135+0000 7f0f06a00640 1 --2- 172.21.15.37:0/718610765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b926430f0 0x555b92643570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.135+0000 7f0f06a00640 1 -- 172.21.15.37:0/718610765 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b92643dd0 msgr2=0x555b92642b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:06.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.135+0000 7f0f06a00640 1 --2- 172.21.15.37:0/718610765 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b92643dd0 0x555b92642b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.135+0000 7f0f06a00640 1 -- 172.21.15.37:0/718610765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b9253a690 con 0x555b9264c460 2024-03-28T18:12:06.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.135+0000 7f0f06a00640 1 --2- 172.21.15.37:0/718610765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b9264c460 0x555b9264e850 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0ef40ee670 tx=0x7f0ef40f91a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96c7ac1dc2cd80e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:06.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.135+0000 7f0effe00640 1 --2- 172.21.15.37:0/4175513774 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f0eec0a2b10 0x7f0eec0a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:06.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.135+0000 7f0ef3e00640 1 -- 172.21.15.37:0/718610765 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef40b97b0 con 0x555b9264c460 2024-03-28T18:12:06.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.135+0000 7f0ef3e00640 1 -- 172.21.15.37:0/718610765 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ef40fe030 con 0x555b9264c460 2024-03-28T18:12:06.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.135+0000 7f0ef3e00640 1 -- 172.21.15.37:0/718610765 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef40bc960 con 0x555b9264c460 2024-03-28T18:12:06.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.135+0000 7f0effe00640 1 -- 172.21.15.37:0/4175513774 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0ef40f0fe0 con 0x7f0f08101ee0 2024-03-28T18:12:06.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.135+0000 7f0f07e00640 1 --2- 172.21.15.37:0/4175513774 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f0eec0a2b10 0x7f0eec0a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:06.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.136+0000 7f0f10e6f740 1 -- 172.21.15.37:0/718610765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b9264c460 msgr2=0x555b9264e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:06.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.136+0000 7f0f10e6f740 1 --2- 172.21.15.37:0/718610765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b9264c460 0x555b9264e850 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0ef40ee670 tx=0x7f0ef40f91a0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.136+0000 7f0f07e00640 1 --2- 172.21.15.37:0/4175513774 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f0eec0a2b10 0x7f0eec0a4fd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0f00005fd0 tx=0x7f0f00009860 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:06.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.136+0000 7f0f10e6f740 1 -- 172.21.15.37:0/718610765 shutdown_connections 2024-03-28T18:12:06.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.136+0000 7f0f10e6f740 1 --2- 172.21.15.37:0/718610765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b9264c460 0x555b9264e850 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.136+0000 7f0f10e6f740 1 --2- 172.21.15.37:0/718610765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b926430f0 0x555b92643570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.136+0000 7f0f10e6f740 1 --2- 172.21.15.37:0/718610765 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b92643dd0 0x555b92642b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.136+0000 7f0f10e6f740 1 -- 172.21.15.37:0/718610765 >> 172.21.15.37:0/718610765 conn(0x555b9260a200 msgr2=0x555b9263db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:06.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.137+0000 7f0f10e6f740 1 -- 172.21.15.37:0/718610765 shutdown_connections 2024-03-28T18:12:06.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.137+0000 7f0f10e6f740 1 -- 172.21.15.37:0/718610765 wait complete. 2024-03-28T18:12:06.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.137+0000 7f0f10e6f740 1 -- start start 2024-03-28T18:12:06.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.137+0000 7f0f10e6f740 1 Processor -- start 2024-03-28T18:12:06.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.138+0000 7f0f10e6f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b926430f0 0x555b9264bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:06.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.138+0000 7f0f10e6f740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b92643dd0 0x555b92648ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:06.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.138+0000 7f0f10e6f740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b9264c460 0x555b92649400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:06.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.138+0000 7f0f10e6f740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b9264f090 con 0x555b9264c460 2024-03-28T18:12:06.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.138+0000 7f0f10e6f740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b92562e80 con 0x555b92643dd0 2024-03-28T18:12:06.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.138+0000 7f0f10e6f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b9264f210 con 0x555b926430f0 2024-03-28T18:12:06.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.138+0000 7f0f10e6f740 10 client.0 _ll_register_callbacks cb 0x555b92575130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:06.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.138+0000 7f0f07400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b926430f0 0x555b9264bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:06.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.138+0000 7f0f07e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b9264c460 0x555b92649400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:06.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.138+0000 7f0f07400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b926430f0 0x555b9264bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:53828/0 (socket says 172.21.15.37:53828) 2024-03-28T18:12:06.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.138+0000 7f0f07400640 1 -- 172.21.15.37:0/2681756980 learned_addr learned my addr 172.21.15.37:0/2681756980 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:06.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.138+0000 7f0f06a00640 1 --2- 172.21.15.37:0/2681756980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b92643dd0 0x555b92648ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:06.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.139+0000 7f0f07e00640 1 -- 172.21.15.37:0/2681756980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b926430f0 msgr2=0x555b9264bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:06.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.139+0000 7f0f07e00640 1 --2- 172.21.15.37:0/2681756980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b926430f0 0x555b9264bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.139+0000 7f0f07e00640 1 -- 172.21.15.37:0/2681756980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b92643dd0 msgr2=0x555b92648ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:06.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.139+0000 7f0f07e00640 1 --2- 172.21.15.37:0/2681756980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b92643dd0 0x555b92648ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.139+0000 7f0f07e00640 1 -- 172.21.15.37:0/2681756980 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555b923857d0 con 0x555b9264c460 2024-03-28T18:12:06.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.139+0000 7f0f07e00640 1 --2- 172.21.15.37:0/2681756980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b9264c460 0x555b92649400 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0f00003e80 tx=0x7f0f00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f18d5ca2294776bc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:06.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.139+0000 7f0ef0c00640 10 client.0 ms_handle_connect on v2:172.21.15.37:3300/0 2024-03-28T18:12:06.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.139+0000 7f0ef0c00640 1 -- 172.21.15.37:0/2681756980 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f00024070 con 0x555b9264c460 2024-03-28T18:12:06.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.139+0000 7f0ef0c00640 1 -- 172.21.15.37:0/2681756980 <== mon.0 v2:172.21.15.37:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0f00010aa0 con 0x555b9264c460 2024-03-28T18:12:06.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.139+0000 7f0f10e6f740 1 -- 172.21.15.37:0/2681756980 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ef40fe030 con 0x555b9264c460 2024-03-28T18:12:06.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.140+0000 7f0ef0c00640 1 -- 172.21.15.37:0/2681756980 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f00030050 con 0x555b9264c460 2024-03-28T18:12:06.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.140+0000 7f0f10e6f740 10 client.15495 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:06.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.140+0000 7f0f10e6f740 1 -- 172.21.15.37:0/2681756980 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555b922bb3f0 con 0x555b9264c460 2024-03-28T18:12:06.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.140+0000 7f0ef0c00640 1 -- 172.21.15.37:0/2681756980 <== mon.0 v2:172.21.15.37:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ef40fe030 con 0x555b9264c460 2024-03-28T18:12:06.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.140+0000 7f0ef0c00640 1 -- 172.21.15.37:0/2681756980 <== mon.0 v2:172.21.15.37:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f0f000283b0 con 0x555b9264c460 2024-03-28T18:12:06.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.140+0000 7f0f10e6f740 10 client.15495 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:06.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.140+0000 7f0f10e6f740 10 client.15495 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:12:06.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.140+0000 7f0f10e6f740 20 client.15495 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:06.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.140+0000 7f0f10e6f740 1 --2- 172.21.15.37:0/2681756980 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x555b9265c0a0 0x555b9265e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:06.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.140+0000 7f0f10e6f740 4 client.15495 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:12:06.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.140+0000 7f0f10e6f740 1 -- 172.21.15.37:0/2681756980 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555b925b4790 con 0x555b9265c0a0 2024-03-28T18:12:06.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.141+0000 7f0f07400640 1 --2- 172.21.15.37:0/2681756980 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x555b9265c0a0 0x555b9265e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:06.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.142+0000 7f0f07400640 1 --2- 172.21.15.37:0/2681756980 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x555b9265c0a0 0x555b9265e530 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9822185cec55097a server_cookie=6686921fffae4ea9 in_seq=0 out_seq=0 2024-03-28T18:12:06.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.142+0000 7f0ef0c00640 10 client.15495 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:06.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.143+0000 7f0ef0c00640 1 -- 172.21.15.37:0/2681756980 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555b925b4790 con 0x555b9265c0a0 2024-03-28T18:12:06.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.143+0000 7f0ef0c00640 10 client.15495 handle_command_reply: tid=0 2024-03-28T18:12:06.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.143+0000 7f0f10e6f740 1 client.15495 shutdown 2024-03-28T18:12:06.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.143+0000 7f0f10e6f740 1 -- 172.21.15.37:0/2681756980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b9264c460 msgr2=0x555b92649400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:06.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.143+0000 7f0f10e6f740 1 --2- 172.21.15.37:0/2681756980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b9264c460 0x555b92649400 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0f00003e80 tx=0x7f0f00004a40 comp rx=0 tx=0).stop 2024-03-28T18:12:06.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.143+0000 7f0f10e6f740 1 -- 172.21.15.37:0/2681756980 shutdown_connections 2024-03-28T18:12:06.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.143+0000 7f0f10e6f740 1 --2- 172.21.15.37:0/2681756980 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x555b9265c0a0 0x555b9265e530 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.143+0000 7f0f10e6f740 1 --2- 172.21.15.37:0/2681756980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b92643dd0 0x555b92648ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.143+0000 7f0f10e6f740 1 --2- 172.21.15.37:0/2681756980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b9264c460 0x555b92649400 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.143+0000 7f0f10e6f740 1 --2- 172.21.15.37:0/2681756980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b926430f0 0x555b9264bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.143+0000 7f0f10e6f740 1 -- 172.21.15.37:0/2681756980 >> 172.21.15.37:0/2681756980 conn(0x555b9260a200 msgr2=0x555b9263f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:06.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.143+0000 7f0ef0c00640 0 client.15495 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:06.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.143+0000 7f0f10e6f740 1 -- 172.21.15.37:0/2681756980 shutdown_connections 2024-03-28T18:12:06.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.143+0000 7f0f10e6f740 1 -- 172.21.15.37:0/2681756980 wait complete. 2024-03-28T18:12:06.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.144+0000 7f0f10e6f740 20 client.15495 trim_cache size 0 max 16384 2024-03-28T18:12:06.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.162+0000 7f0f10e6f740 1 Processor -- start 2024-03-28T18:12:06.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.162+0000 7f0f10e6f740 1 -- start start 2024-03-28T18:12:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.163+0000 7f0f10e6f740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b926430f0 0x555b9264b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.163+0000 7f0f10e6f740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b92643dd0 0x555b9264bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.163+0000 7f0f10e6f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b9264c460 0x555b9264d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.163+0000 7f0f10e6f740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f081b16b0 con 0x555b926430f0 2024-03-28T18:12:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.163+0000 7f0f10e6f740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f081af1b0 con 0x555b92643dd0 2024-03-28T18:12:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.163+0000 7f0f10e6f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f081ad250 con 0x555b9264c460 2024-03-28T18:12:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.163+0000 7f0f07400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b926430f0 0x555b9264b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.163+0000 7f0f07e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b9264c460 0x555b9264d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.163+0000 7f0f07400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b926430f0 0x555b9264b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57944/0 (socket says 172.21.15.37:57944) 2024-03-28T18:12:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.163+0000 7f0f07400640 1 -- 172.21.15.37:0/881474693 learned_addr learned my addr 172.21.15.37:0/881474693 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.163+0000 7f0f06a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b92643dd0 0x555b9264bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.163+0000 7f0f06a00640 1 -- 172.21.15.37:0/881474693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b9264c460 msgr2=0x555b9264d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.163+0000 7f0f06a00640 1 --2- 172.21.15.37:0/881474693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b9264c460 0x555b9264d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.163+0000 7f0f06a00640 1 -- 172.21.15.37:0/881474693 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b926430f0 msgr2=0x555b9264b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.163+0000 7f0f06a00640 1 --2- 172.21.15.37:0/881474693 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b926430f0 0x555b9264b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.163+0000 7f0f06a00640 1 -- 172.21.15.37:0/881474693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b9227b0f0 con 0x555b92643dd0 2024-03-28T18:12:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.163+0000 7f0f07e00640 1 --2- 172.21.15.37:0/881474693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b9264c460 0x555b9264d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.164+0000 7f0f06a00640 1 --2- 172.21.15.37:0/881474693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b92643dd0 0x555b9264bb30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4101000 tx=0x7f0ef4000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c07d6b5d5234f5d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.164+0000 7f0ef3400640 1 -- 172.21.15.37:0/881474693 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef40bcc90 con 0x555b92643dd0 2024-03-28T18:12:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.164+0000 7f0ef3400640 1 -- 172.21.15.37:0/881474693 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ef40fe040 con 0x555b92643dd0 2024-03-28T18:12:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.164+0000 7f0ef3400640 1 -- 172.21.15.37:0/881474693 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef40bce70 con 0x555b92643dd0 2024-03-28T18:12:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.165+0000 7f0f10e6f740 1 -- 172.21.15.37:0/881474693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b92643dd0 msgr2=0x555b9264bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.165+0000 7f0f10e6f740 1 --2- 172.21.15.37:0/881474693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b92643dd0 0x555b9264bb30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4101000 tx=0x7f0ef4000bc0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.165+0000 7f0f10e6f740 1 -- 172.21.15.37:0/881474693 shutdown_connections 2024-03-28T18:12:06.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.165+0000 7f0f10e6f740 1 --2- 172.21.15.37:0/881474693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b92643dd0 0x555b9264bb30 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.165+0000 7f0f10e6f740 1 --2- 172.21.15.37:0/881474693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b9264c460 0x555b9264d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.165+0000 7f0f10e6f740 1 --2- 172.21.15.37:0/881474693 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b926430f0 0x555b9264b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.165+0000 7f0f10e6f740 1 -- 172.21.15.37:0/881474693 >> 172.21.15.37:0/881474693 conn(0x555b9260a200 msgr2=0x555b92647790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:06.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.165+0000 7f0f10e6f740 1 -- 172.21.15.37:0/881474693 shutdown_connections 2024-03-28T18:12:06.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.165+0000 7f0f10e6f740 1 -- 172.21.15.37:0/881474693 wait complete. 2024-03-28T18:12:06.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.165+0000 7f0f10e6f740 1 -- start start 2024-03-28T18:12:06.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.166+0000 7f0f10e6f740 1 Processor -- start 2024-03-28T18:12:06.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.166+0000 7f0f10e6f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b926430f0 0x555b926457d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:06.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.166+0000 7f0f10e6f740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b92643dd0 0x555b92645d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:06.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.166+0000 7f0f10e6f740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b9264c460 0x555b92646250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:06.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.166+0000 7f0f10e6f740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f081ad850 con 0x555b92643dd0 2024-03-28T18:12:06.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.166+0000 7f0f10e6f740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f081aeb50 con 0x555b9264c460 2024-03-28T18:12:06.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.166+0000 7f0f10e6f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f081ade90 con 0x555b926430f0 2024-03-28T18:12:06.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.166+0000 7f0f10e6f740 10 client.0 _ll_register_callbacks cb 0x7f0f00009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:06.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.166+0000 7f0f07400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b926430f0 0x555b926457d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:06.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.166+0000 7f0f07e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b9264c460 0x555b92646250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:06.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.166+0000 7f0f07400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b926430f0 0x555b926457d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:53834/0 (socket says 172.21.15.37:53834) 2024-03-28T18:12:06.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.166+0000 7f0f07400640 1 -- 172.21.15.37:0/1787905756 learned_addr learned my addr 172.21.15.37:0/1787905756 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:06.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.166+0000 7f0f06a00640 1 --2- 172.21.15.37:0/1787905756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b92643dd0 0x555b92645d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:06.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.167+0000 7f0f07400640 1 -- 172.21.15.37:0/1787905756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b9264c460 msgr2=0x555b92646250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:06.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.167+0000 7f0f07400640 1 --2- 172.21.15.37:0/1787905756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b9264c460 0x555b92646250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.167+0000 7f0f07400640 1 -- 172.21.15.37:0/1787905756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b92643dd0 msgr2=0x555b92645d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:06.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.167+0000 7f0f07400640 1 --2- 172.21.15.37:0/1787905756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b92643dd0 0x555b92645d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.167+0000 7f0f07400640 1 -- 172.21.15.37:0/1787905756 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555b926b6c60 con 0x555b926430f0 2024-03-28T18:12:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.167+0000 7f0f07400640 1 --2- 172.21.15.37:0/1787905756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b926430f0 0x555b926457d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8004370 tx=0x7f0ef8009e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ddaadd8082d2f4e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.167+0000 7f0ef0c00640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:12:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.167+0000 7f0ef0c00640 1 -- 172.21.15.37:0/1787905756 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef801eb60 con 0x555b926430f0 2024-03-28T18:12:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.167+0000 7f0ef0c00640 1 -- 172.21.15.37:0/1787905756 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0ef801ed00 con 0x555b926430f0 2024-03-28T18:12:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.167+0000 7f0f10e6f740 1 -- 172.21.15.37:0/1787905756 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ef40fe040 con 0x555b926430f0 2024-03-28T18:12:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.168+0000 7f0ef0c00640 1 -- 172.21.15.37:0/1787905756 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ef802e050 con 0x555b926430f0 2024-03-28T18:12:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.168+0000 7f0f10e6f740 10 client.25687 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.168+0000 7f0f10e6f740 1 -- 172.21.15.37:0/1787905756 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555b926b7fb0 con 0x555b926430f0 2024-03-28T18:12:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.168+0000 7f0ef0c00640 1 -- 172.21.15.37:0/1787905756 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ef40fe040 con 0x555b926430f0 2024-03-28T18:12:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.168+0000 7f0ef0c00640 1 -- 172.21.15.37:0/1787905756 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f0ef80035f0 con 0x555b926430f0 2024-03-28T18:12:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.168+0000 7f0f10e6f740 10 client.25687 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.168+0000 7f0f10e6f740 10 client.25687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:12:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.168+0000 7f0f10e6f740 20 client.25687 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.168+0000 7f0f10e6f740 1 --2- 172.21.15.37:0/1787905756 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x555b926b8710 0x555b926bab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:06.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.168+0000 7f0f10e6f740 4 client.25687 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:12:06.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.168+0000 7f0f10e6f740 1 -- 172.21.15.37:0/1787905756 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555b925aeae0 con 0x555b926b8710 2024-03-28T18:12:06.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.169+0000 7f0f06a00640 1 --2- 172.21.15.37:0/1787905756 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x555b926b8710 0x555b926bab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.169+0000 7f0f06a00640 1 --2- 172.21.15.37:0/1787905756 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x555b926b8710 0x555b926bab30 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95df16ec7b3a0b23 server_cookie=866299ac4d25c51b in_seq=0 out_seq=0 2024-03-28T18:12:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.169+0000 7f0ef0c00640 10 client.25687 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.170+0000 7f0ef0c00640 1 -- 172.21.15.37:0/1787905756 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x555b925aeae0 con 0x555b926b8710 2024-03-28T18:12:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.170+0000 7f0ef0c00640 10 client.25687 handle_command_reply: tid=0 2024-03-28T18:12:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.170+0000 7f0f10e6f740 1 client.25687 shutdown 2024-03-28T18:12:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.170+0000 7f0f10e6f740 1 -- 172.21.15.37:0/1787905756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b926430f0 msgr2=0x555b926457d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.170+0000 7f0f10e6f740 1 --2- 172.21.15.37:0/1787905756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b926430f0 0x555b926457d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8004370 tx=0x7f0ef8009e00 comp rx=0 tx=0).stop 2024-03-28T18:12:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.170+0000 7f0f10e6f740 1 -- 172.21.15.37:0/1787905756 shutdown_connections 2024-03-28T18:12:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.170+0000 7f0f10e6f740 1 --2- 172.21.15.37:0/1787905756 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x555b926b8710 0x555b926bab30 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.170+0000 7f0f10e6f740 1 --2- 172.21.15.37:0/1787905756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555b9264c460 0x555b92646250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.170+0000 7f0f10e6f740 1 --2- 172.21.15.37:0/1787905756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555b92643dd0 0x555b92645d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.170+0000 7f0f10e6f740 1 --2- 172.21.15.37:0/1787905756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b926430f0 0x555b926457d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.170+0000 7f0ef0c00640 0 client.25687 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:06.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.170+0000 7f0f10e6f740 1 -- 172.21.15.37:0/1787905756 >> 172.21.15.37:0/1787905756 conn(0x555b9260a200 msgr2=0x555b9263ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:06.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.170+0000 7f0f10e6f740 1 -- 172.21.15.37:0/1787905756 shutdown_connections 2024-03-28T18:12:06.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.170+0000 7f0f10e6f740 1 -- 172.21.15.37:0/1787905756 wait complete. 2024-03-28T18:12:06.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.170+0000 7f0f10e6f740 20 client.25687 trim_cache size 0 max 16384 2024-03-28T18:12:06.173 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:12:06.173 INFO:teuthology.orchestra.run.smithi037.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:12:06.173 INFO:teuthology.orchestra.run.smithi037.stdout: "scrubs": { 2024-03-28T18:12:06.173 INFO:teuthology.orchestra.run.smithi037.stdout: "de771fd3-8dc5-4e66-a43b-8207502d730b": { 2024-03-28T18:12:06.173 INFO:teuthology.orchestra.run.smithi037.stdout: "path": "/", 2024-03-28T18:12:06.173 INFO:teuthology.orchestra.run.smithi037.stdout: "tag": "de771fd3-8dc5-4e66-a43b-8207502d730b", 2024-03-28T18:12:06.174 INFO:teuthology.orchestra.run.smithi037.stdout: "options": "recursive,force" 2024-03-28T18:12:06.174 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-03-28T18:12:06.174 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-03-28T18:12:06.174 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:12:06.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.171+0000 7f0f0cc00640 1 -- 172.21.15.37:0/4175513774 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f0eec0a2b10 msgr2=0x7f0eec0a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:06.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.171+0000 7f0f0cc00640 1 --2- 172.21.15.37:0/4175513774 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f0eec0a2b10 0x7f0eec0a4fd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0f00005fd0 tx=0x7f0f00009860 comp rx=0 tx=0).stop 2024-03-28T18:12:06.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.171+0000 7f0f0cc00640 1 -- 172.21.15.37:0/4175513774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f08101ee0 msgr2=0x7f0f0819a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.171+0000 7f0f0cc00640 1 --2- 172.21.15.37:0/4175513774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f08101ee0 0x7f0f0819a4c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4036040 tx=0x7f0ef4004360 comp rx=0 tx=0).stop 2024-03-28T18:12:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.171+0000 7f0f0cc00640 1 -- 172.21.15.37:0/4175513774 shutdown_connections 2024-03-28T18:12:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.171+0000 7f0f0cc00640 1 --2- 172.21.15.37:0/4175513774 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f0eec0a2b10 0x7f0eec0a4fd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.171+0000 7f0f0cc00640 1 --2- 172.21.15.37:0/4175513774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f08101ee0 0x7f0f0819a4c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.171+0000 7f0f0cc00640 1 --2- 172.21.15.37:0/4175513774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f081075e0 0x7f0f0819e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.171+0000 7f0f0cc00640 1 --2- 172.21.15.37:0/4175513774 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f08101520 0x7f0f08199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.171+0000 7f0f0cc00640 1 -- 172.21.15.37:0/4175513774 >> 172.21.15.37:0/4175513774 conn(0x7f0f080fd010 msgr2=0x7f0f080741a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.171+0000 7f0f0cc00640 1 -- 172.21.15.37:0/4175513774 shutdown_connections 2024-03-28T18:12:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:06.171+0000 7f0f0cc00640 1 -- 172.21.15.37:0/4175513774 wait complete. 2024-03-28T18:12:06.188 INFO:tasks.cephfs.filesystem:scrub status for tag:de771fd3-8dc5-4e66-a43b-8207502d730b - {'path': '/', 'tag': 'de771fd3-8dc5-4e66-a43b-8207502d730b', 'options': 'recursive,force'} 2024-03-28T18:12:07.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:06 smithi059 ceph-mon[36190]: scrub queued for path: / 2024-03-28T18:12:07.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:06 smithi059 ceph-mon[36190]: scrub summary: idle+waiting paths [/] 2024-03-28T18:12:07.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:06 smithi059 ceph-mon[36190]: scrub summary: active paths [/] 2024-03-28T18:12:07.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:06 smithi059 ceph-mon[36190]: scrub summary: idle+waiting paths [/] 2024-03-28T18:12:07.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:06 smithi170 ceph-mon[36280]: scrub queued for path: / 2024-03-28T18:12:07.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:06 smithi170 ceph-mon[36280]: scrub summary: idle+waiting paths [/] 2024-03-28T18:12:07.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:06 smithi170 ceph-mon[36280]: scrub summary: active paths [/] 2024-03-28T18:12:07.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:06 smithi170 ceph-mon[36280]: scrub summary: idle+waiting paths [/] 2024-03-28T18:12:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:06 smithi037 ceph-mon[31460]: scrub queued for path: / 2024-03-28T18:12:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:06 smithi037 ceph-mon[31460]: scrub summary: idle+waiting paths [/] 2024-03-28T18:12:07.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:06 smithi037 ceph-mon[31460]: scrub summary: active paths [/] 2024-03-28T18:12:07.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:06 smithi037 ceph-mon[31460]: scrub summary: idle+waiting paths [/] 2024-03-28T18:12:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:07 smithi059 ceph-mon[36190]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-03-28T18:12:08.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:07 smithi170 ceph-mon[36280]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-03-28T18:12:08.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:07 smithi037 ceph-mon[31460]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-03-28T18:12:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:09 smithi059 ceph-mon[36190]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-28T18:12:10.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:09 smithi170 ceph-mon[36280]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-28T18:12:10.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:09 smithi037 ceph-mon[31460]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-28T18:12:11.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:10 smithi059 ceph-mon[36190]: scrub summary: idle 2024-03-28T18:12:11.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:10 smithi170 ceph-mon[36280]: scrub summary: idle 2024-03-28T18:12:11.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:10 smithi037 ceph-mon[31460]: scrub summary: idle 2024-03-28T18:12:12.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:11 smithi059 ceph-mon[36190]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-28T18:12:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:11 smithi170 ceph-mon[36280]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-28T18:12:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:11 smithi037 ceph-mon[31460]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-28T18:12:14.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:13 smithi059 ceph-mon[36190]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T18:12:14.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:13 smithi170 ceph-mon[36280]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T18:12:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:13 smithi037 ceph-mon[31460]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T18:12:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:15 smithi059 ceph-mon[36190]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T18:12:16.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:15 smithi170 ceph-mon[36280]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T18:12:16.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:15 smithi037 ceph-mon[31460]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T18:12:18.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:17 smithi059 ceph-mon[36190]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-28T18:12:18.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:17 smithi170 ceph-mon[36280]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-28T18:12:18.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:17 smithi037 ceph-mon[31460]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-28T18:12:20.009 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:19 smithi037 ceph-mon[31460]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-28T18:12:20.009 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:19 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:12:20.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:19 smithi059 ceph-mon[36190]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-28T18:12:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:19 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:12:20.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:19 smithi170 ceph-mon[36280]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-28T18:12:20.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:19 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:12:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:21 smithi059 ceph-mon[36190]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T18:12:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:21 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:12:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:21 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:12:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:21 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:12:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:21 smithi170 ceph-mon[36280]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T18:12:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:21 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:12:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:21 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:12:22.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:21 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:12:22.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:21 smithi037 ceph-mon[31460]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T18:12:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:21 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:12:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:21 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:12:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:21 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:12:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:23 smithi059 ceph-mon[36190]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-28T18:12:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:23 smithi170 ceph-mon[36280]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-28T18:12:24.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:23 smithi037 ceph-mon[31460]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-28T18:12:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:25 smithi059 ceph-mon[36190]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:26.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:25 smithi170 ceph-mon[36280]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:26.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:25 smithi037 ceph-mon[31460]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:27 smithi059 ceph-mon[36190]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:12:28.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:27 smithi170 ceph-mon[36280]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:12:28.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:27 smithi037 ceph-mon[31460]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:12:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:29 smithi059 ceph-mon[36190]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T18:12:30.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:29 smithi170 ceph-mon[36280]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T18:12:30.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:29 smithi037 ceph-mon[31460]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T18:12:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:31 smithi059 ceph-mon[36190]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:32.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:31 smithi170 ceph-mon[36280]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:32.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:31 smithi037 ceph-mon[31460]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:34.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:33 smithi059 ceph-mon[36190]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:12:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:33 smithi170 ceph-mon[36280]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:12:34.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:33 smithi037 ceph-mon[31460]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:12:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:35 smithi059 ceph-mon[36190]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:36.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:35 smithi170 ceph-mon[36280]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:36.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:35 smithi037 ceph-mon[31460]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:36.189 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:12:36.189 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:12:36.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.429+0000 7f66db200640 1 -- 172.21.15.37:0/1962752569 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66dc101530 msgr2=0x7f66dc1019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:36.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.429+0000 7f66db200640 1 --2- 172.21.15.37:0/1962752569 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66dc101530 0x7f66dc1019b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f66c4009930 tx=0x7f66c402bc30 comp rx=0 tx=0).stop 2024-03-28T18:12:36.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.429+0000 7f66db200640 1 -- 172.21.15.37:0/1962752569 shutdown_connections 2024-03-28T18:12:36.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.429+0000 7f66db200640 1 --2- 172.21.15.37:0/1962752569 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66dc101ef0 0x7f66dc10e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.429+0000 7f66db200640 1 --2- 172.21.15.37:0/1962752569 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66dc101530 0x7f66dc1019b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.429+0000 7f66db200640 1 --2- 172.21.15.37:0/1962752569 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66dc1075f0 0x7f66dc1079d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.429+0000 7f66db200640 1 -- 172.21.15.37:0/1962752569 >> 172.21.15.37:0/1962752569 conn(0x7f66dc0fd040 msgr2=0x7f66dc0ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.429+0000 7f66db200640 1 -- 172.21.15.37:0/1962752569 shutdown_connections 2024-03-28T18:12:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.429+0000 7f66db200640 1 -- 172.21.15.37:0/1962752569 wait complete. 2024-03-28T18:12:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.430+0000 7f66db200640 1 Processor -- start 2024-03-28T18:12:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.430+0000 7f66db200640 1 -- start start 2024-03-28T18:12:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.431+0000 7f66db200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66dc101530 0x7f66dc199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.431+0000 7f66db200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66dc101ef0 0x7f66dc19a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.431+0000 7f66db200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66dc1075f0 0x7f66dc19e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.431+0000 7f66db200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc111920 con 0x7f66dc101530 2024-03-28T18:12:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.431+0000 7f66db200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc1117a0 con 0x7f66dc101ef0 2024-03-28T18:12:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.431+0000 7f66db200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc111aa0 con 0x7f66dc1075f0 2024-03-28T18:12:36.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.431+0000 7f66d3e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66dc101530 0x7f66dc199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:36.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.431+0000 7f66d9e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66dc101ef0 0x7f66dc19a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:36.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.431+0000 7f66d9e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66dc101ef0 0x7f66dc19a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:44130/0 (socket says 172.21.15.37:44130) 2024-03-28T18:12:36.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.431+0000 7f66d9e00640 1 -- 172.21.15.37:0/2145630230 learned_addr learned my addr 172.21.15.37:0/2145630230 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:36.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.431+0000 7f66da800640 1 --2- 172.21.15.37:0/2145630230 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66dc1075f0 0x7f66dc19e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:36.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.432+0000 7f66d9e00640 1 -- 172.21.15.37:0/2145630230 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66dc1075f0 msgr2=0x7f66dc19e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:36.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.432+0000 7f66d9e00640 1 --2- 172.21.15.37:0/2145630230 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66dc1075f0 0x7f66dc19e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.432+0000 7f66d9e00640 1 -- 172.21.15.37:0/2145630230 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66dc101530 msgr2=0x7f66dc199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:36.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.432+0000 7f66d9e00640 1 --2- 172.21.15.37:0/2145630230 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66dc101530 0x7f66dc199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.432+0000 7f66d9e00640 1 -- 172.21.15.37:0/2145630230 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66dc19efa0 con 0x7f66dc101ef0 2024-03-28T18:12:36.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.432+0000 7f66d9e00640 1 --2- 172.21.15.37:0/2145630230 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66dc101ef0 0x7f66dc19a4f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f66c4009900 tx=0x7f66c4004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:36.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.432+0000 7f66d2a00640 1 -- 172.21.15.37:0/2145630230 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c4043020 con 0x7f66dc101ef0 2024-03-28T18:12:36.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.432+0000 7f66d2a00640 1 -- 172.21.15.37:0/2145630230 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66c40054e0 con 0x7f66dc101ef0 2024-03-28T18:12:36.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.432+0000 7f66d2a00640 1 -- 172.21.15.37:0/2145630230 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c4003d00 con 0x7f66dc101ef0 2024-03-28T18:12:36.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.433+0000 7f66db200640 1 -- 172.21.15.37:0/2145630230 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66dc19f230 con 0x7f66dc101ef0 2024-03-28T18:12:36.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.433+0000 7f66db200640 1 -- 172.21.15.37:0/2145630230 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66dc1a6b20 con 0x7f66dc101ef0 2024-03-28T18:12:36.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.437+0000 7f66e3794740 1 Processor -- start 2024-03-28T18:12:36.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.437+0000 7f66d2a00640 1 -- 172.21.15.37:0/2145630230 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f66c4046020 con 0x7f66dc101ef0 2024-03-28T18:12:36.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.438+0000 7f66e3794740 1 -- start start 2024-03-28T18:12:36.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.438+0000 7f66e3794740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x565076097dd0 0x565076096b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.438+0000 7f66e3794740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5650760970f0 0x565076097570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.438+0000 7f66e3794740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5650760a0460 0x5650760a2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.438+0000 7f66e3794740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x565075ffb250 con 0x565076097dd0 2024-03-28T18:12:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.438+0000 7f66e3794740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x565075fa3a30 con 0x5650760970f0 2024-03-28T18:12:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.438+0000 7f66e3794740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x565075ff72f0 con 0x5650760a0460 2024-03-28T18:12:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.439+0000 7f66d3e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5650760970f0 0x565076097570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.439+0000 7f66d3e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5650760970f0 0x565076097570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:44138/0 (socket says 172.21.15.37:44138) 2024-03-28T18:12:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.439+0000 7f66d3e00640 1 -- 172.21.15.37:0/4096859967 learned_addr learned my addr 172.21.15.37:0/4096859967 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.439+0000 7f66da800640 1 --2- 172.21.15.37:0/4096859967 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x565076097dd0 0x565076096b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.439+0000 7f66d3e00640 1 -- 172.21.15.37:0/4096859967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5650760a0460 msgr2=0x5650760a2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.439+0000 7f66d3e00640 1 --2- 172.21.15.37:0/4096859967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5650760a0460 0x5650760a2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.440+0000 7f66d3e00640 1 -- 172.21.15.37:0/4096859967 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x565076097dd0 msgr2=0x565076096b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.440+0000 7f66d3e00640 1 --2- 172.21.15.37:0/4096859967 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x565076097dd0 0x565076096b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.440+0000 7f66d3e00640 1 -- 172.21.15.37:0/4096859967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565075f8e690 con 0x5650760970f0 2024-03-28T18:12:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.440+0000 7f66d2a00640 1 --2- 172.21.15.37:0/2145630230 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f66c00a2b10 0x7f66c00a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.440+0000 7f66d2a00640 1 -- 172.21.15.37:0/2145630230 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f66c40f0b00 con 0x7f66dc101ef0 2024-03-28T18:12:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.440+0000 7f66da800640 1 --2- 172.21.15.37:0/2145630230 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f66c00a2b10 0x7f66c00a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.440+0000 7f66d3e00640 1 --2- 172.21.15.37:0/4096859967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5650760970f0 0x565076097570 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f66cc00b710 tx=0x7f66cc004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4d8af2b28486a1f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:36.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.440+0000 7f66cb000640 1 -- 172.21.15.37:0/4096859967 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66cc003bd0 con 0x5650760970f0 2024-03-28T18:12:36.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.441+0000 7f66cb000640 1 -- 172.21.15.37:0/4096859967 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66cc015070 con 0x5650760970f0 2024-03-28T18:12:36.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.441+0000 7f66cb000640 1 -- 172.21.15.37:0/4096859967 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66cc004e60 con 0x5650760970f0 2024-03-28T18:12:36.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.441+0000 7f66e3794740 1 -- 172.21.15.37:0/4096859967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5650760970f0 msgr2=0x565076097570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:36.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.441+0000 7f66e3794740 1 --2- 172.21.15.37:0/4096859967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5650760970f0 0x565076097570 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f66cc00b710 tx=0x7f66cc004610 comp rx=0 tx=0).stop 2024-03-28T18:12:36.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.442+0000 7f66e3794740 1 -- 172.21.15.37:0/4096859967 shutdown_connections 2024-03-28T18:12:36.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.442+0000 7f66e3794740 1 --2- 172.21.15.37:0/4096859967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5650760970f0 0x565076097570 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.442+0000 7f66e3794740 1 --2- 172.21.15.37:0/4096859967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5650760a0460 0x5650760a2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.442+0000 7f66e3794740 1 --2- 172.21.15.37:0/4096859967 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x565076097dd0 0x565076096b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.442+0000 7f66e3794740 1 -- 172.21.15.37:0/4096859967 >> 172.21.15.37:0/4096859967 conn(0x56507605e200 msgr2=0x565076091b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:36.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.442+0000 7f66da800640 1 --2- 172.21.15.37:0/2145630230 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f66c00a2b10 0x7f66c00a4fd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f66d4005fd0 tx=0x7f66d40096a0 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:36.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.442+0000 7f66e3794740 1 -- 172.21.15.37:0/4096859967 shutdown_connections 2024-03-28T18:12:36.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.442+0000 7f66e3794740 1 -- 172.21.15.37:0/4096859967 wait complete. 2024-03-28T18:12:36.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.443+0000 7f66e3794740 1 -- start start 2024-03-28T18:12:36.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.443+0000 7f66e3794740 1 Processor -- start 2024-03-28T18:12:36.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.444+0000 7f66e3794740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5650760970f0 0x56507609fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.444+0000 7f66e3794740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x565076097dd0 0x56507609cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.444+0000 7f66e3794740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5650760a0460 0x56507609d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.444+0000 7f66e3794740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650760a32d0 con 0x5650760a0460 2024-03-28T18:12:36.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.444+0000 7f66e3794740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x565075fb6e80 con 0x5650760970f0 2024-03-28T18:12:36.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.444+0000 7f66e3794740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650760a3450 con 0x565076097dd0 2024-03-28T18:12:36.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.444+0000 7f66e3794740 10 client.0 _ll_register_callbacks cb 0x565075fc9130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:36.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.444+0000 7f66d9e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x565076097dd0 0x56507609cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:36.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.444+0000 7f66d3e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5650760970f0 0x56507609fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:36.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.444+0000 7f66d3e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5650760970f0 0x56507609fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:44140/0 (socket says 172.21.15.37:44140) 2024-03-28T18:12:36.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.444+0000 7f66d9e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x565076097dd0 0x56507609cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:49858/0 (socket says 172.21.15.37:49858) 2024-03-28T18:12:36.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.444+0000 7f66d3e00640 1 -- 172.21.15.37:0/40562453 learned_addr learned my addr 172.21.15.37:0/40562453 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:36.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.445+0000 7f66d3e00640 1 -- 172.21.15.37:0/40562453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x565076097dd0 msgr2=0x56507609cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:36.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.445+0000 7f66d3e00640 1 --2- 172.21.15.37:0/40562453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x565076097dd0 0x56507609cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.445+0000 7f66da800640 1 --2- 172.21.15.37:0/40562453 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5650760a0460 0x56507609d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:36.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.445+0000 7f66d3e00640 1 -- 172.21.15.37:0/40562453 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5650760a0460 msgr2=0x56507609d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:36.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.445+0000 7f66d3e00640 1 --2- 172.21.15.37:0/40562453 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5650760a0460 0x56507609d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.445+0000 7f66d3e00640 1 -- 172.21.15.37:0/40562453 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565075dd97d0 con 0x5650760970f0 2024-03-28T18:12:36.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.445+0000 7f66da800640 1 --2- 172.21.15.37:0/40562453 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5650760a0460 0x56507609d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:12:36.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.446+0000 7f66d3e00640 1 --2- 172.21.15.37:0/40562453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5650760970f0 0x56507609fbf0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f66cc0039c0 tx=0x7f66cc00bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b529f848d831e71f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:36.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.446+0000 7f66bf400640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:12:36.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.446+0000 7f66bf400640 1 -- 172.21.15.37:0/40562453 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66cc003e60 con 0x5650760970f0 2024-03-28T18:12:36.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.446+0000 7f66bf400640 1 -- 172.21.15.37:0/40562453 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f66cc011440 con 0x5650760970f0 2024-03-28T18:12:36.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.446+0000 7f66e3794740 1 -- 172.21.15.37:0/40562453 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f66cc015070 con 0x5650760970f0 2024-03-28T18:12:36.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.446+0000 7f66bf400640 1 -- 172.21.15.37:0/40562453 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66cc011640 con 0x5650760970f0 2024-03-28T18:12:36.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.446+0000 7f66e3794740 10 client.25676 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:36.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.446+0000 7f66e3794740 1 -- 172.21.15.37:0/40562453 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x565075d0f3f0 con 0x5650760970f0 2024-03-28T18:12:36.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.447+0000 7f66bf400640 1 -- 172.21.15.37:0/40562453 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66cc015070 con 0x5650760970f0 2024-03-28T18:12:36.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.448+0000 7f66bf400640 1 -- 172.21.15.37:0/40562453 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f66cc0259f0 con 0x5650760970f0 2024-03-28T18:12:36.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.448+0000 7f66e3794740 10 client.25676 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:36.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.448+0000 7f66e3794740 10 client.25676 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:12:36.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.448+0000 7f66e3794740 20 client.25676 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:36.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.448+0000 7f66e3794740 1 --2- 172.21.15.37:0/40562453 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x5650760b03a0 0x5650760b2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.448+0000 7f66e3794740 4 client.25676 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:12:36.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.448+0000 7f66e3794740 1 -- 172.21.15.37:0/40562453 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565076008790 con 0x5650760b03a0 2024-03-28T18:12:36.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.448+0000 7f66d9e00640 1 --2- 172.21.15.37:0/40562453 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x5650760b03a0 0x5650760b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:36.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.449+0000 7f66d9e00640 1 --2- 172.21.15.37:0/40562453 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x5650760b03a0 0x5650760b2790 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5cd95fe21310caf server_cookie=bad579723edd7f6f in_seq=0 out_seq=0 2024-03-28T18:12:36.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.449+0000 7f66bf400640 10 client.25676 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:36.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.450+0000 7f66bf400640 1 -- 172.21.15.37:0/40562453 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x565076008790 con 0x5650760b03a0 2024-03-28T18:12:36.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.450+0000 7f66bf400640 10 client.25676 handle_command_reply: tid=0 2024-03-28T18:12:36.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.450+0000 7f66e3794740 1 client.25676 shutdown 2024-03-28T18:12:36.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.450+0000 7f66e3794740 1 -- 172.21.15.37:0/40562453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5650760970f0 msgr2=0x56507609fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:36.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.450+0000 7f66e3794740 1 --2- 172.21.15.37:0/40562453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5650760970f0 0x56507609fbf0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f66cc0039c0 tx=0x7f66cc00bdb0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.451+0000 7f66e3794740 1 -- 172.21.15.37:0/40562453 shutdown_connections 2024-03-28T18:12:36.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.451+0000 7f66e3794740 1 --2- 172.21.15.37:0/40562453 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x5650760b03a0 0x5650760b2790 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.451+0000 7f66e3794740 1 --2- 172.21.15.37:0/40562453 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5650760a0460 0x56507609d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.451+0000 7f66e3794740 1 --2- 172.21.15.37:0/40562453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x565076097dd0 0x56507609cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.451+0000 7f66e3794740 1 --2- 172.21.15.37:0/40562453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5650760970f0 0x56507609fbf0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.451+0000 7f66e3794740 1 -- 172.21.15.37:0/40562453 >> 172.21.15.37:0/40562453 conn(0x56507605e200 msgr2=0x565076093890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:36.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.451+0000 7f66bf400640 0 client.25676 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:36.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.451+0000 7f66e3794740 1 -- 172.21.15.37:0/40562453 shutdown_connections 2024-03-28T18:12:36.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.451+0000 7f66e3794740 1 -- 172.21.15.37:0/40562453 wait complete. 2024-03-28T18:12:36.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.451+0000 7f66e3794740 20 client.25676 trim_cache size 0 max 16384 2024-03-28T18:12:36.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.469+0000 7f66e3794740 1 Processor -- start 2024-03-28T18:12:36.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.469+0000 7f66e3794740 1 -- start start 2024-03-28T18:12:36.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.469+0000 7f66e3794740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5650760970f0 0x56507609c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.469+0000 7f66e3794740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x565076097dd0 0x565076090a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.469+0000 7f66e3794740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5650760a0460 0x56507609f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.469+0000 7f66e3794740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc1b1690 con 0x5650760970f0 2024-03-28T18:12:36.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.469+0000 7f66e3794740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc1af190 con 0x565076097dd0 2024-03-28T18:12:36.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.469+0000 7f66e3794740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc1ad230 con 0x5650760a0460 2024-03-28T18:12:36.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.470+0000 7f66d9e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x565076097dd0 0x565076090a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:36.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.470+0000 7f66d9e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x565076097dd0 0x565076090a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:44148/0 (socket says 172.21.15.37:44148) 2024-03-28T18:12:36.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.470+0000 7f66d3e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5650760970f0 0x56507609c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:36.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.470+0000 7f66d9e00640 1 -- 172.21.15.37:0/1267142672 learned_addr learned my addr 172.21.15.37:0/1267142672 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:36.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.470+0000 7f66da800640 1 --2- 172.21.15.37:0/1267142672 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5650760a0460 0x56507609f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:36.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.470+0000 7f66d9e00640 1 -- 172.21.15.37:0/1267142672 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5650760a0460 msgr2=0x56507609f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:36.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.471+0000 7f66d9e00640 1 --2- 172.21.15.37:0/1267142672 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5650760a0460 0x56507609f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.471+0000 7f66d9e00640 1 -- 172.21.15.37:0/1267142672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5650760970f0 msgr2=0x56507609c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:36.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.471+0000 7f66d9e00640 1 --2- 172.21.15.37:0/1267142672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5650760970f0 0x56507609c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.471+0000 7f66d9e00640 1 -- 172.21.15.37:0/1267142672 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565075ccf0f0 con 0x565076097dd0 2024-03-28T18:12:36.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.471+0000 7f66d3e00640 1 --2- 172.21.15.37:0/1267142672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5650760970f0 0x56507609c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:36.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.471+0000 7f66d9e00640 1 --2- 172.21.15.37:0/1267142672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x565076097dd0 0x565076090a40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f66c40b9450 tx=0x7f66c4004ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c899b77fa558a79c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:36.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.471+0000 7f66da800640 1 --2- 172.21.15.37:0/1267142672 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5650760a0460 0x56507609f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:36.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.471+0000 7f66ca600640 1 -- 172.21.15.37:0/1267142672 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c4004c80 con 0x565076097dd0 2024-03-28T18:12:36.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.471+0000 7f66ca600640 1 -- 172.21.15.37:0/1267142672 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66c4005690 con 0x565076097dd0 2024-03-28T18:12:36.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.471+0000 7f66ca600640 1 -- 172.21.15.37:0/1267142672 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c40c1140 con 0x565076097dd0 2024-03-28T18:12:36.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.472+0000 7f66e3794740 1 -- 172.21.15.37:0/1267142672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x565076097dd0 msgr2=0x565076090a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:36.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.472+0000 7f66e3794740 1 --2- 172.21.15.37:0/1267142672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x565076097dd0 0x565076090a40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f66c40b9450 tx=0x7f66c4004ee0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.472+0000 7f66e3794740 1 -- 172.21.15.37:0/1267142672 shutdown_connections 2024-03-28T18:12:36.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.472+0000 7f66e3794740 1 --2- 172.21.15.37:0/1267142672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x565076097dd0 0x565076090a40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.472+0000 7f66e3794740 1 --2- 172.21.15.37:0/1267142672 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5650760a0460 0x56507609f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.472+0000 7f66e3794740 1 --2- 172.21.15.37:0/1267142672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5650760970f0 0x56507609c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.472+0000 7f66e3794740 1 -- 172.21.15.37:0/1267142672 >> 172.21.15.37:0/1267142672 conn(0x56507605e200 msgr2=0x565076090330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:36.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.472+0000 7f66e3794740 1 -- 172.21.15.37:0/1267142672 shutdown_connections 2024-03-28T18:12:36.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.472+0000 7f66e3794740 1 -- 172.21.15.37:0/1267142672 wait complete. 2024-03-28T18:12:36.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.473+0000 7f66e3794740 1 -- start start 2024-03-28T18:12:36.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.473+0000 7f66e3794740 1 Processor -- start 2024-03-28T18:12:36.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.474+0000 7f66e3794740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5650760970f0 0x56507609bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.474+0000 7f66e3794740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x565076097dd0 0x56507609c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.474+0000 7f66e3794740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5650760a0460 0x5650760a7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.474+0000 7f66e3794740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc1ad830 con 0x565076097dd0 2024-03-28T18:12:36.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.474+0000 7f66e3794740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc1aeb30 con 0x5650760970f0 2024-03-28T18:12:36.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.474+0000 7f66e3794740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc1ade70 con 0x5650760a0460 2024-03-28T18:12:36.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.474+0000 7f66e3794740 10 client.0 _ll_register_callbacks cb 0x7f66d4009ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:36.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.474+0000 7f66d9e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x565076097dd0 0x56507609c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:36.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.474+0000 7f66d9e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x565076097dd0 0x56507609c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36210/0 (socket says 172.21.15.37:36210) 2024-03-28T18:12:36.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.474+0000 7f66d9e00640 1 -- 172.21.15.37:0/3456494066 learned_addr learned my addr 172.21.15.37:0/3456494066 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:36.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.474+0000 7f66d3e00640 1 --2- 172.21.15.37:0/3456494066 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5650760970f0 0x56507609bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:36.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.474+0000 7f66d3e00640 1 -- 172.21.15.37:0/3456494066 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5650760a0460 msgr2=0x5650760a7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:36.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.474+0000 7f66d3e00640 1 --2- 172.21.15.37:0/3456494066 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5650760a0460 0x5650760a7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.474+0000 7f66d3e00640 1 -- 172.21.15.37:0/3456494066 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x565076097dd0 msgr2=0x56507609c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:36.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.474+0000 7f66d3e00640 1 --2- 172.21.15.37:0/3456494066 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x565076097dd0 0x56507609c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.474+0000 7f66d3e00640 1 -- 172.21.15.37:0/3456494066 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5650760a7c50 con 0x5650760970f0 2024-03-28T18:12:36.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.475+0000 7f66d3e00640 1 --2- 172.21.15.37:0/3456494066 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5650760970f0 0x56507609bf30 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f66cc011fd0 tx=0x7f66cc0254b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8b98286336e0d96 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:36.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.475+0000 7f66bf400640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:12:36.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.475+0000 7f66bf400640 1 -- 172.21.15.37:0/3456494066 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66cc032070 con 0x5650760970f0 2024-03-28T18:12:36.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.475+0000 7f66e3794740 1 -- 172.21.15.37:0/3456494066 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565076002ae0 con 0x5650760970f0 2024-03-28T18:12:36.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.475+0000 7f66bf400640 1 -- 172.21.15.37:0/3456494066 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f66cc025970 con 0x5650760970f0 2024-03-28T18:12:36.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.475+0000 7f66bf400640 1 -- 172.21.15.37:0/3456494066 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66cc025b70 con 0x5650760970f0 2024-03-28T18:12:36.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.475+0000 7f66e3794740 10 client.25682 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:36.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.475+0000 7f66e3794740 1 -- 172.21.15.37:0/3456494066 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x565076125470 con 0x5650760970f0 2024-03-28T18:12:36.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.476+0000 7f66bf400640 1 -- 172.21.15.37:0/3456494066 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x565076002ae0 con 0x5650760970f0 2024-03-28T18:12:36.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.476+0000 7f66bf400640 1 -- 172.21.15.37:0/3456494066 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f66cc003e60 con 0x5650760970f0 2024-03-28T18:12:36.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.477+0000 7f66e3794740 10 client.25682 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:36.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.477+0000 7f66e3794740 10 client.25682 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:12:36.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.477+0000 7f66e3794740 20 client.25682 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:36.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.477+0000 7f66e3794740 1 --2- 172.21.15.37:0/3456494066 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x565076125a20 0x565076127e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.477+0000 7f66e3794740 4 client.25682 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:12:36.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.477+0000 7f66e3794740 1 -- 172.21.15.37:0/3456494066 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x565075f803a0 con 0x565076125a20 2024-03-28T18:12:36.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.477+0000 7f66d9e00640 1 --2- 172.21.15.37:0/3456494066 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x565076125a20 0x565076127e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:36.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.477+0000 7f66d9e00640 1 --2- 172.21.15.37:0/3456494066 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x565076125a20 0x565076127e10 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=576cf31405e2e00e server_cookie=1d71480afa9cd408 in_seq=0 out_seq=0 2024-03-28T18:12:36.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.478+0000 7f66bf400640 10 client.25682 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:36.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.478+0000 7f66bf400640 1 -- 172.21.15.37:0/3456494066 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x565075f803a0 con 0x565076125a20 2024-03-28T18:12:36.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.478+0000 7f66bf400640 10 client.25682 handle_command_reply: tid=0 2024-03-28T18:12:36.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.478+0000 7f66e3794740 1 client.25682 shutdown 2024-03-28T18:12:36.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.478+0000 7f66e3794740 1 -- 172.21.15.37:0/3456494066 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5650760970f0 msgr2=0x56507609bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:36.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.478+0000 7f66e3794740 1 --2- 172.21.15.37:0/3456494066 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5650760970f0 0x56507609bf30 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f66cc011fd0 tx=0x7f66cc0254b0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.478+0000 7f66e3794740 1 -- 172.21.15.37:0/3456494066 shutdown_connections 2024-03-28T18:12:36.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.478+0000 7f66e3794740 1 --2- 172.21.15.37:0/3456494066 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x565076125a20 0x565076127e10 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.478+0000 7f66e3794740 1 --2- 172.21.15.37:0/3456494066 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5650760a0460 0x5650760a7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.478+0000 7f66e3794740 1 --2- 172.21.15.37:0/3456494066 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x565076097dd0 0x56507609c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.478+0000 7f66e3794740 1 --2- 172.21.15.37:0/3456494066 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5650760970f0 0x56507609bf30 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.478+0000 7f66e3794740 1 -- 172.21.15.37:0/3456494066 >> 172.21.15.37:0/3456494066 conn(0x56507605e200 msgr2=0x565076092b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:36.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.478+0000 7f66bf400640 0 client.25682 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:36.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.479+0000 7f66e3794740 1 -- 172.21.15.37:0/3456494066 shutdown_connections 2024-03-28T18:12:36.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.479+0000 7f66e3794740 1 -- 172.21.15.37:0/3456494066 wait complete. 2024-03-28T18:12:36.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.479+0000 7f66e3794740 20 client.25682 trim_cache size 0 max 16384 2024-03-28T18:12:36.481 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:12:36.481 INFO:teuthology.orchestra.run.smithi037.stdout: "status": "no active scrubs running", 2024-03-28T18:12:36.481 INFO:teuthology.orchestra.run.smithi037.stdout: "scrubs": {} 2024-03-28T18:12:36.481 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:12:36.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.480+0000 7f66db200640 1 -- 172.21.15.37:0/2145630230 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f66c00a2b10 msgr2=0x7f66c00a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:36.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.480+0000 7f66db200640 1 --2- 172.21.15.37:0/2145630230 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f66c00a2b10 0x7f66c00a4fd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f66d4005fd0 tx=0x7f66d40096a0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.480+0000 7f66db200640 1 -- 172.21.15.37:0/2145630230 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66dc101ef0 msgr2=0x7f66dc19a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:36.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.480+0000 7f66db200640 1 --2- 172.21.15.37:0/2145630230 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66dc101ef0 0x7f66dc19a4f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f66c4009900 tx=0x7f66c4004430 comp rx=0 tx=0).stop 2024-03-28T18:12:36.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.480+0000 7f66db200640 1 -- 172.21.15.37:0/2145630230 shutdown_connections 2024-03-28T18:12:36.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.480+0000 7f66db200640 1 --2- 172.21.15.37:0/2145630230 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f66c00a2b10 0x7f66c00a4fd0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.480+0000 7f66db200640 1 --2- 172.21.15.37:0/2145630230 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66dc101ef0 0x7f66dc19a4f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.480+0000 7f66db200640 1 --2- 172.21.15.37:0/2145630230 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66dc1075f0 0x7f66dc19e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.480+0000 7f66db200640 1 --2- 172.21.15.37:0/2145630230 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66dc101530 0x7f66dc199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.480+0000 7f66db200640 1 -- 172.21.15.37:0/2145630230 >> 172.21.15.37:0/2145630230 conn(0x7f66dc0fd040 msgr2=0x7f66dc0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:36.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.480+0000 7f66db200640 1 -- 172.21.15.37:0/2145630230 shutdown_connections 2024-03-28T18:12:36.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.480+0000 7f66db200640 1 -- 172.21.15.37:0/2145630230 wait complete. 2024-03-28T18:12:36.498 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:12:36.498 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:12:36.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.733+0000 7f31cb200640 1 -- 172.21.15.37:0/4259273915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31c4107590 msgr2=0x7f31c4107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:36.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.733+0000 7f31cb200640 1 --2- 172.21.15.37:0/4259273915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31c4107590 0x7f31c4107970 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f31b8009900 tx=0x7f31b802bc30 comp rx=0 tx=0).stop 2024-03-28T18:12:36.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.733+0000 7f31cb200640 1 -- 172.21.15.37:0/4259273915 shutdown_connections 2024-03-28T18:12:36.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.733+0000 7f31cb200640 1 --2- 172.21.15.37:0/4259273915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31c40691c0 0x7f31c410e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.733+0000 7f31cb200640 1 --2- 172.21.15.37:0/4259273915 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31c4068800 0x7f31c4068c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.733+0000 7f31cb200640 1 --2- 172.21.15.37:0/4259273915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31c4107590 0x7f31c4107970 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.733+0000 7f31cb200640 1 -- 172.21.15.37:0/4259273915 >> 172.21.15.37:0/4259273915 conn(0x7f31c4075fb0 msgr2=0x7f31c40763c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:36.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.733+0000 7f31cb200640 1 -- 172.21.15.37:0/4259273915 shutdown_connections 2024-03-28T18:12:36.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.734+0000 7f31cb200640 1 -- 172.21.15.37:0/4259273915 wait complete. 2024-03-28T18:12:36.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.734+0000 7f31cb200640 1 Processor -- start 2024-03-28T18:12:36.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.735+0000 7f31cb200640 1 -- start start 2024-03-28T18:12:36.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.735+0000 7f31cb200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31c4068800 0x7f31c4199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.735+0000 7f31cb200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31c40691c0 0x7f31c419a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.735+0000 7f31cb200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31c4107590 0x7f31c419e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.735+0000 7f31cb200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c41118a0 con 0x7f31c4068800 2024-03-28T18:12:36.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.735+0000 7f31cb200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c4111720 con 0x7f31c40691c0 2024-03-28T18:12:36.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.735+0000 7f31cb200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c4111a20 con 0x7f31c4107590 2024-03-28T18:12:36.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.735+0000 7f31c9e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31c4068800 0x7f31c4199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:36.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.735+0000 7f31c9e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31c4068800 0x7f31c4199e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36230/0 (socket says 172.21.15.37:36230) 2024-03-28T18:12:36.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.736+0000 7f31c9e00640 1 -- 172.21.15.37:0/1226724944 learned_addr learned my addr 172.21.15.37:0/1226724944 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:36.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.736+0000 7f31c9400640 1 --2- 172.21.15.37:0/1226724944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31c40691c0 0x7f31c419a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:36.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.736+0000 7f31ca800640 1 --2- 172.21.15.37:0/1226724944 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31c4107590 0x7f31c419e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:36.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.736+0000 7f31c9e00640 1 -- 172.21.15.37:0/1226724944 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31c4107590 msgr2=0x7f31c419e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:36.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.736+0000 7f31c9e00640 1 --2- 172.21.15.37:0/1226724944 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31c4107590 0x7f31c419e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.736+0000 7f31c9e00640 1 -- 172.21.15.37:0/1226724944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31c40691c0 msgr2=0x7f31c419a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:36.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.736+0000 7f31c9e00640 1 --2- 172.21.15.37:0/1226724944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31c40691c0 0x7f31c419a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:36.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.736+0000 7f31c9e00640 1 -- 172.21.15.37:0/1226724944 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c419ee70 con 0x7f31c4068800 2024-03-28T18:12:36.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.736+0000 7f31c9400640 1 --2- 172.21.15.37:0/1226724944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31c40691c0 0x7f31c419a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:36.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.737+0000 7f31c9e00640 1 --2- 172.21.15.37:0/1226724944 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31c4068800 0x7f31c4199e80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f31b802efd0 tx=0x7f31b8004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:36.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.737+0000 7f31bea00640 1 -- 172.21.15.37:0/1226724944 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b8041070 con 0x7f31c4068800 2024-03-28T18:12:36.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.737+0000 7f31cb200640 1 -- 172.21.15.37:0/1226724944 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c419f100 con 0x7f31c4068800 2024-03-28T18:12:36.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.737+0000 7f31cb200640 1 -- 172.21.15.37:0/1226724944 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c41a69f0 con 0x7f31c4068800 2024-03-28T18:12:36.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.737+0000 7f31bea00640 1 -- 172.21.15.37:0/1226724944 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31b80054e0 con 0x7f31c4068800 2024-03-28T18:12:36.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.737+0000 7f31bea00640 1 -- 172.21.15.37:0/1226724944 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b804b060 con 0x7f31c4068800 2024-03-28T18:12:36.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.740+0000 7f31cb200640 1 -- 172.21.15.37:0/1226724944 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a8005290 con 0x7f31c4068800 2024-03-28T18:12:36.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.741+0000 7f31bea00640 1 -- 172.21.15.37:0/1226724944 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f31b8044070 con 0x7f31c4068800 2024-03-28T18:12:36.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.743+0000 7f31bea00640 1 --2- 172.21.15.37:0/1226724944 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f31ac0a2b10 0x7f31ac0a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:36.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.743+0000 7f31bea00640 1 -- 172.21.15.37:0/1226724944 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f31b80f0a50 con 0x7f31c4068800 2024-03-28T18:12:36.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.744+0000 7f31c9400640 1 --2- 172.21.15.37:0/1226724944 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f31ac0a2b10 0x7f31ac0a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:36.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.745+0000 7f31c9400640 1 --2- 172.21.15.37:0/1226724944 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f31ac0a2b10 0x7f31ac0a4fd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f31c410e7c0 tx=0x7f31b400b390 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:36.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:36.748+0000 7f31bea00640 1 -- 172.21.15.37:0/1226724944 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f31b80bd050 con 0x7f31c4068800 2024-03-28T18:12:37.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.224+0000 7f31cb200640 1 -- 172.21.15.37:0/1226724944 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31a8005850 con 0x7f31c4068800 2024-03-28T18:12:37.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.226+0000 7f31bea00640 1 -- 172.21.15.37:0/1226724944 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11383 (secure 0 0 0) 0x7f31b80c10f0 con 0x7f31c4068800 2024-03-28T18:12:37.226 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:12:37.226 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1926270371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6838","nonce":1926270371},{"type":"v1","addr":"172.21.15.37:6839","nonce":1926270371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:08:19.968894+0000","modified":"2024-03-28T18:10:49.565429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24580,"mds_1":24569,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6837/4034607523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":4034607523},{"type":"v1","addr":"172.21.15.59:6837","nonce":4034607523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6839/1575518279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":1575518279},{"type":"v1","addr":"172.21.15.170:6839","nonce":1575518279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6839/2091629078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":2091629078},{"type":"v1","addr":"172.21.15.59:6839","nonce":2091629078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24580":{"gid":24580,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.37:6841/1923263622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6840","nonce":1923263622},{"type":"v1","addr":"172.21.15.37:6841","nonce":1923263622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24580,"qdb_cluster":[24580,24550,24569]},"id":1}]} 2024-03-28T18:12:37.227 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 23 2024-03-28T18:12:37.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.229+0000 7f31cb200640 1 -- 172.21.15.37:0/1226724944 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f31ac0a2b10 msgr2=0x7f31ac0a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:37.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.229+0000 7f31cb200640 1 --2- 172.21.15.37:0/1226724944 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f31ac0a2b10 0x7f31ac0a4fd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f31c410e7c0 tx=0x7f31b400b390 comp rx=0 tx=0).stop 2024-03-28T18:12:37.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.229+0000 7f31cb200640 1 -- 172.21.15.37:0/1226724944 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31c4068800 msgr2=0x7f31c4199e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:37.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.229+0000 7f31cb200640 1 --2- 172.21.15.37:0/1226724944 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31c4068800 0x7f31c4199e80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f31b802efd0 tx=0x7f31b8004430 comp rx=0 tx=0).stop 2024-03-28T18:12:37.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.229+0000 7f31cb200640 1 -- 172.21.15.37:0/1226724944 shutdown_connections 2024-03-28T18:12:37.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.229+0000 7f31cb200640 1 --2- 172.21.15.37:0/1226724944 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f31ac0a2b10 0x7f31ac0a4fd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.229+0000 7f31cb200640 1 --2- 172.21.15.37:0/1226724944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31c40691c0 0x7f31c419a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.229+0000 7f31cb200640 1 --2- 172.21.15.37:0/1226724944 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31c4107590 0x7f31c419e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.229+0000 7f31cb200640 1 --2- 172.21.15.37:0/1226724944 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31c4068800 0x7f31c4199e80 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.229+0000 7f31cb200640 1 -- 172.21.15.37:0/1226724944 >> 172.21.15.37:0/1226724944 conn(0x7f31c4075fb0 msgr2=0x7f31c4100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:37.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.229+0000 7f31cb200640 1 -- 172.21.15.37:0/1226724944 shutdown_connections 2024-03-28T18:12:37.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.229+0000 7f31cb200640 1 -- 172.21.15.37:0/1226724944 wait complete. 2024-03-28T18:12:37.247 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:12:37.247 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:12:37.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.475+0000 7fd8c9c00640 1 -- 172.21.15.37:0/4016207383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8c4073a50 msgr2=0x7fd8c4073e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:37.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.475+0000 7fd8c9c00640 1 --2- 172.21.15.37:0/4016207383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8c4073a50 0x7fd8c4073e30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009900 tx=0x7fd8b402bcd0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.475+0000 7fd8c9c00640 1 -- 172.21.15.37:0/4016207383 shutdown_connections 2024-03-28T18:12:37.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.475+0000 7fd8c9c00640 1 --2- 172.21.15.37:0/4016207383 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8c410ddd0 0x7fd8c41101c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.475+0000 7fd8c9c00640 1 --2- 172.21.15.37:0/4016207383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c4074370 0x7fd8c410d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.475+0000 7fd8c9c00640 1 --2- 172.21.15.37:0/4016207383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8c4073a50 0x7fd8c4073e30 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.475+0000 7fd8c9c00640 1 -- 172.21.15.37:0/4016207383 >> 172.21.15.37:0/4016207383 conn(0x7fd8c40fd040 msgr2=0x7fd8c40ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:37.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.475+0000 7fd8c9c00640 1 -- 172.21.15.37:0/4016207383 shutdown_connections 2024-03-28T18:12:37.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.475+0000 7fd8c9c00640 1 -- 172.21.15.37:0/4016207383 wait complete. 2024-03-28T18:12:37.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.476+0000 7fd8c9c00640 1 Processor -- start 2024-03-28T18:12:37.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.477+0000 7fd8c9c00640 1 -- start start 2024-03-28T18:12:37.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.477+0000 7fd8c9c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8c4073a50 0x7fd8c419a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.477+0000 7fd8c9c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8c4074370 0x7fd8c419a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.477+0000 7fd8c9c00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c410ddd0 0x7fd8c419e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.477+0000 7fd8c9c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c4113b40 con 0x7fd8c4073a50 2024-03-28T18:12:37.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.477+0000 7fd8c3400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8c4074370 0x7fd8c419a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.477+0000 7fd8c9c00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c41139c0 con 0x7fd8c4074370 2024-03-28T18:12:37.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.477+0000 7fd8c3e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8c4073a50 0x7fd8c419a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.477+0000 7fd8c9c00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c4113cc0 con 0x7fd8c410ddd0 2024-03-28T18:12:37.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.477+0000 7fd8c3400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8c4074370 0x7fd8c419a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:44178/0 (socket says 172.21.15.37:44178) 2024-03-28T18:12:37.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.477+0000 7fd8c3400640 1 -- 172.21.15.37:0/3315931785 learned_addr learned my addr 172.21.15.37:0/3315931785 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:37.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.478+0000 7fd8c9200640 1 --2- 172.21.15.37:0/3315931785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c410ddd0 0x7fd8c419e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.478+0000 7fd8c3400640 1 -- 172.21.15.37:0/3315931785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c410ddd0 msgr2=0x7fd8c419e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:37.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.478+0000 7fd8c3400640 1 --2- 172.21.15.37:0/3315931785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c410ddd0 0x7fd8c419e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.478+0000 7fd8c3400640 1 -- 172.21.15.37:0/3315931785 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8c4073a50 msgr2=0x7fd8c419a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:37.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.478+0000 7fd8c3400640 1 --2- 172.21.15.37:0/3315931785 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8c4073a50 0x7fd8c419a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.478+0000 7fd8c3400640 1 -- 172.21.15.37:0/3315931785 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c419eff0 con 0x7fd8c4074370 2024-03-28T18:12:37.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.478+0000 7fd8c3e00640 1 --2- 172.21.15.37:0/3315931785 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8c4073a50 0x7fd8c419a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:12:37.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.478+0000 7fd8c3400640 1 --2- 172.21.15.37:0/3315931785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8c4074370 0x7fd8c419a540 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd8b000b520 tx=0x7fd8b000b9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:37.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.478+0000 7fd8c0c00640 1 -- 172.21.15.37:0/3315931785 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0004020 con 0x7fd8c4074370 2024-03-28T18:12:37.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.479+0000 7fd8c0c00640 1 -- 172.21.15.37:0/3315931785 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8b00041c0 con 0x7fd8c4074370 2024-03-28T18:12:37.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.479+0000 7fd8c0c00640 1 -- 172.21.15.37:0/3315931785 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b000a950 con 0x7fd8c4074370 2024-03-28T18:12:37.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.479+0000 7fd8c9c00640 1 -- 172.21.15.37:0/3315931785 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c419f2e0 con 0x7fd8c4074370 2024-03-28T18:12:37.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.479+0000 7fd8c9c00640 1 -- 172.21.15.37:0/3315931785 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c4101420 con 0x7fd8c4074370 2024-03-28T18:12:37.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.482+0000 7fd8cdf6d740 1 Processor -- start 2024-03-28T18:12:37.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.483+0000 7fd8cdf6d740 1 -- start start 2024-03-28T18:12:37.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.484+0000 7fd8cdf6d740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f4710 0x563d8d3f4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.484+0000 7fd8cdf6d740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d3f50c0 0x563d8d40a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.484+0000 7fd8cdf6d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d40ae20 0x563d8d40b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.484+0000 7fd8cdf6d740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d8d3582c0 con 0x563d8d3f4710 2024-03-28T18:12:37.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.484+0000 7fd8cdf6d740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d8d300c90 con 0x563d8d3f50c0 2024-03-28T18:12:37.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.484+0000 7fd8cdf6d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d8d354360 con 0x563d8d40ae20 2024-03-28T18:12:37.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.484+0000 7fd8c3e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d3f50c0 0x563d8d40a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.484+0000 7fd8c3e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d3f50c0 0x563d8d40a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:44182/0 (socket says 172.21.15.37:44182) 2024-03-28T18:12:37.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.484+0000 7fd8c3e00640 1 -- 172.21.15.37:0/960978771 learned_addr learned my addr 172.21.15.37:0/960978771 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:37.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.484+0000 7fd8c9200640 1 --2- 172.21.15.37:0/960978771 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f4710 0x563d8d3f4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.485+0000 7fd8c9200640 1 -- 172.21.15.37:0/960978771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d40ae20 msgr2=0x563d8d40b2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:12:37.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.485+0000 7fd8c9200640 1 --2- 172.21.15.37:0/960978771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d40ae20 0x563d8d40b2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.485+0000 7fd8c9200640 1 -- 172.21.15.37:0/960978771 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d3f50c0 msgr2=0x563d8d40a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:37.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.485+0000 7fd8c9200640 1 --2- 172.21.15.37:0/960978771 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d3f50c0 0x563d8d40a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.485+0000 7fd8c9200640 1 -- 172.21.15.37:0/960978771 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d8d325080 con 0x563d8d3f4710 2024-03-28T18:12:37.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.485+0000 7fd8c9200640 1 --2- 172.21.15.37:0/960978771 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f4710 0x563d8d3f4af0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc009e40 tx=0x7fd8bc00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2232ba02d7d9b299 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:37.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.485+0000 7fd8c0c00640 1 -- 172.21.15.37:0/3315931785 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7fd8b000aaf0 con 0x7fd8c4074370 2024-03-28T18:12:37.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.486+0000 7fd8b9600640 1 -- 172.21.15.37:0/960978771 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc015070 con 0x563d8d3f4710 2024-03-28T18:12:37.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.486+0000 7fd8b9600640 1 -- 172.21.15.37:0/960978771 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8bc01ab40 con 0x563d8d3f4710 2024-03-28T18:12:37.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.486+0000 7fd8b9600640 1 -- 172.21.15.37:0/960978771 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc011bb0 con 0x563d8d3f4710 2024-03-28T18:12:37.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.487+0000 7fd8c0c00640 1 --2- 172.21.15.37:0/3315931785 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fd8a80a2b90 0x7fd8a80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.488+0000 7fd8c0c00640 1 -- 172.21.15.37:0/3315931785 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd8b00ce7a0 con 0x7fd8c4074370 2024-03-28T18:12:37.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.488+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/960978771 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f4710 msgr2=0x563d8d3f4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:37.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.488+0000 7fd8cdf6d740 1 --2- 172.21.15.37:0/960978771 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f4710 0x563d8d3f4af0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc009e40 tx=0x7fd8bc00f730 comp rx=0 tx=0).stop 2024-03-28T18:12:37.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.488+0000 7fd8c3e00640 1 --2- 172.21.15.37:0/3315931785 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fd8a80a2b90 0x7fd8a80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.488+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/960978771 shutdown_connections 2024-03-28T18:12:37.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.488+0000 7fd8cdf6d740 1 --2- 172.21.15.37:0/960978771 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d3f50c0 0x563d8d40a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.488+0000 7fd8cdf6d740 1 --2- 172.21.15.37:0/960978771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d40ae20 0x563d8d40b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.488+0000 7fd8cdf6d740 1 --2- 172.21.15.37:0/960978771 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f4710 0x563d8d3f4af0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.488+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/960978771 >> 172.21.15.37:0/960978771 conn(0x563d8d3eef80 msgr2=0x563d8d3ef360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:37.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.489+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/960978771 shutdown_connections 2024-03-28T18:12:37.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.489+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/960978771 wait complete. 2024-03-28T18:12:37.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.490+0000 7fd8c3e00640 1 --2- 172.21.15.37:0/3315931785 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fd8a80a2b90 0x7fd8a80a5050 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4005ab0 tx=0x7fd8b4005a20 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:37.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.490+0000 7fd8cdf6d740 1 -- start start 2024-03-28T18:12:37.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.490+0000 7fd8cdf6d740 1 Processor -- start 2024-03-28T18:12:37.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.491+0000 7fd8cdf6d740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d3f4710 0x563d8d411250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.491+0000 7fd8cdf6d740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f50c0 0x563d8d40e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.491+0000 7fd8cdf6d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d40ae20 0x563d8d40eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.491+0000 7fd8cdf6d740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d8d411da0 con 0x563d8d3f50c0 2024-03-28T18:12:37.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.491+0000 7fd8cdf6d740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d8d3140d0 con 0x563d8d3f4710 2024-03-28T18:12:37.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.491+0000 7fd8cdf6d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d8d411f20 con 0x563d8d40ae20 2024-03-28T18:12:37.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.491+0000 7fd8cdf6d740 10 client.0 _ll_register_callbacks cb 0x563d8d318e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:37.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.491+0000 7fd8c9200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d40ae20 0x563d8d40eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.491+0000 7fd8c9200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d40ae20 0x563d8d40eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:49956/0 (socket says 172.21.15.37:49956) 2024-03-28T18:12:37.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.491+0000 7fd8c9200640 1 -- 172.21.15.37:0/656201301 learned_addr learned my addr 172.21.15.37:0/656201301 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:37.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.491+0000 7fd8c3400640 1 --2- 172.21.15.37:0/656201301 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f50c0 0x563d8d40e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.492+0000 7fd8c3e00640 1 --2- 172.21.15.37:0/656201301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d3f4710 0x563d8d411250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.492+0000 7fd8c9200640 1 -- 172.21.15.37:0/656201301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d3f4710 msgr2=0x563d8d411250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:37.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.492+0000 7fd8c9200640 1 --2- 172.21.15.37:0/656201301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d3f4710 0x563d8d411250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.492+0000 7fd8c9200640 1 -- 172.21.15.37:0/656201301 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f50c0 msgr2=0x563d8d40e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:37.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.492+0000 7fd8c9200640 1 --2- 172.21.15.37:0/656201301 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f50c0 0x563d8d40e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.492+0000 7fd8c9200640 1 -- 172.21.15.37:0/656201301 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d8d06c3f0 con 0x563d8d40ae20 2024-03-28T18:12:37.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.492+0000 7fd8c3400640 1 --2- 172.21.15.37:0/656201301 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f50c0 0x563d8d40e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:12:37.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.492+0000 7fd8c9200640 1 --2- 172.21.15.37:0/656201301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d40ae20 0x563d8d40eb80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc0114f0 tx=0x7fd8bc00ff00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7edefc1dc77a050e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:37.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.492+0000 7fd8ae000640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:12:37.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.492+0000 7fd8ae000640 1 -- 172.21.15.37:0/656201301 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc028020 con 0x563d8d40ae20 2024-03-28T18:12:37.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.493+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/656201301 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd8bc01ab40 con 0x563d8d40ae20 2024-03-28T18:12:37.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.493+0000 7fd8ae000640 1 -- 172.21.15.37:0/656201301 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd8bc003b70 con 0x563d8d40ae20 2024-03-28T18:12:37.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.493+0000 7fd8ae000640 1 -- 172.21.15.37:0/656201301 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8bc015050 con 0x563d8d40ae20 2024-03-28T18:12:37.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.493+0000 7fd8cdf6d740 10 client.25714 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:37.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.493+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/656201301 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563d8d1367d0 con 0x563d8d40ae20 2024-03-28T18:12:37.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.493+0000 7fd8ae000640 1 -- 172.21.15.37:0/656201301 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8bc01ab40 con 0x563d8d40ae20 2024-03-28T18:12:37.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.493+0000 7fd8ae000640 1 -- 172.21.15.37:0/656201301 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd8bc023070 con 0x563d8d40ae20 2024-03-28T18:12:37.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.494+0000 7fd8cdf6d740 10 client.25714 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:37.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.494+0000 7fd8cdf6d740 10 client.25714 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-28T18:12:37.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.494+0000 7fd8cdf6d740 20 client.25714 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:37.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.494+0000 7fd8cdf6d740 1 --2- 172.21.15.37:0/656201301 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x563d8d41ebe0 0x563d8d420fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.494+0000 7fd8cdf6d740 4 client.25714 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:12:37.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.494+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/656201301 --> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563d8d365800 con 0x563d8d41ebe0 2024-03-28T18:12:37.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.494+0000 7fd8c3e00640 1 --2- 172.21.15.37:0/656201301 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x563d8d41ebe0 0x563d8d420fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.495+0000 7fd8c3e00640 1 --2- 172.21.15.37:0/656201301 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x563d8d41ebe0 0x563d8d420fd0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=28c6b658f8c0edda server_cookie=2fbeaae05e8efaab in_seq=0 out_seq=0 2024-03-28T18:12:37.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.495+0000 7fd8ae000640 10 client.25714 ms_handle_connect on v2:172.21.15.170:6838/1575518279 2024-03-28T18:12:37.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.496+0000 7fd8ae000640 1 -- 172.21.15.37:0/656201301 <== mds.2 v2:172.21.15.170:6838/1575518279 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563d8d365800 con 0x563d8d41ebe0 2024-03-28T18:12:37.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.496+0000 7fd8ae000640 10 client.25714 handle_command_reply: tid=0 2024-03-28T18:12:37.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.496+0000 7fd8cdf6d740 1 client.25714 shutdown 2024-03-28T18:12:37.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.496+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/656201301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d40ae20 msgr2=0x563d8d40eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:37.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.496+0000 7fd8cdf6d740 1 --2- 172.21.15.37:0/656201301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d40ae20 0x563d8d40eb80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc0114f0 tx=0x7fd8bc00ff00 comp rx=0 tx=0).stop 2024-03-28T18:12:37.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.496+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/656201301 shutdown_connections 2024-03-28T18:12:37.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.496+0000 7fd8cdf6d740 1 --2- 172.21.15.37:0/656201301 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x563d8d41ebe0 0x563d8d420fd0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.496+0000 7fd8cdf6d740 1 --2- 172.21.15.37:0/656201301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d40ae20 0x563d8d40eb80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.496+0000 7fd8cdf6d740 1 --2- 172.21.15.37:0/656201301 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f50c0 0x563d8d40e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.496+0000 7fd8cdf6d740 1 --2- 172.21.15.37:0/656201301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d3f4710 0x563d8d411250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.496+0000 7fd8ae000640 0 client.25714 ms_handle_reset on v2:172.21.15.170:6838/1575518279 2024-03-28T18:12:37.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.496+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/656201301 >> 172.21.15.37:0/656201301 conn(0x563d8d3eef80 msgr2=0x563d8d3f0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:37.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.497+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/656201301 shutdown_connections 2024-03-28T18:12:37.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.497+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/656201301 wait complete. 2024-03-28T18:12:37.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.497+0000 7fd8cdf6d740 20 client.25714 trim_cache size 0 max 16384 2024-03-28T18:12:37.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.523+0000 7fd8cdf6d740 1 Processor -- start 2024-03-28T18:12:37.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.523+0000 7fd8cdf6d740 1 -- start start 2024-03-28T18:12:37.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.523+0000 7fd8cdf6d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d3f4710 0x563d8d410d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.523+0000 7fd8cdf6d740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f50c0 0x563d8d411250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.523+0000 7fd8cdf6d740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d40ae20 0x563d8d408b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.523+0000 7fd8cdf6d740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c41ad3c0 con 0x563d8d3f50c0 2024-03-28T18:12:37.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.523+0000 7fd8cdf6d740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c41aaec0 con 0x563d8d40ae20 2024-03-28T18:12:37.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.523+0000 7fd8cdf6d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c41a8f80 con 0x563d8d3f4710 2024-03-28T18:12:37.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.524+0000 7fd8c3e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d3f4710 0x563d8d410d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.524+0000 7fd8c3e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d3f4710 0x563d8d410d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:49972/0 (socket says 172.21.15.37:49972) 2024-03-28T18:12:37.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.524+0000 7fd8c3e00640 1 -- 172.21.15.37:0/1527504388 learned_addr learned my addr 172.21.15.37:0/1527504388 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:37.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.524+0000 7fd8c3400640 1 --2- 172.21.15.37:0/1527504388 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f50c0 0x563d8d411250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.524+0000 7fd8c9200640 1 --2- 172.21.15.37:0/1527504388 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d40ae20 0x563d8d408b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.524+0000 7fd8c3e00640 1 -- 172.21.15.37:0/1527504388 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d40ae20 msgr2=0x563d8d408b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:37.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.524+0000 7fd8c3e00640 1 --2- 172.21.15.37:0/1527504388 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d40ae20 0x563d8d408b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.524+0000 7fd8c3e00640 1 -- 172.21.15.37:0/1527504388 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f50c0 msgr2=0x563d8d411250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:37.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.524+0000 7fd8c3e00640 1 --2- 172.21.15.37:0/1527504388 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f50c0 0x563d8d411250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.524+0000 7fd8c3e00640 1 -- 172.21.15.37:0/1527504388 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d8d40ab70 con 0x563d8d3f4710 2024-03-28T18:12:37.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.525+0000 7fd8c3e00640 1 --2- 172.21.15.37:0/1527504388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d3f4710 0x563d8d410d10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd8b402c2e0 tx=0x7fd8b403e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67c2907e8e3032be server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:37.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.526+0000 7fd8b8c00640 1 -- 172.21.15.37:0/1527504388 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b404e310 con 0x563d8d3f4710 2024-03-28T18:12:37.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.526+0000 7fd8b8c00640 1 -- 172.21.15.37:0/1527504388 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8b4044040 con 0x563d8d3f4710 2024-03-28T18:12:37.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.526+0000 7fd8b8c00640 1 -- 172.21.15.37:0/1527504388 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4004d30 con 0x563d8d3f4710 2024-03-28T18:12:37.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.527+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/1527504388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d3f4710 msgr2=0x563d8d410d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:37.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.527+0000 7fd8cdf6d740 1 --2- 172.21.15.37:0/1527504388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d3f4710 0x563d8d410d10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd8b402c2e0 tx=0x7fd8b403e000 comp rx=0 tx=0).stop 2024-03-28T18:12:37.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.527+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/1527504388 shutdown_connections 2024-03-28T18:12:37.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.527+0000 7fd8cdf6d740 1 --2- 172.21.15.37:0/1527504388 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d40ae20 0x563d8d408b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.527+0000 7fd8cdf6d740 1 --2- 172.21.15.37:0/1527504388 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f50c0 0x563d8d411250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.527+0000 7fd8cdf6d740 1 --2- 172.21.15.37:0/1527504388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d3f4710 0x563d8d410d10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.527+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/1527504388 >> 172.21.15.37:0/1527504388 conn(0x563d8d3eef80 msgr2=0x563d8d3ee160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:37.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.527+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/1527504388 shutdown_connections 2024-03-28T18:12:37.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.527+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/1527504388 wait complete. 2024-03-28T18:12:37.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.527+0000 7fd8cdf6d740 1 -- start start 2024-03-28T18:12:37.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.528+0000 7fd8cdf6d740 1 Processor -- start 2024-03-28T18:12:37.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.528+0000 7fd8cdf6d740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f50c0 0x563d8d4170e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.528+0000 7fd8cdf6d740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d40ae20 0x563d8d4153e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.528+0000 7fd8cdf6d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d415a30 0x563d8d415eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.528+0000 7fd8cdf6d740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c41a9580 con 0x563d8d3f50c0 2024-03-28T18:12:37.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.528+0000 7fd8cdf6d740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c41aa860 con 0x563d8d40ae20 2024-03-28T18:12:37.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.528+0000 7fd8cdf6d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c41a9ba0 con 0x563d8d415a30 2024-03-28T18:12:37.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.529+0000 7fd8cdf6d740 10 client.0 _ll_register_callbacks cb 0x7fd8bc009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:37.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.529+0000 7fd8c3400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d40ae20 0x563d8d4153e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.529+0000 7fd8c3400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d40ae20 0x563d8d4153e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:44210/0 (socket says 172.21.15.37:44210) 2024-03-28T18:12:37.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.529+0000 7fd8c3400640 1 -- 172.21.15.37:0/3683868865 learned_addr learned my addr 172.21.15.37:0/3683868865 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:37.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.529+0000 7fd8c3e00640 1 --2- 172.21.15.37:0/3683868865 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f50c0 0x563d8d4170e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.529+0000 7fd8c9200640 1 --2- 172.21.15.37:0/3683868865 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d415a30 0x563d8d415eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.529+0000 7fd8c3400640 1 -- 172.21.15.37:0/3683868865 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d415a30 msgr2=0x563d8d415eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:37.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.529+0000 7fd8c3400640 1 --2- 172.21.15.37:0/3683868865 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d415a30 0x563d8d415eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.529+0000 7fd8c3400640 1 -- 172.21.15.37:0/3683868865 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f50c0 msgr2=0x563d8d4170e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:37.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.529+0000 7fd8c3400640 1 --2- 172.21.15.37:0/3683868865 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f50c0 0x563d8d4170e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.529+0000 7fd8c3400640 1 -- 172.21.15.37:0/3683868865 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d8d47a3c0 con 0x563d8d40ae20 2024-03-28T18:12:37.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.529+0000 7fd8c3e00640 1 --2- 172.21.15.37:0/3683868865 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f50c0 0x563d8d4170e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:12:37.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.529+0000 7fd8c9200640 1 --2- 172.21.15.37:0/3683868865 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d415a30 0x563d8d415eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:12:37.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.530+0000 7fd8c3400640 1 --2- 172.21.15.37:0/3683868865 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d40ae20 0x563d8d4153e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0004770 tx=0x7fd8b0009530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2836aaf9994bc52b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:37.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.530+0000 7fd8ae000640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:12:37.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.530+0000 7fd8ae000640 1 -- 172.21.15.37:0/3683868865 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b00a74d0 con 0x563d8d40ae20 2024-03-28T18:12:37.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.530+0000 7fd8ae000640 1 -- 172.21.15.37:0/3683868865 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd8b00a7130 con 0x563d8d40ae20 2024-03-28T18:12:37.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.530+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/3683868865 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd8b4044040 con 0x563d8d40ae20 2024-03-28T18:12:37.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.530+0000 7fd8ae000640 1 -- 172.21.15.37:0/3683868865 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8b000df40 con 0x563d8d40ae20 2024-03-28T18:12:37.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.531+0000 7fd8cdf6d740 10 client.25706 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:37.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.531+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/3683868865 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563d8d47b6c0 con 0x563d8d40ae20 2024-03-28T18:12:37.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.531+0000 7fd8ae000640 1 -- 172.21.15.37:0/3683868865 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8b4044040 con 0x563d8d40ae20 2024-03-28T18:12:37.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.531+0000 7fd8ae000640 1 -- 172.21.15.37:0/3683868865 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd8b00a2240 con 0x563d8d40ae20 2024-03-28T18:12:37.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.531+0000 7fd8cdf6d740 10 client.25706 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:37.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.531+0000 7fd8cdf6d740 10 client.25706 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-28T18:12:37.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.531+0000 7fd8cdf6d740 20 client.25706 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:37.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.531+0000 7fd8cdf6d740 1 --2- 172.21.15.37:0/3683868865 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x563d8d47bdd0 0x563d8d47e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.531+0000 7fd8cdf6d740 4 client.25706 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:12:37.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.531+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/3683868865 --> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563d8d35fb50 con 0x563d8d47bdd0 2024-03-28T18:12:37.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.532+0000 7fd8c3e00640 1 --2- 172.21.15.37:0/3683868865 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x563d8d47bdd0 0x563d8d47e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.533+0000 7fd8c3e00640 1 --2- 172.21.15.37:0/3683868865 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x563d8d47bdd0 0x563d8d47e1f0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e7a0f9fb54777348 server_cookie=aeb45e5acc6e97b3 in_seq=0 out_seq=0 2024-03-28T18:12:37.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.533+0000 7fd8ae000640 10 client.25706 ms_handle_connect on v2:172.21.15.170:6838/1575518279 2024-03-28T18:12:37.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.533+0000 7fd8ae000640 1 -- 172.21.15.37:0/3683868865 <== mds.2 v2:172.21.15.170:6838/1575518279 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563d8d35fb50 con 0x563d8d47bdd0 2024-03-28T18:12:37.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.533+0000 7fd8ae000640 10 client.25706 handle_command_reply: tid=0 2024-03-28T18:12:37.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.533+0000 7fd8cdf6d740 1 client.25706 shutdown 2024-03-28T18:12:37.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.533+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/3683868865 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d40ae20 msgr2=0x563d8d4153e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:37.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.533+0000 7fd8cdf6d740 1 --2- 172.21.15.37:0/3683868865 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d40ae20 0x563d8d4153e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0004770 tx=0x7fd8b0009530 comp rx=0 tx=0).stop 2024-03-28T18:12:37.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.533+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/3683868865 shutdown_connections 2024-03-28T18:12:37.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.533+0000 7fd8cdf6d740 1 --2- 172.21.15.37:0/3683868865 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x563d8d47bdd0 0x563d8d47e1f0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.533+0000 7fd8cdf6d740 1 --2- 172.21.15.37:0/3683868865 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563d8d40ae20 0x563d8d4153e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.533+0000 7fd8cdf6d740 1 --2- 172.21.15.37:0/3683868865 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563d8d415a30 0x563d8d415eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.533+0000 7fd8cdf6d740 1 --2- 172.21.15.37:0/3683868865 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x563d8d3f50c0 0x563d8d4170e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.533+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/3683868865 >> 172.21.15.37:0/3683868865 conn(0x563d8d3eef80 msgr2=0x563d8d3f0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:37.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.533+0000 7fd8ae000640 0 client.25706 ms_handle_reset on v2:172.21.15.170:6838/1575518279 2024-03-28T18:12:37.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.534+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/3683868865 shutdown_connections 2024-03-28T18:12:37.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.534+0000 7fd8cdf6d740 1 -- 172.21.15.37:0/3683868865 wait complete. 2024-03-28T18:12:37.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.534+0000 7fd8cdf6d740 20 client.25706 trim_cache size 0 max 16384 2024-03-28T18:12:37.535 INFO:teuthology.orchestra.run.smithi037.stdout:[] 2024-03-28T18:12:37.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.535+0000 7fd8c9c00640 1 -- 172.21.15.37:0/3315931785 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fd8a80a2b90 msgr2=0x7fd8a80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:37.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.535+0000 7fd8c9c00640 1 --2- 172.21.15.37:0/3315931785 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fd8a80a2b90 0x7fd8a80a5050 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4005ab0 tx=0x7fd8b4005a20 comp rx=0 tx=0).stop 2024-03-28T18:12:37.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.535+0000 7fd8c9c00640 1 -- 172.21.15.37:0/3315931785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8c4074370 msgr2=0x7fd8c419a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:37.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.535+0000 7fd8c9c00640 1 --2- 172.21.15.37:0/3315931785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8c4074370 0x7fd8c419a540 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd8b000b520 tx=0x7fd8b000b9e0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.535+0000 7fd8c9c00640 1 -- 172.21.15.37:0/3315931785 shutdown_connections 2024-03-28T18:12:37.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.535+0000 7fd8c9c00640 1 --2- 172.21.15.37:0/3315931785 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fd8a80a2b90 0x7fd8a80a5050 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.535+0000 7fd8c9c00640 1 --2- 172.21.15.37:0/3315931785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8c4074370 0x7fd8c419a540 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.535+0000 7fd8c9c00640 1 --2- 172.21.15.37:0/3315931785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c410ddd0 0x7fd8c419e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.535+0000 7fd8c9c00640 1 --2- 172.21.15.37:0/3315931785 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8c4073a50 0x7fd8c419a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.535+0000 7fd8c9c00640 1 -- 172.21.15.37:0/3315931785 >> 172.21.15.37:0/3315931785 conn(0x7fd8c40fd040 msgr2=0x7fd8c40fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:37.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.535+0000 7fd8c9c00640 1 -- 172.21.15.37:0/3315931785 shutdown_connections 2024-03-28T18:12:37.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.535+0000 7fd8c9c00640 1 -- 172.21.15.37:0/3315931785 wait complete. 2024-03-28T18:12:37.560 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:12:37.561 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:12:37.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.765+0000 7ffa9e000640 1 -- 172.21.15.37:0/3285462641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa98109730 msgr2=0x7ffa98109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:37.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.765+0000 7ffa9e000640 1 --2- 172.21.15.37:0/3285462641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa98109730 0x7ffa98109b10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c009930 tx=0x7ffa8c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:12:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.766+0000 7ffa9e000640 1 -- 172.21.15.37:0/3285462641 shutdown_connections 2024-03-28T18:12:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.766+0000 7ffa9e000640 1 --2- 172.21.15.37:0/3285462641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa981040e0 0x7ffa981105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.766+0000 7ffa9e000640 1 --2- 172.21.15.37:0/3285462641 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa98103720 0x7ffa98103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.766+0000 7ffa9e000640 1 --2- 172.21.15.37:0/3285462641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa98109730 0x7ffa98109b10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.766+0000 7ffa9e000640 1 -- 172.21.15.37:0/3285462641 >> 172.21.15.37:0/3285462641 conn(0x7ffa980ff1d0 msgr2=0x7ffa981015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.766+0000 7ffa9e000640 1 -- 172.21.15.37:0/3285462641 shutdown_connections 2024-03-28T18:12:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.766+0000 7ffa9e000640 1 -- 172.21.15.37:0/3285462641 wait complete. 2024-03-28T18:12:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.767+0000 7ffa9e000640 1 Processor -- start 2024-03-28T18:12:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.767+0000 7ffa9e000640 1 -- start start 2024-03-28T18:12:37.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.768+0000 7ffa9e000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa98103720 0x7ffa98075bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.768+0000 7ffa9e000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa981040e0 0x7ffa980760f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.768+0000 7ffa9e000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa98109730 0x7ffa98076630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.768+0000 7ffa9e000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa98113b50 con 0x7ffa98103720 2024-03-28T18:12:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.768+0000 7ffa9e000640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa981139d0 con 0x7ffa981040e0 2024-03-28T18:12:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.768+0000 7ffa9e000640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa98113cd0 con 0x7ffa98109730 2024-03-28T18:12:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.768+0000 7ffa97e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa981040e0 0x7ffa980760f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.768+0000 7ffa97e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa981040e0 0x7ffa980760f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:44218/0 (socket says 172.21.15.37:44218) 2024-03-28T18:12:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.768+0000 7ffa97e00640 1 -- 172.21.15.37:0/2460444651 learned_addr learned my addr 172.21.15.37:0/2460444651 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.768+0000 7ffa9d600640 1 --2- 172.21.15.37:0/2460444651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa98109730 0x7ffa98076630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.768+0000 7ffa9cc00640 1 --2- 172.21.15.37:0/2460444651 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa98103720 0x7ffa98075bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.769+0000 7ffa9d600640 1 -- 172.21.15.37:0/2460444651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa981040e0 msgr2=0x7ffa980760f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.769+0000 7ffa9d600640 1 --2- 172.21.15.37:0/2460444651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa981040e0 0x7ffa980760f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.769+0000 7ffa9d600640 1 -- 172.21.15.37:0/2460444651 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa98103720 msgr2=0x7ffa98075bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.769+0000 7ffa9d600640 1 --2- 172.21.15.37:0/2460444651 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa98103720 0x7ffa98075bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.769+0000 7ffa9d600640 1 -- 172.21.15.37:0/2460444651 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa981aa6a0 con 0x7ffa98109730 2024-03-28T18:12:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.769+0000 7ffa97e00640 1 --2- 172.21.15.37:0/2460444651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa981040e0 0x7ffa980760f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:12:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.769+0000 7ffa9d600640 1 --2- 172.21.15.37:0/2460444651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa98109730 0x7ffa98076630 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ffa9000e940 tx=0x7ffa9000ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.769+0000 7ffa95600640 1 -- 172.21.15.37:0/2460444651 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa90004d20 con 0x7ffa98109730 2024-03-28T18:12:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.769+0000 7ffa95600640 1 -- 172.21.15.37:0/2460444651 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa90005020 con 0x7ffa98109730 2024-03-28T18:12:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.769+0000 7ffa95600640 1 -- 172.21.15.37:0/2460444651 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa900116b0 con 0x7ffa98109730 2024-03-28T18:12:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.770+0000 7ffa9e000640 1 -- 172.21.15.37:0/2460444651 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa981aa990 con 0x7ffa98109730 2024-03-28T18:12:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.770+0000 7ffa9e000640 1 -- 172.21.15.37:0/2460444651 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa981aae90 con 0x7ffa98109730 2024-03-28T18:12:37.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.773+0000 7ffaa23ce740 1 Processor -- start 2024-03-28T18:12:37.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.775+0000 7ffa95600640 1 -- 172.21.15.37:0/2460444651 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7ffa900054d0 con 0x7ffa98109730 2024-03-28T18:12:37.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.775+0000 7ffaa23ce740 1 -- start start 2024-03-28T18:12:37.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.776+0000 7ffaa23ce740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6ad8b0 0x560bbe6adc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.776+0000 7ffaa23ce740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ae260 0x560bbe6b6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.776+0000 7ffaa23ce740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6b6de0 0x560bbe6b9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.776+0000 7ffa97e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6ad8b0 0x560bbe6adc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.777+0000 7ffa97e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6ad8b0 0x560bbe6adc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:50012/0 (socket says 172.21.15.37:50012) 2024-03-28T18:12:37.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.777+0000 7ffa9cc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ae260 0x560bbe6b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.777+0000 7ffaa23ce740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bbe6112c0 con 0x560bbe6ae260 2024-03-28T18:12:37.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.777+0000 7ffaa23ce740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bbe5b9c90 con 0x560bbe6b6de0 2024-03-28T18:12:37.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.777+0000 7ffaa23ce740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bbe60d360 con 0x560bbe6ad8b0 2024-03-28T18:12:37.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.777+0000 7ffa9cc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ae260 0x560bbe6b6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36318/0 (socket says 172.21.15.37:36318) 2024-03-28T18:12:37.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.777+0000 7ffa9cc00640 1 -- 172.21.15.37:0/3410759036 learned_addr learned my addr 172.21.15.37:0/3410759036 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:37.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.777+0000 7ffa97e00640 1 --2- 172.21.15.37:0/3410759036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6b6de0 0x560bbe6b9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.777+0000 7ffa97e00640 1 -- 172.21.15.37:0/3410759036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6b6de0 msgr2=0x560bbe6b9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:37.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.777+0000 7ffa97e00640 1 --2- 172.21.15.37:0/3410759036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6b6de0 0x560bbe6b9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.777+0000 7ffa97e00640 1 -- 172.21.15.37:0/3410759036 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ae260 msgr2=0x560bbe6b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:37.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.777+0000 7ffa97e00640 1 --2- 172.21.15.37:0/3410759036 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ae260 0x560bbe6b6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.777+0000 7ffa97e00640 1 -- 172.21.15.37:0/3410759036 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560bbe5de080 con 0x560bbe6ad8b0 2024-03-28T18:12:37.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.777+0000 7ffa9cc00640 1 --2- 172.21.15.37:0/3410759036 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ae260 0x560bbe6b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:12:37.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.778+0000 7ffa97e00640 1 --2- 172.21.15.37:0/3410759036 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6ad8b0 0x560bbe6adc90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ffa98077400 tx=0x7ffa8400b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d898085ee83e771f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:37.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.778+0000 7ffa95600640 1 --2- 172.21.15.37:0/2460444651 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7ffa800a2b10 0x7ffa800a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.778+0000 7ffa89c00640 1 -- 172.21.15.37:0/3410759036 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa84019030 con 0x560bbe6ad8b0 2024-03-28T18:12:37.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.778+0000 7ffa89c00640 1 -- 172.21.15.37:0/3410759036 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa84013110 con 0x560bbe6ad8b0 2024-03-28T18:12:37.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.778+0000 7ffa89c00640 1 -- 172.21.15.37:0/3410759036 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa84004a00 con 0x560bbe6ad8b0 2024-03-28T18:12:37.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.778+0000 7ffa95600640 1 -- 172.21.15.37:0/2460444651 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ffa900cf200 con 0x7ffa98109730 2024-03-28T18:12:37.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.779+0000 7ffa9cc00640 1 --2- 172.21.15.37:0/2460444651 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7ffa800a2b10 0x7ffa800a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.779+0000 7ffaa23ce740 1 -- 172.21.15.37:0/3410759036 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6ad8b0 msgr2=0x560bbe6adc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:37.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.779+0000 7ffaa23ce740 1 --2- 172.21.15.37:0/3410759036 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6ad8b0 0x560bbe6adc90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ffa98077400 tx=0x7ffa8400b9f0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.780+0000 7ffaa23ce740 1 -- 172.21.15.37:0/3410759036 shutdown_connections 2024-03-28T18:12:37.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.780+0000 7ffaa23ce740 1 --2- 172.21.15.37:0/3410759036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6b6de0 0x560bbe6b9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.780+0000 7ffaa23ce740 1 --2- 172.21.15.37:0/3410759036 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ae260 0x560bbe6b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.780+0000 7ffaa23ce740 1 --2- 172.21.15.37:0/3410759036 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6ad8b0 0x560bbe6adc90 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.780+0000 7ffaa23ce740 1 -- 172.21.15.37:0/3410759036 >> 172.21.15.37:0/3410759036 conn(0x560bbe6a7f80 msgr2=0x560bbe6a8360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:37.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.780+0000 7ffaa23ce740 1 -- 172.21.15.37:0/3410759036 shutdown_connections 2024-03-28T18:12:37.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.780+0000 7ffaa23ce740 1 -- 172.21.15.37:0/3410759036 wait complete. 2024-03-28T18:12:37.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.780+0000 7ffa9cc00640 1 --2- 172.21.15.37:0/2460444651 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7ffa800a2b10 0x7ffa800a4fd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c006010 tx=0x7ffa8c000bc0 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:37.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.781+0000 7ffaa23ce740 1 -- start start 2024-03-28T18:12:37.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.781+0000 7ffaa23ce740 1 Processor -- start 2024-03-28T18:12:37.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.781+0000 7ffaa23ce740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6ad8b0 0x560bbe6b67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.781+0000 7ffaa23ce740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ae260 0x560bbe6b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.781+0000 7ffaa23ce740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6b6de0 0x560bbe6b4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.781+0000 7ffaa23ce740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bbe6b9d50 con 0x560bbe6ae260 2024-03-28T18:12:37.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.781+0000 7ffaa23ce740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bbe5cd0d0 con 0x560bbe6b6de0 2024-03-28T18:12:37.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.781+0000 7ffaa23ce740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bbe6b9ed0 con 0x560bbe6ad8b0 2024-03-28T18:12:37.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.782+0000 7ffaa23ce740 10 client.0 _ll_register_callbacks cb 0x560bbe5d1e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:37.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.782+0000 7ffa9cc00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6ad8b0 0x560bbe6b67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.782+0000 7ffa9cc00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6ad8b0 0x560bbe6b67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:50026/0 (socket says 172.21.15.37:50026) 2024-03-28T18:12:37.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.782+0000 7ffa9cc00640 1 -- 172.21.15.37:0/2179142702 learned_addr learned my addr 172.21.15.37:0/2179142702 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:37.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.782+0000 7ffa97e00640 1 --2- 172.21.15.37:0/2179142702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ae260 0x560bbe6b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.782+0000 7ffa9d600640 1 --2- 172.21.15.37:0/2179142702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6b6de0 0x560bbe6b4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.782+0000 7ffa9cc00640 1 -- 172.21.15.37:0/2179142702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6b6de0 msgr2=0x560bbe6b4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:37.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.782+0000 7ffa9cc00640 1 --2- 172.21.15.37:0/2179142702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6b6de0 0x560bbe6b4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.782+0000 7ffa9cc00640 1 -- 172.21.15.37:0/2179142702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ae260 msgr2=0x560bbe6b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:37.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.782+0000 7ffa9cc00640 1 --2- 172.21.15.37:0/2179142702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ae260 0x560bbe6b3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.782+0000 7ffa9cc00640 1 -- 172.21.15.37:0/2179142702 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560bbe3253f0 con 0x560bbe6ad8b0 2024-03-28T18:12:37.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.782+0000 7ffa9d600640 1 --2- 172.21.15.37:0/2179142702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6b6de0 0x560bbe6b4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:37.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.783+0000 7ffa9cc00640 1 --2- 172.21.15.37:0/2179142702 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6ad8b0 0x560bbe6b67d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c02c170 tx=0x7ffa8c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9c9281903ad1d93 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:37.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.783+0000 7ffa7e000640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:12:37.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.783+0000 7ffa7e000640 1 -- 172.21.15.37:0/2179142702 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c050060 con 0x560bbe6ad8b0 2024-03-28T18:12:37.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.783+0000 7ffa7e000640 1 -- 172.21.15.37:0/2179142702 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ffa8c0046c0 con 0x560bbe6ad8b0 2024-03-28T18:12:37.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.783+0000 7ffaa23ce740 1 -- 172.21.15.37:0/2179142702 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffa84013110 con 0x560bbe6ad8b0 2024-03-28T18:12:37.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.784+0000 7ffa7e000640 1 -- 172.21.15.37:0/2179142702 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa8c0048c0 con 0x560bbe6ad8b0 2024-03-28T18:12:37.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.784+0000 7ffaa23ce740 10 client.25732 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:37.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.784+0000 7ffaa23ce740 1 -- 172.21.15.37:0/2179142702 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560bbe3ef7d0 con 0x560bbe6ad8b0 2024-03-28T18:12:37.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.784+0000 7ffa7e000640 1 -- 172.21.15.37:0/2179142702 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa84013110 con 0x560bbe6ad8b0 2024-03-28T18:12:37.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.784+0000 7ffa7e000640 1 -- 172.21.15.37:0/2179142702 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ffa8c040470 con 0x560bbe6ad8b0 2024-03-28T18:12:37.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.784+0000 7ffaa23ce740 10 client.25732 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:37.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.784+0000 7ffaa23ce740 10 client.25732 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-28T18:12:37.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.785+0000 7ffaa23ce740 20 client.25732 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:37.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.785+0000 7ffaa23ce740 1 --2- 172.21.15.37:0/2179142702 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x560bbe6c6c30 0x560bbe6c9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.785+0000 7ffaa23ce740 4 client.25732 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:12:37.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.785+0000 7ffaa23ce740 1 -- 172.21.15.37:0/2179142702 --> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560bbe61e800 con 0x560bbe6c6c30 2024-03-28T18:12:37.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.785+0000 7ffa97e00640 1 --2- 172.21.15.37:0/2179142702 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x560bbe6c6c30 0x560bbe6c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.786+0000 7ffa97e00640 1 --2- 172.21.15.37:0/2179142702 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x560bbe6c6c30 0x560bbe6c9060 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aba18dbec83e31ce server_cookie=d3c8909e59fe530b in_seq=0 out_seq=0 2024-03-28T18:12:37.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.786+0000 7ffa7e000640 10 client.25732 ms_handle_connect on v2:172.21.15.59:6838/2091629078 2024-03-28T18:12:37.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.786+0000 7ffa7e000640 1 -- 172.21.15.37:0/2179142702 <== mds.1 v2:172.21.15.59:6838/2091629078 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560bbe61e800 con 0x560bbe6c6c30 2024-03-28T18:12:37.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.786+0000 7ffa7e000640 10 client.25732 handle_command_reply: tid=0 2024-03-28T18:12:37.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.786+0000 7ffaa23ce740 1 client.25732 shutdown 2024-03-28T18:12:37.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.787+0000 7ffaa23ce740 1 -- 172.21.15.37:0/2179142702 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6ad8b0 msgr2=0x560bbe6b67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:37.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.787+0000 7ffaa23ce740 1 --2- 172.21.15.37:0/2179142702 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6ad8b0 0x560bbe6b67d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c02c170 tx=0x7ffa8c009260 comp rx=0 tx=0).stop 2024-03-28T18:12:37.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.787+0000 7ffaa23ce740 1 -- 172.21.15.37:0/2179142702 shutdown_connections 2024-03-28T18:12:37.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.787+0000 7ffaa23ce740 1 --2- 172.21.15.37:0/2179142702 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x560bbe6c6c30 0x560bbe6c9060 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.787+0000 7ffaa23ce740 1 --2- 172.21.15.37:0/2179142702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6b6de0 0x560bbe6b4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.787+0000 7ffaa23ce740 1 --2- 172.21.15.37:0/2179142702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ae260 0x560bbe6b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.787+0000 7ffaa23ce740 1 --2- 172.21.15.37:0/2179142702 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6ad8b0 0x560bbe6b67d0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.787+0000 7ffaa23ce740 1 -- 172.21.15.37:0/2179142702 >> 172.21.15.37:0/2179142702 conn(0x560bbe6a7f80 msgr2=0x560bbe6a9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:37.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.787+0000 7ffa7e000640 0 client.25732 ms_handle_reset on v2:172.21.15.59:6838/2091629078 2024-03-28T18:12:37.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.787+0000 7ffaa23ce740 1 -- 172.21.15.37:0/2179142702 shutdown_connections 2024-03-28T18:12:37.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.787+0000 7ffaa23ce740 1 -- 172.21.15.37:0/2179142702 wait complete. 2024-03-28T18:12:37.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.787+0000 7ffaa23ce740 20 client.25732 trim_cache size 0 max 16384 2024-03-28T18:12:37.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.813+0000 7ffaa23ce740 1 Processor -- start 2024-03-28T18:12:37.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.814+0000 7ffaa23ce740 1 -- start start 2024-03-28T18:12:37.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.814+0000 7ffaa23ce740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ad8b0 0x560bbe6ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.814+0000 7ffaa23ce740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6ae260 0x560bbe6c3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.814+0000 7ffaa23ce740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6b6de0 0x560bbe6b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.814+0000 7ffaa23ce740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa981b5be0 con 0x560bbe6ad8b0 2024-03-28T18:12:37.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.814+0000 7ffa97e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6ae260 0x560bbe6c3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.814+0000 7ffaa23ce740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa981b36e0 con 0x560bbe6ae260 2024-03-28T18:12:37.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.814+0000 7ffaa23ce740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa981b1780 con 0x560bbe6b6de0 2024-03-28T18:12:37.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.815+0000 7ffa97e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6ae260 0x560bbe6c3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:44244/0 (socket says 172.21.15.37:44244) 2024-03-28T18:12:37.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.815+0000 7ffa97e00640 1 -- 172.21.15.37:0/4026941523 learned_addr learned my addr 172.21.15.37:0/4026941523 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:37.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.815+0000 7ffa9d600640 1 --2- 172.21.15.37:0/4026941523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6b6de0 0x560bbe6b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.815+0000 7ffa9cc00640 1 --2- 172.21.15.37:0/4026941523 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ad8b0 0x560bbe6ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.815+0000 7ffa9d600640 1 -- 172.21.15.37:0/4026941523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6ae260 msgr2=0x560bbe6c3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:37.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.815+0000 7ffa9d600640 1 --2- 172.21.15.37:0/4026941523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6ae260 0x560bbe6c3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.815+0000 7ffa9d600640 1 -- 172.21.15.37:0/4026941523 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ad8b0 msgr2=0x560bbe6ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:37.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.815+0000 7ffa9d600640 1 --2- 172.21.15.37:0/4026941523 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ad8b0 0x560bbe6ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.815+0000 7ffa9d600640 1 -- 172.21.15.37:0/4026941523 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560bbe6b2030 con 0x560bbe6b6de0 2024-03-28T18:12:37.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.815+0000 7ffa9d600640 1 --2- 172.21.15.37:0/4026941523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6b6de0 0x560bbe6b2990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ffa900cc7a0 tx=0x7ffa90005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e13ae634bed721d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:37.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.816+0000 7ffa89200640 1 -- 172.21.15.37:0/4026941523 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9001f040 con 0x560bbe6b6de0 2024-03-28T18:12:37.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.816+0000 7ffa89200640 1 -- 172.21.15.37:0/4026941523 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa9009b5e0 con 0x560bbe6b6de0 2024-03-28T18:12:37.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.816+0000 7ffa89200640 1 -- 172.21.15.37:0/4026941523 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9009ac90 con 0x560bbe6b6de0 2024-03-28T18:12:37.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.817+0000 7ffaa23ce740 1 -- 172.21.15.37:0/4026941523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6b6de0 msgr2=0x560bbe6b2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:37.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.817+0000 7ffaa23ce740 1 --2- 172.21.15.37:0/4026941523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6b6de0 0x560bbe6b2990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ffa900cc7a0 tx=0x7ffa90005f90 comp rx=0 tx=0).stop 2024-03-28T18:12:37.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.817+0000 7ffaa23ce740 1 -- 172.21.15.37:0/4026941523 shutdown_connections 2024-03-28T18:12:37.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.817+0000 7ffaa23ce740 1 --2- 172.21.15.37:0/4026941523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6ae260 0x560bbe6c3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.817+0000 7ffaa23ce740 1 --2- 172.21.15.37:0/4026941523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6b6de0 0x560bbe6b2990 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.817+0000 7ffaa23ce740 1 --2- 172.21.15.37:0/4026941523 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ad8b0 0x560bbe6ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.817+0000 7ffaa23ce740 1 -- 172.21.15.37:0/4026941523 >> 172.21.15.37:0/4026941523 conn(0x560bbe6a7f80 msgr2=0x560bbe6b2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:37.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.817+0000 7ffaa23ce740 1 -- 172.21.15.37:0/4026941523 shutdown_connections 2024-03-28T18:12:37.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.817+0000 7ffaa23ce740 1 -- 172.21.15.37:0/4026941523 wait complete. 2024-03-28T18:12:37.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.818+0000 7ffaa23ce740 1 -- start start 2024-03-28T18:12:37.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.818+0000 7ffaa23ce740 1 Processor -- start 2024-03-28T18:12:37.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.818+0000 7ffaa23ce740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ad8b0 0x560bbe6cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.819+0000 7ffaa23ce740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6ae260 0x560bbe6cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.819+0000 7ffaa23ce740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6b6de0 0x560bbe6cc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.819+0000 7ffaa23ce740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa981b1d80 con 0x560bbe6ad8b0 2024-03-28T18:12:37.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.819+0000 7ffaa23ce740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa981b3080 con 0x560bbe6b6de0 2024-03-28T18:12:37.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.819+0000 7ffaa23ce740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa981b23c0 con 0x560bbe6ae260 2024-03-28T18:12:37.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.819+0000 7ffaa23ce740 10 client.0 _ll_register_callbacks cb 0x7ffa8c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:37.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.819+0000 7ffa9cc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ad8b0 0x560bbe6cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.819+0000 7ffa9cc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ad8b0 0x560bbe6cbe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36350/0 (socket says 172.21.15.37:36350) 2024-03-28T18:12:37.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.819+0000 7ffa9cc00640 1 -- 172.21.15.37:0/1841011216 learned_addr learned my addr 172.21.15.37:0/1841011216 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:37.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.819+0000 7ffa9d600640 1 --2- 172.21.15.37:0/1841011216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6b6de0 0x560bbe6cc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.819+0000 7ffa97e00640 1 --2- 172.21.15.37:0/1841011216 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6ae260 0x560bbe6cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.819+0000 7ffa9d600640 1 -- 172.21.15.37:0/1841011216 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6ae260 msgr2=0x560bbe6cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:37.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.819+0000 7ffa9d600640 1 --2- 172.21.15.37:0/1841011216 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6ae260 0x560bbe6cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.819+0000 7ffa9d600640 1 -- 172.21.15.37:0/1841011216 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ad8b0 msgr2=0x560bbe6cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:37.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.819+0000 7ffa9d600640 1 --2- 172.21.15.37:0/1841011216 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ad8b0 0x560bbe6cbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.819+0000 7ffa9d600640 1 -- 172.21.15.37:0/1841011216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560bbe721630 con 0x560bbe6b6de0 2024-03-28T18:12:37.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.820+0000 7ffa9d600640 1 --2- 172.21.15.37:0/1841011216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6b6de0 0x560bbe6cc8c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ffa9009f8f0 tx=0x7ffa90097260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccbe336183492130 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:37.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.820+0000 7ffa7e000640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:12:37.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.820+0000 7ffa7e000640 1 -- 172.21.15.37:0/1841011216 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa900b8020 con 0x560bbe6b6de0 2024-03-28T18:12:37.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.820+0000 7ffa7e000640 1 -- 172.21.15.37:0/1841011216 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ffa900b4290 con 0x560bbe6b6de0 2024-03-28T18:12:37.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.820+0000 7ffaa23ce740 1 -- 172.21.15.37:0/1841011216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffa9009b5e0 con 0x560bbe6b6de0 2024-03-28T18:12:37.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.821+0000 7ffa7e000640 1 -- 172.21.15.37:0/1841011216 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa900b3070 con 0x560bbe6b6de0 2024-03-28T18:12:37.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.821+0000 7ffaa23ce740 10 client.25724 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:37.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.821+0000 7ffaa23ce740 1 -- 172.21.15.37:0/1841011216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560bbe722910 con 0x560bbe6b6de0 2024-03-28T18:12:37.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.821+0000 7ffa7e000640 1 -- 172.21.15.37:0/1841011216 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa9009b5e0 con 0x560bbe6b6de0 2024-03-28T18:12:37.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.821+0000 7ffa7e000640 1 -- 172.21.15.37:0/1841011216 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ffa9000f040 con 0x560bbe6b6de0 2024-03-28T18:12:37.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.822+0000 7ffaa23ce740 10 client.25724 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:37.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.822+0000 7ffaa23ce740 10 client.25724 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-28T18:12:37.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.822+0000 7ffaa23ce740 20 client.25724 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:37.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.822+0000 7ffaa23ce740 1 --2- 172.21.15.37:0/1841011216 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x560bbe722f30 0x560bbe725320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:37.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.822+0000 7ffaa23ce740 4 client.25724 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:12:37.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.822+0000 7ffaa23ce740 1 -- 172.21.15.37:0/1841011216 --> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560bbe618b50 con 0x560bbe722f30 2024-03-28T18:12:37.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.823+0000 7ffa9cc00640 1 --2- 172.21.15.37:0/1841011216 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x560bbe722f30 0x560bbe725320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:37.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.823+0000 7ffa7e000640 10 client.25724 ms_handle_connect on v2:172.21.15.59:6838/2091629078 2024-03-28T18:12:37.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.823+0000 7ffa9cc00640 1 --2- 172.21.15.37:0/1841011216 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x560bbe722f30 0x560bbe725320 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3fdc038d0b6b3435 server_cookie=5b1747cbbe798901 in_seq=0 out_seq=0 2024-03-28T18:12:37.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.823+0000 7ffa7e000640 1 -- 172.21.15.37:0/1841011216 <== mds.1 v2:172.21.15.59:6838/2091629078 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560bbe618b50 con 0x560bbe722f30 2024-03-28T18:12:37.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.823+0000 7ffa7e000640 10 client.25724 handle_command_reply: tid=0 2024-03-28T18:12:37.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.823+0000 7ffaa23ce740 1 client.25724 shutdown 2024-03-28T18:12:37.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.824+0000 7ffaa23ce740 1 -- 172.21.15.37:0/1841011216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6b6de0 msgr2=0x560bbe6cc8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:37.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.824+0000 7ffaa23ce740 1 --2- 172.21.15.37:0/1841011216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6b6de0 0x560bbe6cc8c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ffa9009f8f0 tx=0x7ffa90097260 comp rx=0 tx=0).stop 2024-03-28T18:12:37.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.824+0000 7ffaa23ce740 1 -- 172.21.15.37:0/1841011216 shutdown_connections 2024-03-28T18:12:37.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.824+0000 7ffaa23ce740 1 --2- 172.21.15.37:0/1841011216 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x560bbe722f30 0x560bbe725320 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.824+0000 7ffaa23ce740 1 --2- 172.21.15.37:0/1841011216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560bbe6b6de0 0x560bbe6cc8c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.824+0000 7ffaa23ce740 1 --2- 172.21.15.37:0/1841011216 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560bbe6ae260 0x560bbe6cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.824+0000 7ffa7e000640 0 client.25724 ms_handle_reset on v2:172.21.15.59:6838/2091629078 2024-03-28T18:12:37.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.824+0000 7ffaa23ce740 1 --2- 172.21.15.37:0/1841011216 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x560bbe6ad8b0 0x560bbe6cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.824+0000 7ffaa23ce740 1 -- 172.21.15.37:0/1841011216 >> 172.21.15.37:0/1841011216 conn(0x560bbe6a7f80 msgr2=0x560bbe6b2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:37.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.824+0000 7ffaa23ce740 1 -- 172.21.15.37:0/1841011216 shutdown_connections 2024-03-28T18:12:37.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.824+0000 7ffaa23ce740 1 -- 172.21.15.37:0/1841011216 wait complete. 2024-03-28T18:12:37.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.824+0000 7ffaa23ce740 20 client.25724 trim_cache size 0 max 16384 2024-03-28T18:12:37.826 INFO:teuthology.orchestra.run.smithi037.stdout:[] 2024-03-28T18:12:37.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.825+0000 7ffa9e000640 1 -- 172.21.15.37:0/2460444651 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7ffa800a2b10 msgr2=0x7ffa800a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:37.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.825+0000 7ffa9e000640 1 --2- 172.21.15.37:0/2460444651 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7ffa800a2b10 0x7ffa800a4fd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c006010 tx=0x7ffa8c000bc0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.825+0000 7ffa9e000640 1 -- 172.21.15.37:0/2460444651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa98109730 msgr2=0x7ffa98076630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:37.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.825+0000 7ffa9e000640 1 --2- 172.21.15.37:0/2460444651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa98109730 0x7ffa98076630 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ffa9000e940 tx=0x7ffa9000ee00 comp rx=0 tx=0).stop 2024-03-28T18:12:37.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.825+0000 7ffa9e000640 1 -- 172.21.15.37:0/2460444651 shutdown_connections 2024-03-28T18:12:37.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.825+0000 7ffa9e000640 1 --2- 172.21.15.37:0/2460444651 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7ffa800a2b10 0x7ffa800a4fd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.825+0000 7ffa9e000640 1 --2- 172.21.15.37:0/2460444651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa981040e0 0x7ffa980760f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.825+0000 7ffa9e000640 1 --2- 172.21.15.37:0/2460444651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa98109730 0x7ffa98076630 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.825+0000 7ffa9e000640 1 --2- 172.21.15.37:0/2460444651 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa98103720 0x7ffa98075bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:37.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.825+0000 7ffa9e000640 1 -- 172.21.15.37:0/2460444651 >> 172.21.15.37:0/2460444651 conn(0x7ffa980ff1d0 msgr2=0x7ffa980ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:37.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.826+0000 7ffa9e000640 1 -- 172.21.15.37:0/2460444651 shutdown_connections 2024-03-28T18:12:37.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:37.826+0000 7ffa9e000640 1 -- 172.21.15.37:0/2460444651 wait complete. 2024-03-28T18:12:37.848 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:37 smithi037 ceph-mon[31460]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:12:37.848 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:37 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/1226724944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:12:37.850 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:12:37.850 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:12:38.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:37 smithi059 ceph-mon[36190]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:12:38.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:37 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/1226724944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:12:38.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:37 smithi170 ceph-mon[36280]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:12:38.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:37 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/1226724944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:12:38.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.079+0000 7f4a36a00640 1 -- 172.21.15.37:0/2195708396 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a301004c0 msgr2=0x7f4a301008a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:38.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.079+0000 7f4a36a00640 1 --2- 172.21.15.37:0/2195708396 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a301004c0 0x7f4a301008a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4a24009900 tx=0x7f4a2402bd10 comp rx=0 tx=0).stop 2024-03-28T18:12:38.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.080+0000 7f4a36a00640 1 -- 172.21.15.37:0/2195708396 shutdown_connections 2024-03-28T18:12:38.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.080+0000 7f4a36a00640 1 --2- 172.21.15.37:0/2195708396 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a301058c0 0x7f4a30105d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.080+0000 7f4a36a00640 1 --2- 172.21.15.37:0/2195708396 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a30100de0 0x7f4a301052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.080+0000 7f4a36a00640 1 --2- 172.21.15.37:0/2195708396 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a301004c0 0x7f4a301008a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.080+0000 7f4a36a00640 1 -- 172.21.15.37:0/2195708396 >> 172.21.15.37:0/2195708396 conn(0x7f4a300fbef0 msgr2=0x7f4a300fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:38.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.080+0000 7f4a36a00640 1 -- 172.21.15.37:0/2195708396 shutdown_connections 2024-03-28T18:12:38.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.080+0000 7f4a36a00640 1 -- 172.21.15.37:0/2195708396 wait complete. 2024-03-28T18:12:38.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.081+0000 7f4a36a00640 1 Processor -- start 2024-03-28T18:12:38.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.082+0000 7f4a36a00640 1 -- start start 2024-03-28T18:12:38.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.082+0000 7f4a36a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a301004c0 0x7f4a30114660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:38.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.082+0000 7f4a36a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a30100de0 0x7f4a30114ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:38.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.082+0000 7f4a36a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a301058c0 0x7f4a3010d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:38.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.082+0000 7f4a36a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a30117f50 con 0x7f4a301058c0 2024-03-28T18:12:38.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.082+0000 7f4a36a00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a30117dd0 con 0x7f4a301004c0 2024-03-28T18:12:38.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.082+0000 7f4a36a00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a301180d0 con 0x7f4a30100de0 2024-03-28T18:12:38.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.083+0000 7f4a35600640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a301004c0 0x7f4a30114660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:38.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.083+0000 7f4a35600640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a301004c0 0x7f4a30114660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:44264/0 (socket says 172.21.15.37:44264) 2024-03-28T18:12:38.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.083+0000 7f4a35600640 1 -- 172.21.15.37:0/661675599 learned_addr learned my addr 172.21.15.37:0/661675599 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:38.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.083+0000 7f4a34c00640 1 --2- 172.21.15.37:0/661675599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a30100de0 0x7f4a30114ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:38.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.083+0000 7f4a36000640 1 --2- 172.21.15.37:0/661675599 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a301058c0 0x7f4a3010d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:38.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.083+0000 7f4a35600640 1 -- 172.21.15.37:0/661675599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a30100de0 msgr2=0x7f4a30114ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:38.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.083+0000 7f4a35600640 1 --2- 172.21.15.37:0/661675599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a30100de0 0x7f4a30114ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.083+0000 7f4a35600640 1 -- 172.21.15.37:0/661675599 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a301058c0 msgr2=0x7f4a3010d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:38.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.083+0000 7f4a35600640 1 --2- 172.21.15.37:0/661675599 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a301058c0 0x7f4a3010d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.083+0000 7f4a35600640 1 -- 172.21.15.37:0/661675599 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3010e1d0 con 0x7f4a301004c0 2024-03-28T18:12:38.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.083+0000 7f4a34c00640 1 --2- 172.21.15.37:0/661675599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a30100de0 0x7f4a30114ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:38.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.083+0000 7f4a36000640 1 --2- 172.21.15.37:0/661675599 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a301058c0 0x7f4a3010d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:12:38.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.084+0000 7f4a35600640 1 --2- 172.21.15.37:0/661675599 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a301004c0 0x7f4a30114660 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4a24000c00 tx=0x7f4a240043c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:38.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.084+0000 7f4a2e000640 1 -- 172.21.15.37:0/661675599 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a24042020 con 0x7f4a301004c0 2024-03-28T18:12:38.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.084+0000 7f4a36a00640 1 -- 172.21.15.37:0/661675599 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a3010e460 con 0x7f4a301004c0 2024-03-28T18:12:38.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.084+0000 7f4a2e000640 1 -- 172.21.15.37:0/661675599 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a24005000 con 0x7f4a301004c0 2024-03-28T18:12:38.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.084+0000 7f4a2e000640 1 -- 172.21.15.37:0/661675599 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a24003b20 con 0x7f4a301004c0 2024-03-28T18:12:38.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.085+0000 7f4a36a00640 1 -- 172.21.15.37:0/661675599 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a30112470 con 0x7f4a301004c0 2024-03-28T18:12:38.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.089+0000 7f4a3ac73740 1 Processor -- start 2024-03-28T18:12:38.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.089+0000 7f4a3ac73740 1 -- start start 2024-03-28T18:12:38.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.090+0000 7f4a3ac73740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab318b0 0x55a1bab31c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:38.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.090+0000 7f4a3ac73740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab32260 0x55a1bab3a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:38.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.090+0000 7f4a3ac73740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab3ade0 0x55a1bab3d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:38.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.090+0000 7f4a2e000640 1 -- 172.21.15.37:0/661675599 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f4a24045020 con 0x7f4a301004c0 2024-03-28T18:12:38.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.090+0000 7f4a34c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab3ade0 0x55a1bab3d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:38.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.090+0000 7f4a34c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab3ade0 0x55a1bab3d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:44276/0 (socket says 172.21.15.37:44276) 2024-03-28T18:12:38.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.090+0000 7f4a34c00640 1 -- 172.21.15.37:0/47554809 learned_addr learned my addr 172.21.15.37:0/47554809 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:38.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.090+0000 7f4a3ac73740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1baa952c0 con 0x55a1bab32260 2024-03-28T18:12:38.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.091+0000 7f4a3ac73740 1 -- 172.21.15.37:0/47554809 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1baa3dc90 con 0x55a1bab3ade0 2024-03-28T18:12:38.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.091+0000 7f4a3ac73740 1 -- 172.21.15.37:0/47554809 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1baa91360 con 0x55a1bab318b0 2024-03-28T18:12:38.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.091+0000 7f4a36000640 1 --2- 172.21.15.37:0/47554809 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab318b0 0x55a1bab31c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:38.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.091+0000 7f4a35600640 1 --2- 172.21.15.37:0/47554809 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab32260 0x55a1bab3a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:38.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.092+0000 7f4a36000640 1 -- 172.21.15.37:0/47554809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab3ade0 msgr2=0x55a1bab3d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:38.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.092+0000 7f4a36000640 1 --2- 172.21.15.37:0/47554809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab3ade0 0x55a1bab3d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.092+0000 7f4a36000640 1 -- 172.21.15.37:0/47554809 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab32260 msgr2=0x55a1bab3a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.092+0000 7f4a36000640 1 --2- 172.21.15.37:0/47554809 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab32260 0x55a1bab3a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.092+0000 7f4a36000640 1 -- 172.21.15.37:0/47554809 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a1baa62080 con 0x55a1bab318b0 2024-03-28T18:12:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.092+0000 7f4a35600640 1 --2- 172.21.15.37:0/47554809 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab32260 0x55a1bab3a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.092+0000 7f4a34c00640 1 --2- 172.21.15.37:0/47554809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab3ade0 0x55a1bab3d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.092+0000 7f4a36000640 1 --2- 172.21.15.37:0/47554809 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab318b0 0x55a1bab31c90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4a28009e30 tx=0x7f4a2800f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6cb683e4d269433 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.092+0000 7f4a2e000640 1 --2- 172.21.15.37:0/661675599 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f4a180a2b10 0x7f4a180a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.093+0000 7f4a22600640 1 -- 172.21.15.37:0/47554809 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a28011640 con 0x55a1bab318b0 2024-03-28T18:12:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.093+0000 7f4a2e000640 1 -- 172.21.15.37:0/661675599 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4a240f0c70 con 0x7f4a301004c0 2024-03-28T18:12:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.093+0000 7f4a22600640 1 -- 172.21.15.37:0/47554809 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a28017070 con 0x55a1bab318b0 2024-03-28T18:12:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.093+0000 7f4a22600640 1 -- 172.21.15.37:0/47554809 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a28010bb0 con 0x55a1bab318b0 2024-03-28T18:12:38.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.093+0000 7f4a35600640 1 --2- 172.21.15.37:0/661675599 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f4a180a2b10 0x7f4a180a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:38.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.094+0000 7f4a3ac73740 1 -- 172.21.15.37:0/47554809 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab318b0 msgr2=0x55a1bab31c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:38.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.094+0000 7f4a3ac73740 1 --2- 172.21.15.37:0/47554809 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab318b0 0x55a1bab31c90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4a28009e30 tx=0x7f4a2800f730 comp rx=0 tx=0).stop 2024-03-28T18:12:38.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.094+0000 7f4a3ac73740 1 -- 172.21.15.37:0/47554809 shutdown_connections 2024-03-28T18:12:38.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.094+0000 7f4a3ac73740 1 --2- 172.21.15.37:0/47554809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab3ade0 0x55a1bab3d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.094+0000 7f4a3ac73740 1 --2- 172.21.15.37:0/47554809 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab32260 0x55a1bab3a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.094+0000 7f4a3ac73740 1 --2- 172.21.15.37:0/47554809 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab318b0 0x55a1bab31c90 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.094+0000 7f4a3ac73740 1 -- 172.21.15.37:0/47554809 >> 172.21.15.37:0/47554809 conn(0x55a1bab2bf80 msgr2=0x55a1bab2c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:38.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.094+0000 7f4a35600640 1 --2- 172.21.15.37:0/661675599 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f4a180a2b10 0x7f4a180a4fd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4a240ee3e0 tx=0x7f4a240dd040 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:38.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.095+0000 7f4a3ac73740 1 -- 172.21.15.37:0/47554809 shutdown_connections 2024-03-28T18:12:38.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.095+0000 7f4a3ac73740 1 -- 172.21.15.37:0/47554809 wait complete. 2024-03-28T18:12:38.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.095+0000 7f4a3ac73740 1 -- start start 2024-03-28T18:12:38.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.096+0000 7f4a3ac73740 1 Processor -- start 2024-03-28T18:12:38.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.096+0000 7f4a3ac73740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab318b0 0x55a1bab3a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:38.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.096+0000 7f4a3ac73740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab32260 0x55a1bab37aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:38.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.096+0000 7f4a3ac73740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab3ade0 0x55a1bab380b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:38.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.096+0000 7f4a3ac73740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1bab3dd50 con 0x55a1bab318b0 2024-03-28T18:12:38.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.096+0000 7f4a3ac73740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1baa510d0 con 0x55a1bab3ade0 2024-03-28T18:12:38.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.096+0000 7f4a3ac73740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1bab3ded0 con 0x55a1bab32260 2024-03-28T18:12:38.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.096+0000 7f4a3ac73740 10 client.0 _ll_register_callbacks cb 0x55a1baa55e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:38.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.097+0000 7f4a36000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab318b0 0x55a1bab3a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:38.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.097+0000 7f4a36000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab318b0 0x55a1bab3a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36396/0 (socket says 172.21.15.37:36396) 2024-03-28T18:12:38.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.097+0000 7f4a36000640 1 -- 172.21.15.37:0/569421657 learned_addr learned my addr 172.21.15.37:0/569421657 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:38.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.097+0000 7f4a34c00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab32260 0x55a1bab37aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:38.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.097+0000 7f4a36000640 1 --2- 172.21.15.37:0/569421657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab3ade0 0x55a1bab380b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:38.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.097+0000 7f4a34c00640 1 -- 172.21.15.37:0/569421657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab3ade0 msgr2=0x55a1bab380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:38.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.097+0000 7f4a34c00640 1 --2- 172.21.15.37:0/569421657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab3ade0 0x55a1bab380b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.097+0000 7f4a34c00640 1 -- 172.21.15.37:0/569421657 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab318b0 msgr2=0x55a1bab3a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:38.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.097+0000 7f4a34c00640 1 --2- 172.21.15.37:0/569421657 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab318b0 0x55a1bab3a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.097+0000 7f4a34c00640 1 -- 172.21.15.37:0/569421657 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a1ba7a93f0 con 0x55a1bab32260 2024-03-28T18:12:38.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.098+0000 7f4a34c00640 1 --2- 172.21.15.37:0/569421657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab32260 0x55a1bab37aa0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c011730 tx=0x7f4a1c011bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65d76badac60995c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:38.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.098+0000 7f4a16a00640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:12:38.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.098+0000 7f4a16a00640 1 -- 172.21.15.37:0/569421657 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c010c60 con 0x55a1bab32260 2024-03-28T18:12:38.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.098+0000 7f4a3ac73740 1 -- 172.21.15.37:0/569421657 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4a28017070 con 0x55a1bab32260 2024-03-28T18:12:38.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.098+0000 7f4a16a00640 1 -- 172.21.15.37:0/569421657 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4a1c010e00 con 0x55a1bab32260 2024-03-28T18:12:38.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.099+0000 7f4a16a00640 1 -- 172.21.15.37:0/569421657 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a1c005020 con 0x55a1bab32260 2024-03-28T18:12:38.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.099+0000 7f4a3ac73740 10 client.25750 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:38.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.099+0000 7f4a3ac73740 1 -- 172.21.15.37:0/569421657 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a1ba8737d0 con 0x55a1bab32260 2024-03-28T18:12:38.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.099+0000 7f4a16a00640 1 -- 172.21.15.37:0/569421657 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a28017070 con 0x55a1bab32260 2024-03-28T18:12:38.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.099+0000 7f4a16a00640 1 -- 172.21.15.37:0/569421657 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f4a1c023430 con 0x55a1bab32260 2024-03-28T18:12:38.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.099+0000 7f4a3ac73740 10 client.25750 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:38.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.099+0000 7f4a3ac73740 10 client.25750 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:12:38.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.100+0000 7f4a3ac73740 20 client.25750 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:38.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.100+0000 7f4a3ac73740 1 --2- 172.21.15.37:0/569421657 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55a1bab4ac00 0x55a1bab4cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:38.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.100+0000 7f4a3ac73740 4 client.25750 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:12:38.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.100+0000 7f4a3ac73740 1 -- 172.21.15.37:0/569421657 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a1baaa2800 con 0x55a1bab4ac00 2024-03-28T18:12:38.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.100+0000 7f4a36000640 1 --2- 172.21.15.37:0/569421657 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55a1bab4ac00 0x55a1bab4cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:38.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.101+0000 7f4a36000640 1 --2- 172.21.15.37:0/569421657 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55a1bab4ac00 0x55a1bab4cff0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e65066e037e61b21 server_cookie=77e5817c199ba512 in_seq=0 out_seq=0 2024-03-28T18:12:38.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.101+0000 7f4a16a00640 10 client.25750 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:38.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.102+0000 7f4a16a00640 1 -- 172.21.15.37:0/569421657 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a1baaa2800 con 0x55a1bab4ac00 2024-03-28T18:12:38.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.102+0000 7f4a16a00640 10 client.25750 handle_command_reply: tid=0 2024-03-28T18:12:38.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.102+0000 7f4a3ac73740 1 client.25750 shutdown 2024-03-28T18:12:38.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.102+0000 7f4a3ac73740 1 -- 172.21.15.37:0/569421657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab32260 msgr2=0x55a1bab37aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:38.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.102+0000 7f4a3ac73740 1 --2- 172.21.15.37:0/569421657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab32260 0x55a1bab37aa0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c011730 tx=0x7f4a1c011bf0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.103+0000 7f4a3ac73740 1 -- 172.21.15.37:0/569421657 shutdown_connections 2024-03-28T18:12:38.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.103+0000 7f4a3ac73740 1 --2- 172.21.15.37:0/569421657 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55a1bab4ac00 0x55a1bab4cff0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.103+0000 7f4a3ac73740 1 --2- 172.21.15.37:0/569421657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab3ade0 0x55a1bab380b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.103+0000 7f4a3ac73740 1 --2- 172.21.15.37:0/569421657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab32260 0x55a1bab37aa0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.103+0000 7f4a3ac73740 1 --2- 172.21.15.37:0/569421657 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab318b0 0x55a1bab3a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.103+0000 7f4a3ac73740 1 -- 172.21.15.37:0/569421657 >> 172.21.15.37:0/569421657 conn(0x55a1bab2bf80 msgr2=0x55a1bab2dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:38.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.103+0000 7f4a16a00640 0 client.25750 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:38.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.103+0000 7f4a3ac73740 1 -- 172.21.15.37:0/569421657 shutdown_connections 2024-03-28T18:12:38.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.103+0000 7f4a3ac73740 1 -- 172.21.15.37:0/569421657 wait complete. 2024-03-28T18:12:38.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.103+0000 7f4a3ac73740 20 client.25750 trim_cache size 0 max 16384 2024-03-28T18:12:38.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.130+0000 7f4a3ac73740 1 Processor -- start 2024-03-28T18:12:38.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.130+0000 7f4a3ac73740 1 -- start start 2024-03-28T18:12:38.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.130+0000 7f4a3ac73740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab318b0 0x55a1bab2afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:38.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.130+0000 7f4a3ac73740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab32260 0x55a1bab3a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:38.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.130+0000 7f4a3ac73740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab3ade0 0x55a1bab3a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:38.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.130+0000 7f4a3ac73740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a301be360 con 0x55a1bab3ade0 2024-03-28T18:12:38.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.130+0000 7f4a3ac73740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a301bbe60 con 0x55a1bab32260 2024-03-28T18:12:38.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.130+0000 7f4a3ac73740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a301b9f00 con 0x55a1bab318b0 2024-03-28T18:12:38.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.131+0000 7f4a34c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab32260 0x55a1bab3a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:38.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.131+0000 7f4a34c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab32260 0x55a1bab3a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:44284/0 (socket says 172.21.15.37:44284) 2024-03-28T18:12:38.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.131+0000 7f4a34c00640 1 -- 172.21.15.37:0/1818294551 learned_addr learned my addr 172.21.15.37:0/1818294551 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:38.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.131+0000 7f4a36000640 1 --2- 172.21.15.37:0/1818294551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab318b0 0x55a1bab2afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:38.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.131+0000 7f4a36000640 1 --2- 172.21.15.37:0/1818294551 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab3ade0 0x55a1bab3a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:38.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.131+0000 7f4a36000640 1 -- 172.21.15.37:0/1818294551 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab32260 msgr2=0x55a1bab3a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:38.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.131+0000 7f4a36000640 1 --2- 172.21.15.37:0/1818294551 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab32260 0x55a1bab3a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.131+0000 7f4a36000640 1 -- 172.21.15.37:0/1818294551 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab3ade0 msgr2=0x55a1bab3a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:38.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.131+0000 7f4a36000640 1 --2- 172.21.15.37:0/1818294551 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab3ade0 0x55a1bab3a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.131+0000 7f4a36000640 1 -- 172.21.15.37:0/1818294551 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a1bab38420 con 0x55a1bab318b0 2024-03-28T18:12:38.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.132+0000 7f4a36000640 1 --2- 172.21.15.37:0/1818294551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab318b0 0x55a1bab2afe0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4a28005c40 tx=0x7f4a28009720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c3ebe39457bce57 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:38.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.132+0000 7f4a21c00640 1 -- 172.21.15.37:0/1818294551 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a28034070 con 0x55a1bab318b0 2024-03-28T18:12:38.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.132+0000 7f4a21c00640 1 -- 172.21.15.37:0/1818294551 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a28017030 con 0x55a1bab318b0 2024-03-28T18:12:38.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.132+0000 7f4a21c00640 1 -- 172.21.15.37:0/1818294551 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a28026040 con 0x55a1bab318b0 2024-03-28T18:12:38.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.133+0000 7f4a3ac73740 1 -- 172.21.15.37:0/1818294551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab318b0 msgr2=0x55a1bab2afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:38.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.133+0000 7f4a3ac73740 1 --2- 172.21.15.37:0/1818294551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab318b0 0x55a1bab2afe0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4a28005c40 tx=0x7f4a28009720 comp rx=0 tx=0).stop 2024-03-28T18:12:38.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.133+0000 7f4a3ac73740 1 -- 172.21.15.37:0/1818294551 shutdown_connections 2024-03-28T18:12:38.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.133+0000 7f4a3ac73740 1 --2- 172.21.15.37:0/1818294551 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab32260 0x55a1bab3a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.133+0000 7f4a3ac73740 1 --2- 172.21.15.37:0/1818294551 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab3ade0 0x55a1bab3a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.133+0000 7f4a3ac73740 1 --2- 172.21.15.37:0/1818294551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab318b0 0x55a1bab2afe0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.133+0000 7f4a3ac73740 1 -- 172.21.15.37:0/1818294551 >> 172.21.15.37:0/1818294551 conn(0x55a1bab2bf80 msgr2=0x55a1bab2a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:38.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.133+0000 7f4a3ac73740 1 -- 172.21.15.37:0/1818294551 shutdown_connections 2024-03-28T18:12:38.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.133+0000 7f4a3ac73740 1 -- 172.21.15.37:0/1818294551 wait complete. 2024-03-28T18:12:38.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.134+0000 7f4a3ac73740 1 -- start start 2024-03-28T18:12:38.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.134+0000 7f4a3ac73740 1 Processor -- start 2024-03-28T18:12:38.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.134+0000 7f4a3ac73740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab318b0 0x55a1bab40860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:38.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.134+0000 7f4a3ac73740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab32260 0x55a1bab40da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:38.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.135+0000 7f4a3ac73740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab3ade0 0x55a1baba5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:38.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.135+0000 7f4a3ac73740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a301ba500 con 0x55a1bab318b0 2024-03-28T18:12:38.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.135+0000 7f4a3ac73740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a301bb800 con 0x55a1bab32260 2024-03-28T18:12:38.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.135+0000 7f4a3ac73740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a301bab40 con 0x55a1bab3ade0 2024-03-28T18:12:38.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.135+0000 7f4a3ac73740 10 client.0 _ll_register_callbacks cb 0x7f4a1c00bca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:38.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.135+0000 7f4a34c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab32260 0x55a1bab40da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:38.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.135+0000 7f4a34c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab32260 0x55a1bab40da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:44288/0 (socket says 172.21.15.37:44288) 2024-03-28T18:12:38.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.135+0000 7f4a34c00640 1 -- 172.21.15.37:0/1020814443 learned_addr learned my addr 172.21.15.37:0/1020814443 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:38.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.135+0000 7f4a36000640 1 --2- 172.21.15.37:0/1020814443 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab318b0 0x55a1bab40860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:38.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.135+0000 7f4a36000640 1 --2- 172.21.15.37:0/1020814443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab3ade0 0x55a1baba5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:38.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.135+0000 7f4a34c00640 1 -- 172.21.15.37:0/1020814443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab3ade0 msgr2=0x55a1baba5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:38.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.136+0000 7f4a34c00640 1 --2- 172.21.15.37:0/1020814443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab3ade0 0x55a1baba5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.136+0000 7f4a34c00640 1 -- 172.21.15.37:0/1020814443 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab318b0 msgr2=0x55a1bab40860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:38.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.136+0000 7f4a34c00640 1 --2- 172.21.15.37:0/1020814443 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab318b0 0x55a1bab40860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.136+0000 7f4a34c00640 1 -- 172.21.15.37:0/1020814443 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a1bab41640 con 0x55a1bab32260 2024-03-28T18:12:38.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.136+0000 7f4a36000640 1 --2- 172.21.15.37:0/1020814443 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab318b0 0x55a1bab40860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:12:38.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.136+0000 7f4a34c00640 1 --2- 172.21.15.37:0/1020814443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab32260 0x55a1bab40da0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c011730 tx=0x7f4a1c023660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5164b7f91142f310 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:38.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.136+0000 7f4a16a00640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:12:38.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.136+0000 7f4a16a00640 1 -- 172.21.15.37:0/1020814443 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c010c60 con 0x55a1bab32260 2024-03-28T18:12:38.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.136+0000 7f4a3ac73740 1 -- 172.21.15.37:0/1020814443 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4a28017030 con 0x55a1bab32260 2024-03-28T18:12:38.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.136+0000 7f4a16a00640 1 -- 172.21.15.37:0/1020814443 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4a1c017070 con 0x55a1bab32260 2024-03-28T18:12:38.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.137+0000 7f4a16a00640 1 -- 172.21.15.37:0/1020814443 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a1c010e00 con 0x55a1bab32260 2024-03-28T18:12:38.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.137+0000 7f4a3ac73740 10 client.25742 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:38.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.137+0000 7f4a3ac73740 1 -- 172.21.15.37:0/1020814443 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a1baba6ab0 con 0x55a1bab32260 2024-03-28T18:12:38.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.137+0000 7f4a16a00640 1 -- 172.21.15.37:0/1020814443 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a28017030 con 0x55a1bab32260 2024-03-28T18:12:38.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.138+0000 7f4a16a00640 1 -- 172.21.15.37:0/1020814443 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f4a1c004020 con 0x55a1bab32260 2024-03-28T18:12:38.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.138+0000 7f4a3ac73740 10 client.25742 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:38.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.138+0000 7f4a3ac73740 10 client.25742 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:12:38.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.138+0000 7f4a3ac73740 20 client.25742 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:38.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.138+0000 7f4a3ac73740 1 --2- 172.21.15.37:0/1020814443 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55a1baba71d0 0x55a1baba95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:38.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.138+0000 7f4a3ac73740 4 client.25742 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:12:38.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.138+0000 7f4a3ac73740 1 -- 172.21.15.37:0/1020814443 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a1baa9cb50 con 0x55a1baba71d0 2024-03-28T18:12:38.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.138+0000 7f4a36000640 1 --2- 172.21.15.37:0/1020814443 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55a1baba71d0 0x55a1baba95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:38.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.139+0000 7f4a36000640 1 --2- 172.21.15.37:0/1020814443 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55a1baba71d0 0x55a1baba95f0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a9af3b05b801c3d server_cookie=70916f5d7608c53d in_seq=0 out_seq=0 2024-03-28T18:12:38.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.139+0000 7f4a16a00640 10 client.25742 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:38.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.139+0000 7f4a16a00640 1 -- 172.21.15.37:0/1020814443 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a1baa9cb50 con 0x55a1baba71d0 2024-03-28T18:12:38.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.139+0000 7f4a16a00640 10 client.25742 handle_command_reply: tid=0 2024-03-28T18:12:38.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.139+0000 7f4a3ac73740 1 client.25742 shutdown 2024-03-28T18:12:38.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.139+0000 7f4a3ac73740 1 -- 172.21.15.37:0/1020814443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab32260 msgr2=0x55a1bab40da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:38.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.139+0000 7f4a3ac73740 1 --2- 172.21.15.37:0/1020814443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab32260 0x55a1bab40da0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c011730 tx=0x7f4a1c023660 comp rx=0 tx=0).stop 2024-03-28T18:12:38.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.139+0000 7f4a3ac73740 1 -- 172.21.15.37:0/1020814443 shutdown_connections 2024-03-28T18:12:38.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.139+0000 7f4a3ac73740 1 --2- 172.21.15.37:0/1020814443 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55a1baba71d0 0x55a1baba95f0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.139+0000 7f4a3ac73740 1 --2- 172.21.15.37:0/1020814443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a1bab32260 0x55a1bab40da0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.139+0000 7f4a3ac73740 1 --2- 172.21.15.37:0/1020814443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a1bab3ade0 0x55a1baba5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.139+0000 7f4a3ac73740 1 --2- 172.21.15.37:0/1020814443 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a1bab318b0 0x55a1bab40860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.139+0000 7f4a3ac73740 1 -- 172.21.15.37:0/1020814443 >> 172.21.15.37:0/1020814443 conn(0x55a1bab2bf80 msgr2=0x55a1bab2db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:38.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.140+0000 7f4a16a00640 0 client.25742 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:38.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.140+0000 7f4a3ac73740 1 -- 172.21.15.37:0/1020814443 shutdown_connections 2024-03-28T18:12:38.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.140+0000 7f4a3ac73740 1 -- 172.21.15.37:0/1020814443 wait complete. 2024-03-28T18:12:38.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.140+0000 7f4a3ac73740 20 client.25742 trim_cache size 0 max 16384 2024-03-28T18:12:38.142 INFO:teuthology.orchestra.run.smithi037.stdout:[] 2024-03-28T18:12:38.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.141+0000 7f4a36a00640 1 -- 172.21.15.37:0/661675599 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f4a180a2b10 msgr2=0x7f4a180a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:38.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.141+0000 7f4a36a00640 1 --2- 172.21.15.37:0/661675599 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f4a180a2b10 0x7f4a180a4fd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4a240ee3e0 tx=0x7f4a240dd040 comp rx=0 tx=0).stop 2024-03-28T18:12:38.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.141+0000 7f4a36a00640 1 -- 172.21.15.37:0/661675599 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a301004c0 msgr2=0x7f4a30114660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:38.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.141+0000 7f4a36a00640 1 --2- 172.21.15.37:0/661675599 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a301004c0 0x7f4a30114660 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4a24000c00 tx=0x7f4a240043c0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.141+0000 7f4a36a00640 1 -- 172.21.15.37:0/661675599 shutdown_connections 2024-03-28T18:12:38.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.141+0000 7f4a36a00640 1 --2- 172.21.15.37:0/661675599 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f4a180a2b10 0x7f4a180a4fd0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.141+0000 7f4a36a00640 1 --2- 172.21.15.37:0/661675599 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a301058c0 0x7f4a3010d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.141+0000 7f4a36a00640 1 --2- 172.21.15.37:0/661675599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a30100de0 0x7f4a30114ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.141+0000 7f4a36a00640 1 --2- 172.21.15.37:0/661675599 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a301004c0 0x7f4a30114660 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:38.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.141+0000 7f4a36a00640 1 -- 172.21.15.37:0/661675599 >> 172.21.15.37:0/661675599 conn(0x7f4a300fbef0 msgr2=0x7f4a30103270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:38.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.141+0000 7f4a36a00640 1 -- 172.21.15.37:0/661675599 shutdown_connections 2024-03-28T18:12:38.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:38.141+0000 7f4a36a00640 1 -- 172.21.15.37:0/661675599 wait complete. 2024-03-28T18:12:39.167 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:12:39.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:12:39.167 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:12:39.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.451+0000 7f724f200640 1 -- 172.21.15.37:0/2386121610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72480764f0 msgr2=0x7f7248111fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:39.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.451+0000 7f724f200640 1 --2- 172.21.15.37:0/2386121610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72480764f0 0x7f7248111fa0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7234009930 tx=0x7f723402bc30 comp rx=0 tx=0).stop 2024-03-28T18:12:39.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.452+0000 7f724f200640 1 -- 172.21.15.37:0/2386121610 shutdown_connections 2024-03-28T18:12:39.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.452+0000 7f724f200640 1 --2- 172.21.15.37:0/2386121610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72480764f0 0x7f7248111fa0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.452+0000 7f724f200640 1 --2- 172.21.15.37:0/2386121610 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f72481124e0 0x7f72481148d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.452+0000 7f724f200640 1 --2- 172.21.15.37:0/2386121610 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7248075bd0 0x7f7248075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.452+0000 7f724f200640 1 -- 172.21.15.37:0/2386121610 >> 172.21.15.37:0/2386121610 conn(0x7f72480ff310 msgr2=0x7f7248101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:39.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.452+0000 7f724f200640 1 -- 172.21.15.37:0/2386121610 shutdown_connections 2024-03-28T18:12:39.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.452+0000 7f724f200640 1 -- 172.21.15.37:0/2386121610 wait complete. 2024-03-28T18:12:39.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.453+0000 7f724f200640 1 Processor -- start 2024-03-28T18:12:39.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.453+0000 7f724f200640 1 -- start start 2024-03-28T18:12:39.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.453+0000 7f724f200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7248075bd0 0x7f724819e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.453+0000 7f724f200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72480764f0 0x7f724819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.453+0000 7f724f200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72481124e0 0x7f72481a2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.453+0000 7f724f200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7248118090 con 0x7f7248075bd0 2024-03-28T18:12:39.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.453+0000 7f724f200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7248117f10 con 0x7f72480764f0 2024-03-28T18:12:39.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.453+0000 7f724f200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7248118210 con 0x7f72481124e0 2024-03-28T18:12:39.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.454+0000 7f7247e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7248075bd0 0x7f724819e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.454+0000 7f7247e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7248075bd0 0x7f724819e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36438/0 (socket says 172.21.15.37:36438) 2024-03-28T18:12:39.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.454+0000 7f7247e00640 1 -- 172.21.15.37:0/782818350 learned_addr learned my addr 172.21.15.37:0/782818350 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:39.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.454+0000 7f7247400640 1 --2- 172.21.15.37:0/782818350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72480764f0 0x7f724819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.454+0000 7f724d200640 1 --2- 172.21.15.37:0/782818350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72481124e0 0x7f72481a2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.454+0000 7f7247e00640 1 -- 172.21.15.37:0/782818350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72481124e0 msgr2=0x7f72481a2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:39.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.454+0000 7f7247e00640 1 --2- 172.21.15.37:0/782818350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72481124e0 0x7f72481a2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.454+0000 7f7247e00640 1 -- 172.21.15.37:0/782818350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72480764f0 msgr2=0x7f724819ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:39.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.454+0000 7f7247e00640 1 --2- 172.21.15.37:0/782818350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72480764f0 0x7f724819ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.454+0000 7f7247e00640 1 -- 172.21.15.37:0/782818350 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72481a3550 con 0x7f7248075bd0 2024-03-28T18:12:39.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.454+0000 7f724d200640 1 --2- 172.21.15.37:0/782818350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72481124e0 0x7f72481a2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:12:39.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.455+0000 7f7247e00640 1 --2- 172.21.15.37:0/782818350 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7248075bd0 0x7f724819e590 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f723800e9f0 tx=0x7f723800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:39.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.455+0000 7f7244c00640 1 -- 172.21.15.37:0/782818350 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f723800cd80 con 0x7f7248075bd0 2024-03-28T18:12:39.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.455+0000 7f7244c00640 1 -- 172.21.15.37:0/782818350 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7238010910 con 0x7f7248075bd0 2024-03-28T18:12:39.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.455+0000 7f7244c00640 1 -- 172.21.15.37:0/782818350 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7238018a00 con 0x7f7248075bd0 2024-03-28T18:12:39.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.455+0000 7f724f200640 1 -- 172.21.15.37:0/782818350 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72481a3840 con 0x7f7248075bd0 2024-03-28T18:12:39.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.455+0000 7f724f200640 1 -- 172.21.15.37:0/782818350 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7248103750 con 0x7f7248075bd0 2024-03-28T18:12:39.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.460+0000 7f7251f71740 1 Processor -- start 2024-03-28T18:12:39.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.460+0000 7f7251f71740 1 -- start start 2024-03-28T18:12:39.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.461+0000 7f7251f71740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8ae3080 0x555ad8ae3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.461+0000 7f7244c00640 1 -- 172.21.15.37:0/782818350 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f7238010ac0 con 0x7f7248075bd0 2024-03-28T18:12:39.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.461+0000 7f7251f71740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8aec040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.461+0000 7f7251f71740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8aec580 0x555ad8aee970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.462+0000 7f7251f71740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ad8a47eb0 con 0x555ad8aec580 2024-03-28T18:12:39.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.462+0000 7f724d200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8ae3080 0x555ad8ae3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.462+0000 7f7251f71740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ad89f0a60 con 0x555ad8ae39a0 2024-03-28T18:12:39.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.462+0000 7f7251f71740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ad8a43f50 con 0x555ad8ae3080 2024-03-28T18:12:39.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.462+0000 7f7247e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8aec040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.462+0000 7f724d200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8ae3080 0x555ad8ae3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:50114/0 (socket says 172.21.15.37:50114) 2024-03-28T18:12:39.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.462+0000 7f724d200640 1 -- 172.21.15.37:0/2962412218 learned_addr learned my addr 172.21.15.37:0/2962412218 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:39.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.463+0000 7f7247400640 1 --2- 172.21.15.37:0/2962412218 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8aec580 0x555ad8aee970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.463+0000 7f7244c00640 1 --2- 172.21.15.37:0/782818350 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f722c0a2b10 0x7f722c0a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.463+0000 7f7244c00640 1 -- 172.21.15.37:0/782818350 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f72380cf4b0 con 0x7f7248075bd0 2024-03-28T18:12:39.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.463+0000 7f7247400640 1 -- 172.21.15.37:0/2962412218 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8ae3080 msgr2=0x555ad8ae3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:39.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.463+0000 7f7247400640 1 --2- 172.21.15.37:0/2962412218 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8ae3080 0x555ad8ae3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.463+0000 7f7247400640 1 -- 172.21.15.37:0/2962412218 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 msgr2=0x555ad8aec040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:39.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.463+0000 7f7247400640 1 --2- 172.21.15.37:0/2962412218 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8aec040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.463+0000 7f7247400640 1 -- 172.21.15.37:0/2962412218 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ad88247e0 con 0x555ad8aec580 2024-03-28T18:12:39.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.463+0000 7f7247e00640 1 --2- 172.21.15.37:0/2962412218 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8aec040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:39.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.464+0000 7f7247400640 1 --2- 172.21.15.37:0/2962412218 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8aec580 0x555ad8aee970 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7234036040 tx=0x7f72340095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fc1910c5b2e0766 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:39.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.464+0000 7f724d200640 1 --2- 172.21.15.37:0/782818350 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f722c0a2b10 0x7f722c0a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.465+0000 7f723d600640 1 -- 172.21.15.37:0/2962412218 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7234045070 con 0x555ad8aec580 2024-03-28T18:12:39.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.465+0000 7f723d600640 1 -- 172.21.15.37:0/2962412218 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f723402c880 con 0x555ad8aec580 2024-03-28T18:12:39.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.465+0000 7f723d600640 1 -- 172.21.15.37:0/2962412218 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7234004b60 con 0x555ad8aec580 2024-03-28T18:12:39.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.466+0000 7f7251f71740 1 -- 172.21.15.37:0/2962412218 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8aec580 msgr2=0x555ad8aee970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:39.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.466+0000 7f724d200640 1 --2- 172.21.15.37:0/782818350 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f722c0a2b10 0x7f722c0a4fd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f72400124b0 tx=0x7f7240004a40 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:39.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.466+0000 7f7251f71740 1 --2- 172.21.15.37:0/2962412218 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8aec580 0x555ad8aee970 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7234036040 tx=0x7f72340095e0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.466+0000 7f7251f71740 1 -- 172.21.15.37:0/2962412218 shutdown_connections 2024-03-28T18:12:39.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.466+0000 7f7251f71740 1 --2- 172.21.15.37:0/2962412218 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8aec040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.466+0000 7f7251f71740 1 --2- 172.21.15.37:0/2962412218 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8aec580 0x555ad8aee970 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.466+0000 7f7251f71740 1 --2- 172.21.15.37:0/2962412218 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8ae3080 0x555ad8ae3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.466+0000 7f7251f71740 1 -- 172.21.15.37:0/2962412218 >> 172.21.15.37:0/2962412218 conn(0x555ad8add7b0 msgr2=0x555ad8addb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:39.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.466+0000 7f7251f71740 1 -- 172.21.15.37:0/2962412218 shutdown_connections 2024-03-28T18:12:39.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.466+0000 7f7251f71740 1 -- 172.21.15.37:0/2962412218 wait complete. 2024-03-28T18:12:39.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.467+0000 7f7251f71740 1 -- start start 2024-03-28T18:12:39.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.467+0000 7f7251f71740 1 Processor -- start 2024-03-28T18:12:39.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.468+0000 7f7251f71740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8ae3080 0x555ad8aebec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.468+0000 7f7251f71740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8ae9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.468+0000 7f7251f71740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8aec580 0x555ad8ae9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.468+0000 7f7251f71740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ad8aef310 con 0x555ad8ae3080 2024-03-28T18:12:39.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.468+0000 7f7251f71740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ad8a03eb0 con 0x555ad8ae39a0 2024-03-28T18:12:39.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.468+0000 7f7251f71740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ad8aef490 con 0x555ad8aec580 2024-03-28T18:12:39.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.468+0000 7f7251f71740 10 client.0 _ll_register_callbacks cb 0x555ad89f8150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:39.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.468+0000 7f724d200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8aec580 0x555ad8ae9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.468+0000 7f7247e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8ae3080 0x555ad8aebec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.468+0000 7f724d200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8aec580 0x555ad8ae9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:50116/0 (socket says 172.21.15.37:50116) 2024-03-28T18:12:39.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.468+0000 7f724d200640 1 -- 172.21.15.37:0/4032799274 learned_addr learned my addr 172.21.15.37:0/4032799274 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:39.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.468+0000 7f7247400640 1 --2- 172.21.15.37:0/4032799274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8ae9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.469+0000 7f724d200640 1 -- 172.21.15.37:0/4032799274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 msgr2=0x555ad8ae9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:39.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.469+0000 7f724d200640 1 --2- 172.21.15.37:0/4032799274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8ae9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.469+0000 7f724d200640 1 -- 172.21.15.37:0/4032799274 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8ae3080 msgr2=0x555ad8aebec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:39.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.469+0000 7f724d200640 1 --2- 172.21.15.37:0/4032799274 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8ae3080 0x555ad8aebec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.469+0000 7f724d200640 1 -- 172.21.15.37:0/4032799274 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555ad8884f70 con 0x555ad8aec580 2024-03-28T18:12:39.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.469+0000 7f724d200640 1 --2- 172.21.15.37:0/4032799274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8aec580 0x555ad8ae9790 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f724000f8e0 tx=0x7f7240004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84ca35c9dbfd867 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:39.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.469+0000 7f7232000640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:12:39.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.469+0000 7f7247400640 1 --2- 172.21.15.37:0/4032799274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8ae9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:39.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.469+0000 7f7232000640 1 -- 172.21.15.37:0/4032799274 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724002a060 con 0x555ad8aec580 2024-03-28T18:12:39.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.469+0000 7f7232000640 1 -- 172.21.15.37:0/4032799274 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7240017070 con 0x555ad8aec580 2024-03-28T18:12:39.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.469+0000 7f7251f71740 1 -- 172.21.15.37:0/4032799274 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f723402c880 con 0x555ad8aec580 2024-03-28T18:12:39.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.470+0000 7f7232000640 1 -- 172.21.15.37:0/4032799274 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f724001dbe0 con 0x555ad8aec580 2024-03-28T18:12:39.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.470+0000 7f7251f71740 10 client.25765 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:39.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.470+0000 7f7251f71740 1 -- 172.21.15.37:0/4032799274 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555ad871c170 con 0x555ad8aec580 2024-03-28T18:12:39.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.470+0000 7f7232000640 1 -- 172.21.15.37:0/4032799274 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f723402c880 con 0x555ad8aec580 2024-03-28T18:12:39.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.470+0000 7f7232000640 1 -- 172.21.15.37:0/4032799274 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f72400103d0 con 0x555ad8aec580 2024-03-28T18:12:39.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.470+0000 7f7251f71740 10 client.25765 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:39.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.470+0000 7f7251f71740 10 client.25765 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:12:39.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.470+0000 7f7251f71740 20 client.25765 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:39.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.471+0000 7f7251f71740 1 --2- 172.21.15.37:0/4032799274 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x555ad8afc630 0x555ad8afea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.471+0000 7f7251f71740 4 client.25765 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:12:39.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.471+0000 7f7251f71740 1 -- 172.21.15.37:0/4032799274 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555ad8a553f0 con 0x555ad8afc630 2024-03-28T18:12:39.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.471+0000 7f7247e00640 1 --2- 172.21.15.37:0/4032799274 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x555ad8afc630 0x555ad8afea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.472+0000 7f7247e00640 1 --2- 172.21.15.37:0/4032799274 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x555ad8afc630 0x555ad8afea20 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fab7b23df953133b server_cookie=54ca83d5a401b9c9 in_seq=0 out_seq=0 2024-03-28T18:12:39.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.472+0000 7f7232000640 10 client.25765 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:39.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.473+0000 7f7232000640 1 -- 172.21.15.37:0/4032799274 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555ad8a553f0 con 0x555ad8afc630 2024-03-28T18:12:39.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.473+0000 7f7232000640 10 client.25765 handle_command_reply: tid=0 2024-03-28T18:12:39.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.473+0000 7f7251f71740 1 client.25765 shutdown 2024-03-28T18:12:39.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.473+0000 7f7251f71740 1 -- 172.21.15.37:0/4032799274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8aec580 msgr2=0x555ad8ae9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:39.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.473+0000 7f7251f71740 1 --2- 172.21.15.37:0/4032799274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8aec580 0x555ad8ae9790 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f724000f8e0 tx=0x7f7240004ab0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.473+0000 7f7251f71740 1 -- 172.21.15.37:0/4032799274 shutdown_connections 2024-03-28T18:12:39.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.473+0000 7f7251f71740 1 --2- 172.21.15.37:0/4032799274 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x555ad8afc630 0x555ad8afea20 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.473+0000 7f7251f71740 1 --2- 172.21.15.37:0/4032799274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8ae9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.473+0000 7f7251f71740 1 --2- 172.21.15.37:0/4032799274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8aec580 0x555ad8ae9790 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.473+0000 7f7251f71740 1 --2- 172.21.15.37:0/4032799274 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8ae3080 0x555ad8aebec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.473+0000 7f7232000640 0 client.25765 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:39.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.473+0000 7f7251f71740 1 -- 172.21.15.37:0/4032799274 >> 172.21.15.37:0/4032799274 conn(0x555ad8add7b0 msgr2=0x555ad8adf500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:39.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.474+0000 7f7251f71740 1 -- 172.21.15.37:0/4032799274 shutdown_connections 2024-03-28T18:12:39.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.474+0000 7f7251f71740 1 -- 172.21.15.37:0/4032799274 wait complete. 2024-03-28T18:12:39.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.474+0000 7f7251f71740 20 client.25765 trim_cache size 0 max 16384 2024-03-28T18:12:39.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.491+0000 7f7251f71740 1 Processor -- start 2024-03-28T18:12:39.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.491+0000 7f7251f71740 1 -- start start 2024-03-28T18:12:39.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.492+0000 7f7251f71740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8ae3080 0x555ad8aeb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.492+0000 7f7251f71740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8aebec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.492+0000 7f7251f71740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8aec580 0x555ad8af7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.492+0000 7f7251f71740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72481be660 con 0x555ad8aec580 2024-03-28T18:12:39.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.492+0000 7f7251f71740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72481bc160 con 0x555ad8ae39a0 2024-03-28T18:12:39.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.492+0000 7f7251f71740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724810b980 con 0x555ad8ae3080 2024-03-28T18:12:39.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.492+0000 7f7247400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8aebec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.492+0000 7f7247e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8ae3080 0x555ad8aeb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.492+0000 7f7247400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8aebec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:44322/0 (socket says 172.21.15.37:44322) 2024-03-28T18:12:39.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.492+0000 7f7247400640 1 -- 172.21.15.37:0/1624023342 learned_addr learned my addr 172.21.15.37:0/1624023342 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:39.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.492+0000 7f724d200640 1 --2- 172.21.15.37:0/1624023342 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8aec580 0x555ad8af7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.493+0000 7f7247400640 1 -- 172.21.15.37:0/1624023342 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8ae3080 msgr2=0x555ad8aeb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:39.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.493+0000 7f7247400640 1 --2- 172.21.15.37:0/1624023342 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8ae3080 0x555ad8aeb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.493+0000 7f7247400640 1 -- 172.21.15.37:0/1624023342 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8aec580 msgr2=0x555ad8af7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:39.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.493+0000 7f7247400640 1 --2- 172.21.15.37:0/1624023342 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8aec580 0x555ad8af7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.493+0000 7f7247400640 1 -- 172.21.15.37:0/1624023342 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ad8ae45a0 con 0x555ad8ae39a0 2024-03-28T18:12:39.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.493+0000 7f7247e00640 1 --2- 172.21.15.37:0/1624023342 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8ae3080 0x555ad8aeb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:39.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.493+0000 7f7247400640 1 --2- 172.21.15.37:0/1624023342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8aebec0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f72340047d0 tx=0x7f7234004800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a28bb993fc5d14b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:39.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.493+0000 7f723cc00640 1 -- 172.21.15.37:0/1624023342 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7234045040 con 0x555ad8ae39a0 2024-03-28T18:12:39.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.493+0000 7f723cc00640 1 -- 172.21.15.37:0/1624023342 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f723402c880 con 0x555ad8ae39a0 2024-03-28T18:12:39.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.493+0000 7f723cc00640 1 -- 172.21.15.37:0/1624023342 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f723403f440 con 0x555ad8ae39a0 2024-03-28T18:12:39.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.494+0000 7f7251f71740 1 -- 172.21.15.37:0/1624023342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 msgr2=0x555ad8aebec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:39.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.494+0000 7f7251f71740 1 --2- 172.21.15.37:0/1624023342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8aebec0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f72340047d0 tx=0x7f7234004800 comp rx=0 tx=0).stop 2024-03-28T18:12:39.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.494+0000 7f7251f71740 1 -- 172.21.15.37:0/1624023342 shutdown_connections 2024-03-28T18:12:39.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.494+0000 7f7251f71740 1 --2- 172.21.15.37:0/1624023342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8aebec0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.494+0000 7f7251f71740 1 --2- 172.21.15.37:0/1624023342 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8aec580 0x555ad8af7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.494+0000 7f7251f71740 1 --2- 172.21.15.37:0/1624023342 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8ae3080 0x555ad8aeb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.494+0000 7f7251f71740 1 -- 172.21.15.37:0/1624023342 >> 172.21.15.37:0/1624023342 conn(0x555ad8add7b0 msgr2=0x555ad8ae9370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:39.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.494+0000 7f7251f71740 1 -- 172.21.15.37:0/1624023342 shutdown_connections 2024-03-28T18:12:39.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.494+0000 7f7251f71740 1 -- 172.21.15.37:0/1624023342 wait complete. 2024-03-28T18:12:39.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.495+0000 7f7251f71740 1 -- start start 2024-03-28T18:12:39.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.495+0000 7f7251f71740 1 Processor -- start 2024-03-28T18:12:39.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.495+0000 7f7251f71740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8ae3080 0x555ad8ae60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.495+0000 7f7251f71740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8ae65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.495+0000 7f7251f71740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8aec580 0x555ad8b50e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.495+0000 7f7251f71740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724810bf80 con 0x555ad8ae3080 2024-03-28T18:12:39.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.495+0000 7f7251f71740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724810d280 con 0x555ad8ae39a0 2024-03-28T18:12:39.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.495+0000 7f7251f71740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724810c5c0 con 0x555ad8aec580 2024-03-28T18:12:39.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.495+0000 7f7251f71740 10 client.0 _ll_register_callbacks cb 0x7f7240009d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:39.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.496+0000 7f7247e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8ae3080 0x555ad8ae60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.496+0000 7f7247400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8ae65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.496+0000 7f7247400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8ae65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:44332/0 (socket says 172.21.15.37:44332) 2024-03-28T18:12:39.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.496+0000 7f7247400640 1 -- 172.21.15.37:0/3468859021 learned_addr learned my addr 172.21.15.37:0/3468859021 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:39.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.496+0000 7f724d200640 1 --2- 172.21.15.37:0/3468859021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8aec580 0x555ad8b50e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.496+0000 7f7247400640 1 -- 172.21.15.37:0/3468859021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8aec580 msgr2=0x555ad8b50e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:39.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.496+0000 7f7247400640 1 --2- 172.21.15.37:0/3468859021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8aec580 0x555ad8b50e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.496+0000 7f7247400640 1 -- 172.21.15.37:0/3468859021 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8ae3080 msgr2=0x555ad8ae60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:39.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.496+0000 7f7247400640 1 --2- 172.21.15.37:0/3468859021 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8ae3080 0x555ad8ae60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.496+0000 7f7247400640 1 -- 172.21.15.37:0/3468859021 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555ad8aeece0 con 0x555ad8ae39a0 2024-03-28T18:12:39.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.496+0000 7f724d200640 1 --2- 172.21.15.37:0/3468859021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8aec580 0x555ad8b50e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:39.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.497+0000 7f7247400640 1 --2- 172.21.15.37:0/3468859021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8ae65e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f723402c170 tx=0x7f72340095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d65bb3b3637a38c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:39.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.497+0000 7f7232000640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:12:39.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.497+0000 7f7232000640 1 -- 172.21.15.37:0/3468859021 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7234045040 con 0x555ad8ae39a0 2024-03-28T18:12:39.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.497+0000 7f7232000640 1 -- 172.21.15.37:0/3468859021 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f723404a450 con 0x555ad8ae39a0 2024-03-28T18:12:39.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.497+0000 7f7251f71740 1 -- 172.21.15.37:0/3468859021 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f723402c880 con 0x555ad8ae39a0 2024-03-28T18:12:39.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.497+0000 7f7232000640 1 -- 172.21.15.37:0/3468859021 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f723402cd40 con 0x555ad8ae39a0 2024-03-28T18:12:39.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.497+0000 7f7251f71740 10 client.25760 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:39.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.497+0000 7f7251f71740 1 -- 172.21.15.37:0/3468859021 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555ad8b58650 con 0x555ad8ae39a0 2024-03-28T18:12:39.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.497+0000 7f7232000640 1 -- 172.21.15.37:0/3468859021 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f723402c880 con 0x555ad8ae39a0 2024-03-28T18:12:39.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.498+0000 7f7232000640 1 -- 172.21.15.37:0/3468859021 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f723403f440 con 0x555ad8ae39a0 2024-03-28T18:12:39.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.498+0000 7f7251f71740 10 client.25760 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:39.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.498+0000 7f7251f71740 10 client.25760 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:12:39.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.498+0000 7f7251f71740 20 client.25760 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:39.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.498+0000 7f7251f71740 1 --2- 172.21.15.37:0/3468859021 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x555ad8b58880 0x555ad8b5ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.498+0000 7f7251f71740 4 client.25760 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:12:39.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.498+0000 7f7251f71740 1 -- 172.21.15.37:0/3468859021 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x555ad8a4f740 con 0x555ad8b58880 2024-03-28T18:12:39.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.498+0000 7f7247e00640 1 --2- 172.21.15.37:0/3468859021 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x555ad8b58880 0x555ad8b5ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.499+0000 7f7247e00640 1 --2- 172.21.15.37:0/3468859021 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x555ad8b58880 0x555ad8b5ac70 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5d80c86bee21cb server_cookie=ff5c9a8e33df70c4 in_seq=0 out_seq=0 2024-03-28T18:12:39.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.499+0000 7f7232000640 10 client.25760 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:39.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.501+0000 7f7232000640 1 -- 172.21.15.37:0/3468859021 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x555ad8a4f740 con 0x555ad8b58880 2024-03-28T18:12:39.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.501+0000 7f7232000640 10 client.25760 handle_command_reply: tid=0 2024-03-28T18:12:39.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.501+0000 7f7251f71740 1 client.25760 shutdown 2024-03-28T18:12:39.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.501+0000 7f7251f71740 1 -- 172.21.15.37:0/3468859021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 msgr2=0x555ad8ae65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:39.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.501+0000 7f7251f71740 1 --2- 172.21.15.37:0/3468859021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8ae65e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f723402c170 tx=0x7f72340095e0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.502+0000 7f7251f71740 1 -- 172.21.15.37:0/3468859021 shutdown_connections 2024-03-28T18:12:39.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.502+0000 7f7251f71740 1 --2- 172.21.15.37:0/3468859021 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x555ad8b58880 0x555ad8b5ac70 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.502+0000 7f7251f71740 1 --2- 172.21.15.37:0/3468859021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555ad8ae39a0 0x555ad8ae65e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.502+0000 7f7251f71740 1 --2- 172.21.15.37:0/3468859021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555ad8aec580 0x555ad8b50e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.502+0000 7f7251f71740 1 --2- 172.21.15.37:0/3468859021 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x555ad8ae3080 0x555ad8ae60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.502+0000 7f7251f71740 1 -- 172.21.15.37:0/3468859021 >> 172.21.15.37:0/3468859021 conn(0x555ad8add7b0 msgr2=0x555ad8adbfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:39.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.502+0000 7f7232000640 0 client.25760 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:39.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.502+0000 7f7251f71740 1 -- 172.21.15.37:0/3468859021 shutdown_connections 2024-03-28T18:12:39.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.502+0000 7f7251f71740 1 -- 172.21.15.37:0/3468859021 wait complete. 2024-03-28T18:12:39.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.502+0000 7f7251f71740 20 client.25760 trim_cache size 0 max 16384 2024-03-28T18:12:39.504 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:12:39.505 INFO:teuthology.orchestra.run.smithi037.stdout: "return_code": 0, 2024-03-28T18:12:39.505 INFO:teuthology.orchestra.run.smithi037.stdout: "scrub_tag": "d178a304-cf1e-48ee-a9a1-2a9f48cccf7f", 2024-03-28T18:12:39.505 INFO:teuthology.orchestra.run.smithi037.stdout: "mode": "asynchronous" 2024-03-28T18:12:39.505 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:12:39.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.505+0000 7f724f200640 1 -- 172.21.15.37:0/782818350 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f722c0a2b10 msgr2=0x7f722c0a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:39.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.505+0000 7f724f200640 1 --2- 172.21.15.37:0/782818350 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f722c0a2b10 0x7f722c0a4fd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f72400124b0 tx=0x7f7240004a40 comp rx=0 tx=0).stop 2024-03-28T18:12:39.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.505+0000 7f724f200640 1 -- 172.21.15.37:0/782818350 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7248075bd0 msgr2=0x7f724819e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:39.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.505+0000 7f724f200640 1 --2- 172.21.15.37:0/782818350 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7248075bd0 0x7f724819e590 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f723800e9f0 tx=0x7f723800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.505+0000 7f724f200640 1 -- 172.21.15.37:0/782818350 shutdown_connections 2024-03-28T18:12:39.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.505+0000 7f724f200640 1 --2- 172.21.15.37:0/782818350 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f722c0a2b10 0x7f722c0a4fd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.505+0000 7f724f200640 1 --2- 172.21.15.37:0/782818350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72480764f0 0x7f724819ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.505+0000 7f724f200640 1 --2- 172.21.15.37:0/782818350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72481124e0 0x7f72481a2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.505+0000 7f724f200640 1 --2- 172.21.15.37:0/782818350 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7248075bd0 0x7f724819e590 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.505+0000 7f724f200640 1 -- 172.21.15.37:0/782818350 >> 172.21.15.37:0/782818350 conn(0x7f72480ff310 msgr2=0x7f7248113c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:39.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.505+0000 7f724f200640 1 -- 172.21.15.37:0/782818350 shutdown_connections 2024-03-28T18:12:39.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.505+0000 7f724f200640 1 -- 172.21.15.37:0/782818350 wait complete. 2024-03-28T18:12:39.523 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:12:39.523 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:12:39.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.717+0000 7f89b4e00640 1 -- 172.21.15.37:0/1815072668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89b0101530 msgr2=0x7f89b01019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:39.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.717+0000 7f89b4e00640 1 --2- 172.21.15.37:0/1815072668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89b0101530 0x7f89b01019b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f899c009930 tx=0x7f899c02bc20 comp rx=0 tx=0).stop 2024-03-28T18:12:39.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.717+0000 7f89b4e00640 1 -- 172.21.15.37:0/1815072668 shutdown_connections 2024-03-28T18:12:39.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.717+0000 7f89b4e00640 1 --2- 172.21.15.37:0/1815072668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89b0101530 0x7f89b01019b0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.717+0000 7f89b4e00640 1 --2- 172.21.15.37:0/1815072668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89b0101ef0 0x7f89b010e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.717+0000 7f89b4e00640 1 --2- 172.21.15.37:0/1815072668 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89b01075f0 0x7f89b01079d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.717+0000 7f89b4e00640 1 -- 172.21.15.37:0/1815072668 >> 172.21.15.37:0/1815072668 conn(0x7f89b00fd040 msgr2=0x7f89b00ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:39.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.717+0000 7f89b4e00640 1 -- 172.21.15.37:0/1815072668 shutdown_connections 2024-03-28T18:12:39.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.717+0000 7f89b4e00640 1 -- 172.21.15.37:0/1815072668 wait complete. 2024-03-28T18:12:39.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.718+0000 7f89b4e00640 1 Processor -- start 2024-03-28T18:12:39.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.718+0000 7f89b4e00640 1 -- start start 2024-03-28T18:12:39.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.718+0000 7f89b4e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89b0101530 0x7f89b0197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.718+0000 7f89b4e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89b0101ef0 0x7f89b01982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.718+0000 7f89b4e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89b01075f0 0x7f89b019c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.719+0000 7f89b4e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b0111900 con 0x7f89b0101ef0 2024-03-28T18:12:39.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.719+0000 7f89b4e00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b0111780 con 0x7f89b0101530 2024-03-28T18:12:39.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.719+0000 7f89b4e00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b0111a80 con 0x7f89b01075f0 2024-03-28T18:12:39.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.719+0000 7f89afe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89b01075f0 0x7f89b019c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.719+0000 7f89afe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89b01075f0 0x7f89b019c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:50154/0 (socket says 172.21.15.37:50154) 2024-03-28T18:12:39.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.719+0000 7f89afe00640 1 -- 172.21.15.37:0/1717729294 learned_addr learned my addr 172.21.15.37:0/1717729294 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:39.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.719+0000 7f89af400640 1 --2- 172.21.15.37:0/1717729294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89b0101530 0x7f89b0197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.719+0000 7f89aea00640 1 --2- 172.21.15.37:0/1717729294 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89b0101ef0 0x7f89b01982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.720+0000 7f89af400640 1 -- 172.21.15.37:0/1717729294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89b01075f0 msgr2=0x7f89b019c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:39.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.720+0000 7f89af400640 1 --2- 172.21.15.37:0/1717729294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89b01075f0 0x7f89b019c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.720+0000 7f89af400640 1 -- 172.21.15.37:0/1717729294 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89b0101ef0 msgr2=0x7f89b01982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:39.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.720+0000 7f89af400640 1 --2- 172.21.15.37:0/1717729294 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89b0101ef0 0x7f89b01982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.720+0000 7f89af400640 1 -- 172.21.15.37:0/1717729294 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b019cd40 con 0x7f89b0101530 2024-03-28T18:12:39.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.720+0000 7f89afe00640 1 --2- 172.21.15.37:0/1717729294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89b01075f0 0x7f89b019c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:39.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.720+0000 7f89aea00640 1 --2- 172.21.15.37:0/1717729294 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89b0101ef0 0x7f89b01982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:39.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.721+0000 7f89af400640 1 --2- 172.21.15.37:0/1717729294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89b0101530 0x7f89b0197d80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f89a000d6d0 tx=0x7f89a000db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:39.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.721+0000 7f89a7e00640 1 -- 172.21.15.37:0/1717729294 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a0017070 con 0x7f89b0101530 2024-03-28T18:12:39.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.721+0000 7f89a7e00640 1 -- 172.21.15.37:0/1717729294 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89a0009820 con 0x7f89b0101530 2024-03-28T18:12:39.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.721+0000 7f89a7e00640 1 -- 172.21.15.37:0/1717729294 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a0004d00 con 0x7f89b0101530 2024-03-28T18:12:39.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.721+0000 7f89b4e00640 1 -- 172.21.15.37:0/1717729294 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b019d030 con 0x7f89b0101530 2024-03-28T18:12:39.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.722+0000 7f89b4e00640 1 -- 172.21.15.37:0/1717729294 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b00738d0 con 0x7f89b0101530 2024-03-28T18:12:39.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.725+0000 7f89b91e9740 1 Processor -- start 2024-03-28T18:12:39.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.725+0000 7f89b91e9740 1 -- start start 2024-03-28T18:12:39.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.725+0000 7f89b91e9740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f74dd0 0x564c90f73b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.725+0000 7f89b91e9740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f740f0 0x564c90f74570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.725+0000 7f89b91e9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f7d460 0x564c90f7f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.725+0000 7f89b91e9740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c90ed8250 con 0x564c90f740f0 2024-03-28T18:12:39.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.725+0000 7f89b91e9740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c90e80a30 con 0x564c90f74dd0 2024-03-28T18:12:39.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.725+0000 7f89b91e9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c90ed42f0 con 0x564c90f7d460 2024-03-28T18:12:39.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.726+0000 7f89afe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f74dd0 0x564c90f73b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.726+0000 7f89afe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f74dd0 0x564c90f73b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:44374/0 (socket says 172.21.15.37:44374) 2024-03-28T18:12:39.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.726+0000 7f89afe00640 1 -- 172.21.15.37:0/258714631 learned_addr learned my addr 172.21.15.37:0/258714631 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:39.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.726+0000 7f89afe00640 1 -- 172.21.15.37:0/258714631 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f7d460 msgr2=0x564c90f7f850 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:12:39.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.726+0000 7f89afe00640 1 --2- 172.21.15.37:0/258714631 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f7d460 0x564c90f7f850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.726+0000 7f89afe00640 1 -- 172.21.15.37:0/258714631 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f740f0 msgr2=0x564c90f74570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:12:39.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.726+0000 7f89afe00640 1 --2- 172.21.15.37:0/258714631 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f740f0 0x564c90f74570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.726+0000 7f89afe00640 1 -- 172.21.15.37:0/258714631 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c90e6b690 con 0x564c90f74dd0 2024-03-28T18:12:39.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.726+0000 7f89a7e00640 1 -- 172.21.15.37:0/1717729294 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f89a00099d0 con 0x7f89b0101530 2024-03-28T18:12:39.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.727+0000 7f89afe00640 1 --2- 172.21.15.37:0/258714631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f74dd0 0x564c90f73b20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f89a80118b0 tx=0x7f89a8011d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a20e994d50ad6aad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:39.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.728+0000 7f89a4c00640 1 -- 172.21.15.37:0/258714631 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a801e070 con 0x564c90f74dd0 2024-03-28T18:12:39.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.728+0000 7f89a4c00640 1 -- 172.21.15.37:0/258714631 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89a8019410 con 0x564c90f74dd0 2024-03-28T18:12:39.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.728+0000 7f89a4c00640 1 -- 172.21.15.37:0/258714631 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a800fea0 con 0x564c90f74dd0 2024-03-28T18:12:39.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.728+0000 7f89a7e00640 1 --2- 172.21.15.37:0/1717729294 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f89940a2b40 0x7f89940a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.728+0000 7f89a7e00640 1 -- 172.21.15.37:0/1717729294 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f89a00ceb50 con 0x7f89b0101530 2024-03-28T18:12:39.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.728+0000 7f89b91e9740 1 -- 172.21.15.37:0/258714631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f74dd0 msgr2=0x564c90f73b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:39.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.728+0000 7f89b91e9740 1 --2- 172.21.15.37:0/258714631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f74dd0 0x564c90f73b20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f89a80118b0 tx=0x7f89a8011d70 comp rx=0 tx=0).stop 2024-03-28T18:12:39.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.729+0000 7f89af400640 1 --2- 172.21.15.37:0/1717729294 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f89940a2b40 0x7f89940a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.729+0000 7f89b91e9740 1 -- 172.21.15.37:0/258714631 shutdown_connections 2024-03-28T18:12:39.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.729+0000 7f89b91e9740 1 --2- 172.21.15.37:0/258714631 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f7d460 0x564c90f7f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.729+0000 7f89b91e9740 1 --2- 172.21.15.37:0/258714631 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f740f0 0x564c90f74570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.729+0000 7f89b91e9740 1 --2- 172.21.15.37:0/258714631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f74dd0 0x564c90f73b20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.729+0000 7f89b91e9740 1 -- 172.21.15.37:0/258714631 >> 172.21.15.37:0/258714631 conn(0x564c90f3b200 msgr2=0x564c90f6eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:39.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.730+0000 7f89af400640 1 --2- 172.21.15.37:0/1717729294 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f89940a2b40 0x7f89940a5000 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f89a00cc1b0 tx=0x7f89a00057f0 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:39.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.730+0000 7f89b91e9740 1 -- 172.21.15.37:0/258714631 shutdown_connections 2024-03-28T18:12:39.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.730+0000 7f89b91e9740 1 -- 172.21.15.37:0/258714631 wait complete. 2024-03-28T18:12:39.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.730+0000 7f89b91e9740 1 -- start start 2024-03-28T18:12:39.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.730+0000 7f89b91e9740 1 Processor -- start 2024-03-28T18:12:39.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.731+0000 7f89b91e9740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f740f0 0x564c90f7cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.731+0000 7f89b91e9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f74dd0 0x564c90f79f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.731+0000 7f89b91e9740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f7d460 0x564c90f7a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.731+0000 7f89b91e9740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c90f802d0 con 0x564c90f740f0 2024-03-28T18:12:39.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.731+0000 7f89b91e9740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c90e93e80 con 0x564c90f7d460 2024-03-28T18:12:39.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.731+0000 7f89b91e9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c90f80450 con 0x564c90f74dd0 2024-03-28T18:12:39.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.731+0000 7f89b91e9740 10 client.0 _ll_register_callbacks cb 0x564c90ea6130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:39.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.732+0000 7f89aea00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f74dd0 0x564c90f79f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.732+0000 7f89aea00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f74dd0 0x564c90f79f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:50178/0 (socket says 172.21.15.37:50178) 2024-03-28T18:12:39.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.732+0000 7f89aea00640 1 -- 172.21.15.37:0/2867529605 learned_addr learned my addr 172.21.15.37:0/2867529605 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:39.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.732+0000 7f89afe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f7d460 0x564c90f7a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.732+0000 7f89afe00640 1 --2- 172.21.15.37:0/2867529605 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f740f0 0x564c90f7cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.733+0000 7f89aea00640 1 -- 172.21.15.37:0/2867529605 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f7d460 msgr2=0x564c90f7a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:39.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.733+0000 7f89aea00640 1 --2- 172.21.15.37:0/2867529605 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f7d460 0x564c90f7a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.733+0000 7f89aea00640 1 -- 172.21.15.37:0/2867529605 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f740f0 msgr2=0x564c90f7cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:39.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.733+0000 7f89aea00640 1 --2- 172.21.15.37:0/2867529605 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f740f0 0x564c90f7cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.733+0000 7f89aea00640 1 -- 172.21.15.37:0/2867529605 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564c90cb67d0 con 0x564c90f74dd0 2024-03-28T18:12:39.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.733+0000 7f89afe00640 1 --2- 172.21.15.37:0/2867529605 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f740f0 0x564c90f7cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:39.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.734+0000 7f89aea00640 1 --2- 172.21.15.37:0/2867529605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f74dd0 0x564c90f79f40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f899c02c230 tx=0x7f899c004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbe1d0701cf3440e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:39.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.734+0000 7f8999600640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:12:39.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.735+0000 7f8999600640 1 -- 172.21.15.37:0/2867529605 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899c045070 con 0x564c90f74dd0 2024-03-28T18:12:39.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.735+0000 7f8999600640 1 -- 172.21.15.37:0/2867529605 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f899c0043c0 con 0x564c90f74dd0 2024-03-28T18:12:39.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.735+0000 7f89b91e9740 1 -- 172.21.15.37:0/2867529605 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f89a8019410 con 0x564c90f74dd0 2024-03-28T18:12:39.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.735+0000 7f8999600640 1 -- 172.21.15.37:0/2867529605 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f899c051050 con 0x564c90f74dd0 2024-03-28T18:12:39.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.735+0000 7f89b91e9740 10 client.25780 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:39.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.735+0000 7f89b91e9740 1 -- 172.21.15.37:0/2867529605 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564c90bec3f0 con 0x564c90f74dd0 2024-03-28T18:12:39.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.735+0000 7f8999600640 1 -- 172.21.15.37:0/2867529605 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89a8019410 con 0x564c90f74dd0 2024-03-28T18:12:39.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.735+0000 7f8999600640 1 -- 172.21.15.37:0/2867529605 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f899c0035f0 con 0x564c90f74dd0 2024-03-28T18:12:39.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.736+0000 7f89b91e9740 10 client.25780 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:39.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.736+0000 7f89b91e9740 10 client.25780 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:12:39.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.736+0000 7f89b91e9740 20 client.25780 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:39.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.736+0000 7f89b91e9740 1 --2- 172.21.15.37:0/2867529605 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x564c90f8d340 0x564c90f8f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.736+0000 7f89b91e9740 4 client.25780 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:12:39.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.736+0000 7f89b91e9740 1 -- 172.21.15.37:0/2867529605 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564c90ee5790 con 0x564c90f8d340 2024-03-28T18:12:39.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.736+0000 7f89afe00640 1 --2- 172.21.15.37:0/2867529605 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x564c90f8d340 0x564c90f8f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.737+0000 7f89afe00640 1 --2- 172.21.15.37:0/2867529605 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x564c90f8d340 0x564c90f8f730 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e158a077fdbf83f9 server_cookie=de6acd40f1ec131 in_seq=0 out_seq=0 2024-03-28T18:12:39.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.737+0000 7f8999600640 10 client.25780 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:39.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.739+0000 7f8999600640 1 -- 172.21.15.37:0/2867529605 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564c90ee5790 con 0x564c90f8d340 2024-03-28T18:12:39.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.739+0000 7f8999600640 10 client.25780 handle_command_reply: tid=0 2024-03-28T18:12:39.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.739+0000 7f89b91e9740 1 client.25780 shutdown 2024-03-28T18:12:39.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.739+0000 7f89b91e9740 1 -- 172.21.15.37:0/2867529605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f74dd0 msgr2=0x564c90f79f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:39.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.739+0000 7f89b91e9740 1 --2- 172.21.15.37:0/2867529605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f74dd0 0x564c90f79f40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f899c02c230 tx=0x7f899c004040 comp rx=0 tx=0).stop 2024-03-28T18:12:39.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.739+0000 7f89b91e9740 1 -- 172.21.15.37:0/2867529605 shutdown_connections 2024-03-28T18:12:39.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.739+0000 7f89b91e9740 1 --2- 172.21.15.37:0/2867529605 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x564c90f8d340 0x564c90f8f730 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.739+0000 7f89b91e9740 1 --2- 172.21.15.37:0/2867529605 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f7d460 0x564c90f7a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.739+0000 7f89b91e9740 1 --2- 172.21.15.37:0/2867529605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f74dd0 0x564c90f79f40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.739+0000 7f89b91e9740 1 --2- 172.21.15.37:0/2867529605 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f740f0 0x564c90f7cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.739+0000 7f89b91e9740 1 -- 172.21.15.37:0/2867529605 >> 172.21.15.37:0/2867529605 conn(0x564c90f3b200 msgr2=0x564c90f70890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:39.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.739+0000 7f8999600640 0 client.25780 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:39.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.740+0000 7f89b91e9740 1 -- 172.21.15.37:0/2867529605 shutdown_connections 2024-03-28T18:12:39.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.740+0000 7f89b91e9740 1 -- 172.21.15.37:0/2867529605 wait complete. 2024-03-28T18:12:39.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.740+0000 7f89b91e9740 20 client.25780 trim_cache size 0 max 16384 2024-03-28T18:12:39.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.757+0000 7f89b91e9740 1 Processor -- start 2024-03-28T18:12:39.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.758+0000 7f89b91e9740 1 -- start start 2024-03-28T18:12:39.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.758+0000 7f89b91e9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f740f0 0x564c90f7c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.758+0000 7f89b91e9740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f74dd0 0x564c90f7cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.758+0000 7f89b91e9740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f7d460 0x564c90f7e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.758+0000 7f89b91e9740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b01b5b00 con 0x564c90f74dd0 2024-03-28T18:12:39.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.758+0000 7f89b91e9740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b01b3600 con 0x564c90f7d460 2024-03-28T18:12:39.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.758+0000 7f89b91e9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b01b16a0 con 0x564c90f740f0 2024-03-28T18:12:39.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.758+0000 7f89afe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f740f0 0x564c90f7c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.758+0000 7f89afe00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f740f0 0x564c90f7c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:50192/0 (socket says 172.21.15.37:50192) 2024-03-28T18:12:39.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.758+0000 7f89afe00640 1 -- 172.21.15.37:0/706915180 learned_addr learned my addr 172.21.15.37:0/706915180 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:39.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.758+0000 7f89afe00640 1 --2- 172.21.15.37:0/706915180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f7d460 0x564c90f7e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.758+0000 7f89aea00640 1 --2- 172.21.15.37:0/706915180 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f74dd0 0x564c90f7cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.759+0000 7f89afe00640 1 -- 172.21.15.37:0/706915180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f7d460 msgr2=0x564c90f7e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:39.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.759+0000 7f89afe00640 1 --2- 172.21.15.37:0/706915180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f7d460 0x564c90f7e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.759+0000 7f89afe00640 1 -- 172.21.15.37:0/706915180 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f74dd0 msgr2=0x564c90f7cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:39.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.759+0000 7f89afe00640 1 --2- 172.21.15.37:0/706915180 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f74dd0 0x564c90f7cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.759+0000 7f89afe00640 1 -- 172.21.15.37:0/706915180 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c90bac0f0 con 0x564c90f740f0 2024-03-28T18:12:39.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.759+0000 7f89afe00640 1 --2- 172.21.15.37:0/706915180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f740f0 0x564c90f7c670 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f89a801c040 tx=0x7f89a8009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18dd38a263ede1a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:39.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.759+0000 7f899be00640 1 -- 172.21.15.37:0/706915180 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a801e040 con 0x564c90f740f0 2024-03-28T18:12:39.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.759+0000 7f899be00640 1 -- 172.21.15.37:0/706915180 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89a802b0e0 con 0x564c90f740f0 2024-03-28T18:12:39.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.759+0000 7f899be00640 1 -- 172.21.15.37:0/706915180 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a802b850 con 0x564c90f740f0 2024-03-28T18:12:39.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.760+0000 7f89b91e9740 1 -- 172.21.15.37:0/706915180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f740f0 msgr2=0x564c90f7c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:39.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.760+0000 7f89b91e9740 1 --2- 172.21.15.37:0/706915180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f740f0 0x564c90f7c670 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f89a801c040 tx=0x7f89a8009500 comp rx=0 tx=0).stop 2024-03-28T18:12:39.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.760+0000 7f89b91e9740 1 -- 172.21.15.37:0/706915180 shutdown_connections 2024-03-28T18:12:39.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.760+0000 7f89b91e9740 1 --2- 172.21.15.37:0/706915180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f7d460 0x564c90f7e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.760+0000 7f89b91e9740 1 --2- 172.21.15.37:0/706915180 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f74dd0 0x564c90f7cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.760+0000 7f89b91e9740 1 --2- 172.21.15.37:0/706915180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f740f0 0x564c90f7c670 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.760+0000 7f89b91e9740 1 -- 172.21.15.37:0/706915180 >> 172.21.15.37:0/706915180 conn(0x564c90f3b200 msgr2=0x564c90f6e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:39.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.760+0000 7f89b91e9740 1 -- 172.21.15.37:0/706915180 shutdown_connections 2024-03-28T18:12:39.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.760+0000 7f89b91e9740 1 -- 172.21.15.37:0/706915180 wait complete. 2024-03-28T18:12:39.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.761+0000 7f89b91e9740 1 -- start start 2024-03-28T18:12:39.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.761+0000 7f89b91e9740 1 Processor -- start 2024-03-28T18:12:39.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.761+0000 7f89b91e9740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f740f0 0x564c90f771e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.762+0000 7f89b91e9740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f74dd0 0x564c90f77720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.762+0000 7f89b91e9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f7d460 0x564c90f87f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.762+0000 7f89b91e9740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b01b1ca0 con 0x564c90f74dd0 2024-03-28T18:12:39.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.762+0000 7f89b91e9740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b01b2fa0 con 0x564c90f740f0 2024-03-28T18:12:39.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.762+0000 7f89b91e9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b01b22e0 con 0x564c90f7d460 2024-03-28T18:12:39.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.762+0000 7f89b91e9740 10 client.0 _ll_register_callbacks cb 0x7f899c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:12:39.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.762+0000 7f89afe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f740f0 0x564c90f771e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.762+0000 7f89afe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f740f0 0x564c90f771e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:44404/0 (socket says 172.21.15.37:44404) 2024-03-28T18:12:39.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.762+0000 7f89afe00640 1 -- 172.21.15.37:0/3733231117 learned_addr learned my addr 172.21.15.37:0/3733231117 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:12:39.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.762+0000 7f89aea00640 1 --2- 172.21.15.37:0/3733231117 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f74dd0 0x564c90f77720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.762+0000 7f89afe00640 1 --2- 172.21.15.37:0/3733231117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f7d460 0x564c90f87f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.762+0000 7f89afe00640 1 -- 172.21.15.37:0/3733231117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f7d460 msgr2=0x564c90f87f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:39.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.762+0000 7f89afe00640 1 --2- 172.21.15.37:0/3733231117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f7d460 0x564c90f87f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.762+0000 7f89afe00640 1 -- 172.21.15.37:0/3733231117 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f74dd0 msgr2=0x564c90f77720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:12:39.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.762+0000 7f89afe00640 1 --2- 172.21.15.37:0/3733231117 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f74dd0 0x564c90f77720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.762+0000 7f89afe00640 1 -- 172.21.15.37:0/3733231117 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564c90f884a0 con 0x564c90f740f0 2024-03-28T18:12:39.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.763+0000 7f89aea00640 1 --2- 172.21.15.37:0/3733231117 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f74dd0 0x564c90f77720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:12:39.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.763+0000 7f89afe00640 1 --2- 172.21.15.37:0/3733231117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f740f0 0x564c90f771e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f89a8019c60 tx=0x7f89a802b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77857c1cdb1a7a3a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:12:39.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.763+0000 7f8999600640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:12:39.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.763+0000 7f8999600640 1 -- 172.21.15.37:0/3733231117 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a801e030 con 0x564c90f740f0 2024-03-28T18:12:39.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.763+0000 7f8999600640 1 -- 172.21.15.37:0/3733231117 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f89a8016070 con 0x564c90f740f0 2024-03-28T18:12:39.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.763+0000 7f89b91e9740 1 -- 172.21.15.37:0/3733231117 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f89a802b0e0 con 0x564c90f740f0 2024-03-28T18:12:39.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.763+0000 7f8999600640 1 -- 172.21.15.37:0/3733231117 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89a802b310 con 0x564c90f740f0 2024-03-28T18:12:39.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.763+0000 7f89b91e9740 10 client.25778 fetch_fsmap learned FSMap version 23 2024-03-28T18:12:39.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.764+0000 7f89b91e9740 1 -- 172.21.15.37:0/3733231117 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564c90fe9290 con 0x564c90f740f0 2024-03-28T18:12:39.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.764+0000 7f8999600640 1 -- 172.21.15.37:0/3733231117 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89a802b0e0 con 0x564c90f740f0 2024-03-28T18:12:39.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.764+0000 7f8999600640 1 -- 172.21.15.37:0/3733231117 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f89a8024040 con 0x564c90f740f0 2024-03-28T18:12:39.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.764+0000 7f89b91e9740 10 client.25778 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T18:12:39.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.764+0000 7f89b91e9740 10 client.25778 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:12:39.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.764+0000 7f89b91e9740 20 client.25778 populate_metadata read hostname 'smithi037' 2024-03-28T18:12:39.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.764+0000 7f89b91e9740 1 --2- 172.21.15.37:0/3733231117 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x564c90fe95c0 0x564c90feb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:12:39.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.764+0000 7f89b91e9740 4 client.25778 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:12:39.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.764+0000 7f89b91e9740 1 -- 172.21.15.37:0/3733231117 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564c90edfae0 con 0x564c90fe95c0 2024-03-28T18:12:39.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.765+0000 7f89aea00640 1 --2- 172.21.15.37:0/3733231117 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x564c90fe95c0 0x564c90feb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:12:39.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.765+0000 7f89aea00640 1 --2- 172.21.15.37:0/3733231117 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x564c90fe95c0 0x564c90feb9b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80d516cbc8450904 server_cookie=8b0fba4c3ee31827 in_seq=0 out_seq=0 2024-03-28T18:12:39.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.765+0000 7f8999600640 10 client.25778 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:39.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.766+0000 7f8999600640 1 -- 172.21.15.37:0/3733231117 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x564c90edfae0 con 0x564c90fe95c0 2024-03-28T18:12:39.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.766+0000 7f8999600640 10 client.25778 handle_command_reply: tid=0 2024-03-28T18:12:39.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.766+0000 7f89b91e9740 1 client.25778 shutdown 2024-03-28T18:12:39.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.766+0000 7f89b91e9740 1 -- 172.21.15.37:0/3733231117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f740f0 msgr2=0x564c90f771e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:39.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.766+0000 7f89b91e9740 1 --2- 172.21.15.37:0/3733231117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f740f0 0x564c90f771e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f89a8019c60 tx=0x7f89a802b040 comp rx=0 tx=0).stop 2024-03-28T18:12:39.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.766+0000 7f89b91e9740 1 -- 172.21.15.37:0/3733231117 shutdown_connections 2024-03-28T18:12:39.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.766+0000 7f89b91e9740 1 --2- 172.21.15.37:0/3733231117 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x564c90fe95c0 0x564c90feb9b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.766+0000 7f89b91e9740 1 --2- 172.21.15.37:0/3733231117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564c90f7d460 0x564c90f87f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.766+0000 7f89b91e9740 1 --2- 172.21.15.37:0/3733231117 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564c90f74dd0 0x564c90f77720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.766+0000 7f89b91e9740 1 --2- 172.21.15.37:0/3733231117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564c90f740f0 0x564c90f771e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.766+0000 7f89b91e9740 1 -- 172.21.15.37:0/3733231117 >> 172.21.15.37:0/3733231117 conn(0x564c90f3b200 msgr2=0x564c90f787f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:39.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.766+0000 7f8999600640 0 client.25778 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:12:39.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.767+0000 7f89b91e9740 1 -- 172.21.15.37:0/3733231117 shutdown_connections 2024-03-28T18:12:39.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.767+0000 7f89b91e9740 1 -- 172.21.15.37:0/3733231117 wait complete. 2024-03-28T18:12:39.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.767+0000 7f89b91e9740 20 client.25778 trim_cache size 0 max 16384 2024-03-28T18:12:39.771 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:12:39.771 INFO:teuthology.orchestra.run.smithi037.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:12:39.771 INFO:teuthology.orchestra.run.smithi037.stdout: "scrubs": { 2024-03-28T18:12:39.771 INFO:teuthology.orchestra.run.smithi037.stdout: "d178a304-cf1e-48ee-a9a1-2a9f48cccf7f": { 2024-03-28T18:12:39.771 INFO:teuthology.orchestra.run.smithi037.stdout: "path": "/", 2024-03-28T18:12:39.772 INFO:teuthology.orchestra.run.smithi037.stdout: "tag": "d178a304-cf1e-48ee-a9a1-2a9f48cccf7f", 2024-03-28T18:12:39.772 INFO:teuthology.orchestra.run.smithi037.stdout: "options": "recursive,force" 2024-03-28T18:12:39.772 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-03-28T18:12:39.772 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-03-28T18:12:39.772 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:12:39.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.769+0000 7f89b4e00640 1 -- 172.21.15.37:0/1717729294 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f89940a2b40 msgr2=0x7f89940a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:39.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.769+0000 7f89b4e00640 1 --2- 172.21.15.37:0/1717729294 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f89940a2b40 0x7f89940a5000 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f89a00cc1b0 tx=0x7f89a00057f0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.769+0000 7f89b4e00640 1 -- 172.21.15.37:0/1717729294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89b0101530 msgr2=0x7f89b0197d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:12:39.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.769+0000 7f89b4e00640 1 --2- 172.21.15.37:0/1717729294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89b0101530 0x7f89b0197d80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f89a000d6d0 tx=0x7f89a000db90 comp rx=0 tx=0).stop 2024-03-28T18:12:39.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.769+0000 7f89b4e00640 1 -- 172.21.15.37:0/1717729294 shutdown_connections 2024-03-28T18:12:39.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.769+0000 7f89b4e00640 1 --2- 172.21.15.37:0/1717729294 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f89940a2b40 0x7f89940a5000 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.769+0000 7f89b4e00640 1 --2- 172.21.15.37:0/1717729294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89b01075f0 0x7f89b019c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.769+0000 7f89b4e00640 1 --2- 172.21.15.37:0/1717729294 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89b0101ef0 0x7f89b01982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.769+0000 7f89b4e00640 1 --2- 172.21.15.37:0/1717729294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89b0101530 0x7f89b0197d80 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:12:39.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.769+0000 7f89b4e00640 1 -- 172.21.15.37:0/1717729294 >> 172.21.15.37:0/1717729294 conn(0x7f89b00fd040 msgr2=0x7f89b00fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:12:39.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.769+0000 7f89b4e00640 1 -- 172.21.15.37:0/1717729294 shutdown_connections 2024-03-28T18:12:39.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:12:39.769+0000 7f89b4e00640 1 -- 172.21.15.37:0/1717729294 wait complete. 2024-03-28T18:12:39.774 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:39 smithi037 ceph-mon[31460]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:39.774 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:39 smithi037 ceph-mon[31460]: scrub queued for path: / 2024-03-28T18:12:39.774 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:39 smithi037 ceph-mon[31460]: scrub summary: idle+waiting paths [/] 2024-03-28T18:12:39.774 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:39 smithi037 ceph-mon[31460]: scrub summary: active paths [/] 2024-03-28T18:12:39.774 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:39 smithi037 ceph-mon[31460]: scrub summary: idle+waiting paths [/] 2024-03-28T18:12:39.794 INFO:tasks.cephfs.filesystem:scrub status for tag:d178a304-cf1e-48ee-a9a1-2a9f48cccf7f - {'path': '/', 'tag': 'd178a304-cf1e-48ee-a9a1-2a9f48cccf7f', 'options': 'recursive,force'} 2024-03-28T18:12:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:39 smithi059 ceph-mon[36190]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:39 smithi059 ceph-mon[36190]: scrub queued for path: / 2024-03-28T18:12:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:39 smithi059 ceph-mon[36190]: scrub summary: idle+waiting paths [/] 2024-03-28T18:12:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:39 smithi059 ceph-mon[36190]: scrub summary: active paths [/] 2024-03-28T18:12:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:39 smithi059 ceph-mon[36190]: scrub summary: idle+waiting paths [/] 2024-03-28T18:12:40.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:39 smithi170 ceph-mon[36280]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:40.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:39 smithi170 ceph-mon[36280]: scrub queued for path: / 2024-03-28T18:12:40.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:39 smithi170 ceph-mon[36280]: scrub summary: idle+waiting paths [/] 2024-03-28T18:12:40.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:39 smithi170 ceph-mon[36280]: scrub summary: active paths [/] 2024-03-28T18:12:40.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:39 smithi170 ceph-mon[36280]: scrub summary: idle+waiting paths [/] 2024-03-28T18:12:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:40 smithi059 ceph-mon[36190]: scrub summary: idle 2024-03-28T18:12:41.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:40 smithi170 ceph-mon[36280]: scrub summary: idle 2024-03-28T18:12:41.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:40 smithi037 ceph-mon[31460]: scrub summary: idle 2024-03-28T18:12:42.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:41 smithi059 ceph-mon[36190]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:42.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:41 smithi170 ceph-mon[36280]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:42.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:41 smithi037 ceph-mon[31460]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:44.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:43 smithi037 ceph-mon[31460]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 8 op/s 2024-03-28T18:12:44.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:43 smithi037 ceph-mon[31460]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-28T18:12:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:43 smithi059 ceph-mon[36190]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 8 op/s 2024-03-28T18:12:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:43 smithi059 ceph-mon[36190]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-28T18:12:44.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:43 smithi170 ceph-mon[36280]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 8 op/s 2024-03-28T18:12:44.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:43 smithi170 ceph-mon[36280]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-28T18:12:46.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:45 smithi037 ceph-mon[31460]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-28T18:12:46.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:45 smithi059 ceph-mon[36190]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-28T18:12:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:45 smithi170 ceph-mon[36280]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-28T18:12:48.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:47 smithi037 ceph-mon[31460]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T18:12:48.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:47 smithi059 ceph-mon[36190]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T18:12:48.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:47 smithi170 ceph-mon[36280]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T18:12:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:49 smithi037 ceph-mon[31460]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-28T18:12:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:49 smithi059 ceph-mon[36190]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-28T18:12:50.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:49 smithi170 ceph-mon[36280]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-28T18:12:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:51 smithi037 ceph-mon[31460]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-28T18:12:52.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:51 smithi059 ceph-mon[36190]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-28T18:12:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:51 smithi170 ceph-mon[36280]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-28T18:12:53.319 INFO:tasks.workunit.client.0.smithi037.stderr:Updating files: 56% (6973/12271) Updating files: 57% (6995/12271) Updating files: 58% (7118/12271) Updating files: 59% (7240/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 89% (10931/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-28T18:12:53.785 INFO:tasks.workunit.client.0.smithi037.stderr:Note: switching to '3be9a5c9bc793e11e2800a8c0c696e8b46742033'. 2024-03-28T18:12:53.785 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-03-28T18:12:53.785 INFO:tasks.workunit.client.0.smithi037.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-28T18:12:53.785 INFO:tasks.workunit.client.0.smithi037.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-28T18:12:53.785 INFO:tasks.workunit.client.0.smithi037.stderr:state without impacting any branches by switching back to a branch. 2024-03-28T18:12:53.785 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-03-28T18:12:53.785 INFO:tasks.workunit.client.0.smithi037.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-28T18:12:53.785 INFO:tasks.workunit.client.0.smithi037.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-28T18:12:53.785 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-03-28T18:12:53.785 INFO:tasks.workunit.client.0.smithi037.stderr: git switch -c 2024-03-28T18:12:53.785 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-03-28T18:12:53.786 INFO:tasks.workunit.client.0.smithi037.stderr:Or undo this operation with: 2024-03-28T18:12:53.786 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-03-28T18:12:53.786 INFO:tasks.workunit.client.0.smithi037.stderr: git switch - 2024-03-28T18:12:53.786 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-03-28T18:12:53.786 INFO:tasks.workunit.client.0.smithi037.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-28T18:12:53.786 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-03-28T18:12:53.786 INFO:tasks.workunit.client.0.smithi037.stderr:HEAD is now at 3be9a5c9bc7 dummy commit #1 2024-03-28T18:12:53.794 DEBUG:teuthology.orchestra.run.smithi037:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-28T18:12:53.858 INFO:tasks.workunit.client.0.smithi037.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-28T18:12:53.865 INFO:tasks.workunit.client.0.smithi037.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T18:12:53.865 INFO:tasks.workunit.client.0.smithi037.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-28T18:12:54.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:53 smithi037 ceph-mon[31460]: pgmap v54: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T18:12:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:53 smithi059 ceph-mon[36190]: pgmap v54: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T18:12:54.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:53 smithi170 ceph-mon[36280]: pgmap v54: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T18:12:54.791 INFO:tasks.workunit.client.0.smithi037.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-28T18:12:54.952 INFO:tasks.workunit.client.0.smithi037.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-28T18:12:55.041 INFO:tasks.workunit.client.0.smithi037.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T18:12:55.046 INFO:tasks.workunit.client.0.smithi037.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T18:12:55.047 INFO:tasks.workunit.client.0.smithi037.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-28T18:12:55.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:54 smithi037 ceph-mon[31460]: pgmap v55: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T18:12:55.128 INFO:tasks.workunit.client.0.smithi037.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T18:12:55.140 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-28T18:12:55.140 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-28T18:12:55.188 INFO:tasks.workunit:Running workunits matching suites/fsx.sh on client.0... 2024-03-28T18:12:55.189 INFO:tasks.workunit:Running workunit suites/fsx.sh... 2024-03-28T18:12:55.189 DEBUG:teuthology.orchestra.run.smithi037:workunit test suites/fsx.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh 2024-03-28T18:12:55.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:54 smithi059 ceph-mon[36190]: pgmap v55: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T18:12:55.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:54 smithi170 ceph-mon[36280]: pgmap v55: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T18:12:55.302 INFO:tasks.workunit.client.0.smithi037.stderr:+ set -e 2024-03-28T18:12:55.303 INFO:tasks.workunit.client.0.smithi037.stderr:+ git clone https://git.ceph.com/xfstests-dev.git 2024-03-28T18:12:55.309 INFO:tasks.workunit.client.0.smithi037.stderr:Cloning into 'xfstests-dev'... 2024-03-28T18:12:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:57 smithi059 ceph-mon[36190]: pgmap v56: 97 pgs: 97 active+clean; 1.5 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-28T18:12:57.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:57 smithi170 ceph-mon[36280]: pgmap v56: 97 pgs: 97 active+clean; 1.5 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-28T18:12:57.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:57 smithi037 ceph-mon[31460]: pgmap v56: 97 pgs: 97 active+clean; 1.5 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-28T18:12:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:12:59 smithi059 ceph-mon[36190]: pgmap v57: 97 pgs: 97 active+clean; 1.5 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-28T18:12:59.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:12:59 smithi170 ceph-mon[36280]: pgmap v57: 97 pgs: 97 active+clean; 1.5 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-28T18:12:59.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:12:59 smithi037 ceph-mon[31460]: pgmap v57: 97 pgs: 97 active+clean; 1.5 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-28T18:13:01.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:01 smithi059 ceph-mon[36190]: pgmap v58: 97 pgs: 97 active+clean; 1.5 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-28T18:13:01.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:01 smithi170 ceph-mon[36280]: pgmap v58: 97 pgs: 97 active+clean; 1.5 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-28T18:13:01.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:01 smithi037 ceph-mon[31460]: pgmap v58: 97 pgs: 97 active+clean; 1.5 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-28T18:13:02.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:02 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:13:02.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:02 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:13:02.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:02 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:13:03.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:03 smithi059 ceph-mon[36190]: pgmap v59: 97 pgs: 97 active+clean; 15 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 51 op/s 2024-03-28T18:13:03.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:03 smithi170 ceph-mon[36280]: pgmap v59: 97 pgs: 97 active+clean; 15 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 51 op/s 2024-03-28T18:13:03.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:03 smithi037 ceph-mon[31460]: pgmap v59: 97 pgs: 97 active+clean; 15 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 51 op/s 2024-03-28T18:13:05.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:05 smithi059 ceph-mon[36190]: pgmap v60: 97 pgs: 97 active+clean; 15 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 48 op/s 2024-03-28T18:13:05.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:05 smithi170 ceph-mon[36280]: pgmap v60: 97 pgs: 97 active+clean; 15 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 48 op/s 2024-03-28T18:13:05.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:05 smithi037 ceph-mon[31460]: pgmap v60: 97 pgs: 97 active+clean; 15 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 48 op/s 2024-03-28T18:13:07.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:07 smithi059 ceph-mon[36190]: pgmap v61: 97 pgs: 97 active+clean; 16 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 107 op/s 2024-03-28T18:13:07.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:07 smithi170 ceph-mon[36280]: pgmap v61: 97 pgs: 97 active+clean; 16 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 107 op/s 2024-03-28T18:13:07.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:07 smithi037 ceph-mon[31460]: pgmap v61: 97 pgs: 97 active+clean; 16 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 107 op/s 2024-03-28T18:13:09.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:09 smithi059 ceph-mon[36190]: pgmap v62: 97 pgs: 97 active+clean; 16 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 101 op/s 2024-03-28T18:13:09.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:09 smithi170 ceph-mon[36280]: pgmap v62: 97 pgs: 97 active+clean; 16 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 101 op/s 2024-03-28T18:13:09.794 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:13:09.795 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:13:09.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:09 smithi037 ceph-mon[31460]: pgmap v62: 97 pgs: 97 active+clean; 16 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 101 op/s 2024-03-28T18:13:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.027+0000 7f4360a00640 1 -- 172.21.15.37:0/2889913273 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f435c073a70 msgr2=0x7f435c073e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.027+0000 7f4360a00640 1 --2- 172.21.15.37:0/2889913273 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f435c073a70 0x7f435c073e50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f434c009860 tx=0x7f434c02bc10 comp rx=0 tx=0).stop 2024-03-28T18:13:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.028+0000 7f4360a00640 1 -- 172.21.15.37:0/2889913273 shutdown_connections 2024-03-28T18:13:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.028+0000 7f4360a00640 1 --2- 172.21.15.37:0/2889913273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f435c10ded0 0x7f435c1102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.028+0000 7f4360a00640 1 --2- 172.21.15.37:0/2889913273 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f435c074390 0x7f435c10d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.028+0000 7f4360a00640 1 --2- 172.21.15.37:0/2889913273 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f435c073a70 0x7f435c073e50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.028+0000 7f4360a00640 1 -- 172.21.15.37:0/2889913273 >> 172.21.15.37:0/2889913273 conn(0x7f435c078aa0 msgr2=0x7f435c078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.028+0000 7f4360a00640 1 -- 172.21.15.37:0/2889913273 shutdown_connections 2024-03-28T18:13:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.028+0000 7f4360a00640 1 -- 172.21.15.37:0/2889913273 wait complete. 2024-03-28T18:13:10.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.029+0000 7f4360a00640 1 Processor -- start 2024-03-28T18:13:10.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.029+0000 7f4360a00640 1 -- start start 2024-03-28T18:13:10.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.030+0000 7f4360a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f435c073a70 0x7f435c19a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.030+0000 7f4360a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f435c074390 0x7f435c19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.030+0000 7f4360a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f435c10ded0 0x7f435c19ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.030+0000 7f4360a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c113c30 con 0x7f435c074390 2024-03-28T18:13:10.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.030+0000 7f4360a00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c113ab0 con 0x7f435c073a70 2024-03-28T18:13:10.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.030+0000 7f4360a00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c113db0 con 0x7f435c10ded0 2024-03-28T18:13:10.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.030+0000 7f4359e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f435c073a70 0x7f435c19a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.030+0000 7f4359e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f435c073a70 0x7f435c19a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:34864/0 (socket says 172.21.15.37:34864) 2024-03-28T18:13:10.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.030+0000 7f4359e00640 1 -- 172.21.15.37:0/2054133386 learned_addr learned my addr 172.21.15.37:0/2054133386 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:10.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.030+0000 7f4359400640 1 --2- 172.21.15.37:0/2054133386 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f435c074390 0x7f435c19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.031+0000 7f435a800640 1 --2- 172.21.15.37:0/2054133386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f435c10ded0 0x7f435c19ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.031+0000 7f4359e00640 1 -- 172.21.15.37:0/2054133386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f435c10ded0 msgr2=0x7f435c19ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.031+0000 7f4359e00640 1 --2- 172.21.15.37:0/2054133386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f435c10ded0 0x7f435c19ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.031+0000 7f4359e00640 1 -- 172.21.15.37:0/2054133386 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f435c074390 msgr2=0x7f435c19a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.031+0000 7f4359e00640 1 --2- 172.21.15.37:0/2054133386 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f435c074390 0x7f435c19a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.031+0000 7f4359e00640 1 -- 172.21.15.37:0/2054133386 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c19f180 con 0x7f435c073a70 2024-03-28T18:13:10.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.031+0000 7f435a800640 1 --2- 172.21.15.37:0/2054133386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f435c10ded0 0x7f435c19ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:13:10.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.031+0000 7f4359400640 1 --2- 172.21.15.37:0/2054133386 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f435c074390 0x7f435c19a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:13:10.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.031+0000 7f4359e00640 1 --2- 172.21.15.37:0/2054133386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f435c073a70 0x7f435c19a190 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f434c037040 tx=0x7f434c0344c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:10.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.032+0000 7f4352a00640 1 -- 172.21.15.37:0/2054133386 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c041020 con 0x7f435c073a70 2024-03-28T18:13:10.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.032+0000 7f4360a00640 1 -- 172.21.15.37:0/2054133386 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f435c19f410 con 0x7f435c073a70 2024-03-28T18:13:10.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.032+0000 7f4352a00640 1 -- 172.21.15.37:0/2054133386 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f434c02ca90 con 0x7f435c073a70 2024-03-28T18:13:10.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.032+0000 7f4352a00640 1 -- 172.21.15.37:0/2054133386 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c039040 con 0x7f435c073a70 2024-03-28T18:13:10.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.032+0000 7f4360a00640 1 -- 172.21.15.37:0/2054133386 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f435c1a6d00 con 0x7f435c073a70 2024-03-28T18:13:10.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.036+0000 7f43637f9740 1 Processor -- start 2024-03-28T18:13:10.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.036+0000 7f43637f9740 1 -- start start 2024-03-28T18:13:10.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.037+0000 7f4352a00640 1 -- 172.21.15.37:0/2054133386 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f434c044020 con 0x7f435c073a70 2024-03-28T18:13:10.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.037+0000 7f43637f9740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14537c00 0x556a14536950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.037+0000 7f43637f9740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14536f20 0x556a145373a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.037+0000 7f43637f9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a14542680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.037+0000 7f43637f9740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a1449b250 con 0x556a14537c00 2024-03-28T18:13:10.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.037+0000 7f43637f9740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a14443a30 con 0x556a14536f20 2024-03-28T18:13:10.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.037+0000 7f43637f9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a144972f0 con 0x556a14540290 2024-03-28T18:13:10.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.037+0000 7f4359400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a14542680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.037+0000 7f4359e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14536f20 0x556a145373a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.037+0000 7f4359400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a14542680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56540/0 (socket says 172.21.15.37:56540) 2024-03-28T18:13:10.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.037+0000 7f435a800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14537c00 0x556a14536950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.037+0000 7f4359400640 1 -- 172.21.15.37:0/1851932859 learned_addr learned my addr 172.21.15.37:0/1851932859 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:10.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.038+0000 7f4359400640 1 -- 172.21.15.37:0/1851932859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14536f20 msgr2=0x556a145373a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:10.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.038+0000 7f4359400640 1 --2- 172.21.15.37:0/1851932859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14536f20 0x556a145373a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.038+0000 7f4359400640 1 -- 172.21.15.37:0/1851932859 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14537c00 msgr2=0x556a14536950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:10.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.038+0000 7f4359400640 1 --2- 172.21.15.37:0/1851932859 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14537c00 0x556a14536950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.038+0000 7f4359400640 1 -- 172.21.15.37:0/1851932859 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556a1442e690 con 0x556a14540290 2024-03-28T18:13:10.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.038+0000 7f4359e00640 1 --2- 172.21.15.37:0/1851932859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14536f20 0x556a145373a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:13:10.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.038+0000 7f4359400640 1 --2- 172.21.15.37:0/1851932859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a14542680 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4348009cf0 tx=0x7f43480117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b246e9bdd2320c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:10.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.038+0000 7f4347000640 1 -- 172.21.15.37:0/1851932859 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4348012450 con 0x556a14540290 2024-03-28T18:13:10.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.038+0000 7f4347000640 1 -- 172.21.15.37:0/1851932859 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4348012ad0 con 0x556a14540290 2024-03-28T18:13:10.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.038+0000 7f4347000640 1 -- 172.21.15.37:0/1851932859 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434800fc60 con 0x556a14540290 2024-03-28T18:13:10.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.039+0000 7f4352a00640 1 --2- 172.21.15.37:0/2054133386 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f43400a29f0 0x7f43400a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.039+0000 7f4352a00640 1 -- 172.21.15.37:0/2054133386 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f434c0f04e0 con 0x7f435c073a70 2024-03-28T18:13:10.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.039+0000 7f435a800640 1 --2- 172.21.15.37:0/2054133386 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f43400a29f0 0x7f43400a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.039+0000 7f43637f9740 1 -- 172.21.15.37:0/1851932859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 msgr2=0x556a14542680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.039+0000 7f43637f9740 1 --2- 172.21.15.37:0/1851932859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a14542680 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4348009cf0 tx=0x7f43480117b0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.039+0000 7f43637f9740 1 -- 172.21.15.37:0/1851932859 shutdown_connections 2024-03-28T18:13:10.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.040+0000 7f43637f9740 1 --2- 172.21.15.37:0/1851932859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14536f20 0x556a145373a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.040+0000 7f43637f9740 1 --2- 172.21.15.37:0/1851932859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a14542680 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.040+0000 7f43637f9740 1 --2- 172.21.15.37:0/1851932859 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14537c00 0x556a14536950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.040+0000 7f43637f9740 1 -- 172.21.15.37:0/1851932859 >> 172.21.15.37:0/1851932859 conn(0x556a144fe200 msgr2=0x556a14531970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:10.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.040+0000 7f43637f9740 1 -- 172.21.15.37:0/1851932859 shutdown_connections 2024-03-28T18:13:10.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.040+0000 7f435a800640 1 --2- 172.21.15.37:0/2054133386 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f43400a29f0 0x7f43400a4eb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f435c06af40 tx=0x7f4354011040 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:10.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.040+0000 7f43637f9740 1 -- 172.21.15.37:0/1851932859 wait complete. 2024-03-28T18:13:10.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.041+0000 7f43637f9740 1 -- start start 2024-03-28T18:13:10.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.041+0000 7f43637f9740 1 Processor -- start 2024-03-28T18:13:10.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.042+0000 7f43637f9740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14536f20 0x556a1453f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.042+0000 7f43637f9740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14537c00 0x556a1453cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.042+0000 7f43637f9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a1453d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.042+0000 7f43637f9740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a14543100 con 0x556a14536f20 2024-03-28T18:13:10.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.042+0000 7f43637f9740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a14456e80 con 0x556a14537c00 2024-03-28T18:13:10.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.042+0000 7f43637f9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a14543280 con 0x556a14540290 2024-03-28T18:13:10.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.042+0000 7f43637f9740 10 client.0 _ll_register_callbacks cb 0x556a14469130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:13:10.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.042+0000 7f4359400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14537c00 0x556a1453cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.042+0000 7f4359400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14537c00 0x556a1453cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:34882/0 (socket says 172.21.15.37:34882) 2024-03-28T18:13:10.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.042+0000 7f4359400640 1 -- 172.21.15.37:0/112121891 learned_addr learned my addr 172.21.15.37:0/112121891 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:10.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.042+0000 7f4359e00640 1 --2- 172.21.15.37:0/112121891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14536f20 0x556a1453f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.042+0000 7f435a800640 1 --2- 172.21.15.37:0/112121891 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a1453d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.042+0000 7f4359400640 1 -- 172.21.15.37:0/112121891 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 msgr2=0x556a1453d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:10.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.042+0000 7f4359400640 1 --2- 172.21.15.37:0/112121891 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a1453d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.042+0000 7f4359400640 1 -- 172.21.15.37:0/112121891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14536f20 msgr2=0x556a1453f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:10.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.042+0000 7f4359400640 1 --2- 172.21.15.37:0/112121891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14536f20 0x556a1453f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.042+0000 7f4359400640 1 -- 172.21.15.37:0/112121891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556a142797d0 con 0x556a14537c00 2024-03-28T18:13:10.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.043+0000 7f4359400640 1 --2- 172.21.15.37:0/112121891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14537c00 0x556a1453cc50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4348012010 tx=0x7f434800f4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8123e58d32d9d1c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:10.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.043+0000 7f433b400640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:13:10.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.043+0000 7f433b400640 1 -- 172.21.15.37:0/112121891 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4348028020 con 0x556a14537c00 2024-03-28T18:13:10.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.043+0000 7f433b400640 1 -- 172.21.15.37:0/112121891 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4348004040 con 0x556a14537c00 2024-03-28T18:13:10.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.043+0000 7f43637f9740 1 -- 172.21.15.37:0/112121891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4348012ad0 con 0x556a14537c00 2024-03-28T18:13:10.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.044+0000 7f433b400640 1 -- 172.21.15.37:0/112121891 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4348033050 con 0x556a14537c00 2024-03-28T18:13:10.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.044+0000 7f43637f9740 10 client.25790 fetch_fsmap learned FSMap version 24 2024-03-28T18:13:10.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.044+0000 7f43637f9740 1 -- 172.21.15.37:0/112121891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x556a141af3f0 con 0x556a14537c00 2024-03-28T18:13:10.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.044+0000 7f433b400640 1 -- 172.21.15.37:0/112121891 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4348012ad0 con 0x556a14537c00 2024-03-28T18:13:10.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.044+0000 7f433b400640 1 -- 172.21.15.37:0/112121891 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f434800fd00 con 0x556a14537c00 2024-03-28T18:13:10.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.044+0000 7f43637f9740 10 client.25790 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T18:13:10.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.044+0000 7f43637f9740 10 client.25790 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:13:10.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.044+0000 7f43637f9740 20 client.25790 populate_metadata read hostname 'smithi037' 2024-03-28T18:13:10.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.045+0000 7f43637f9740 1 --2- 172.21.15.37:0/112121891 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556a145500c0 0x556a145524b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.045+0000 7f43637f9740 4 client.25790 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:13:10.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.045+0000 7f43637f9740 1 -- 172.21.15.37:0/112121891 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556a144a8790 con 0x556a145500c0 2024-03-28T18:13:10.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.045+0000 7f4359e00640 1 --2- 172.21.15.37:0/112121891 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556a145500c0 0x556a145524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.045+0000 7f4359e00640 1 --2- 172.21.15.37:0/112121891 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556a145500c0 0x556a145524b0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2be51f7cef58c175 server_cookie=196a2f642ef27c46 in_seq=0 out_seq=0 2024-03-28T18:13:10.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.045+0000 7f433b400640 10 client.25790 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:13:10.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.048+0000 7f433b400640 1 -- 172.21.15.37:0/112121891 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556a144a8790 con 0x556a145500c0 2024-03-28T18:13:10.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.048+0000 7f433b400640 10 client.25790 handle_command_reply: tid=0 2024-03-28T18:13:10.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.048+0000 7f43637f9740 1 client.25790 shutdown 2024-03-28T18:13:10.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.048+0000 7f43637f9740 1 -- 172.21.15.37:0/112121891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14537c00 msgr2=0x556a1453cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.048+0000 7f43637f9740 1 --2- 172.21.15.37:0/112121891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14537c00 0x556a1453cc50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4348012010 tx=0x7f434800f4b0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.049+0000 7f43637f9740 1 -- 172.21.15.37:0/112121891 shutdown_connections 2024-03-28T18:13:10.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.049+0000 7f43637f9740 1 --2- 172.21.15.37:0/112121891 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556a145500c0 0x556a145524b0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.049+0000 7f43637f9740 1 --2- 172.21.15.37:0/112121891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14537c00 0x556a1453cc50 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.049+0000 7f433b400640 0 client.25790 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:13:10.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.049+0000 7f43637f9740 1 --2- 172.21.15.37:0/112121891 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a1453d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.049+0000 7f43637f9740 1 --2- 172.21.15.37:0/112121891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14536f20 0x556a1453f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.049+0000 7f43637f9740 1 -- 172.21.15.37:0/112121891 >> 172.21.15.37:0/112121891 conn(0x556a144fe200 msgr2=0x556a145336c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:10.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.049+0000 7f43637f9740 1 -- 172.21.15.37:0/112121891 shutdown_connections 2024-03-28T18:13:10.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.049+0000 7f43637f9740 1 -- 172.21.15.37:0/112121891 wait complete. 2024-03-28T18:13:10.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.049+0000 7f43637f9740 20 client.25790 trim_cache size 0 max 16384 2024-03-28T18:13:10.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.071+0000 7f43637f9740 1 Processor -- start 2024-03-28T18:13:10.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.071+0000 7f43637f9740 1 -- start start 2024-03-28T18:13:10.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.071+0000 7f43637f9740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14536f20 0x556a14541680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.071+0000 7f43637f9740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14537c00 0x556a14541bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.071+0000 7f43637f9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a1453f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.071+0000 7f43637f9740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c1b1980 con 0x556a14537c00 2024-03-28T18:13:10.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.071+0000 7f43637f9740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c1af480 con 0x556a14536f20 2024-03-28T18:13:10.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.071+0000 7f43637f9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c1ad520 con 0x556a14540290 2024-03-28T18:13:10.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.072+0000 7f4359400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14537c00 0x556a14541bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.072+0000 7f435a800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a1453f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.071+0000 7f4359e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14536f20 0x556a14541680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.072+0000 7f435a800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a1453f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56558/0 (socket says 172.21.15.37:56558) 2024-03-28T18:13:10.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.072+0000 7f4359400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14537c00 0x556a14541bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60048/0 (socket says 172.21.15.37:60048) 2024-03-28T18:13:10.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.072+0000 7f435a800640 1 -- 172.21.15.37:0/3760892867 learned_addr learned my addr 172.21.15.37:0/3760892867 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:10.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.072+0000 7f435a800640 1 -- 172.21.15.37:0/3760892867 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14536f20 msgr2=0x556a14541680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:10.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.072+0000 7f435a800640 1 --2- 172.21.15.37:0/3760892867 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14536f20 0x556a14541680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.072+0000 7f435a800640 1 -- 172.21.15.37:0/3760892867 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14537c00 msgr2=0x556a14541bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:10.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.072+0000 7f435a800640 1 --2- 172.21.15.37:0/3760892867 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14537c00 0x556a14541bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.072+0000 7f435a800640 1 -- 172.21.15.37:0/3760892867 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556a1416f0f0 con 0x556a14540290 2024-03-28T18:13:10.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.072+0000 7f435a800640 1 --2- 172.21.15.37:0/3760892867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a1453f380 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f435401b670 tx=0x7f4354017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ada6e3b798c778db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:10.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.072+0000 7f4346600640 1 -- 172.21.15.37:0/3760892867 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4354021040 con 0x556a14540290 2024-03-28T18:13:10.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.073+0000 7f4346600640 1 -- 172.21.15.37:0/3760892867 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4354003e50 con 0x556a14540290 2024-03-28T18:13:10.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.073+0000 7f4346600640 1 -- 172.21.15.37:0/3760892867 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43540198a0 con 0x556a14540290 2024-03-28T18:13:10.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.073+0000 7f43637f9740 1 -- 172.21.15.37:0/3760892867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 msgr2=0x556a1453f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.073+0000 7f43637f9740 1 --2- 172.21.15.37:0/3760892867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a1453f380 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f435401b670 tx=0x7f4354017000 comp rx=0 tx=0).stop 2024-03-28T18:13:10.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.073+0000 7f43637f9740 1 -- 172.21.15.37:0/3760892867 shutdown_connections 2024-03-28T18:13:10.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.073+0000 7f43637f9740 1 --2- 172.21.15.37:0/3760892867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a1453f380 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.073+0000 7f43637f9740 1 --2- 172.21.15.37:0/3760892867 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14537c00 0x556a14541bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.073+0000 7f43637f9740 1 --2- 172.21.15.37:0/3760892867 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14536f20 0x556a14541680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.074+0000 7f43637f9740 1 -- 172.21.15.37:0/3760892867 >> 172.21.15.37:0/3760892867 conn(0x556a144fe200 msgr2=0x556a1453b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:10.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.074+0000 7f43637f9740 1 -- 172.21.15.37:0/3760892867 shutdown_connections 2024-03-28T18:13:10.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.074+0000 7f43637f9740 1 -- 172.21.15.37:0/3760892867 wait complete. 2024-03-28T18:13:10.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.074+0000 7f43637f9740 1 -- start start 2024-03-28T18:13:10.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.075+0000 7f43637f9740 1 Processor -- start 2024-03-28T18:13:10.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.075+0000 7f43637f9740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14536f20 0x556a14539a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.075+0000 7f43637f9740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14537c00 0x556a14539f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.075+0000 7f43637f9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a145b2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.075+0000 7f43637f9740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c1adb20 con 0x556a14537c00 2024-03-28T18:13:10.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.075+0000 7f43637f9740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c1aee20 con 0x556a14536f20 2024-03-28T18:13:10.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.075+0000 7f43637f9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c1ae160 con 0x556a14540290 2024-03-28T18:13:10.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.075+0000 7f43637f9740 10 client.0 _ll_register_callbacks cb 0x7f43540099e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:13:10.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.076+0000 7f4359e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14536f20 0x556a14539a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.076+0000 7f4359e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14536f20 0x556a14539a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:34910/0 (socket says 172.21.15.37:34910) 2024-03-28T18:13:10.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.076+0000 7f435a800640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a145b2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.076+0000 7f4359e00640 1 -- 172.21.15.37:0/378939412 learned_addr learned my addr 172.21.15.37:0/378939412 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:10.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.076+0000 7f4359e00640 1 -- 172.21.15.37:0/378939412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 msgr2=0x556a145b2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:10.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.076+0000 7f4359e00640 1 --2- 172.21.15.37:0/378939412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a145b2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.076+0000 7f4359e00640 1 -- 172.21.15.37:0/378939412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14537c00 msgr2=0x556a14539f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:10.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.077+0000 7f4359400640 1 --2- 172.21.15.37:0/378939412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14537c00 0x556a14539f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.078+0000 7f4359e00640 1 --2- 172.21.15.37:0/378939412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14537c00 0x556a14539f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.078+0000 7f4359e00640 1 -- 172.21.15.37:0/378939412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556a145a3760 con 0x556a14536f20 2024-03-28T18:13:10.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.078+0000 7f435a800640 1 --2- 172.21.15.37:0/378939412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a145b2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:13:10.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.078+0000 7f4359400640 1 --2- 172.21.15.37:0/378939412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14537c00 0x556a14539f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:13:10.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.079+0000 7f4359e00640 1 --2- 172.21.15.37:0/378939412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14536f20 0x556a14539a30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f434c000c00 tx=0x7f434c005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ec3bdcc8eb732a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:10.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.079+0000 7f433b400640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:13:10.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.079+0000 7f433b400640 1 -- 172.21.15.37:0/378939412 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c0f3ca0 con 0x556a14536f20 2024-03-28T18:13:10.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.079+0000 7f433b400640 1 -- 172.21.15.37:0/378939412 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f434c0edee0 con 0x556a14536f20 2024-03-28T18:13:10.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.079+0000 7f43637f9740 1 -- 172.21.15.37:0/378939412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4354003e50 con 0x556a14536f20 2024-03-28T18:13:10.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.079+0000 7f433b400640 1 -- 172.21.15.37:0/378939412 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f434c0bcc80 con 0x556a14536f20 2024-03-28T18:13:10.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.079+0000 7f43637f9740 10 client.25796 fetch_fsmap learned FSMap version 24 2024-03-28T18:13:10.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.079+0000 7f43637f9740 1 -- 172.21.15.37:0/378939412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x556a145a4a90 con 0x556a14536f20 2024-03-28T18:13:10.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.080+0000 7f433b400640 1 -- 172.21.15.37:0/378939412 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4354003e50 con 0x556a14536f20 2024-03-28T18:13:10.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.080+0000 7f433b400640 1 -- 172.21.15.37:0/378939412 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f434c03c030 con 0x556a14536f20 2024-03-28T18:13:10.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.080+0000 7f43637f9740 10 client.25796 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T18:13:10.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.080+0000 7f43637f9740 10 client.25796 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:13:10.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.080+0000 7f43637f9740 20 client.25796 populate_metadata read hostname 'smithi037' 2024-03-28T18:13:10.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.080+0000 7f43637f9740 1 --2- 172.21.15.37:0/378939412 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556a145a51c0 0x556a145a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.080+0000 7f43637f9740 4 client.25796 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:13:10.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.080+0000 7f43637f9740 1 -- 172.21.15.37:0/378939412 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556a144a2ae0 con 0x556a145a51c0 2024-03-28T18:13:10.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.080+0000 7f4359400640 1 --2- 172.21.15.37:0/378939412 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556a145a51c0 0x556a145a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.081+0000 7f4359400640 1 --2- 172.21.15.37:0/378939412 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556a145a51c0 0x556a145a75e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=219d8c695664f721 server_cookie=f7412442ce669420 in_seq=0 out_seq=0 2024-03-28T18:13:10.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.081+0000 7f433b400640 10 client.25796 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:13:10.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.082+0000 7f433b400640 1 -- 172.21.15.37:0/378939412 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x556a144a2ae0 con 0x556a145a51c0 2024-03-28T18:13:10.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.082+0000 7f433b400640 10 client.25796 handle_command_reply: tid=0 2024-03-28T18:13:10.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.082+0000 7f43637f9740 1 client.25796 shutdown 2024-03-28T18:13:10.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.082+0000 7f43637f9740 1 -- 172.21.15.37:0/378939412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14536f20 msgr2=0x556a14539a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.082+0000 7f43637f9740 1 --2- 172.21.15.37:0/378939412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14536f20 0x556a14539a30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f434c000c00 tx=0x7f434c005740 comp rx=0 tx=0).stop 2024-03-28T18:13:10.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.083+0000 7f43637f9740 1 -- 172.21.15.37:0/378939412 shutdown_connections 2024-03-28T18:13:10.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.083+0000 7f43637f9740 1 --2- 172.21.15.37:0/378939412 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x556a145a51c0 0x556a145a75e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.083+0000 7f43637f9740 1 --2- 172.21.15.37:0/378939412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556a14540290 0x556a145b2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.083+0000 7f43637f9740 1 --2- 172.21.15.37:0/378939412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x556a14537c00 0x556a14539f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.083+0000 7f43637f9740 1 --2- 172.21.15.37:0/378939412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556a14536f20 0x556a14539a30 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.083+0000 7f43637f9740 1 -- 172.21.15.37:0/378939412 >> 172.21.15.37:0/378939412 conn(0x556a144fe200 msgr2=0x556a14530200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:10.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.083+0000 7f433b400640 0 client.25796 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:13:10.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.083+0000 7f43637f9740 1 -- 172.21.15.37:0/378939412 shutdown_connections 2024-03-28T18:13:10.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.083+0000 7f43637f9740 1 -- 172.21.15.37:0/378939412 wait complete. 2024-03-28T18:13:10.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.083+0000 7f43637f9740 20 client.25796 trim_cache size 0 max 16384 2024-03-28T18:13:10.084 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:13:10.084 INFO:teuthology.orchestra.run.smithi037.stdout: "status": "no active scrubs running", 2024-03-28T18:13:10.084 INFO:teuthology.orchestra.run.smithi037.stdout: "scrubs": {} 2024-03-28T18:13:10.084 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:13:10.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.084+0000 7f4360a00640 1 -- 172.21.15.37:0/2054133386 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f43400a29f0 msgr2=0x7f43400a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.084+0000 7f4360a00640 1 --2- 172.21.15.37:0/2054133386 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f43400a29f0 0x7f43400a4eb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f435c06af40 tx=0x7f4354011040 comp rx=0 tx=0).stop 2024-03-28T18:13:10.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.084+0000 7f4360a00640 1 -- 172.21.15.37:0/2054133386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f435c073a70 msgr2=0x7f435c19a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.084+0000 7f4360a00640 1 --2- 172.21.15.37:0/2054133386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f435c073a70 0x7f435c19a190 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f434c037040 tx=0x7f434c0344c0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.084+0000 7f4360a00640 1 -- 172.21.15.37:0/2054133386 shutdown_connections 2024-03-28T18:13:10.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.084+0000 7f4360a00640 1 --2- 172.21.15.37:0/2054133386 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f43400a29f0 0x7f43400a4eb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.084+0000 7f4360a00640 1 --2- 172.21.15.37:0/2054133386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f435c10ded0 0x7f435c19ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.084+0000 7f4360a00640 1 --2- 172.21.15.37:0/2054133386 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f435c074390 0x7f435c19a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.084+0000 7f4360a00640 1 --2- 172.21.15.37:0/2054133386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f435c073a70 0x7f435c19a190 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.084+0000 7f4360a00640 1 -- 172.21.15.37:0/2054133386 >> 172.21.15.37:0/2054133386 conn(0x7f435c078aa0 msgr2=0x7f435c0fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:10.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.084+0000 7f4360a00640 1 -- 172.21.15.37:0/2054133386 shutdown_connections 2024-03-28T18:13:10.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.084+0000 7f4360a00640 1 -- 172.21.15.37:0/2054133386 wait complete. 2024-03-28T18:13:10.104 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:13:10.104 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:13:10.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.313+0000 7f3c9ea00640 1 -- 172.21.15.37:0/1584038062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c98109690 msgr2=0x7f3c98109a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.313+0000 7f3c9ea00640 1 --2- 172.21.15.37:0/1584038062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c98109690 0x7f3c98109a70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c009900 tx=0x7f3c8c02bc40 comp rx=0 tx=0).stop 2024-03-28T18:13:10.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.314+0000 7f3c9ea00640 1 -- 172.21.15.37:0/1584038062 shutdown_connections 2024-03-28T18:13:10.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.314+0000 7f3c9ea00640 1 --2- 172.21.15.37:0/1584038062 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c98104040 0x7f3c98110550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.314+0000 7f3c9ea00640 1 --2- 172.21.15.37:0/1584038062 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c98103680 0x7f3c98103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.314+0000 7f3c9ea00640 1 --2- 172.21.15.37:0/1584038062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c98109690 0x7f3c98109a70 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.314+0000 7f3c9ea00640 1 -- 172.21.15.37:0/1584038062 >> 172.21.15.37:0/1584038062 conn(0x7f3c980ff130 msgr2=0x7f3c98101550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:10.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.314+0000 7f3c9ea00640 1 -- 172.21.15.37:0/1584038062 shutdown_connections 2024-03-28T18:13:10.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.314+0000 7f3c9ea00640 1 -- 172.21.15.37:0/1584038062 wait complete. 2024-03-28T18:13:10.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.315+0000 7f3c9ea00640 1 Processor -- start 2024-03-28T18:13:10.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.315+0000 7f3c9ea00640 1 -- start start 2024-03-28T18:13:10.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.315+0000 7f3c9ea00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c98103680 0x7f3c98199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.315+0000 7f3c9ea00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c98104040 0x7f3c9819a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.315+0000 7f3c9ea00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c98109690 0x7f3c9819e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.316+0000 7f3c9ea00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c98113af0 con 0x7f3c98109690 2024-03-28T18:13:10.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.316+0000 7f3c9ea00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c98113970 con 0x7f3c98104040 2024-03-28T18:13:10.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.316+0000 7f3c9ea00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c98113c70 con 0x7f3c98103680 2024-03-28T18:13:10.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.316+0000 7f3c9d600640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c98103680 0x7f3c98199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.316+0000 7f3c9d600640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c98103680 0x7f3c98199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56586/0 (socket says 172.21.15.37:56586) 2024-03-28T18:13:10.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.316+0000 7f3c9cc00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c98104040 0x7f3c9819a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.316+0000 7f3c9d600640 1 -- 172.21.15.37:0/2018398521 learned_addr learned my addr 172.21.15.37:0/2018398521 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:10.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.316+0000 7f3c9e000640 1 --2- 172.21.15.37:0/2018398521 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c98109690 0x7f3c9819e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.316+0000 7f3c9cc00640 1 -- 172.21.15.37:0/2018398521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c98103680 msgr2=0x7f3c98199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.316+0000 7f3c9cc00640 1 --2- 172.21.15.37:0/2018398521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c98103680 0x7f3c98199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.316+0000 7f3c9cc00640 1 -- 172.21.15.37:0/2018398521 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c98109690 msgr2=0x7f3c9819e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.316+0000 7f3c9cc00640 1 --2- 172.21.15.37:0/2018398521 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c98109690 0x7f3c9819e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.316+0000 7f3c9cc00640 1 -- 172.21.15.37:0/2018398521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9819ef40 con 0x7f3c98104040 2024-03-28T18:13:10.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.317+0000 7f3c9d600640 1 --2- 172.21.15.37:0/2018398521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c98103680 0x7f3c98199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:13:10.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.317+0000 7f3c9cc00640 1 --2- 172.21.15.37:0/2018398521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c98104040 0x7f3c9819a490 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3c8400e9f0 tx=0x7f3c8400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:10.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.317+0000 7f3c96000640 1 -- 172.21.15.37:0/2018398521 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8400ccc0 con 0x7f3c98104040 2024-03-28T18:13:10.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.317+0000 7f3c96000640 1 -- 172.21.15.37:0/2018398521 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c84010910 con 0x7f3c98104040 2024-03-28T18:13:10.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.317+0000 7f3c96000640 1 -- 172.21.15.37:0/2018398521 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c84018960 con 0x7f3c98104040 2024-03-28T18:13:10.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.317+0000 7f3c9ea00640 1 -- 172.21.15.37:0/2018398521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9819f230 con 0x7f3c98104040 2024-03-28T18:13:10.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.317+0000 7f3c9ea00640 1 -- 172.21.15.37:0/2018398521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c981a6ac0 con 0x7f3c98104040 2024-03-28T18:13:10.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.319+0000 7f3c9ea00640 1 -- 172.21.15.37:0/2018398521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c98068800 con 0x7f3c98104040 2024-03-28T18:13:10.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.321+0000 7f3c96000640 1 -- 172.21.15.37:0/2018398521 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f3c84010ac0 con 0x7f3c98104040 2024-03-28T18:13:10.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.322+0000 7f3c96000640 1 --2- 172.21.15.37:0/2018398521 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f3c800a2a30 0x7f3c800a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.323+0000 7f3c96000640 1 -- 172.21.15.37:0/2018398521 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3c840cf190 con 0x7f3c98104040 2024-03-28T18:13:10.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.323+0000 7f3c9d600640 1 --2- 172.21.15.37:0/2018398521 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f3c800a2a30 0x7f3c800a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.323+0000 7f3c9d600640 1 --2- 172.21.15.37:0/2018398521 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f3c800a2a30 0x7f3c800a4ef0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c02c860 tx=0x7f3c8c005b50 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:10.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.327+0000 7f3c96000640 1 -- 172.21.15.37:0/2018398521 <== mon.2 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3c8409b870 con 0x7f3c98104040 2024-03-28T18:13:10.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.657+0000 7f3c9ea00640 1 -- 172.21.15.37:0/2018398521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c9810b660 con 0x7f3c98104040 2024-03-28T18:13:10.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.658+0000 7f3c96000640 1 -- 172.21.15.37:0/2018398521 <== mon.2 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11386 (secure 0 0 0) 0x7f3c8409f910 con 0x7f3c98104040 2024-03-28T18:13:10.658 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:13:10.658 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1926270371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6838","nonce":1926270371},{"type":"v1","addr":"172.21.15.37:6839","nonce":1926270371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:08:19.968894+0000","modified":"2024-03-28T18:13:00.596304+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24580,"mds_1":24569,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6837/4034607523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":4034607523},{"type":"v1","addr":"172.21.15.59:6837","nonce":4034607523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6839/1575518279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":1575518279},{"type":"v1","addr":"172.21.15.170:6839","nonce":1575518279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6839/2091629078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":2091629078},{"type":"v1","addr":"172.21.15.59:6839","nonce":2091629078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24580":{"gid":24580,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.37:6841/1923263622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6840","nonce":1923263622},{"type":"v1","addr":"172.21.15.37:6841","nonce":1923263622}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24580,"qdb_cluster":[24569,24580,24550]},"id":1}]} 2024-03-28T18:13:10.659 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 24 2024-03-28T18:13:10.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.661+0000 7f3c9ea00640 1 -- 172.21.15.37:0/2018398521 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f3c800a2a30 msgr2=0x7f3c800a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.661+0000 7f3c9ea00640 1 --2- 172.21.15.37:0/2018398521 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f3c800a2a30 0x7f3c800a4ef0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c02c860 tx=0x7f3c8c005b50 comp rx=0 tx=0).stop 2024-03-28T18:13:10.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.661+0000 7f3c9ea00640 1 -- 172.21.15.37:0/2018398521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c98104040 msgr2=0x7f3c9819a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.661+0000 7f3c9ea00640 1 --2- 172.21.15.37:0/2018398521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c98104040 0x7f3c9819a490 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3c8400e9f0 tx=0x7f3c8400eeb0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.661+0000 7f3c9ea00640 1 -- 172.21.15.37:0/2018398521 shutdown_connections 2024-03-28T18:13:10.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.661+0000 7f3c9ea00640 1 --2- 172.21.15.37:0/2018398521 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f3c800a2a30 0x7f3c800a4ef0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.661+0000 7f3c9ea00640 1 --2- 172.21.15.37:0/2018398521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c98104040 0x7f3c9819a490 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.661+0000 7f3c9ea00640 1 --2- 172.21.15.37:0/2018398521 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c98109690 0x7f3c9819e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.661+0000 7f3c9ea00640 1 --2- 172.21.15.37:0/2018398521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c98103680 0x7f3c98199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.661+0000 7f3c9ea00640 1 -- 172.21.15.37:0/2018398521 >> 172.21.15.37:0/2018398521 conn(0x7f3c980ff130 msgr2=0x7f3c980ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:10.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.661+0000 7f3c9ea00640 1 -- 172.21.15.37:0/2018398521 shutdown_connections 2024-03-28T18:13:10.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.661+0000 7f3c9ea00640 1 -- 172.21.15.37:0/2018398521 wait complete. 2024-03-28T18:13:10.686 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:13:10.686 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:13:10.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.909+0000 7fce8de00640 1 -- 172.21.15.37:0/2401453011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce88107650 msgr2=0x7fce88107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.909+0000 7fce8de00640 1 --2- 172.21.15.37:0/2401453011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce88107650 0x7fce88107a30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fce78009930 tx=0x7fce7802bc30 comp rx=0 tx=0).stop 2024-03-28T18:13:10.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.909+0000 7fce8de00640 1 -- 172.21.15.37:0/2401453011 shutdown_connections 2024-03-28T18:13:10.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.909+0000 7fce8de00640 1 --2- 172.21.15.37:0/2401453011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce88101640 0x7fce88101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.909+0000 7fce8de00640 1 --2- 172.21.15.37:0/2401453011 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce88102000 0x7fce8810e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.909+0000 7fce8de00640 1 --2- 172.21.15.37:0/2401453011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce88107650 0x7fce88107a30 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.909+0000 7fce8de00640 1 -- 172.21.15.37:0/2401453011 >> 172.21.15.37:0/2401453011 conn(0x7fce880fd110 msgr2=0x7fce880ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:10.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.909+0000 7fce8de00640 1 -- 172.21.15.37:0/2401453011 shutdown_connections 2024-03-28T18:13:10.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.909+0000 7fce8de00640 1 -- 172.21.15.37:0/2401453011 wait complete. 2024-03-28T18:13:10.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.910+0000 7fce8de00640 1 Processor -- start 2024-03-28T18:13:10.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.910+0000 7fce8de00640 1 -- start start 2024-03-28T18:13:10.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.911+0000 7fce8de00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce88101640 0x7fce8819a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.911+0000 7fce8de00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce88102000 0x7fce8819a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.911+0000 7fce8de00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce88107650 0x7fce8819ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.911+0000 7fce8de00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce88111a70 con 0x7fce88107650 2024-03-28T18:13:10.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.911+0000 7fce8de00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce881118f0 con 0x7fce88102000 2024-03-28T18:13:10.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.911+0000 7fce8de00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce88111bf0 con 0x7fce88101640 2024-03-28T18:13:10.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.911+0000 7fce86a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce88101640 0x7fce8819a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.911+0000 7fce86a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce88101640 0x7fce8819a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56614/0 (socket says 172.21.15.37:56614) 2024-03-28T18:13:10.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.911+0000 7fce86a00640 1 -- 172.21.15.37:0/1672477107 learned_addr learned my addr 172.21.15.37:0/1672477107 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:10.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.911+0000 7fce86000640 1 --2- 172.21.15.37:0/1672477107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce88102000 0x7fce8819a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.911+0000 7fce87400640 1 --2- 172.21.15.37:0/1672477107 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce88107650 0x7fce8819ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.912+0000 7fce86a00640 1 -- 172.21.15.37:0/1672477107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce88102000 msgr2=0x7fce8819a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.912+0000 7fce86a00640 1 --2- 172.21.15.37:0/1672477107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce88102000 0x7fce8819a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.912+0000 7fce86a00640 1 -- 172.21.15.37:0/1672477107 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce88107650 msgr2=0x7fce8819ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.912+0000 7fce86a00640 1 --2- 172.21.15.37:0/1672477107 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce88107650 0x7fce8819ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.912+0000 7fce86a00640 1 -- 172.21.15.37:0/1672477107 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8819f170 con 0x7fce88101640 2024-03-28T18:13:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.912+0000 7fce87400640 1 --2- 172.21.15.37:0/1672477107 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce88107650 0x7fce8819ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:13:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.912+0000 7fce86000640 1 --2- 172.21.15.37:0/1672477107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce88102000 0x7fce8819a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:13:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.912+0000 7fce86a00640 1 --2- 172.21.15.37:0/1672477107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce88101640 0x7fce8819a180 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fce78009900 tx=0x7fce78004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.912+0000 7fce8ca00640 1 -- 172.21.15.37:0/1672477107 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce78043020 con 0x7fce88101640 2024-03-28T18:13:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.912+0000 7fce8ca00640 1 -- 172.21.15.37:0/1672477107 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce780054e0 con 0x7fce88101640 2024-03-28T18:13:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.912+0000 7fce8de00640 1 -- 172.21.15.37:0/1672477107 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce8819f400 con 0x7fce88101640 2024-03-28T18:13:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.912+0000 7fce8ca00640 1 -- 172.21.15.37:0/1672477107 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce78003bb0 con 0x7fce88101640 2024-03-28T18:13:10.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.913+0000 7fce8de00640 1 -- 172.21.15.37:0/1672477107 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce881a6cf0 con 0x7fce88101640 2024-03-28T18:13:10.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.917+0000 7fce90bef740 1 Processor -- start 2024-03-28T18:13:10.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.917+0000 7fce90bef740 1 -- start start 2024-03-28T18:13:10.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.917+0000 7fce8ca00640 1 -- 172.21.15.37:0/1672477107 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fce78046020 con 0x7fce88101640 2024-03-28T18:13:10.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.918+0000 7fce90bef740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06e36d0 0x5627e06e3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.918+0000 7fce90bef740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06e4080 0x5627e06ec630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.918+0000 7fce90bef740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06ecc00 0x5627e06ef050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.918+0000 7fce90bef740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e06472c0 con 0x5627e06ecc00 2024-03-28T18:13:10.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.918+0000 7fce90bef740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e05efc90 con 0x5627e06e4080 2024-03-28T18:13:10.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.918+0000 7fce90bef740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e0643360 con 0x5627e06e36d0 2024-03-28T18:13:10.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.918+0000 7fce86a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06e4080 0x5627e06ec630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.918+0000 7fce87400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06e36d0 0x5627e06e3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.918+0000 7fce86a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06e4080 0x5627e06ec630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:34958/0 (socket says 172.21.15.37:34958) 2024-03-28T18:13:10.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.918+0000 7fce86a00640 1 -- 172.21.15.37:0/1665645407 learned_addr learned my addr 172.21.15.37:0/1665645407 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:10.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.918+0000 7fce87400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06e36d0 0x5627e06e3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56618/0 (socket says 172.21.15.37:56618) 2024-03-28T18:13:10.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.918+0000 7fce86000640 1 --2- 172.21.15.37:0/1665645407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06ecc00 0x5627e06ef050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.918+0000 7fce86a00640 1 -- 172.21.15.37:0/1665645407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06e36d0 msgr2=0x5627e06e3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:10.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.918+0000 7fce86a00640 1 --2- 172.21.15.37:0/1665645407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06e36d0 0x5627e06e3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.918+0000 7fce86a00640 1 -- 172.21.15.37:0/1665645407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06ecc00 msgr2=0x5627e06ef050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:10.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.918+0000 7fce86a00640 1 --2- 172.21.15.37:0/1665645407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06ecc00 0x5627e06ef050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.918+0000 7fce86a00640 1 -- 172.21.15.37:0/1665645407 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627e0614080 con 0x5627e06e4080 2024-03-28T18:13:10.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.919+0000 7fce86a00640 1 --2- 172.21.15.37:0/1665645407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06e4080 0x5627e06ec630 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fce780ee520 tx=0x7fce780f4dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b947703d1d1c0b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:10.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.919+0000 7fce73e00640 1 -- 172.21.15.37:0/1665645407 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce780f7ac0 con 0x5627e06e4080 2024-03-28T18:13:10.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.919+0000 7fce73e00640 1 -- 172.21.15.37:0/1665645407 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce780f7c60 con 0x5627e06e4080 2024-03-28T18:13:10.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.919+0000 7fce73e00640 1 -- 172.21.15.37:0/1665645407 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce78101630 con 0x5627e06e4080 2024-03-28T18:13:10.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.919+0000 7fce8ca00640 1 --2- 172.21.15.37:0/1672477107 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fce6c0a29f0 0x7fce6c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.920+0000 7fce8ca00640 1 -- 172.21.15.37:0/1672477107 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fce780f0db0 con 0x7fce88101640 2024-03-28T18:13:10.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.920+0000 7fce87400640 1 --2- 172.21.15.37:0/1672477107 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fce6c0a29f0 0x7fce6c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.920+0000 7fce90bef740 1 -- 172.21.15.37:0/1665645407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06e4080 msgr2=0x5627e06ec630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.920+0000 7fce90bef740 1 --2- 172.21.15.37:0/1665645407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06e4080 0x5627e06ec630 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fce780ee520 tx=0x7fce780f4dd0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.920+0000 7fce90bef740 1 -- 172.21.15.37:0/1665645407 shutdown_connections 2024-03-28T18:13:10.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.920+0000 7fce90bef740 1 --2- 172.21.15.37:0/1665645407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06e4080 0x5627e06ec630 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.920+0000 7fce90bef740 1 --2- 172.21.15.37:0/1665645407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06ecc00 0x5627e06ef050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.920+0000 7fce90bef740 1 --2- 172.21.15.37:0/1665645407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06e36d0 0x5627e06e3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.920+0000 7fce90bef740 1 -- 172.21.15.37:0/1665645407 >> 172.21.15.37:0/1665645407 conn(0x5627e06ddda0 msgr2=0x5627e06de180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:10.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.921+0000 7fce90bef740 1 -- 172.21.15.37:0/1665645407 shutdown_connections 2024-03-28T18:13:10.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.921+0000 7fce90bef740 1 -- 172.21.15.37:0/1665645407 wait complete. 2024-03-28T18:13:10.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.921+0000 7fce87400640 1 --2- 172.21.15.37:0/1672477107 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fce6c0a29f0 0x7fce6c0a4eb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fce80005fd0 tx=0x7fce80009500 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:10.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.921+0000 7fce90bef740 1 -- start start 2024-03-28T18:13:10.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.921+0000 7fce90bef740 1 Processor -- start 2024-03-28T18:13:10.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.922+0000 7fce90bef740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06e36d0 0x5627e06ec510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.922+0000 7fce90bef740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06e4080 0x5627e06e98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.922+0000 7fce90bef740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06ecc00 0x5627e06e9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.922+0000 7fce90bef740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e06ef930 con 0x5627e06ecc00 2024-03-28T18:13:10.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.922+0000 7fce90bef740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e06030d0 con 0x5627e06e36d0 2024-03-28T18:13:10.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.922+0000 7fce90bef740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e06efab0 con 0x5627e06e4080 2024-03-28T18:13:10.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.922+0000 7fce90bef740 10 client.0 _ll_register_callbacks cb 0x5627e0607e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:13:10.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.922+0000 7fce87400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06ecc00 0x5627e06e9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.922+0000 7fce86a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06e36d0 0x5627e06ec510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.922+0000 7fce86a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06e36d0 0x5627e06ec510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:34974/0 (socket says 172.21.15.37:34974) 2024-03-28T18:13:10.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.922+0000 7fce86a00640 1 -- 172.21.15.37:0/3952309967 learned_addr learned my addr 172.21.15.37:0/3952309967 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:10.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.922+0000 7fce86000640 1 --2- 172.21.15.37:0/3952309967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06e4080 0x5627e06e98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.923+0000 7fce86a00640 1 -- 172.21.15.37:0/3952309967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06e4080 msgr2=0x5627e06e98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:10.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.923+0000 7fce86a00640 1 --2- 172.21.15.37:0/3952309967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06e4080 0x5627e06e98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.923+0000 7fce86a00640 1 -- 172.21.15.37:0/3952309967 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06ecc00 msgr2=0x5627e06e9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:10.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.923+0000 7fce86a00640 1 --2- 172.21.15.37:0/3952309967 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06ecc00 0x5627e06e9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.923+0000 7fce86a00640 1 -- 172.21.15.37:0/3952309967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5627e035b3f0 con 0x5627e06e36d0 2024-03-28T18:13:10.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.923+0000 7fce87400640 1 --2- 172.21.15.37:0/3952309967 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06ecc00 0x5627e06e9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:13:10.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.923+0000 7fce86a00640 1 --2- 172.21.15.37:0/3952309967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06e36d0 0x5627e06ec510 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fce780b9950 tx=0x7fce78004ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fae94d0bea1cfe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:10.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.923+0000 7fce70c00640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:13:10.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.923+0000 7fce70c00640 1 -- 172.21.15.37:0/3952309967 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce780b9790 con 0x5627e06e36d0 2024-03-28T18:13:10.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.923+0000 7fce70c00640 1 -- 172.21.15.37:0/3952309967 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fce78003d50 con 0x5627e06e36d0 2024-03-28T18:13:10.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.923+0000 7fce90bef740 1 -- 172.21.15.37:0/3952309967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fce780f7c60 con 0x5627e06e36d0 2024-03-28T18:13:10.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.924+0000 7fce70c00640 1 -- 172.21.15.37:0/3952309967 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce780f7ac0 con 0x5627e06e36d0 2024-03-28T18:13:10.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.924+0000 7fce90bef740 10 client.25814 fetch_fsmap learned FSMap version 24 2024-03-28T18:13:10.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.924+0000 7fce90bef740 1 -- 172.21.15.37:0/3952309967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5627e04257d0 con 0x5627e06e36d0 2024-03-28T18:13:10.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.924+0000 7fce70c00640 1 -- 172.21.15.37:0/3952309967 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce780f7c60 con 0x5627e06e36d0 2024-03-28T18:13:10.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.924+0000 7fce70c00640 1 -- 172.21.15.37:0/3952309967 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fce780b9b60 con 0x5627e06e36d0 2024-03-28T18:13:10.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.924+0000 7fce90bef740 10 client.25814 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T18:13:10.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.924+0000 7fce90bef740 10 client.25814 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-28T18:13:10.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.924+0000 7fce90bef740 20 client.25814 populate_metadata read hostname 'smithi037' 2024-03-28T18:13:10.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.924+0000 7fce90bef740 1 --2- 172.21.15.37:0/3952309967 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5627e06fc640 0x5627e06fea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.924+0000 7fce90bef740 4 client.25814 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:13:10.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.924+0000 7fce90bef740 1 -- 172.21.15.37:0/3952309967 --> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5627e0654800 con 0x5627e06fc640 2024-03-28T18:13:10.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.925+0000 7fce86000640 1 --2- 172.21.15.37:0/3952309967 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5627e06fc640 0x5627e06fea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.925+0000 7fce86000640 1 --2- 172.21.15.37:0/3952309967 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5627e06fc640 0x5627e06fea30 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=26decf86409a39e7 server_cookie=d05082ddea40d457 in_seq=0 out_seq=0 2024-03-28T18:13:10.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.925+0000 7fce70c00640 10 client.25814 ms_handle_connect on v2:172.21.15.170:6838/1575518279 2024-03-28T18:13:10.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.926+0000 7fce70c00640 1 -- 172.21.15.37:0/3952309967 <== mds.2 v2:172.21.15.170:6838/1575518279 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5627e0654800 con 0x5627e06fc640 2024-03-28T18:13:10.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.926+0000 7fce70c00640 10 client.25814 handle_command_reply: tid=0 2024-03-28T18:13:10.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.926+0000 7fce90bef740 1 client.25814 shutdown 2024-03-28T18:13:10.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.926+0000 7fce90bef740 1 -- 172.21.15.37:0/3952309967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06e36d0 msgr2=0x5627e06ec510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.926+0000 7fce90bef740 1 --2- 172.21.15.37:0/3952309967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06e36d0 0x5627e06ec510 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fce780b9950 tx=0x7fce78004ee0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.927+0000 7fce90bef740 1 -- 172.21.15.37:0/3952309967 shutdown_connections 2024-03-28T18:13:10.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.927+0000 7fce90bef740 1 --2- 172.21.15.37:0/3952309967 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5627e06fc640 0x5627e06fea30 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.927+0000 7fce90bef740 1 --2- 172.21.15.37:0/3952309967 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06ecc00 0x5627e06e9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.927+0000 7fce90bef740 1 --2- 172.21.15.37:0/3952309967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06e4080 0x5627e06e98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.927+0000 7fce90bef740 1 --2- 172.21.15.37:0/3952309967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06e36d0 0x5627e06ec510 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.927+0000 7fce90bef740 1 -- 172.21.15.37:0/3952309967 >> 172.21.15.37:0/3952309967 conn(0x5627e06ddda0 msgr2=0x5627e06dfaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:10.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.927+0000 7fce70c00640 0 client.25814 ms_handle_reset on v2:172.21.15.170:6838/1575518279 2024-03-28T18:13:10.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.927+0000 7fce90bef740 1 -- 172.21.15.37:0/3952309967 shutdown_connections 2024-03-28T18:13:10.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.927+0000 7fce90bef740 1 -- 172.21.15.37:0/3952309967 wait complete. 2024-03-28T18:13:10.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.928+0000 7fce90bef740 20 client.25814 trim_cache size 0 max 16384 2024-03-28T18:13:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.943+0000 7fce90bef740 1 Processor -- start 2024-03-28T18:13:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.944+0000 7fce90bef740 1 -- start start 2024-03-28T18:13:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.944+0000 7fce90bef740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06e36d0 0x5627e06f8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.944+0000 7fce90bef740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06e4080 0x5627e06ebfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.944+0000 7fce90bef740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06ecc00 0x5627e06ec510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.944+0000 7fce90bef740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce881b1910 con 0x5627e06ecc00 2024-03-28T18:13:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.944+0000 7fce90bef740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce881af410 con 0x5627e06e36d0 2024-03-28T18:13:10.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.944+0000 7fce90bef740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce881ad4b0 con 0x5627e06e4080 2024-03-28T18:13:10.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.944+0000 7fce86000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06e4080 0x5627e06ebfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.944+0000 7fce86a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06e36d0 0x5627e06f8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.944+0000 7fce86000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06e4080 0x5627e06ebfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56640/0 (socket says 172.21.15.37:56640) 2024-03-28T18:13:10.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.944+0000 7fce86000640 1 -- 172.21.15.37:0/4214248281 learned_addr learned my addr 172.21.15.37:0/4214248281 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:10.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.944+0000 7fce87400640 1 --2- 172.21.15.37:0/4214248281 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06ecc00 0x5627e06ec510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.945+0000 7fce86a00640 1 -- 172.21.15.37:0/4214248281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06e4080 msgr2=0x5627e06ebfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:10.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.945+0000 7fce86a00640 1 --2- 172.21.15.37:0/4214248281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06e4080 0x5627e06ebfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.945+0000 7fce86a00640 1 -- 172.21.15.37:0/4214248281 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06ecc00 msgr2=0x5627e06ec510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:10.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.945+0000 7fce86a00640 1 --2- 172.21.15.37:0/4214248281 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06ecc00 0x5627e06ec510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.945+0000 7fce86a00640 1 -- 172.21.15.37:0/4214248281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627e06ea1b0 con 0x5627e06e36d0 2024-03-28T18:13:10.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.945+0000 7fce86a00640 1 --2- 172.21.15.37:0/4214248281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06e36d0 0x5627e06f8be0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fce780bd480 tx=0x7fce780f4dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c32e6853aad03b55 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:10.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.945+0000 7fce73400640 1 -- 172.21.15.37:0/4214248281 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce780bce50 con 0x5627e06e36d0 2024-03-28T18:13:10.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.945+0000 7fce73400640 1 -- 172.21.15.37:0/4214248281 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce780f7c60 con 0x5627e06e36d0 2024-03-28T18:13:10.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.945+0000 7fce73400640 1 -- 172.21.15.37:0/4214248281 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce780bd030 con 0x5627e06e36d0 2024-03-28T18:13:10.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.946+0000 7fce90bef740 1 -- 172.21.15.37:0/4214248281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06e36d0 msgr2=0x5627e06f8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.946+0000 7fce90bef740 1 --2- 172.21.15.37:0/4214248281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06e36d0 0x5627e06f8be0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fce780bd480 tx=0x7fce780f4dd0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.946+0000 7fce90bef740 1 -- 172.21.15.37:0/4214248281 shutdown_connections 2024-03-28T18:13:10.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.946+0000 7fce90bef740 1 --2- 172.21.15.37:0/4214248281 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06ecc00 0x5627e06ec510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.946+0000 7fce90bef740 1 --2- 172.21.15.37:0/4214248281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06e4080 0x5627e06ebfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.946+0000 7fce90bef740 1 --2- 172.21.15.37:0/4214248281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06e36d0 0x5627e06f8be0 secure :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fce780bd480 tx=0x7fce780f4dd0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.946+0000 7fce90bef740 1 -- 172.21.15.37:0/4214248281 >> 172.21.15.37:0/4214248281 conn(0x5627e06ddda0 msgr2=0x5627e06dceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:10.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.947+0000 7fce90bef740 1 -- 172.21.15.37:0/4214248281 shutdown_connections 2024-03-28T18:13:10.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.948+0000 7fce90bef740 1 -- 172.21.15.37:0/4214248281 wait complete. 2024-03-28T18:13:10.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.948+0000 7fce90bef740 1 -- start start 2024-03-28T18:13:10.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.948+0000 7fce90bef740 1 Processor -- start 2024-03-28T18:13:10.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.949+0000 7fce90bef740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06e4080 0x5627e0701430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.949+0000 7fce90bef740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06ecc00 0x5627e0701970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.949+0000 7fce90bef740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06ff840 0x5627e06ffcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.949+0000 7fce90bef740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce881adab0 con 0x5627e06e4080 2024-03-28T18:13:10.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.949+0000 7fce90bef740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce881aedb0 con 0x5627e06ecc00 2024-03-28T18:13:10.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.949+0000 7fce90bef740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce881ae0f0 con 0x5627e06ff840 2024-03-28T18:13:10.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.949+0000 7fce90bef740 10 client.0 _ll_register_callbacks cb 0x7fce800099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:13:10.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.949+0000 7fce87400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06ff840 0x5627e06ffcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.949+0000 7fce86a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06e4080 0x5627e0701430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.949+0000 7fce87400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06ff840 0x5627e06ffcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56650/0 (socket says 172.21.15.37:56650) 2024-03-28T18:13:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.949+0000 7fce87400640 1 -- 172.21.15.37:0/4001215978 learned_addr learned my addr 172.21.15.37:0/4001215978 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.950+0000 7fce86000640 1 --2- 172.21.15.37:0/4001215978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06ecc00 0x5627e0701970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.950+0000 7fce87400640 1 -- 172.21.15.37:0/4001215978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06ecc00 msgr2=0x5627e0701970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.950+0000 7fce87400640 1 --2- 172.21.15.37:0/4001215978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06ecc00 0x5627e0701970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.950+0000 7fce87400640 1 -- 172.21.15.37:0/4001215978 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06e4080 msgr2=0x5627e0701430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.950+0000 7fce87400640 1 --2- 172.21.15.37:0/4001215978 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06e4080 0x5627e0701430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.950+0000 7fce87400640 1 -- 172.21.15.37:0/4001215978 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5627e07004c0 con 0x5627e06ff840 2024-03-28T18:13:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.950+0000 7fce86000640 1 --2- 172.21.15.37:0/4001215978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06ecc00 0x5627e0701970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:13:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.950+0000 7fce87400640 1 --2- 172.21.15.37:0/4001215978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06ff840 0x5627e06ffcc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fce8001ff40 tx=0x7fce80004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74178e0d9c81b7ba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.950+0000 7fce70c00640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:13:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.951+0000 7fce70c00640 1 -- 172.21.15.37:0/4001215978 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8002f060 con 0x5627e06ff840 2024-03-28T18:13:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.951+0000 7fce70c00640 1 -- 172.21.15.37:0/4001215978 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fce8000e5d0 con 0x5627e06ff840 2024-03-28T18:13:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.951+0000 7fce90bef740 1 -- 172.21.15.37:0/4001215978 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fce780f7c60 con 0x5627e06ff840 2024-03-28T18:13:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.951+0000 7fce70c00640 1 -- 172.21.15.37:0/4001215978 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce8001de60 con 0x5627e06ff840 2024-03-28T18:13:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.951+0000 7fce90bef740 10 client.25828 fetch_fsmap learned FSMap version 24 2024-03-28T18:13:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.951+0000 7fce90bef740 1 -- 172.21.15.37:0/4001215978 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5627e0758c70 con 0x5627e06ff840 2024-03-28T18:13:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.951+0000 7fce70c00640 1 -- 172.21.15.37:0/4001215978 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce780f7c60 con 0x5627e06ff840 2024-03-28T18:13:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.952+0000 7fce70c00640 1 -- 172.21.15.37:0/4001215978 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fce80029070 con 0x5627e06ff840 2024-03-28T18:13:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.952+0000 7fce90bef740 10 client.25828 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T18:13:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.952+0000 7fce90bef740 10 client.25828 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-28T18:13:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.952+0000 7fce90bef740 20 client.25828 populate_metadata read hostname 'smithi037' 2024-03-28T18:13:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.952+0000 7fce90bef740 1 --2- 172.21.15.37:0/4001215978 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5627e07593d0 0x5627e075b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.952+0000 7fce90bef740 4 client.25828 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:13:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.952+0000 7fce90bef740 1 -- 172.21.15.37:0/4001215978 --> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5627e064eb50 con 0x5627e07593d0 2024-03-28T18:13:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.953+0000 7fce86a00640 1 --2- 172.21.15.37:0/4001215978 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5627e07593d0 0x5627e075b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.953+0000 7fce86a00640 1 --2- 172.21.15.37:0/4001215978 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5627e07593d0 0x5627e075b7f0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cbec7cb9f6329d6e server_cookie=79dbc4fb2f556871 in_seq=0 out_seq=0 2024-03-28T18:13:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.953+0000 7fce70c00640 10 client.25828 ms_handle_connect on v2:172.21.15.170:6838/1575518279 2024-03-28T18:13:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.953+0000 7fce70c00640 1 -- 172.21.15.37:0/4001215978 <== mds.2 v2:172.21.15.170:6838/1575518279 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5627e064eb50 con 0x5627e07593d0 2024-03-28T18:13:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.953+0000 7fce70c00640 10 client.25828 handle_command_reply: tid=0 2024-03-28T18:13:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.953+0000 7fce90bef740 1 client.25828 shutdown 2024-03-28T18:13:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.954+0000 7fce90bef740 1 -- 172.21.15.37:0/4001215978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06ff840 msgr2=0x5627e06ffcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.954+0000 7fce90bef740 1 --2- 172.21.15.37:0/4001215978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06ff840 0x5627e06ffcc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fce8001ff40 tx=0x7fce80004a40 comp rx=0 tx=0).stop 2024-03-28T18:13:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.954+0000 7fce90bef740 1 -- 172.21.15.37:0/4001215978 shutdown_connections 2024-03-28T18:13:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.954+0000 7fce90bef740 1 --2- 172.21.15.37:0/4001215978 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5627e07593d0 0x5627e075b7f0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.954+0000 7fce90bef740 1 --2- 172.21.15.37:0/4001215978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5627e06ecc00 0x5627e0701970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.954+0000 7fce90bef740 1 --2- 172.21.15.37:0/4001215978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e06ff840 0x5627e06ffcc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.954+0000 7fce90bef740 1 --2- 172.21.15.37:0/4001215978 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5627e06e4080 0x5627e0701430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.954+0000 7fce90bef740 1 -- 172.21.15.37:0/4001215978 >> 172.21.15.37:0/4001215978 conn(0x5627e06ddda0 msgr2=0x5627e06e4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.954+0000 7fce70c00640 0 client.25828 ms_handle_reset on v2:172.21.15.170:6838/1575518279 2024-03-28T18:13:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.954+0000 7fce90bef740 1 -- 172.21.15.37:0/4001215978 shutdown_connections 2024-03-28T18:13:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.954+0000 7fce90bef740 1 -- 172.21.15.37:0/4001215978 wait complete. 2024-03-28T18:13:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.954+0000 7fce90bef740 20 client.25828 trim_cache size 0 max 16384 2024-03-28T18:13:10.956 INFO:teuthology.orchestra.run.smithi037.stdout:[] 2024-03-28T18:13:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.955+0000 7fce8de00640 1 -- 172.21.15.37:0/1672477107 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fce6c0a29f0 msgr2=0x7fce6c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.955+0000 7fce8de00640 1 --2- 172.21.15.37:0/1672477107 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fce6c0a29f0 0x7fce6c0a4eb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fce80005fd0 tx=0x7fce80009500 comp rx=0 tx=0).stop 2024-03-28T18:13:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.955+0000 7fce8de00640 1 -- 172.21.15.37:0/1672477107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce88101640 msgr2=0x7fce8819a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.955+0000 7fce8de00640 1 --2- 172.21.15.37:0/1672477107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce88101640 0x7fce8819a180 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fce78009900 tx=0x7fce78004360 comp rx=0 tx=0).stop 2024-03-28T18:13:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.955+0000 7fce8de00640 1 -- 172.21.15.37:0/1672477107 shutdown_connections 2024-03-28T18:13:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.955+0000 7fce8de00640 1 --2- 172.21.15.37:0/1672477107 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fce6c0a29f0 0x7fce6c0a4eb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.955+0000 7fce8de00640 1 --2- 172.21.15.37:0/1672477107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce88102000 0x7fce8819a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.955+0000 7fce8de00640 1 --2- 172.21.15.37:0/1672477107 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce88107650 0x7fce8819ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.955+0000 7fce8de00640 1 --2- 172.21.15.37:0/1672477107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce88101640 0x7fce8819a180 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:10.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.955+0000 7fce8de00640 1 -- 172.21.15.37:0/1672477107 >> 172.21.15.37:0/1672477107 conn(0x7fce880fd110 msgr2=0x7fce8810cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:10.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.955+0000 7fce8de00640 1 -- 172.21.15.37:0/1672477107 shutdown_connections 2024-03-28T18:13:10.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:10.955+0000 7fce8de00640 1 -- 172.21.15.37:0/1672477107 wait complete. 2024-03-28T18:13:10.975 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:13:10.975 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:13:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.199+0000 7f7d35600640 1 -- 172.21.15.37:0/3884682061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d30102ff0 msgr2=0x7f7d3010b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.199+0000 7f7d35600640 1 --2- 172.21.15.37:0/3884682061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d30102ff0 0x7f7d3010b4d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7d18009900 tx=0x7f7d1802bcd0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.199+0000 7f7d35600640 1 -- 172.21.15.37:0/3884682061 shutdown_connections 2024-03-28T18:13:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.199+0000 7f7d35600640 1 --2- 172.21.15.37:0/3884682061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d30102ff0 0x7f7d3010b4d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.199+0000 7f7d35600640 1 --2- 172.21.15.37:0/3884682061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d3010baf0 0x7f7d3010df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.199+0000 7f7d35600640 1 --2- 172.21.15.37:0/3884682061 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d301026d0 0x7f7d30102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.199+0000 7f7d35600640 1 -- 172.21.15.37:0/3884682061 >> 172.21.15.37:0/3884682061 conn(0x7f7d300fbef0 msgr2=0x7f7d300fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:11.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.199+0000 7f7d35600640 1 -- 172.21.15.37:0/3884682061 shutdown_connections 2024-03-28T18:13:11.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.199+0000 7f7d35600640 1 -- 172.21.15.37:0/3884682061 wait complete. 2024-03-28T18:13:11.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.200+0000 7f7d35600640 1 Processor -- start 2024-03-28T18:13:11.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.200+0000 7f7d35600640 1 -- start start 2024-03-28T18:13:11.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.201+0000 7f7d35600640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d301026d0 0x7f7d30197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.201+0000 7f7d35600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d30102ff0 0x7f7d30198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.201+0000 7f7d35600640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d3010baf0 0x7f7d3019c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.201+0000 7f7d35600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d30111900 con 0x7f7d30102ff0 2024-03-28T18:13:11.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.201+0000 7f7d35600640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d30111780 con 0x7f7d301026d0 2024-03-28T18:13:11.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.201+0000 7f7d35600640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d30111a80 con 0x7f7d3010baf0 2024-03-28T18:13:11.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.201+0000 7f7d2fe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d301026d0 0x7f7d30197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.201+0000 7f7d2fe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d301026d0 0x7f7d30197de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:35008/0 (socket says 172.21.15.37:35008) 2024-03-28T18:13:11.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.201+0000 7f7d2fe00640 1 -- 172.21.15.37:0/81461498 learned_addr learned my addr 172.21.15.37:0/81461498 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:11.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.201+0000 7f7d27400640 1 --2- 172.21.15.37:0/81461498 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d30102ff0 0x7f7d30198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.201+0000 7f7d34c00640 1 --2- 172.21.15.37:0/81461498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d3010baf0 0x7f7d3019c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.201+0000 7f7d2fe00640 1 -- 172.21.15.37:0/81461498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d3010baf0 msgr2=0x7f7d3019c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:11.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.201+0000 7f7d2fe00640 1 --2- 172.21.15.37:0/81461498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d3010baf0 0x7f7d3019c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.201+0000 7f7d2fe00640 1 -- 172.21.15.37:0/81461498 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d30102ff0 msgr2=0x7f7d30198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:11.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.201+0000 7f7d2fe00640 1 --2- 172.21.15.37:0/81461498 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d30102ff0 0x7f7d30198320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.201+0000 7f7d2fe00640 1 -- 172.21.15.37:0/81461498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3019cdd0 con 0x7f7d301026d0 2024-03-28T18:13:11.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.201+0000 7f7d27400640 1 --2- 172.21.15.37:0/81461498 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d30102ff0 0x7f7d30198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:13:11.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.202+0000 7f7d2fe00640 1 --2- 172.21.15.37:0/81461498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d301026d0 0x7f7d30197de0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000ebc0 tx=0x7f7d2000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:11.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.202+0000 7f7d2d600640 1 -- 172.21.15.37:0/81461498 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d20010630 con 0x7f7d301026d0 2024-03-28T18:13:11.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.202+0000 7f7d2d600640 1 -- 172.21.15.37:0/81461498 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d20010cb0 con 0x7f7d301026d0 2024-03-28T18:13:11.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.202+0000 7f7d35600640 1 -- 172.21.15.37:0/81461498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3019d0c0 con 0x7f7d301026d0 2024-03-28T18:13:11.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.202+0000 7f7d2d600640 1 -- 172.21.15.37:0/81461498 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d20018bd0 con 0x7f7d301026d0 2024-03-28T18:13:11.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.202+0000 7f7d35600640 1 -- 172.21.15.37:0/81461498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d301a4950 con 0x7f7d301026d0 2024-03-28T18:13:11.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.205+0000 7f7d2d600640 1 -- 172.21.15.37:0/81461498 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f7d200107d0 con 0x7f7d301026d0 2024-03-28T18:13:11.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.205+0000 7f7d39993740 1 Processor -- start 2024-03-28T18:13:11.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.206+0000 7f7d39993740 1 -- start start 2024-03-28T18:13:11.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.206+0000 7f7d39993740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6a8b0 0x55751aa6ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.206+0000 7f7d39993740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa6b260 0x55751aa73810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.206+0000 7f7d39993740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa73de0 0x55751aa76230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.206+0000 7f7d39993740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55751a9ce2c0 con 0x55751aa73de0 2024-03-28T18:13:11.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.206+0000 7f7d39993740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55751a976c90 con 0x55751aa6b260 2024-03-28T18:13:11.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.206+0000 7f7d39993740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55751a9ca360 con 0x55751aa6a8b0 2024-03-28T18:13:11.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.207+0000 7f7d2fe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa6b260 0x55751aa73810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.207+0000 7f7d2fe00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa6b260 0x55751aa73810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:35010/0 (socket says 172.21.15.37:35010) 2024-03-28T18:13:11.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.207+0000 7f7d2fe00640 1 -- 172.21.15.37:0/206851357 learned_addr learned my addr 172.21.15.37:0/206851357 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:11.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.207+0000 7f7d34c00640 1 --2- 172.21.15.37:0/206851357 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6a8b0 0x55751aa6ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.207+0000 7f7d27400640 1 --2- 172.21.15.37:0/206851357 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa73de0 0x55751aa76230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.207+0000 7f7d2fe00640 1 -- 172.21.15.37:0/206851357 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6a8b0 msgr2=0x55751aa6ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:11.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.207+0000 7f7d2fe00640 1 --2- 172.21.15.37:0/206851357 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6a8b0 0x55751aa6ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.207+0000 7f7d2fe00640 1 -- 172.21.15.37:0/206851357 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa73de0 msgr2=0x55751aa76230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:11.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.207+0000 7f7d2fe00640 1 --2- 172.21.15.37:0/206851357 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa73de0 0x55751aa76230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.207+0000 7f7d2fe00640 1 -- 172.21.15.37:0/206851357 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55751a99b080 con 0x55751aa6b260 2024-03-28T18:13:11.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.207+0000 7f7d27400640 1 --2- 172.21.15.37:0/206851357 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa73de0 0x55751aa76230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:13:11.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.208+0000 7f7d2d600640 1 --2- 172.21.15.37:0/81461498 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f7d140a29f0 0x7f7d140a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.208+0000 7f7d2fe00640 1 --2- 172.21.15.37:0/206851357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa6b260 0x55751aa73810 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7d200ccb40 tx=0x7f7d200d5a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eda35d974a334da server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:11.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.208+0000 7f7d2d600640 1 -- 172.21.15.37:0/81461498 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7d200cf4b0 con 0x7f7d301026d0 2024-03-28T18:13:11.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.208+0000 7f7d34c00640 1 --2- 172.21.15.37:0/81461498 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f7d140a29f0 0x7f7d140a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.208+0000 7f7d25200640 1 -- 172.21.15.37:0/206851357 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d20097d80 con 0x55751aa6b260 2024-03-28T18:13:11.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.208+0000 7f7d25200640 1 -- 172.21.15.37:0/206851357 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d2009c1e0 con 0x55751aa6b260 2024-03-28T18:13:11.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.208+0000 7f7d25200640 1 -- 172.21.15.37:0/206851357 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2009b440 con 0x55751aa6b260 2024-03-28T18:13:11.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.209+0000 7f7d39993740 1 -- 172.21.15.37:0/206851357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa6b260 msgr2=0x55751aa73810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:11.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.209+0000 7f7d34c00640 1 --2- 172.21.15.37:0/81461498 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f7d140a29f0 0x7f7d140a4eb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009c70 tx=0x7f7d28005f90 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:11.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.209+0000 7f7d39993740 1 --2- 172.21.15.37:0/206851357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa6b260 0x55751aa73810 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7d200ccb40 tx=0x7f7d200d5a90 comp rx=0 tx=0).stop 2024-03-28T18:13:11.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.209+0000 7f7d39993740 1 -- 172.21.15.37:0/206851357 shutdown_connections 2024-03-28T18:13:11.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.209+0000 7f7d39993740 1 --2- 172.21.15.37:0/206851357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa6b260 0x55751aa73810 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.209+0000 7f7d39993740 1 --2- 172.21.15.37:0/206851357 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa73de0 0x55751aa76230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.209+0000 7f7d39993740 1 --2- 172.21.15.37:0/206851357 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6a8b0 0x55751aa6ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.209+0000 7f7d39993740 1 -- 172.21.15.37:0/206851357 >> 172.21.15.37:0/206851357 conn(0x55751aa64f80 msgr2=0x55751aa65360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:11.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.209+0000 7f7d39993740 1 -- 172.21.15.37:0/206851357 shutdown_connections 2024-03-28T18:13:11.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.209+0000 7f7d39993740 1 -- 172.21.15.37:0/206851357 wait complete. 2024-03-28T18:13:11.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.210+0000 7f7d39993740 1 -- start start 2024-03-28T18:13:11.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.210+0000 7f7d39993740 1 Processor -- start 2024-03-28T18:13:11.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.210+0000 7f7d39993740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa6a8b0 0x55751aa6cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.210+0000 7f7d39993740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6b260 0x55751aa6d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.210+0000 7f7d39993740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa73de0 0x55751aa6d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.210+0000 7f7d39993740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55751aa76b30 con 0x55751aa6a8b0 2024-03-28T18:13:11.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.210+0000 7f7d39993740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55751a98a0d0 con 0x55751aa73de0 2024-03-28T18:13:11.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.210+0000 7f7d39993740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55751aa76cb0 con 0x55751aa6b260 2024-03-28T18:13:11.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.211+0000 7f7d39993740 10 client.0 _ll_register_callbacks cb 0x55751a98ee80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:13:11.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.211+0000 7f7d2fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa6a8b0 0x55751aa6cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.211+0000 7f7d34c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa73de0 0x55751aa6d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.211+0000 7f7d2fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa6a8b0 0x55751aa6cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60150/0 (socket says 172.21.15.37:60150) 2024-03-28T18:13:11.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.211+0000 7f7d2fe00640 1 -- 172.21.15.37:0/922549649 learned_addr learned my addr 172.21.15.37:0/922549649 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:11.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.211+0000 7f7d27400640 1 --2- 172.21.15.37:0/922549649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6b260 0x55751aa6d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.211+0000 7f7d2fe00640 1 -- 172.21.15.37:0/922549649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6b260 msgr2=0x55751aa6d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:11.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.211+0000 7f7d2fe00640 1 --2- 172.21.15.37:0/922549649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6b260 0x55751aa6d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.211+0000 7f7d2fe00640 1 -- 172.21.15.37:0/922549649 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa73de0 msgr2=0x55751aa6d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:11.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.211+0000 7f7d2fe00640 1 --2- 172.21.15.37:0/922549649 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa73de0 0x55751aa6d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.211+0000 7f7d2fe00640 1 -- 172.21.15.37:0/922549649 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55751a6e23f0 con 0x55751aa6a8b0 2024-03-28T18:13:11.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.211+0000 7f7d27400640 1 --2- 172.21.15.37:0/922549649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6b260 0x55751aa6d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:13:11.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.211+0000 7f7d34c00640 1 --2- 172.21.15.37:0/922549649 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa73de0 0x55751aa6d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:13:11.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.212+0000 7f7d2fe00640 1 --2- 172.21.15.37:0/922549649 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa6a8b0 0x55751aa6cca0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7d20097f40 tx=0x7f7d20000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8475c0468b730f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:11.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.212+0000 7f7d1d600640 10 client.0 ms_handle_connect on v2:172.21.15.37:3300/0 2024-03-28T18:13:11.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.212+0000 7f7d1d600640 1 -- 172.21.15.37:0/922549649 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d20010e60 con 0x55751aa6a8b0 2024-03-28T18:13:11.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.212+0000 7f7d1d600640 1 -- 172.21.15.37:0/922549649 <== mon.0 v2:172.21.15.37:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7d200d5160 con 0x55751aa6a8b0 2024-03-28T18:13:11.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.212+0000 7f7d39993740 1 -- 172.21.15.37:0/922549649 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7d2009c1e0 con 0x55751aa6a8b0 2024-03-28T18:13:11.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.212+0000 7f7d1d600640 1 -- 172.21.15.37:0/922549649 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d2009b620 con 0x55751aa6a8b0 2024-03-28T18:13:11.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.212+0000 7f7d39993740 10 client.15642 fetch_fsmap learned FSMap version 24 2024-03-28T18:13:11.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.212+0000 7f7d39993740 1 -- 172.21.15.37:0/922549649 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55751a7ac7d0 con 0x55751aa6a8b0 2024-03-28T18:13:11.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.213+0000 7f7d1d600640 1 -- 172.21.15.37:0/922549649 <== mon.0 v2:172.21.15.37:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d2009c1e0 con 0x55751aa6a8b0 2024-03-28T18:13:11.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.213+0000 7f7d1d600640 1 -- 172.21.15.37:0/922549649 <== mon.0 v2:172.21.15.37:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f7d2009bd40 con 0x55751aa6a8b0 2024-03-28T18:13:11.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.213+0000 7f7d39993740 10 client.15642 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T18:13:11.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.213+0000 7f7d39993740 10 client.15642 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-28T18:13:11.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.213+0000 7f7d39993740 20 client.15642 populate_metadata read hostname 'smithi037' 2024-03-28T18:13:11.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.213+0000 7f7d39993740 1 --2- 172.21.15.37:0/922549649 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55751aa8c2b0 0x55751aa8e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.213+0000 7f7d39993740 4 client.15642 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:13:11.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.213+0000 7f7d39993740 1 -- 172.21.15.37:0/922549649 --> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55751a9db800 con 0x55751aa8c2b0 2024-03-28T18:13:11.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.213+0000 7f7d27400640 1 --2- 172.21.15.37:0/922549649 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55751aa8c2b0 0x55751aa8e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.214+0000 7f7d27400640 1 --2- 172.21.15.37:0/922549649 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55751aa8c2b0 0x55751aa8e6a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c0edeec205f7a077 server_cookie=1d90ceb1a5a1356f in_seq=0 out_seq=0 2024-03-28T18:13:11.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.214+0000 7f7d1d600640 10 client.15642 ms_handle_connect on v2:172.21.15.59:6838/2091629078 2024-03-28T18:13:11.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.214+0000 7f7d1d600640 1 -- 172.21.15.37:0/922549649 <== mds.1 v2:172.21.15.59:6838/2091629078 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55751a9db800 con 0x55751aa8c2b0 2024-03-28T18:13:11.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.214+0000 7f7d1d600640 10 client.15642 handle_command_reply: tid=0 2024-03-28T18:13:11.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.215+0000 7f7d39993740 1 client.15642 shutdown 2024-03-28T18:13:11.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.215+0000 7f7d39993740 1 -- 172.21.15.37:0/922549649 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa6a8b0 msgr2=0x55751aa6cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:11.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.215+0000 7f7d39993740 1 --2- 172.21.15.37:0/922549649 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa6a8b0 0x55751aa6cca0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7d20097f40 tx=0x7f7d20000f40 comp rx=0 tx=0).stop 2024-03-28T18:13:11.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.215+0000 7f7d39993740 1 -- 172.21.15.37:0/922549649 shutdown_connections 2024-03-28T18:13:11.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.215+0000 7f7d39993740 1 --2- 172.21.15.37:0/922549649 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55751aa8c2b0 0x55751aa8e6a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.215+0000 7f7d39993740 1 --2- 172.21.15.37:0/922549649 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa73de0 0x55751aa6d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.215+0000 7f7d39993740 1 --2- 172.21.15.37:0/922549649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6b260 0x55751aa6d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.215+0000 7f7d39993740 1 --2- 172.21.15.37:0/922549649 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa6a8b0 0x55751aa6cca0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.215+0000 7f7d1d600640 0 client.15642 ms_handle_reset on v2:172.21.15.59:6838/2091629078 2024-03-28T18:13:11.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.215+0000 7f7d39993740 1 -- 172.21.15.37:0/922549649 >> 172.21.15.37:0/922549649 conn(0x55751aa64f80 msgr2=0x55751aa66cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:11.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.215+0000 7f7d39993740 1 -- 172.21.15.37:0/922549649 shutdown_connections 2024-03-28T18:13:11.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.215+0000 7f7d39993740 1 -- 172.21.15.37:0/922549649 wait complete. 2024-03-28T18:13:11.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.215+0000 7f7d39993740 20 client.15642 trim_cache size 0 max 16384 2024-03-28T18:13:11.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.236+0000 7f7d39993740 1 Processor -- start 2024-03-28T18:13:11.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.236+0000 7f7d39993740 1 -- start start 2024-03-28T18:13:11.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.237+0000 7f7d39993740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa6a8b0 0x55751aa936d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.237+0000 7f7d39993740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6b260 0x55751aa794d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.237+0000 7f7d39993740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa73de0 0x55751aa79a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.237+0000 7f7d39993740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d301ad370 con 0x55751aa6a8b0 2024-03-28T18:13:11.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.237+0000 7f7d39993740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d301aae70 con 0x55751aa73de0 2024-03-28T18:13:11.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.237+0000 7f7d39993740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d301a8f10 con 0x55751aa6b260 2024-03-28T18:13:11.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.237+0000 7f7d27400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6b260 0x55751aa794d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.237+0000 7f7d2fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa6a8b0 0x55751aa936d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.237+0000 7f7d34c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa73de0 0x55751aa79a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.237+0000 7f7d27400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6b260 0x55751aa794d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56694/0 (socket says 172.21.15.37:56694) 2024-03-28T18:13:11.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.237+0000 7f7d27400640 1 -- 172.21.15.37:0/2089741935 learned_addr learned my addr 172.21.15.37:0/2089741935 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:11.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.237+0000 7f7d27400640 1 -- 172.21.15.37:0/2089741935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa73de0 msgr2=0x55751aa79a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:11.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.237+0000 7f7d27400640 1 --2- 172.21.15.37:0/2089741935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa73de0 0x55751aa79a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.237+0000 7f7d27400640 1 -- 172.21.15.37:0/2089741935 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa6a8b0 msgr2=0x55751aa936d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:11.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.237+0000 7f7d27400640 1 --2- 172.21.15.37:0/2089741935 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa6a8b0 0x55751aa936d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.237+0000 7f7d27400640 1 -- 172.21.15.37:0/2089741935 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55751aa65d10 con 0x55751aa6b260 2024-03-28T18:13:11.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.238+0000 7f7d27400640 1 --2- 172.21.15.37:0/2089741935 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6b260 0x55751aa794d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7d18000c00 tx=0x7f7d18009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e20629cc7d338f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:11.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.238+0000 7f7d1fe00640 1 -- 172.21.15.37:0/2089741935 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d18048020 con 0x55751aa6b260 2024-03-28T18:13:11.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.238+0000 7f7d1fe00640 1 -- 172.21.15.37:0/2089741935 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d1802ce00 con 0x55751aa6b260 2024-03-28T18:13:11.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.238+0000 7f7d1fe00640 1 -- 172.21.15.37:0/2089741935 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d180052a0 con 0x55751aa6b260 2024-03-28T18:13:11.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.239+0000 7f7d39993740 1 -- 172.21.15.37:0/2089741935 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6b260 msgr2=0x55751aa794d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:11.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.239+0000 7f7d39993740 1 --2- 172.21.15.37:0/2089741935 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6b260 0x55751aa794d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7d18000c00 tx=0x7f7d18009560 comp rx=0 tx=0).stop 2024-03-28T18:13:11.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.239+0000 7f7d39993740 1 -- 172.21.15.37:0/2089741935 shutdown_connections 2024-03-28T18:13:11.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.239+0000 7f7d39993740 1 --2- 172.21.15.37:0/2089741935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa73de0 0x55751aa79a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.239+0000 7f7d39993740 1 --2- 172.21.15.37:0/2089741935 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6b260 0x55751aa794d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.239+0000 7f7d39993740 1 --2- 172.21.15.37:0/2089741935 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa6a8b0 0x55751aa936d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.239+0000 7f7d39993740 1 -- 172.21.15.37:0/2089741935 >> 172.21.15.37:0/2089741935 conn(0x55751aa64f80 msgr2=0x55751aa777b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:11.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.239+0000 7f7d39993740 1 -- 172.21.15.37:0/2089741935 shutdown_connections 2024-03-28T18:13:11.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.239+0000 7f7d39993740 1 -- 172.21.15.37:0/2089741935 wait complete. 2024-03-28T18:13:11.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.239+0000 7f7d39993740 1 -- start start 2024-03-28T18:13:11.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.240+0000 7f7d39993740 1 Processor -- start 2024-03-28T18:13:11.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.240+0000 7f7d39993740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa6a8b0 0x55751aa86330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.241+0000 7f7d39993740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6b260 0x55751aa86870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.241+0000 7f7d39993740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa73de0 0x55751aae67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.241+0000 7f7d39993740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d301a9510 con 0x55751aa6a8b0 2024-03-28T18:13:11.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.241+0000 7f7d39993740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d301aa810 con 0x55751aa73de0 2024-03-28T18:13:11.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.241+0000 7f7d39993740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d301a9b50 con 0x55751aa6b260 2024-03-28T18:13:11.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.241+0000 7f7d39993740 10 client.0 _ll_register_callbacks cb 0x7f7d2006e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:13:11.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.241+0000 7f7d27400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6b260 0x55751aa86870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.241+0000 7f7d34c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa73de0 0x55751aae67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.241+0000 7f7d27400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6b260 0x55751aa86870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56706/0 (socket says 172.21.15.37:56706) 2024-03-28T18:13:11.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.241+0000 7f7d27400640 1 -- 172.21.15.37:0/2481566779 learned_addr learned my addr 172.21.15.37:0/2481566779 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:11.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.241+0000 7f7d34c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa73de0 0x55751aae67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:35036/0 (socket says 172.21.15.37:35036) 2024-03-28T18:13:11.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.241+0000 7f7d2fe00640 1 --2- 172.21.15.37:0/2481566779 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa6a8b0 0x55751aa86330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.241+0000 7f7d34c00640 1 -- 172.21.15.37:0/2481566779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6b260 msgr2=0x55751aa86870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:11.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.241+0000 7f7d34c00640 1 --2- 172.21.15.37:0/2481566779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6b260 0x55751aa86870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.241+0000 7f7d34c00640 1 -- 172.21.15.37:0/2481566779 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa6a8b0 msgr2=0x55751aa86330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:11.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.241+0000 7f7d34c00640 1 --2- 172.21.15.37:0/2481566779 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa6a8b0 0x55751aa86330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.241+0000 7f7d34c00640 1 -- 172.21.15.37:0/2481566779 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55751aae6dd0 con 0x55751aa73de0 2024-03-28T18:13:11.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.242+0000 7f7d27400640 1 --2- 172.21.15.37:0/2481566779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6b260 0x55751aa86870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:13:11.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.242+0000 7f7d34c00640 1 --2- 172.21.15.37:0/2481566779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa73de0 0x55751aae67e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7d28012cb0 tx=0x7f7d280046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21085490cfb25df server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:11.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.242+0000 7f7d1d600640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:13:11.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.242+0000 7f7d1d600640 1 -- 172.21.15.37:0/2481566779 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2802f060 con 0x55751aa73de0 2024-03-28T18:13:11.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.242+0000 7f7d1d600640 1 -- 172.21.15.37:0/2481566779 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7d2801f8c0 con 0x55751aa73de0 2024-03-28T18:13:11.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.242+0000 7f7d39993740 1 -- 172.21.15.37:0/2481566779 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7d1802ce00 con 0x55751aa73de0 2024-03-28T18:13:11.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.243+0000 7f7d1d600640 1 -- 172.21.15.37:0/2481566779 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d2801fac0 con 0x55751aa73de0 2024-03-28T18:13:11.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.243+0000 7f7d39993740 10 client.25835 fetch_fsmap learned FSMap version 24 2024-03-28T18:13:11.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.243+0000 7f7d39993740 1 -- 172.21.15.37:0/2481566779 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55751aae8110 con 0x55751aa73de0 2024-03-28T18:13:11.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.243+0000 7f7d1d600640 1 -- 172.21.15.37:0/2481566779 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d1802ce00 con 0x55751aa73de0 2024-03-28T18:13:11.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.243+0000 7f7d1d600640 1 -- 172.21.15.37:0/2481566779 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f7d2802a400 con 0x55751aa73de0 2024-03-28T18:13:11.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.243+0000 7f7d39993740 10 client.25835 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T18:13:11.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.243+0000 7f7d39993740 10 client.25835 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-28T18:13:11.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.243+0000 7f7d39993740 20 client.25835 populate_metadata read hostname 'smithi037' 2024-03-28T18:13:11.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.243+0000 7f7d39993740 1 --2- 172.21.15.37:0/2481566779 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55751aae8790 0x55751aaeabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.243+0000 7f7d39993740 4 client.25835 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:13:11.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.243+0000 7f7d39993740 1 -- 172.21.15.37:0/2481566779 --> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55751a9d5b50 con 0x55751aae8790 2024-03-28T18:13:11.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.244+0000 7f7d2fe00640 1 --2- 172.21.15.37:0/2481566779 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55751aae8790 0x55751aaeabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.244+0000 7f7d2fe00640 1 --2- 172.21.15.37:0/2481566779 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55751aae8790 0x55751aaeabb0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9230c9f73976698b server_cookie=628fab45822dbfef in_seq=0 out_seq=0 2024-03-28T18:13:11.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.244+0000 7f7d1d600640 10 client.25835 ms_handle_connect on v2:172.21.15.59:6838/2091629078 2024-03-28T18:13:11.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.244+0000 7f7d1d600640 1 -- 172.21.15.37:0/2481566779 <== mds.1 v2:172.21.15.59:6838/2091629078 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55751a9d5b50 con 0x55751aae8790 2024-03-28T18:13:11.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.244+0000 7f7d1d600640 10 client.25835 handle_command_reply: tid=0 2024-03-28T18:13:11.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.244+0000 7f7d39993740 1 client.25835 shutdown 2024-03-28T18:13:11.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.245+0000 7f7d39993740 1 -- 172.21.15.37:0/2481566779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa73de0 msgr2=0x55751aae67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:11.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.245+0000 7f7d39993740 1 --2- 172.21.15.37:0/2481566779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa73de0 0x55751aae67e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7d28012cb0 tx=0x7f7d280046c0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.245+0000 7f7d39993740 1 -- 172.21.15.37:0/2481566779 shutdown_connections 2024-03-28T18:13:11.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.245+0000 7f7d39993740 1 --2- 172.21.15.37:0/2481566779 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x55751aae8790 0x55751aaeabb0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.245+0000 7f7d39993740 1 --2- 172.21.15.37:0/2481566779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55751aa73de0 0x55751aae67e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.245+0000 7f7d39993740 1 --2- 172.21.15.37:0/2481566779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55751aa6b260 0x55751aa86870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.245+0000 7f7d39993740 1 --2- 172.21.15.37:0/2481566779 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55751aa6a8b0 0x55751aa86330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.245+0000 7f7d39993740 1 -- 172.21.15.37:0/2481566779 >> 172.21.15.37:0/2481566779 conn(0x55751aa64f80 msgr2=0x55751aa702b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:11.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.245+0000 7f7d1d600640 0 client.25835 ms_handle_reset on v2:172.21.15.59:6838/2091629078 2024-03-28T18:13:11.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.245+0000 7f7d39993740 1 -- 172.21.15.37:0/2481566779 shutdown_connections 2024-03-28T18:13:11.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.245+0000 7f7d39993740 1 -- 172.21.15.37:0/2481566779 wait complete. 2024-03-28T18:13:11.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.245+0000 7f7d39993740 20 client.25835 trim_cache size 0 max 16384 2024-03-28T18:13:11.248 INFO:teuthology.orchestra.run.smithi037.stdout:[] 2024-03-28T18:13:11.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.246+0000 7f7d35600640 1 -- 172.21.15.37:0/81461498 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f7d140a29f0 msgr2=0x7f7d140a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:11.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.246+0000 7f7d35600640 1 --2- 172.21.15.37:0/81461498 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f7d140a29f0 0x7f7d140a4eb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009c70 tx=0x7f7d28005f90 comp rx=0 tx=0).stop 2024-03-28T18:13:11.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.246+0000 7f7d35600640 1 -- 172.21.15.37:0/81461498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d301026d0 msgr2=0x7f7d30197de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:11.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.246+0000 7f7d35600640 1 --2- 172.21.15.37:0/81461498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d301026d0 0x7f7d30197de0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000ebc0 tx=0x7f7d2000c6a0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.246+0000 7f7d35600640 1 -- 172.21.15.37:0/81461498 shutdown_connections 2024-03-28T18:13:11.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.246+0000 7f7d35600640 1 --2- 172.21.15.37:0/81461498 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f7d140a29f0 0x7f7d140a4eb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.246+0000 7f7d35600640 1 --2- 172.21.15.37:0/81461498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d3010baf0 0x7f7d3019c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.246+0000 7f7d35600640 1 --2- 172.21.15.37:0/81461498 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7d30102ff0 0x7f7d30198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.246+0000 7f7d35600640 1 --2- 172.21.15.37:0/81461498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d301026d0 0x7f7d30197de0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.246+0000 7f7d35600640 1 -- 172.21.15.37:0/81461498 >> 172.21.15.37:0/81461498 conn(0x7f7d300fbef0 msgr2=0x7f7d300fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:11.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.247+0000 7f7d35600640 1 -- 172.21.15.37:0/81461498 shutdown_connections 2024-03-28T18:13:11.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.247+0000 7f7d35600640 1 -- 172.21.15.37:0/81461498 wait complete. 2024-03-28T18:13:11.268 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:13:11.268 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:13:11.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.491+0000 7fa10c400640 1 -- 172.21.15.37:0/1683610160 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa104109850 msgr2=0x7fa104109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:11.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.491+0000 7fa10c400640 1 --2- 172.21.15.37:0/1683610160 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa104109850 0x7fa104109c30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8009930 tx=0x7fa0f802bc30 comp rx=0 tx=0).stop 2024-03-28T18:13:11.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.491+0000 7fa10c400640 1 -- 172.21.15.37:0/1683610160 shutdown_connections 2024-03-28T18:13:11.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.491+0000 7fa10c400640 1 --2- 172.21.15.37:0/1683610160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa104103840 0x7fa104103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.491+0000 7fa10c400640 1 --2- 172.21.15.37:0/1683610160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa104104200 0x7fa104110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.491+0000 7fa10c400640 1 --2- 172.21.15.37:0/1683610160 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa104109850 0x7fa104109c30 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.491+0000 7fa10c400640 1 -- 172.21.15.37:0/1683610160 >> 172.21.15.37:0/1683610160 conn(0x7fa1040ff350 msgr2=0x7fa104101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:11.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.491+0000 7fa10c400640 1 -- 172.21.15.37:0/1683610160 shutdown_connections 2024-03-28T18:13:11.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.491+0000 7fa10c400640 1 -- 172.21.15.37:0/1683610160 wait complete. 2024-03-28T18:13:11.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.492+0000 7fa10c400640 1 Processor -- start 2024-03-28T18:13:11.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.492+0000 7fa10c400640 1 -- start start 2024-03-28T18:13:11.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.493+0000 7fa10c400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa104103840 0x7fa10419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.493+0000 7fa10c400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa104104200 0x7fa10419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.493+0000 7fa10c400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa104109850 0x7fa10419ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.493+0000 7fa10c400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa104113c70 con 0x7fa104109850 2024-03-28T18:13:11.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.493+0000 7fa10c400640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa104113af0 con 0x7fa104103840 2024-03-28T18:13:11.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.493+0000 7fa10c400640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa104113df0 con 0x7fa104104200 2024-03-28T18:13:11.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.493+0000 7fa109a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa104103840 0x7fa10419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.493+0000 7fa109a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa104103840 0x7fa10419a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:35056/0 (socket says 172.21.15.37:35056) 2024-03-28T18:13:11.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.493+0000 7fa109a00640 1 -- 172.21.15.37:0/184625407 learned_addr learned my addr 172.21.15.37:0/184625407 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:11.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.493+0000 7fa109000640 1 --2- 172.21.15.37:0/184625407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa104104200 0x7fa10419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.493+0000 7fa109a00640 1 -- 172.21.15.37:0/184625407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa104104200 msgr2=0x7fa10419a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:11.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.493+0000 7fa109a00640 1 --2- 172.21.15.37:0/184625407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa104104200 0x7fa10419a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.493+0000 7fa109a00640 1 -- 172.21.15.37:0/184625407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa104109850 msgr2=0x7fa10419ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:11.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.494+0000 7fa10a400640 1 --2- 172.21.15.37:0/184625407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa104109850 0x7fa10419ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.494+0000 7fa109a00640 1 --2- 172.21.15.37:0/184625407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa104109850 0x7fa10419ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.494+0000 7fa109a00640 1 -- 172.21.15.37:0/184625407 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa10419f190 con 0x7fa104103840 2024-03-28T18:13:11.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.494+0000 7fa10a400640 1 --2- 172.21.15.37:0/184625407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa104109850 0x7fa10419ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:13:11.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.494+0000 7fa109a00640 1 --2- 172.21.15.37:0/184625407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa104103840 0x7fa10419a1f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8009900 tx=0x7fa0f8004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:11.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.494+0000 7fa102000640 1 -- 172.21.15.37:0/184625407 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8043020 con 0x7fa104103840 2024-03-28T18:13:11.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.494+0000 7fa10c400640 1 -- 172.21.15.37:0/184625407 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa10419f420 con 0x7fa104103840 2024-03-28T18:13:11.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.494+0000 7fa102000640 1 -- 172.21.15.37:0/184625407 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0f80054e0 con 0x7fa104103840 2024-03-28T18:13:11.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.495+0000 7fa102000640 1 -- 172.21.15.37:0/184625407 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8003bb0 con 0x7fa104103840 2024-03-28T18:13:11.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.495+0000 7fa10c400640 1 -- 172.21.15.37:0/184625407 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa104075ae0 con 0x7fa104103840 2024-03-28T18:13:11.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.499+0000 7fa102000640 1 -- 172.21.15.37:0/184625407 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fa0f8046020 con 0x7fa104103840 2024-03-28T18:13:11.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.499+0000 7fa10f1bc740 1 Processor -- start 2024-03-28T18:13:11.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.499+0000 7fa10f1bc740 1 -- start start 2024-03-28T18:13:11.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.499+0000 7fa10f1bc740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fddab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.499+0000 7fa10f1bc740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f7d2fde080 0x55f7d2fe6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.499+0000 7fa10f1bc740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fe6c00 0x55f7d2fe9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.499+0000 7fa10f1bc740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7d2f412c0 con 0x55f7d2fde080 2024-03-28T18:13:11.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.499+0000 7fa10f1bc740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7d2ee9c90 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.499+0000 7fa10f1bc740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7d2f3d360 con 0x55f7d2fe6c00 2024-03-28T18:13:11.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.500+0000 7fa10a400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fddab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.500+0000 7fa10a400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fddab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:35062/0 (socket says 172.21.15.37:35062) 2024-03-28T18:13:11.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.500+0000 7fa10a400640 1 -- 172.21.15.37:0/1845743281 learned_addr learned my addr 172.21.15.37:0/1845743281 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:11.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.500+0000 7fa109a00640 1 --2- 172.21.15.37:0/1845743281 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f7d2fde080 0x55f7d2fe6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.500+0000 7fa109000640 1 --2- 172.21.15.37:0/1845743281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fe6c00 0x55f7d2fe9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.500+0000 7fa109000640 1 -- 172.21.15.37:0/1845743281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fdd6d0 msgr2=0x55f7d2fddab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:11.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.500+0000 7fa109000640 1 --2- 172.21.15.37:0/1845743281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fddab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.500+0000 7fa109000640 1 -- 172.21.15.37:0/1845743281 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f7d2fde080 msgr2=0x55f7d2fe6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:11.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.500+0000 7fa109000640 1 --2- 172.21.15.37:0/1845743281 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f7d2fde080 0x55f7d2fe6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.500+0000 7fa109000640 1 -- 172.21.15.37:0/1845743281 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f7d2f0e080 con 0x55f7d2fe6c00 2024-03-28T18:13:11.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.500+0000 7fa10a400640 1 --2- 172.21.15.37:0/1845743281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fddab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:13:11.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.501+0000 7fa109000640 1 --2- 172.21.15.37:0/1845743281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fe6c00 0x55f7d2fe9050 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa0f00041c0 tx=0x7fa0f0005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e6764d2527b8f3f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:11.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.501+0000 7fa102000640 1 --2- 172.21.15.37:0/184625407 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fa0ec0a29f0 0x7fa0ec0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.501+0000 7fa0f6a00640 1 -- 172.21.15.37:0/1845743281 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0024070 con 0x55f7d2fe6c00 2024-03-28T18:13:11.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.501+0000 7fa102000640 1 -- 172.21.15.37:0/184625407 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa0f80f0d10 con 0x7fa104103840 2024-03-28T18:13:11.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.501+0000 7fa0f6a00640 1 -- 172.21.15.37:0/1845743281 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0f0015d50 con 0x55f7d2fe6c00 2024-03-28T18:13:11.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.501+0000 7fa0f6a00640 1 -- 172.21.15.37:0/1845743281 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f001f8b0 con 0x55f7d2fe6c00 2024-03-28T18:13:11.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.501+0000 7fa10a400640 1 --2- 172.21.15.37:0/184625407 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fa0ec0a29f0 0x7fa0ec0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.502+0000 7fa10f1bc740 1 -- 172.21.15.37:0/1845743281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fe6c00 msgr2=0x55f7d2fe9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:11.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.502+0000 7fa10f1bc740 1 --2- 172.21.15.37:0/1845743281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fe6c00 0x55f7d2fe9050 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa0f00041c0 tx=0x7fa0f0005f90 comp rx=0 tx=0).stop 2024-03-28T18:13:11.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.502+0000 7fa10a400640 1 --2- 172.21.15.37:0/184625407 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fa0ec0a29f0 0x7fa0ec0a4eb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc009fd0 tx=0x7fa0fc009500 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.502+0000 7fa10f1bc740 1 -- 172.21.15.37:0/1845743281 shutdown_connections 2024-03-28T18:13:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.502+0000 7fa10f1bc740 1 --2- 172.21.15.37:0/1845743281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fe6c00 0x55f7d2fe9050 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.502+0000 7fa10f1bc740 1 --2- 172.21.15.37:0/1845743281 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f7d2fde080 0x55f7d2fe6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.502+0000 7fa10f1bc740 1 --2- 172.21.15.37:0/1845743281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fddab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.502+0000 7fa10f1bc740 1 -- 172.21.15.37:0/1845743281 >> 172.21.15.37:0/1845743281 conn(0x55f7d2fd7da0 msgr2=0x55f7d2fd8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.502+0000 7fa10f1bc740 1 -- 172.21.15.37:0/1845743281 shutdown_connections 2024-03-28T18:13:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.502+0000 7fa10f1bc740 1 -- 172.21.15.37:0/1845743281 wait complete. 2024-03-28T18:13:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.503+0000 7fa10f1bc740 1 -- start start 2024-03-28T18:13:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.503+0000 7fa10f1bc740 1 Processor -- start 2024-03-28T18:13:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.504+0000 7fa10f1bc740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fe64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.504+0000 7fa10f1bc740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fde080 0x55f7d2fe38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.504+0000 7fa10f1bc740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f7d2fe6c00 0x55f7d2fe3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.504+0000 7fa10f1bc740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7d2fe9b70 con 0x55f7d2fe6c00 2024-03-28T18:13:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.504+0000 7fa10f1bc740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7d2efd0d0 con 0x55f7d2fde080 2024-03-28T18:13:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.504+0000 7fa10f1bc740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7d2fe9cf0 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.504+0000 7fa10f1bc740 10 client.0 _ll_register_callbacks cb 0x55f7d2f01e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:13:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.504+0000 7fa109000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fde080 0x55f7d2fe38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.504+0000 7fa109a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fe64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.504+0000 7fa109000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fde080 0x55f7d2fe38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:35078/0 (socket says 172.21.15.37:35078) 2024-03-28T18:13:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.504+0000 7fa109000640 1 -- 172.21.15.37:0/2663063328 learned_addr learned my addr 172.21.15.37:0/2663063328 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.504+0000 7fa109a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fe64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56728/0 (socket says 172.21.15.37:56728) 2024-03-28T18:13:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.504+0000 7fa10a400640 1 --2- 172.21.15.37:0/2663063328 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f7d2fe6c00 0x55f7d2fe3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.504+0000 7fa109a00640 1 -- 172.21.15.37:0/2663063328 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fde080 msgr2=0x55f7d2fe38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.504+0000 7fa109a00640 1 --2- 172.21.15.37:0/2663063328 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fde080 0x55f7d2fe38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.504+0000 7fa109a00640 1 -- 172.21.15.37:0/2663063328 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f7d2fe6c00 msgr2=0x55f7d2fe3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.504+0000 7fa109a00640 1 --2- 172.21.15.37:0/2663063328 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f7d2fe6c00 0x55f7d2fe3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.504+0000 7fa109a00640 1 -- 172.21.15.37:0/2663063328 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f7d2c553f0 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.505+0000 7fa109a00640 1 --2- 172.21.15.37:0/2663063328 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fe64a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80ee5a0 tx=0x7fa0f8005cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40973d33100a1298 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.505+0000 7fa0eb400640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:13:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.505+0000 7fa0eb400640 1 -- 172.21.15.37:0/2663063328 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f80b9750 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.505+0000 7fa0eb400640 1 -- 172.21.15.37:0/2663063328 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa0f80b9b20 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.505+0000 7fa10f1bc740 1 -- 172.21.15.37:0/2663063328 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa0f0015d50 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.505+0000 7fa0eb400640 1 -- 172.21.15.37:0/2663063328 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0f80bce10 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.505+0000 7fa10f1bc740 10 client.25858 fetch_fsmap learned FSMap version 24 2024-03-28T18:13:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.505+0000 7fa10f1bc740 1 -- 172.21.15.37:0/2663063328 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55f7d2d1f7d0 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.505+0000 7fa0eb400640 1 -- 172.21.15.37:0/2663063328 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0f0015d50 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.506+0000 7fa0eb400640 1 -- 172.21.15.37:0/2663063328 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa0f80eebe0 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.506+0000 7fa10f1bc740 10 client.25858 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T18:13:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.506+0000 7fa10f1bc740 10 client.25858 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:13:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.506+0000 7fa10f1bc740 20 client.25858 populate_metadata read hostname 'smithi037' 2024-03-28T18:13:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.506+0000 7fa10f1bc740 1 --2- 172.21.15.37:0/2663063328 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55f7d2ff6a00 0x55f7d2ff8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.506+0000 7fa10f1bc740 4 client.25858 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:13:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.506+0000 7fa10f1bc740 1 -- 172.21.15.37:0/2663063328 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f7d2f4e800 con 0x55f7d2ff6a00 2024-03-28T18:13:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.506+0000 7fa109000640 1 --2- 172.21.15.37:0/2663063328 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55f7d2ff6a00 0x55f7d2ff8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.507+0000 7fa109000640 1 --2- 172.21.15.37:0/2663063328 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55f7d2ff6a00 0x55f7d2ff8e80 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15c0fa7309f19635 server_cookie=353f9faf731bb126 in_seq=0 out_seq=0 2024-03-28T18:13:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.507+0000 7fa0eb400640 10 client.25858 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:13:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.508+0000 7fa0eb400640 1 -- 172.21.15.37:0/2663063328 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f7d2f4e800 con 0x55f7d2ff6a00 2024-03-28T18:13:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.508+0000 7fa0eb400640 10 client.25858 handle_command_reply: tid=0 2024-03-28T18:13:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.508+0000 7fa10f1bc740 1 client.25858 shutdown 2024-03-28T18:13:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.509+0000 7fa10f1bc740 1 -- 172.21.15.37:0/2663063328 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fdd6d0 msgr2=0x55f7d2fe64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.509+0000 7fa10f1bc740 1 --2- 172.21.15.37:0/2663063328 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fe64a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80ee5a0 tx=0x7fa0f8005cb0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.509+0000 7fa10f1bc740 1 -- 172.21.15.37:0/2663063328 shutdown_connections 2024-03-28T18:13:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.509+0000 7fa10f1bc740 1 --2- 172.21.15.37:0/2663063328 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55f7d2ff6a00 0x55f7d2ff8e80 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.509+0000 7fa10f1bc740 1 --2- 172.21.15.37:0/2663063328 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fde080 0x55f7d2fe38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.509+0000 7fa10f1bc740 1 --2- 172.21.15.37:0/2663063328 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f7d2fe6c00 0x55f7d2fe3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.509+0000 7fa10f1bc740 1 --2- 172.21.15.37:0/2663063328 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fe64a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.509+0000 7fa10f1bc740 1 -- 172.21.15.37:0/2663063328 >> 172.21.15.37:0/2663063328 conn(0x55f7d2fd7da0 msgr2=0x55f7d2fd9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:11.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.509+0000 7fa0eb400640 0 client.25858 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:13:11.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.509+0000 7fa10f1bc740 1 -- 172.21.15.37:0/2663063328 shutdown_connections 2024-03-28T18:13:11.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.509+0000 7fa10f1bc740 1 -- 172.21.15.37:0/2663063328 wait complete. 2024-03-28T18:13:11.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.509+0000 7fa10f1bc740 20 client.25858 trim_cache size 0 max 16384 2024-03-28T18:13:11.515 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:11 smithi037 ceph-mon[31460]: pgmap v63: 97 pgs: 97 active+clean; 16 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 101 op/s 2024-03-28T18:13:11.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.531+0000 7fa10f1bc740 1 Processor -- start 2024-03-28T18:13:11.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.532+0000 7fa10f1bc740 1 -- start start 2024-03-28T18:13:11.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.532+0000 7fa10f1bc740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fe5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.532+0000 7fa10f1bc740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f7d2fde080 0x55f7d2fe64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.532+0000 7fa10f1bc740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fe6c00 0x55f7d2fe0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.532+0000 7fa10f1bc740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1041ad5a0 con 0x55f7d2fde080 2024-03-28T18:13:11.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.532+0000 7fa10f1bc740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1041ab0a0 con 0x55f7d2fe6c00 2024-03-28T18:13:11.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.532+0000 7fa10f1bc740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1041a9140 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.532+0000 7fa109a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fe5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.532+0000 7fa109000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f7d2fde080 0x55f7d2fe64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.532+0000 7fa109a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fe5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56736/0 (socket says 172.21.15.37:56736) 2024-03-28T18:13:11.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.532+0000 7fa109a00640 1 -- 172.21.15.37:0/956887620 learned_addr learned my addr 172.21.15.37:0/956887620 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:11.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.532+0000 7fa10a400640 1 --2- 172.21.15.37:0/956887620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fe6c00 0x55f7d2fe0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.533+0000 7fa109a00640 1 -- 172.21.15.37:0/956887620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fe6c00 msgr2=0x55f7d2fe0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:11.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.533+0000 7fa109a00640 1 --2- 172.21.15.37:0/956887620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fe6c00 0x55f7d2fe0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.533+0000 7fa109a00640 1 -- 172.21.15.37:0/956887620 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f7d2fde080 msgr2=0x55f7d2fe64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:11.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.533+0000 7fa109a00640 1 --2- 172.21.15.37:0/956887620 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f7d2fde080 0x55f7d2fe64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.533+0000 7fa109a00640 1 -- 172.21.15.37:0/956887620 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f7d2fdde20 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.533+0000 7fa109a00640 1 --2- 172.21.15.37:0/956887620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fe5f60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa0f802cb40 tx=0x7fa0f8003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49a97e46869de609 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:11.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.533+0000 7fa0f6000640 1 -- 172.21.15.37:0/956887620 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f80db070 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.533+0000 7fa0f6000640 1 -- 172.21.15.37:0/956887620 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0f8005690 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.533+0000 7fa0f6000640 1 -- 172.21.15.37:0/956887620 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f80bcff0 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.534+0000 7fa10f1bc740 1 -- 172.21.15.37:0/956887620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fdd6d0 msgr2=0x55f7d2fe5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:11.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.534+0000 7fa10f1bc740 1 --2- 172.21.15.37:0/956887620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fe5f60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa0f802cb40 tx=0x7fa0f8003680 comp rx=0 tx=0).stop 2024-03-28T18:13:11.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.534+0000 7fa10f1bc740 1 -- 172.21.15.37:0/956887620 shutdown_connections 2024-03-28T18:13:11.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.534+0000 7fa10f1bc740 1 --2- 172.21.15.37:0/956887620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fe6c00 0x55f7d2fe0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.534+0000 7fa10f1bc740 1 --2- 172.21.15.37:0/956887620 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f7d2fde080 0x55f7d2fe64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.534+0000 7fa10f1bc740 1 --2- 172.21.15.37:0/956887620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fe5f60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.534+0000 7fa10f1bc740 1 -- 172.21.15.37:0/956887620 >> 172.21.15.37:0/956887620 conn(0x55f7d2fd7da0 msgr2=0x55f7d2fd6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:11.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.534+0000 7fa10f1bc740 1 -- 172.21.15.37:0/956887620 shutdown_connections 2024-03-28T18:13:11.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.535+0000 7fa10f1bc740 1 -- 172.21.15.37:0/956887620 wait complete. 2024-03-28T18:13:11.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.535+0000 7fa10f1bc740 1 -- start start 2024-03-28T18:13:11.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.535+0000 7fa10f1bc740 1 Processor -- start 2024-03-28T18:13:11.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.536+0000 7fa10f1bc740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fecb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.536+0000 7fa10f1bc740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f7d2fde080 0x55f7d2fed050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.536+0000 7fa10f1bc740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fe6c00 0x55f7d3051360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.536+0000 7fa10f1bc740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1041a9740 con 0x55f7d2fde080 2024-03-28T18:13:11.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.536+0000 7fa10f1bc740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1041aaa40 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.536+0000 7fa10f1bc740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1041a9d80 con 0x55f7d2fe6c00 2024-03-28T18:13:11.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.536+0000 7fa10f1bc740 10 client.0 _ll_register_callbacks cb 0x7fa0f802ca90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:13:11.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.536+0000 7fa109a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fecb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.536+0000 7fa109000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f7d2fde080 0x55f7d2fed050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.536+0000 7fa10a400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fe6c00 0x55f7d3051360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.536+0000 7fa109a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fecb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:35102/0 (socket says 172.21.15.37:35102) 2024-03-28T18:13:11.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.536+0000 7fa109a00640 1 -- 172.21.15.37:0/269082419 learned_addr learned my addr 172.21.15.37:0/269082419 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:11.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.537+0000 7fa109a00640 1 -- 172.21.15.37:0/269082419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fe6c00 msgr2=0x55f7d3051360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:11.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.537+0000 7fa109a00640 1 --2- 172.21.15.37:0/269082419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fe6c00 0x55f7d3051360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.537+0000 7fa109a00640 1 -- 172.21.15.37:0/269082419 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f7d2fde080 msgr2=0x55f7d2fed050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:11.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.537+0000 7fa109a00640 1 --2- 172.21.15.37:0/269082419 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f7d2fde080 0x55f7d2fed050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.537+0000 7fa109a00640 1 -- 172.21.15.37:0/269082419 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f7d30518f0 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.537+0000 7fa109a00640 1 --2- 172.21.15.37:0/269082419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fecb10 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80bd440 tx=0x7fa0f8004ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5c86b16dabf256d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:11.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.537+0000 7fa0eb400640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:13:11.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.537+0000 7fa0eb400640 1 -- 172.21.15.37:0/269082419 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f80db070 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.537+0000 7fa10f1bc740 1 -- 172.21.15.37:0/269082419 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f7d2f48b50 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.537+0000 7fa0eb400640 1 -- 172.21.15.37:0/269082419 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa0f80eede0 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.537+0000 7fa0eb400640 1 -- 172.21.15.37:0/269082419 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0f80a6050 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.538+0000 7fa10f1bc740 10 client.25853 fetch_fsmap learned FSMap version 24 2024-03-28T18:13:11.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.538+0000 7fa10f1bc740 1 -- 172.21.15.37:0/269082419 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55f7d30518f0 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.538+0000 7fa0eb400640 1 -- 172.21.15.37:0/269082419 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f7d2f48b50 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.538+0000 7fa0eb400640 1 -- 172.21.15.37:0/269082419 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa0f80c57c0 con 0x55f7d2fdd6d0 2024-03-28T18:13:11.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.538+0000 7fa10f1bc740 10 client.25853 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T18:13:11.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.539+0000 7fa10f1bc740 10 client.25853 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:13:11.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.539+0000 7fa10f1bc740 20 client.25853 populate_metadata read hostname 'smithi037' 2024-03-28T18:13:11.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.539+0000 7fa10f1bc740 1 --2- 172.21.15.37:0/269082419 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55f7d3052e00 0x55f7d3055220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:11.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.539+0000 7fa10f1bc740 4 client.25853 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:13:11.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.539+0000 7fa10f1bc740 1 -- 172.21.15.37:0/269082419 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f7d2ec6500 con 0x55f7d3052e00 2024-03-28T18:13:11.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.539+0000 7fa109000640 1 --2- 172.21.15.37:0/269082419 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55f7d3052e00 0x55f7d3055220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:11.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.540+0000 7fa0eb400640 10 client.25853 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:13:11.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.539+0000 7fa109000640 1 --2- 172.21.15.37:0/269082419 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55f7d3052e00 0x55f7d3055220 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36e63d622ff7e196 server_cookie=61a0933c63c3efba in_seq=0 out_seq=0 2024-03-28T18:13:11.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.541+0000 7fa0eb400640 1 -- 172.21.15.37:0/269082419 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f7d2ec6500 con 0x55f7d3052e00 2024-03-28T18:13:11.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.541+0000 7fa0eb400640 10 client.25853 handle_command_reply: tid=0 2024-03-28T18:13:11.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.541+0000 7fa10f1bc740 1 client.25853 shutdown 2024-03-28T18:13:11.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.541+0000 7fa10f1bc740 1 -- 172.21.15.37:0/269082419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fdd6d0 msgr2=0x55f7d2fecb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:11.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.541+0000 7fa10f1bc740 1 --2- 172.21.15.37:0/269082419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fecb10 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80bd440 tx=0x7fa0f8004ee0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.541+0000 7fa10f1bc740 1 -- 172.21.15.37:0/269082419 shutdown_connections 2024-03-28T18:13:11.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.541+0000 7fa10f1bc740 1 --2- 172.21.15.37:0/269082419 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55f7d3052e00 0x55f7d3055220 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.541+0000 7fa10f1bc740 1 --2- 172.21.15.37:0/269082419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f7d2fe6c00 0x55f7d3051360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.541+0000 7fa10f1bc740 1 --2- 172.21.15.37:0/269082419 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f7d2fde080 0x55f7d2fed050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.541+0000 7fa10f1bc740 1 --2- 172.21.15.37:0/269082419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f7d2fdd6d0 0x55f7d2fecb10 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.541+0000 7fa0eb400640 0 client.25853 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:13:11.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.541+0000 7fa10f1bc740 1 -- 172.21.15.37:0/269082419 >> 172.21.15.37:0/269082419 conn(0x55f7d2fd7da0 msgr2=0x55f7d2fde460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:11.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.542+0000 7fa10f1bc740 1 -- 172.21.15.37:0/269082419 shutdown_connections 2024-03-28T18:13:11.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.542+0000 7fa10f1bc740 1 -- 172.21.15.37:0/269082419 wait complete. 2024-03-28T18:13:11.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.542+0000 7fa10f1bc740 20 client.25853 trim_cache size 0 max 16384 2024-03-28T18:13:11.543 INFO:teuthology.orchestra.run.smithi037.stdout:[] 2024-03-28T18:13:11.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.543+0000 7fa10c400640 1 -- 172.21.15.37:0/184625407 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fa0ec0a29f0 msgr2=0x7fa0ec0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:11.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.543+0000 7fa10c400640 1 --2- 172.21.15.37:0/184625407 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fa0ec0a29f0 0x7fa0ec0a4eb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc009fd0 tx=0x7fa0fc009500 comp rx=0 tx=0).stop 2024-03-28T18:13:11.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.543+0000 7fa10c400640 1 -- 172.21.15.37:0/184625407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa104103840 msgr2=0x7fa10419a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:11.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.543+0000 7fa10c400640 1 --2- 172.21.15.37:0/184625407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa104103840 0x7fa10419a1f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8009900 tx=0x7fa0f8004360 comp rx=0 tx=0).stop 2024-03-28T18:13:11.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.543+0000 7fa10c400640 1 -- 172.21.15.37:0/184625407 shutdown_connections 2024-03-28T18:13:11.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.543+0000 7fa10c400640 1 --2- 172.21.15.37:0/184625407 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fa0ec0a29f0 0x7fa0ec0a4eb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.543+0000 7fa10c400640 1 --2- 172.21.15.37:0/184625407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa104109850 0x7fa10419ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.543+0000 7fa10c400640 1 --2- 172.21.15.37:0/184625407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa104104200 0x7fa10419a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.543+0000 7fa10c400640 1 --2- 172.21.15.37:0/184625407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa104103840 0x7fa10419a1f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:11.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.543+0000 7fa10c400640 1 -- 172.21.15.37:0/184625407 >> 172.21.15.37:0/184625407 conn(0x7fa1040ff350 msgr2=0x7fa1040ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:11.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.543+0000 7fa10c400640 1 -- 172.21.15.37:0/184625407 shutdown_connections 2024-03-28T18:13:11.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:11.543+0000 7fa10c400640 1 -- 172.21.15.37:0/184625407 wait complete. 2024-03-28T18:13:11.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:11 smithi059 ceph-mon[36190]: pgmap v63: 97 pgs: 97 active+clean; 16 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 101 op/s 2024-03-28T18:13:11.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:11 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/2018398521' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:13:11.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:11 smithi170 ceph-mon[36280]: pgmap v63: 97 pgs: 97 active+clean; 16 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 101 op/s 2024-03-28T18:13:11.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:11 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/2018398521' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:13:11.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:11 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/2018398521' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:13:12.563 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:13:12.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:13:12.563 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:13:12.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.819+0000 7f3a46c00640 1 -- 172.21.15.37:0/1317868861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a401075e0 msgr2=0x7f3a401079c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:12.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.819+0000 7f3a46c00640 1 --2- 172.21.15.37:0/1317868861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a401075e0 0x7f3a401079c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3a34009930 tx=0x7f3a3402bc30 comp rx=0 tx=0).stop 2024-03-28T18:13:12.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.819+0000 7f3a46c00640 1 -- 172.21.15.37:0/1317868861 shutdown_connections 2024-03-28T18:13:12.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.819+0000 7f3a46c00640 1 --2- 172.21.15.37:0/1317868861 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a40101ee0 0x7f3a4010e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.819+0000 7f3a46c00640 1 --2- 172.21.15.37:0/1317868861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a40101520 0x7f3a401019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.819+0000 7f3a46c00640 1 --2- 172.21.15.37:0/1317868861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a401075e0 0x7f3a401079c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.819+0000 7f3a46c00640 1 -- 172.21.15.37:0/1317868861 >> 172.21.15.37:0/1317868861 conn(0x7f3a400fd010 msgr2=0x7f3a400ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:12.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.819+0000 7f3a46c00640 1 -- 172.21.15.37:0/1317868861 shutdown_connections 2024-03-28T18:13:12.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.819+0000 7f3a46c00640 1 -- 172.21.15.37:0/1317868861 wait complete. 2024-03-28T18:13:12.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.821+0000 7f3a46c00640 1 Processor -- start 2024-03-28T18:13:12.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.821+0000 7f3a46c00640 1 -- start start 2024-03-28T18:13:12.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.822+0000 7f3a46c00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a40101520 0x7f3a40197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:12.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.822+0000 7f3a46c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a40101ee0 0x7f3a40198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:12.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.822+0000 7f3a46c00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a401075e0 0x7f3a4019c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:12.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.823+0000 7f3a46c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a401118c0 con 0x7f3a40101ee0 2024-03-28T18:13:12.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.823+0000 7f3a46c00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a40111740 con 0x7f3a40101520 2024-03-28T18:13:12.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.823+0000 7f3a46c00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a40111a40 con 0x7f3a401075e0 2024-03-28T18:13:12.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.823+0000 7f3a45800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a40101520 0x7f3a40197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:12.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.823+0000 7f3a45800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a40101520 0x7f3a40197e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:35110/0 (socket says 172.21.15.37:35110) 2024-03-28T18:13:12.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.823+0000 7f3a45800640 1 -- 172.21.15.37:0/3754299535 learned_addr learned my addr 172.21.15.37:0/3754299535 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:12.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.823+0000 7f3a44e00640 1 --2- 172.21.15.37:0/3754299535 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a40101ee0 0x7f3a40198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:12.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.823+0000 7f3a46200640 1 --2- 172.21.15.37:0/3754299535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a401075e0 0x7f3a4019c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:12.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.823+0000 7f3a45800640 1 -- 172.21.15.37:0/3754299535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a401075e0 msgr2=0x7f3a4019c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:12.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.823+0000 7f3a45800640 1 --2- 172.21.15.37:0/3754299535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a401075e0 0x7f3a4019c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.823+0000 7f3a45800640 1 -- 172.21.15.37:0/3754299535 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a40101ee0 msgr2=0x7f3a40198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:12.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.823+0000 7f3a45800640 1 --2- 172.21.15.37:0/3754299535 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a40101ee0 0x7f3a40198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.823+0000 7f3a45800640 1 -- 172.21.15.37:0/3754299535 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a4019ce10 con 0x7f3a40101520 2024-03-28T18:13:12.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.823+0000 7f3a46200640 1 --2- 172.21.15.37:0/3754299535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a401075e0 0x7f3a4019c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:13:12.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.823+0000 7f3a44e00640 1 --2- 172.21.15.37:0/3754299535 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a40101ee0 0x7f3a40198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:13:12.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.824+0000 7f3a45800640 1 --2- 172.21.15.37:0/3754299535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a40101520 0x7f3a40197e20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3a34009230 tx=0x7f3a3402cb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:12.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.824+0000 7f3a3e000640 1 -- 172.21.15.37:0/3754299535 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a34041070 con 0x7f3a40101520 2024-03-28T18:13:12.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.824+0000 7f3a3e000640 1 -- 172.21.15.37:0/3754299535 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a34004450 con 0x7f3a40101520 2024-03-28T18:13:12.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.824+0000 7f3a3e000640 1 -- 172.21.15.37:0/3754299535 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a34003e50 con 0x7f3a40101520 2024-03-28T18:13:12.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.824+0000 7f3a46c00640 1 -- 172.21.15.37:0/3754299535 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a4019d0a0 con 0x7f3a40101520 2024-03-28T18:13:12.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.824+0000 7f3a46c00640 1 -- 172.21.15.37:0/3754299535 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a401a4990 con 0x7f3a40101520 2024-03-28T18:13:12.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.828+0000 7f3a4abfd740 1 Processor -- start 2024-03-28T18:13:12.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.828+0000 7f3a4abfd740 1 -- start start 2024-03-28T18:13:12.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.828+0000 7f3a3e000640 1 -- 172.21.15.37:0/3754299535 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f3a34044070 con 0x7f3a40101520 2024-03-28T18:13:12.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.828+0000 7f3a4abfd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33290 0x55a2e3b33670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:12.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.828+0000 7f3a4abfd740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b3c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:12.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.828+0000 7f3a4abfd740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b3c790 0x55a2e3b3eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:12.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.828+0000 7f3a4abfd740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2e3a97eb0 con 0x55a2e3b3c790 2024-03-28T18:13:12.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.828+0000 7f3a4abfd740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2e3a40a60 con 0x55a2e3b33bb0 2024-03-28T18:13:12.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.828+0000 7f3a4abfd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2e3a93f50 con 0x55a2e3b33290 2024-03-28T18:13:12.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.828+0000 7f3a45800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b3c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:12.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.828+0000 7f3a46200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33290 0x55a2e3b33670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:12.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.828+0000 7f3a46200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33290 0x55a2e3b33670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56766/0 (socket says 172.21.15.37:56766) 2024-03-28T18:13:12.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.829+0000 7f3a46200640 1 -- 172.21.15.37:0/3751677843 learned_addr learned my addr 172.21.15.37:0/3751677843 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:12.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.829+0000 7f3a44e00640 1 --2- 172.21.15.37:0/3751677843 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b3c790 0x55a2e3b3eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:12.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.829+0000 7f3a46200640 1 -- 172.21.15.37:0/3751677843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b33bb0 msgr2=0x55a2e3b3c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:12.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.829+0000 7f3a46200640 1 --2- 172.21.15.37:0/3751677843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b3c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.829+0000 7f3a46200640 1 -- 172.21.15.37:0/3751677843 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b3c790 msgr2=0x55a2e3b3eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:12.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.829+0000 7f3a46200640 1 --2- 172.21.15.37:0/3751677843 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b3c790 0x55a2e3b3eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.829+0000 7f3a46200640 1 -- 172.21.15.37:0/3751677843 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2e38747e0 con 0x55a2e3b33290 2024-03-28T18:13:12.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.829+0000 7f3a45800640 1 --2- 172.21.15.37:0/3751677843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b3c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:13:12.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.829+0000 7f3a44e00640 1 --2- 172.21.15.37:0/3751677843 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b3c790 0x55a2e3b3eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:13:12.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.829+0000 7f3a46200640 1 --2- 172.21.15.37:0/3751677843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33290 0x55a2e3b33670 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3a38009fd0 tx=0x7f3a38010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=347bbbb9fcd52ae6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:12.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.830+0000 7f3a32a00640 1 -- 172.21.15.37:0/3751677843 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a38011700 con 0x55a2e3b33290 2024-03-28T18:13:12.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.830+0000 7f3a32a00640 1 -- 172.21.15.37:0/3751677843 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a38011d80 con 0x55a2e3b33290 2024-03-28T18:13:12.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.830+0000 7f3a32a00640 1 -- 172.21.15.37:0/3751677843 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3800e3c0 con 0x55a2e3b33290 2024-03-28T18:13:12.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.830+0000 7f3a3e000640 1 --2- 172.21.15.37:0/3754299535 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f3a280a2b90 0x7f3a280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:12.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.830+0000 7f3a3e000640 1 -- 172.21.15.37:0/3754299535 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3a340f06e0 con 0x7f3a40101520 2024-03-28T18:13:12.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.830+0000 7f3a45800640 1 --2- 172.21.15.37:0/3754299535 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f3a280a2b90 0x7f3a280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:12.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.831+0000 7f3a4abfd740 1 -- 172.21.15.37:0/3751677843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33290 msgr2=0x55a2e3b33670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:12.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.831+0000 7f3a4abfd740 1 --2- 172.21.15.37:0/3751677843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33290 0x55a2e3b33670 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3a38009fd0 tx=0x7f3a38010860 comp rx=0 tx=0).stop 2024-03-28T18:13:12.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.831+0000 7f3a4abfd740 1 -- 172.21.15.37:0/3751677843 shutdown_connections 2024-03-28T18:13:12.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.831+0000 7f3a4abfd740 1 --2- 172.21.15.37:0/3751677843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b3c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.831+0000 7f3a4abfd740 1 --2- 172.21.15.37:0/3751677843 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b3c790 0x55a2e3b3eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.831+0000 7f3a4abfd740 1 --2- 172.21.15.37:0/3751677843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33290 0x55a2e3b33670 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.831+0000 7f3a4abfd740 1 -- 172.21.15.37:0/3751677843 >> 172.21.15.37:0/3751677843 conn(0x55a2e3b2d9c0 msgr2=0x55a2e3b2dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:12.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.831+0000 7f3a4abfd740 1 -- 172.21.15.37:0/3751677843 shutdown_connections 2024-03-28T18:13:12.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.831+0000 7f3a4abfd740 1 -- 172.21.15.37:0/3751677843 wait complete. 2024-03-28T18:13:12.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.831+0000 7f3a45800640 1 --2- 172.21.15.37:0/3754299535 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f3a280a2b90 0x7f3a280a5050 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x55a2e3b32d60 tx=0x7f3a340f6530 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:12.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.832+0000 7f3a4abfd740 1 -- start start 2024-03-28T18:13:12.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.832+0000 7f3a4abfd740 1 Processor -- start 2024-03-28T18:13:12.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.832+0000 7f3a4abfd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33290 0x55a2e3b3c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:12.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.832+0000 7f3a4abfd740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b394d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:12.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.833+0000 7f3a4abfd740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b3c790 0x55a2e3b39a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:12.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.833+0000 7f3a4abfd740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2e3b3f580 con 0x55a2e3b33bb0 2024-03-28T18:13:12.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.833+0000 7f3a4abfd740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2e3a53eb0 con 0x55a2e3b3c790 2024-03-28T18:13:12.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.833+0000 7f3a4abfd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2e3b3f700 con 0x55a2e3b33290 2024-03-28T18:13:12.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.833+0000 7f3a4abfd740 10 client.0 _ll_register_callbacks cb 0x55a2e3a48150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:13:12.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.833+0000 7f3a46200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33290 0x55a2e3b3c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:12.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.833+0000 7f3a46200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33290 0x55a2e3b3c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56770/0 (socket says 172.21.15.37:56770) 2024-03-28T18:13:12.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.833+0000 7f3a46200640 1 -- 172.21.15.37:0/2239277897 learned_addr learned my addr 172.21.15.37:0/2239277897 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:12.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.833+0000 7f3a44e00640 1 --2- 172.21.15.37:0/2239277897 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:12.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.833+0000 7f3a46200640 1 --2- 172.21.15.37:0/2239277897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b3c790 0x55a2e3b39a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:12.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.833+0000 7f3a46200640 1 -- 172.21.15.37:0/2239277897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b3c790 msgr2=0x55a2e3b39a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:12.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.833+0000 7f3a46200640 1 --2- 172.21.15.37:0/2239277897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b3c790 0x55a2e3b39a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.833+0000 7f3a46200640 1 -- 172.21.15.37:0/2239277897 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b33bb0 msgr2=0x55a2e3b394d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:12.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.833+0000 7f3a46200640 1 --2- 172.21.15.37:0/2239277897 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b394d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.833+0000 7f3a46200640 1 -- 172.21.15.37:0/2239277897 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a2e38d4f70 con 0x55a2e3b33290 2024-03-28T18:13:12.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.833+0000 7f3a44e00640 1 --2- 172.21.15.37:0/2239277897 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b394d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:13:12.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.834+0000 7f3a46200640 1 --2- 172.21.15.37:0/2239277897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33290 0x55a2e3b3c140 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3a38011fd0 tx=0x7f3a38013040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c014f5743fc1ac4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:12.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.834+0000 7f3a27400640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:13:12.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.834+0000 7f3a27400640 1 -- 172.21.15.37:0/2239277897 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a38004c50 con 0x55a2e3b33290 2024-03-28T18:13:12.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.834+0000 7f3a27400640 1 -- 172.21.15.37:0/2239277897 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3a38004df0 con 0x55a2e3b33290 2024-03-28T18:13:12.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.834+0000 7f3a4abfd740 1 -- 172.21.15.37:0/2239277897 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a38011d80 con 0x55a2e3b33290 2024-03-28T18:13:12.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.834+0000 7f3a27400640 1 -- 172.21.15.37:0/2239277897 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a38011730 con 0x55a2e3b33290 2024-03-28T18:13:12.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.834+0000 7f3a4abfd740 10 client.25876 fetch_fsmap learned FSMap version 24 2024-03-28T18:13:12.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.834+0000 7f3a4abfd740 1 -- 172.21.15.37:0/2239277897 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55a2e376c170 con 0x55a2e3b33290 2024-03-28T18:13:12.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.835+0000 7f3a27400640 1 -- 172.21.15.37:0/2239277897 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a38011d80 con 0x55a2e3b33290 2024-03-28T18:13:12.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.835+0000 7f3a27400640 1 -- 172.21.15.37:0/2239277897 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f3a38029040 con 0x55a2e3b33290 2024-03-28T18:13:12.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.835+0000 7f3a4abfd740 10 client.25876 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T18:13:12.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.835+0000 7f3a4abfd740 10 client.25876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:13:12.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.835+0000 7f3a4abfd740 20 client.25876 populate_metadata read hostname 'smithi037' 2024-03-28T18:13:12.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.835+0000 7f3a4abfd740 1 --2- 172.21.15.37:0/2239277897 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55a2e3b4c740 0x55a2e3b4eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:12.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.835+0000 7f3a4abfd740 4 client.25876 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:13:12.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.835+0000 7f3a4abfd740 1 -- 172.21.15.37:0/2239277897 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a2e3aa53f0 con 0x55a2e3b4c740 2024-03-28T18:13:12.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.836+0000 7f3a44e00640 1 --2- 172.21.15.37:0/2239277897 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55a2e3b4c740 0x55a2e3b4eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:12.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.837+0000 7f3a44e00640 1 --2- 172.21.15.37:0/2239277897 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55a2e3b4c740 0x55a2e3b4eb30 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0f8f1b30ba8400e server_cookie=5f7cd3cefa798f61 in_seq=0 out_seq=0 2024-03-28T18:13:12.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.837+0000 7f3a27400640 10 client.25876 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:13:12.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.838+0000 7f3a27400640 1 -- 172.21.15.37:0/2239277897 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a2e3aa53f0 con 0x55a2e3b4c740 2024-03-28T18:13:12.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.838+0000 7f3a27400640 10 client.25876 handle_command_reply: tid=0 2024-03-28T18:13:12.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.838+0000 7f3a4abfd740 1 client.25876 shutdown 2024-03-28T18:13:12.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.838+0000 7f3a4abfd740 1 -- 172.21.15.37:0/2239277897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33290 msgr2=0x55a2e3b3c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:12.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.838+0000 7f3a4abfd740 1 --2- 172.21.15.37:0/2239277897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33290 0x55a2e3b3c140 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3a38011fd0 tx=0x7f3a38013040 comp rx=0 tx=0).stop 2024-03-28T18:13:12.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.838+0000 7f3a4abfd740 1 -- 172.21.15.37:0/2239277897 shutdown_connections 2024-03-28T18:13:12.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.838+0000 7f3a4abfd740 1 --2- 172.21.15.37:0/2239277897 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55a2e3b4c740 0x55a2e3b4eb30 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.838+0000 7f3a4abfd740 1 --2- 172.21.15.37:0/2239277897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b3c790 0x55a2e3b39a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.838+0000 7f3a4abfd740 1 --2- 172.21.15.37:0/2239277897 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b394d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.838+0000 7f3a27400640 0 client.25876 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:13:12.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.838+0000 7f3a4abfd740 1 --2- 172.21.15.37:0/2239277897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33290 0x55a2e3b3c140 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.838+0000 7f3a4abfd740 1 -- 172.21.15.37:0/2239277897 >> 172.21.15.37:0/2239277897 conn(0x55a2e3b2d9c0 msgr2=0x55a2e3b2f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:12.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.838+0000 7f3a4abfd740 1 -- 172.21.15.37:0/2239277897 shutdown_connections 2024-03-28T18:13:12.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.838+0000 7f3a4abfd740 1 -- 172.21.15.37:0/2239277897 wait complete. 2024-03-28T18:13:12.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.839+0000 7f3a4abfd740 20 client.25876 trim_cache size 0 max 16384 2024-03-28T18:13:12.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.871+0000 7f3a4abfd740 1 Processor -- start 2024-03-28T18:13:12.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.872+0000 7f3a4abfd740 1 -- start start 2024-03-28T18:13:12.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.872+0000 7f3a4abfd740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b33290 0x55a2e3b2ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:12.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.872+0000 7f3a4abfd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b3bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:12.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.872+0000 7f3a4abfd740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b3c790 0x55a2e3b3c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:12.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.872+0000 7f3a4abfd740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a401af5c0 con 0x55a2e3b33290 2024-03-28T18:13:12.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.872+0000 7f3a4abfd740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a401ad0c0 con 0x55a2e3b3c790 2024-03-28T18:13:12.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.872+0000 7f3a4abfd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a401ab160 con 0x55a2e3b33bb0 2024-03-28T18:13:12.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.872+0000 7f3a44e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b3bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:12.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.872+0000 7f3a44e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b3bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56782/0 (socket says 172.21.15.37:56782) 2024-03-28T18:13:12.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.872+0000 7f3a44e00640 1 -- 172.21.15.37:0/720890476 learned_addr learned my addr 172.21.15.37:0/720890476 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:12.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.873+0000 7f3a46200640 1 --2- 172.21.15.37:0/720890476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b33290 0x55a2e3b2ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:12.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.873+0000 7f3a46200640 1 --2- 172.21.15.37:0/720890476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b3c790 0x55a2e3b3c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:12.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.873+0000 7f3a44e00640 1 -- 172.21.15.37:0/720890476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b3c790 msgr2=0x55a2e3b3c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:12.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.873+0000 7f3a44e00640 1 --2- 172.21.15.37:0/720890476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b3c790 0x55a2e3b3c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.873+0000 7f3a44e00640 1 -- 172.21.15.37:0/720890476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b33290 msgr2=0x55a2e3b2ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:12.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.873+0000 7f3a44e00640 1 --2- 172.21.15.37:0/720890476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b33290 0x55a2e3b2ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.873+0000 7f3a44e00640 1 -- 172.21.15.37:0/720890476 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2e3b339e0 con 0x55a2e3b33bb0 2024-03-28T18:13:12.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.873+0000 7f3a46200640 1 --2- 172.21.15.37:0/720890476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b3c790 0x55a2e3b3c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:13:12.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.873+0000 7f3a44e00640 1 --2- 172.21.15.37:0/720890476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b3bc00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c005d00 tx=0x7f3a2c013b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d95ce4535dc972f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:12.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.873+0000 7f3a32000640 1 -- 172.21.15.37:0/720890476 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c020bc0 con 0x55a2e3b33bb0 2024-03-28T18:13:12.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.874+0000 7f3a32000640 1 -- 172.21.15.37:0/720890476 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a2c020d60 con 0x55a2e3b33bb0 2024-03-28T18:13:12.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.874+0000 7f3a32000640 1 -- 172.21.15.37:0/720890476 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c01f430 con 0x55a2e3b33bb0 2024-03-28T18:13:12.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.875+0000 7f3a4abfd740 1 -- 172.21.15.37:0/720890476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33bb0 msgr2=0x55a2e3b3bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:12.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.875+0000 7f3a4abfd740 1 --2- 172.21.15.37:0/720890476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b3bc00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c005d00 tx=0x7f3a2c013b90 comp rx=0 tx=0).stop 2024-03-28T18:13:12.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.875+0000 7f3a4abfd740 1 -- 172.21.15.37:0/720890476 shutdown_connections 2024-03-28T18:13:12.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.875+0000 7f3a4abfd740 1 --2- 172.21.15.37:0/720890476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b3c790 0x55a2e3b3c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.875+0000 7f3a4abfd740 1 --2- 172.21.15.37:0/720890476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b3bc00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.875+0000 7f3a4abfd740 1 --2- 172.21.15.37:0/720890476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b33290 0x55a2e3b2ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.875+0000 7f3a4abfd740 1 -- 172.21.15.37:0/720890476 >> 172.21.15.37:0/720890476 conn(0x55a2e3b2d9c0 msgr2=0x55a2e3b2f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:12.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.875+0000 7f3a4abfd740 1 -- 172.21.15.37:0/720890476 shutdown_connections 2024-03-28T18:13:12.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.875+0000 7f3a4abfd740 1 -- 172.21.15.37:0/720890476 wait complete. 2024-03-28T18:13:12.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.875+0000 7f3a4abfd740 1 -- start start 2024-03-28T18:13:12.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.876+0000 7f3a4abfd740 1 Processor -- start 2024-03-28T18:13:12.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.876+0000 7f3a4abfd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33290 0x55a2e3b41950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:12.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.876+0000 7f3a4abfd740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b41e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:12.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.876+0000 7f3a4abfd740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b3c790 0x55a2e3b423d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:12.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.876+0000 7f3a4abfd740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a401ab760 con 0x55a2e3b3c790 2024-03-28T18:13:12.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.876+0000 7f3a4abfd740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a401aca60 con 0x55a2e3b33bb0 2024-03-28T18:13:12.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.876+0000 7f3a4abfd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a401abda0 con 0x55a2e3b33290 2024-03-28T18:13:12.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.877+0000 7f3a4abfd740 10 client.0 _ll_register_callbacks cb 0x7f3a340f6400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:13:12.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.877+0000 7f3a44e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b41e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:12.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.877+0000 7f3a44e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b41e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:35160/0 (socket says 172.21.15.37:35160) 2024-03-28T18:13:12.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.877+0000 7f3a44e00640 1 -- 172.21.15.37:0/3976450247 learned_addr learned my addr 172.21.15.37:0/3976450247 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:12.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.877+0000 7f3a46200640 1 --2- 172.21.15.37:0/3976450247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33290 0x55a2e3b41950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:12.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.877+0000 7f3a46200640 1 --2- 172.21.15.37:0/3976450247 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b3c790 0x55a2e3b423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:12.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.877+0000 7f3a44e00640 1 -- 172.21.15.37:0/3976450247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33290 msgr2=0x55a2e3b41950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:12.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.877+0000 7f3a44e00640 1 --2- 172.21.15.37:0/3976450247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33290 0x55a2e3b41950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.877+0000 7f3a44e00640 1 -- 172.21.15.37:0/3976450247 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b3c790 msgr2=0x55a2e3b423d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:12.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.877+0000 7f3a44e00640 1 --2- 172.21.15.37:0/3976450247 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b3c790 0x55a2e3b423d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.877+0000 7f3a44e00640 1 -- 172.21.15.37:0/3976450247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a2e3bac390 con 0x55a2e3b33bb0 2024-03-28T18:13:12.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.878+0000 7f3a44e00640 1 --2- 172.21.15.37:0/3976450247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b41e90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c00a010 tx=0x7f3a2c009bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2960ca597d86f81d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:12.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.878+0000 7f3a27400640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:13:12.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.878+0000 7f3a27400640 1 -- 172.21.15.37:0/3976450247 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c005480 con 0x55a2e3b33bb0 2024-03-28T18:13:12.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.878+0000 7f3a27400640 1 -- 172.21.15.37:0/3976450247 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3a2c004700 con 0x55a2e3b33bb0 2024-03-28T18:13:12.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.878+0000 7f3a4abfd740 1 -- 172.21.15.37:0/3976450247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a2c020d60 con 0x55a2e3b33bb0 2024-03-28T18:13:12.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.878+0000 7f3a27400640 1 -- 172.21.15.37:0/3976450247 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a2c004900 con 0x55a2e3b33bb0 2024-03-28T18:13:12.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.878+0000 7f3a4abfd740 10 client.25868 fetch_fsmap learned FSMap version 24 2024-03-28T18:13:12.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.878+0000 7f3a4abfd740 1 -- 172.21.15.37:0/3976450247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55a2e3bad640 con 0x55a2e3b33bb0 2024-03-28T18:13:12.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.878+0000 7f3a27400640 1 -- 172.21.15.37:0/3976450247 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a2c020d60 con 0x55a2e3b33bb0 2024-03-28T18:13:12.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.879+0000 7f3a27400640 1 -- 172.21.15.37:0/3976450247 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f3a2c031c10 con 0x55a2e3b33bb0 2024-03-28T18:13:12.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.879+0000 7f3a4abfd740 10 client.25868 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T18:13:12.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.879+0000 7f3a4abfd740 10 client.25868 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:13:12.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.879+0000 7f3a4abfd740 20 client.25868 populate_metadata read hostname 'smithi037' 2024-03-28T18:13:12.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.879+0000 7f3a4abfd740 1 --2- 172.21.15.37:0/3976450247 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55a2e3badc90 0x55a2e3bb00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:12.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.879+0000 7f3a4abfd740 4 client.25868 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:13:12.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.879+0000 7f3a4abfd740 1 -- 172.21.15.37:0/3976450247 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55a2e3b39860 con 0x55a2e3badc90 2024-03-28T18:13:12.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.879+0000 7f3a46200640 1 --2- 172.21.15.37:0/3976450247 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55a2e3badc90 0x55a2e3bb00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:12.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.880+0000 7f3a46200640 1 --2- 172.21.15.37:0/3976450247 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55a2e3badc90 0x55a2e3bb00b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acebe6793323488a server_cookie=a1ad9c69933be607 in_seq=0 out_seq=0 2024-03-28T18:13:12.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.880+0000 7f3a27400640 10 client.25868 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:13:12.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.882+0000 7f3a27400640 1 -- 172.21.15.37:0/3976450247 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a2e3b39860 con 0x55a2e3badc90 2024-03-28T18:13:12.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.882+0000 7f3a27400640 10 client.25868 handle_command_reply: tid=0 2024-03-28T18:13:12.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.882+0000 7f3a4abfd740 1 client.25868 shutdown 2024-03-28T18:13:12.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.883+0000 7f3a4abfd740 1 -- 172.21.15.37:0/3976450247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b33bb0 msgr2=0x55a2e3b41e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:12.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.883+0000 7f3a4abfd740 1 --2- 172.21.15.37:0/3976450247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b41e90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c00a010 tx=0x7f3a2c009bf0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.883+0000 7f3a4abfd740 1 -- 172.21.15.37:0/3976450247 shutdown_connections 2024-03-28T18:13:12.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.883+0000 7f3a4abfd740 1 --2- 172.21.15.37:0/3976450247 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55a2e3badc90 0x55a2e3bb00b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.883+0000 7f3a4abfd740 1 --2- 172.21.15.37:0/3976450247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a2e3b33bb0 0x55a2e3b41e90 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.883+0000 7f3a4abfd740 1 --2- 172.21.15.37:0/3976450247 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55a2e3b3c790 0x55a2e3b423d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.883+0000 7f3a4abfd740 1 --2- 172.21.15.37:0/3976450247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a2e3b33290 0x55a2e3b41950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.883+0000 7f3a4abfd740 1 -- 172.21.15.37:0/3976450247 >> 172.21.15.37:0/3976450247 conn(0x55a2e3b2d9c0 msgr2=0x55a2e3b33f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:12.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.883+0000 7f3a27400640 0 client.25868 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:13:12.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.883+0000 7f3a4abfd740 1 -- 172.21.15.37:0/3976450247 shutdown_connections 2024-03-28T18:13:12.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.883+0000 7f3a4abfd740 1 -- 172.21.15.37:0/3976450247 wait complete. 2024-03-28T18:13:12.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.883+0000 7f3a4abfd740 20 client.25868 trim_cache size 0 max 16384 2024-03-28T18:13:12.884 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:13:12.884 INFO:teuthology.orchestra.run.smithi037.stdout: "return_code": 0, 2024-03-28T18:13:12.884 INFO:teuthology.orchestra.run.smithi037.stdout: "scrub_tag": "3d49dbce-f1b2-47ce-aaff-eb788d7d2af7", 2024-03-28T18:13:12.884 INFO:teuthology.orchestra.run.smithi037.stdout: "mode": "asynchronous" 2024-03-28T18:13:12.884 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:13:12.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.884+0000 7f3a46c00640 1 -- 172.21.15.37:0/3754299535 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f3a280a2b90 msgr2=0x7f3a280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:12.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.884+0000 7f3a46c00640 1 --2- 172.21.15.37:0/3754299535 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f3a280a2b90 0x7f3a280a5050 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x55a2e3b32d60 tx=0x7f3a340f6530 comp rx=0 tx=0).stop 2024-03-28T18:13:12.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.884+0000 7f3a46c00640 1 -- 172.21.15.37:0/3754299535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a40101520 msgr2=0x7f3a40197e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:12.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.884+0000 7f3a46c00640 1 --2- 172.21.15.37:0/3754299535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a40101520 0x7f3a40197e20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3a34009230 tx=0x7f3a3402cb30 comp rx=0 tx=0).stop 2024-03-28T18:13:12.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.884+0000 7f3a46c00640 1 -- 172.21.15.37:0/3754299535 shutdown_connections 2024-03-28T18:13:12.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.884+0000 7f3a46c00640 1 --2- 172.21.15.37:0/3754299535 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f3a280a2b90 0x7f3a280a5050 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.884+0000 7f3a46c00640 1 --2- 172.21.15.37:0/3754299535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a401075e0 0x7f3a4019c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.884+0000 7f3a46c00640 1 --2- 172.21.15.37:0/3754299535 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a40101ee0 0x7f3a40198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:12.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.884+0000 7f3a46c00640 1 --2- 172.21.15.37:0/3754299535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a40101520 0x7f3a40197e20 secure :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3a34009230 tx=0x7f3a3402cb30 comp rx=0 tx=0).stop 2024-03-28T18:13:12.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.884+0000 7f3a46c00640 1 -- 172.21.15.37:0/3754299535 >> 172.21.15.37:0/3754299535 conn(0x7f3a400fd010 msgr2=0x7f3a400fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:12.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.885+0000 7f3a46c00640 1 -- 172.21.15.37:0/3754299535 shutdown_connections 2024-03-28T18:13:12.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:12.885+0000 7f3a46c00640 1 -- 172.21.15.37:0/3754299535 wait complete. 2024-03-28T18:13:12.905 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:13:12.905 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:13:13.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.166+0000 7fd56d200640 1 -- 172.21.15.37:0/2731968156 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd568103720 msgr2=0x7fd568103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:13.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.166+0000 7fd56d200640 1 --2- 172.21.15.37:0/2731968156 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd568103720 0x7fd568103ba0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd554009930 tx=0x7fd55402bc20 comp rx=0 tx=0).stop 2024-03-28T18:13:13.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.166+0000 7fd56d200640 1 -- 172.21.15.37:0/2731968156 shutdown_connections 2024-03-28T18:13:13.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.166+0000 7fd56d200640 1 --2- 172.21.15.37:0/2731968156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5681040e0 0x7fd5681105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.166+0000 7fd56d200640 1 --2- 172.21.15.37:0/2731968156 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd568103720 0x7fd568103ba0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.166+0000 7fd56d200640 1 --2- 172.21.15.37:0/2731968156 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd568109730 0x7fd568109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.166+0000 7fd56d200640 1 -- 172.21.15.37:0/2731968156 >> 172.21.15.37:0/2731968156 conn(0x7fd5680ff1b0 msgr2=0x7fd5681015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:13.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.166+0000 7fd56d200640 1 -- 172.21.15.37:0/2731968156 shutdown_connections 2024-03-28T18:13:13.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.166+0000 7fd56d200640 1 -- 172.21.15.37:0/2731968156 wait complete. 2024-03-28T18:13:13.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.168+0000 7fd56d200640 1 Processor -- start 2024-03-28T18:13:13.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.168+0000 7fd56d200640 1 -- start start 2024-03-28T18:13:13.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.168+0000 7fd56d200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd568103720 0x7fd56819a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:13.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.168+0000 7fd56d200640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5681040e0 0x7fd56819a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:13.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.168+0000 7fd56d200640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd568109730 0x7fd56819e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:13.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.169+0000 7fd56d200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd568113b50 con 0x7fd568103720 2024-03-28T18:13:13.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.169+0000 7fd56d200640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5681139d0 con 0x7fd5681040e0 2024-03-28T18:13:13.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.169+0000 7fd56d200640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd568113cd0 con 0x7fd568109730 2024-03-28T18:13:13.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.169+0000 7fd567e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd568109730 0x7fd56819e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:13.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.169+0000 7fd567e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd568109730 0x7fd56819e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56820/0 (socket says 172.21.15.37:56820) 2024-03-28T18:13:13.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.169+0000 7fd567e00640 1 -- 172.21.15.37:0/445213875 learned_addr learned my addr 172.21.15.37:0/445213875 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:13.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.169+0000 7fd567400640 1 --2- 172.21.15.37:0/445213875 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd568103720 0x7fd56819a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:13.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.169+0000 7fd566a00640 1 --2- 172.21.15.37:0/445213875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5681040e0 0x7fd56819a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:13.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.170+0000 7fd566a00640 1 -- 172.21.15.37:0/445213875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd568109730 msgr2=0x7fd56819e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:13.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.170+0000 7fd566a00640 1 --2- 172.21.15.37:0/445213875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd568109730 0x7fd56819e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.170+0000 7fd566a00640 1 -- 172.21.15.37:0/445213875 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd568103720 msgr2=0x7fd56819a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:13.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.170+0000 7fd566a00640 1 --2- 172.21.15.37:0/445213875 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd568103720 0x7fd56819a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.170+0000 7fd566a00640 1 -- 172.21.15.37:0/445213875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56819f080 con 0x7fd5681040e0 2024-03-28T18:13:13.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.170+0000 7fd567400640 1 --2- 172.21.15.37:0/445213875 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd568103720 0x7fd56819a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:13:13.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.170+0000 7fd566a00640 1 --2- 172.21.15.37:0/445213875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5681040e0 0x7fd56819a5d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd554036040 tx=0x7fd554004250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:13.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.170+0000 7fd55fe00640 1 -- 172.21.15.37:0/445213875 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd554043020 con 0x7fd5681040e0 2024-03-28T18:13:13.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.171+0000 7fd56d200640 1 -- 172.21.15.37:0/445213875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd56819f310 con 0x7fd5681040e0 2024-03-28T18:13:13.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.171+0000 7fd56d200640 1 -- 172.21.15.37:0/445213875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd568075ae0 con 0x7fd5681040e0 2024-03-28T18:13:13.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.171+0000 7fd55fe00640 1 -- 172.21.15.37:0/445213875 <== mon.2 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd554005000 con 0x7fd5681040e0 2024-03-28T18:13:13.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.171+0000 7fd55fe00640 1 -- 172.21.15.37:0/445213875 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd554003b20 con 0x7fd5681040e0 2024-03-28T18:13:13.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.176+0000 7fd55fe00640 1 -- 172.21.15.37:0/445213875 <== mon.2 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fd554046020 con 0x7fd5681040e0 2024-03-28T18:13:13.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.178+0000 7fd5715c8740 1 Processor -- start 2024-03-28T18:13:13.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.178+0000 7fd55fe00640 1 --2- 172.21.15.37:0/445213875 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fd54c0a29f0 0x7fd54c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:13.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.178+0000 7fd5715c8740 1 -- start start 2024-03-28T18:13:13.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.178+0000 7fd55fe00640 1 -- 172.21.15.37:0/445213875 <== mon.2 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd5540f0b40 con 0x7fd5681040e0 2024-03-28T18:13:13.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.178+0000 7fd567e00640 1 --2- 172.21.15.37:0/445213875 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fd54c0a29f0 0x7fd54c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:13.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.179+0000 7fd5715c8740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f4219f5dd0 0x55f4219f4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:13.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.179+0000 7fd5715c8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4219f50f0 0x55f4219f5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:13.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.179+0000 7fd5715c8740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219fe460 0x55f421a00850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:13.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.179+0000 7fd5715c8740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f421959250 con 0x55f4219f5dd0 2024-03-28T18:13:13.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.179+0000 7fd5715c8740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f421901a30 con 0x55f4219fe460 2024-03-28T18:13:13.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.179+0000 7fd5715c8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4219552f0 con 0x55f4219f50f0 2024-03-28T18:13:13.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.180+0000 7fd566a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4219f50f0 0x55f4219f5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:13.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.180+0000 7fd567e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219fe460 0x55f421a00850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:13.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.180+0000 7fd566a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4219f50f0 0x55f4219f5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:56826/0 (socket says 172.21.15.37:56826) 2024-03-28T18:13:13.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.180+0000 7fd566a00640 1 -- 172.21.15.37:0/2810701673 learned_addr learned my addr 172.21.15.37:0/2810701673 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:13.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.180+0000 7fd567400640 1 --2- 172.21.15.37:0/2810701673 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f4219f5dd0 0x55f4219f4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:13.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.180+0000 7fd566a00640 1 -- 172.21.15.37:0/2810701673 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219fe460 msgr2=0x55f421a00850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:13.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.180+0000 7fd566a00640 1 --2- 172.21.15.37:0/2810701673 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219fe460 0x55f421a00850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.180+0000 7fd566a00640 1 -- 172.21.15.37:0/2810701673 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f4219f5dd0 msgr2=0x55f4219f4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:13.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.180+0000 7fd567e00640 1 --2- 172.21.15.37:0/445213875 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fd54c0a29f0 0x7fd54c0a4eb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd5600099a0 tx=0x7fd560009500 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:13.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.180+0000 7fd566a00640 1 --2- 172.21.15.37:0/2810701673 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f4219f5dd0 0x55f4219f4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.180+0000 7fd566a00640 1 -- 172.21.15.37:0/2810701673 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4218ec690 con 0x55f4219f50f0 2024-03-28T18:13:13.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.181+0000 7fd566a00640 1 --2- 172.21.15.37:0/2810701673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4219f50f0 0x55f4219f5570 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd5540b9c50 tx=0x7fd55402bc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=446735c5965ae85a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:13.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.181+0000 7fd55cc00640 1 -- 172.21.15.37:0/2810701673 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5540bd280 con 0x55f4219f50f0 2024-03-28T18:13:13.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.182+0000 7fd55cc00640 1 -- 172.21.15.37:0/2810701673 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5540c13d0 con 0x55f4219f50f0 2024-03-28T18:13:13.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.182+0000 7fd55cc00640 1 -- 172.21.15.37:0/2810701673 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5540b98f0 con 0x55f4219f50f0 2024-03-28T18:13:13.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.182+0000 7fd5715c8740 1 -- 172.21.15.37:0/2810701673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4219f50f0 msgr2=0x55f4219f5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:13.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.182+0000 7fd5715c8740 1 --2- 172.21.15.37:0/2810701673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4219f50f0 0x55f4219f5570 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd5540b9c50 tx=0x7fd55402bc20 comp rx=0 tx=0).stop 2024-03-28T18:13:13.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.183+0000 7fd5715c8740 1 -- 172.21.15.37:0/2810701673 shutdown_connections 2024-03-28T18:13:13.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.183+0000 7fd5715c8740 1 --2- 172.21.15.37:0/2810701673 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219fe460 0x55f421a00850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.183+0000 7fd5715c8740 1 --2- 172.21.15.37:0/2810701673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4219f50f0 0x55f4219f5570 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.183+0000 7fd5715c8740 1 --2- 172.21.15.37:0/2810701673 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f4219f5dd0 0x55f4219f4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.183+0000 7fd5715c8740 1 -- 172.21.15.37:0/2810701673 >> 172.21.15.37:0/2810701673 conn(0x55f4219bc200 msgr2=0x55f4219efb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:13.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.183+0000 7fd5715c8740 1 -- 172.21.15.37:0/2810701673 shutdown_connections 2024-03-28T18:13:13.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.183+0000 7fd5715c8740 1 -- 172.21.15.37:0/2810701673 wait complete. 2024-03-28T18:13:13.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.184+0000 7fd5715c8740 1 -- start start 2024-03-28T18:13:13.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.184+0000 7fd5715c8740 1 Processor -- start 2024-03-28T18:13:13.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.184+0000 7fd5715c8740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f4219f50f0 0x55f4219fdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:13.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.184+0000 7fd5715c8740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219f5dd0 0x55f4219faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:13.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.184+0000 7fd5715c8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4219fe460 0x55f4219fb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:13.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.185+0000 7fd5715c8740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f421a01090 con 0x55f4219f50f0 2024-03-28T18:13:13.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.185+0000 7fd5715c8740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f421914e80 con 0x55f4219f5dd0 2024-03-28T18:13:13.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.185+0000 7fd5715c8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f421a01210 con 0x55f4219fe460 2024-03-28T18:13:13.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.185+0000 7fd5715c8740 10 client.0 _ll_register_callbacks cb 0x55f421927130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:13:13.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.185+0000 7fd567400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f4219f50f0 0x55f4219fdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:13.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.185+0000 7fd567e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4219fe460 0x55f4219fb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:13.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.185+0000 7fd567400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f4219f50f0 0x55f4219fdb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60318/0 (socket says 172.21.15.37:60318) 2024-03-28T18:13:13.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.185+0000 7fd567400640 1 -- 172.21.15.37:0/3527906361 learned_addr learned my addr 172.21.15.37:0/3527906361 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:13.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.185+0000 7fd566a00640 1 --2- 172.21.15.37:0/3527906361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219f5dd0 0x55f4219faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:13.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.185+0000 7fd567400640 1 -- 172.21.15.37:0/3527906361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4219fe460 msgr2=0x55f4219fb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:13.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.185+0000 7fd567400640 1 --2- 172.21.15.37:0/3527906361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4219fe460 0x55f4219fb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.186+0000 7fd567400640 1 -- 172.21.15.37:0/3527906361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219f5dd0 msgr2=0x55f4219faf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:13.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.186+0000 7fd567400640 1 --2- 172.21.15.37:0/3527906361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219f5dd0 0x55f4219faf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.186+0000 7fd567400640 1 -- 172.21.15.37:0/3527906361 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f4217377d0 con 0x55f4219f50f0 2024-03-28T18:13:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.186+0000 7fd567e00640 1 --2- 172.21.15.37:0/3527906361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4219fe460 0x55f4219fb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:13:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.186+0000 7fd567400640 1 --2- 172.21.15.37:0/3527906361 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f4219f50f0 0x55f4219fdb70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd558012cf0 tx=0x7fd55800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f31d8282606addd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.186+0000 7fd551600640 10 client.0 ms_handle_connect on v2:172.21.15.37:3300/0 2024-03-28T18:13:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.187+0000 7fd551600640 1 -- 172.21.15.37:0/3527906361 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd558012110 con 0x55f4219f50f0 2024-03-28T18:13:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.187+0000 7fd551600640 1 -- 172.21.15.37:0/3527906361 <== mon.0 v2:172.21.15.37:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd558004050 con 0x55f4219f50f0 2024-03-28T18:13:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.187+0000 7fd5715c8740 1 -- 172.21.15.37:0/3527906361 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd5540c13d0 con 0x55f4219f50f0 2024-03-28T18:13:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.187+0000 7fd551600640 1 -- 172.21.15.37:0/3527906361 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5580122b0 con 0x55f4219f50f0 2024-03-28T18:13:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.187+0000 7fd5715c8740 10 client.15693 fetch_fsmap learned FSMap version 24 2024-03-28T18:13:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.187+0000 7fd5715c8740 1 -- 172.21.15.37:0/3527906361 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55f42166d3f0 con 0x55f4219f50f0 2024-03-28T18:13:13.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.188+0000 7fd551600640 1 -- 172.21.15.37:0/3527906361 <== mon.0 v2:172.21.15.37:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5540c13d0 con 0x55f4219f50f0 2024-03-28T18:13:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.189+0000 7fd551600640 1 -- 172.21.15.37:0/3527906361 <== mon.0 v2:172.21.15.37:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd558025450 con 0x55f4219f50f0 2024-03-28T18:13:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.189+0000 7fd5715c8740 10 client.15693 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T18:13:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.189+0000 7fd5715c8740 10 client.15693 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:13:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.189+0000 7fd5715c8740 20 client.15693 populate_metadata read hostname 'smithi037' 2024-03-28T18:13:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.189+0000 7fd5715c8740 1 --2- 172.21.15.37:0/3527906361 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55f421a0e040 0x55f421a10430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.189+0000 7fd5715c8740 4 client.15693 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:13:13.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.189+0000 7fd5715c8740 1 -- 172.21.15.37:0/3527906361 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f421966790 con 0x55f421a0e040 2024-03-28T18:13:13.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.190+0000 7fd566a00640 1 --2- 172.21.15.37:0/3527906361 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55f421a0e040 0x55f421a10430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:13.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.191+0000 7fd551600640 10 client.15693 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:13:13.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.191+0000 7fd566a00640 1 --2- 172.21.15.37:0/3527906361 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55f421a0e040 0x55f421a10430 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a01fe5859bae5a87 server_cookie=92143d71954d38b1 in_seq=0 out_seq=0 2024-03-28T18:13:13.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.193+0000 7fd551600640 1 -- 172.21.15.37:0/3527906361 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f421966790 con 0x55f421a0e040 2024-03-28T18:13:13.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.193+0000 7fd551600640 10 client.15693 handle_command_reply: tid=0 2024-03-28T18:13:13.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.193+0000 7fd5715c8740 1 client.15693 shutdown 2024-03-28T18:13:13.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.193+0000 7fd5715c8740 1 -- 172.21.15.37:0/3527906361 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f4219f50f0 msgr2=0x55f4219fdb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:13.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.193+0000 7fd5715c8740 1 --2- 172.21.15.37:0/3527906361 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f4219f50f0 0x55f4219fdb70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd558012cf0 tx=0x7fd55800f790 comp rx=0 tx=0).stop 2024-03-28T18:13:13.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.193+0000 7fd5715c8740 1 -- 172.21.15.37:0/3527906361 shutdown_connections 2024-03-28T18:13:13.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.193+0000 7fd5715c8740 1 --2- 172.21.15.37:0/3527906361 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55f421a0e040 0x55f421a10430 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.193+0000 7fd5715c8740 1 --2- 172.21.15.37:0/3527906361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219f5dd0 0x55f4219faf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.193+0000 7fd5715c8740 1 --2- 172.21.15.37:0/3527906361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4219fe460 0x55f4219fb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.193+0000 7fd5715c8740 1 --2- 172.21.15.37:0/3527906361 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f4219f50f0 0x55f4219fdb70 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.193+0000 7fd5715c8740 1 -- 172.21.15.37:0/3527906361 >> 172.21.15.37:0/3527906361 conn(0x55f4219bc200 msgr2=0x55f4219f1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:13.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.193+0000 7fd551600640 0 client.15693 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:13:13.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.194+0000 7fd5715c8740 1 -- 172.21.15.37:0/3527906361 shutdown_connections 2024-03-28T18:13:13.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.194+0000 7fd5715c8740 1 -- 172.21.15.37:0/3527906361 wait complete. 2024-03-28T18:13:13.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.194+0000 7fd5715c8740 20 client.15693 trim_cache size 0 max 16384 2024-03-28T18:13:13.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.224+0000 7fd5715c8740 1 Processor -- start 2024-03-28T18:13:13.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.225+0000 7fd5715c8740 1 -- start start 2024-03-28T18:13:13.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.225+0000 7fd5715c8740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219f5dd0 0x55f4219fd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:13.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.225+0000 7fd5715c8740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f4219fe460 0x55f4219fdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:13.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.225+0000 7fd5715c8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f421a15ce0 0x55f4219ff850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:13.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.225+0000 7fd5715c8740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5681b5c30 con 0x55f4219fe460 2024-03-28T18:13:13.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.225+0000 7fd5715c8740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5681b3730 con 0x55f4219f5dd0 2024-03-28T18:13:13.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.225+0000 7fd5715c8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5681b17f0 con 0x55f421a15ce0 2024-03-28T18:13:13.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.225+0000 7fd567400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219f5dd0 0x55f4219fd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:13.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.225+0000 7fd567400640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219f5dd0 0x55f4219fd630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:35218/0 (socket says 172.21.15.37:35218) 2024-03-28T18:13:13.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.225+0000 7fd567400640 1 -- 172.21.15.37:0/2717970541 learned_addr learned my addr 172.21.15.37:0/2717970541 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:13.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.225+0000 7fd567e00640 1 --2- 172.21.15.37:0/2717970541 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f421a15ce0 0x55f4219ff850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:13.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.225+0000 7fd566a00640 1 --2- 172.21.15.37:0/2717970541 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f4219fe460 0x55f4219fdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:13.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.226+0000 7fd567e00640 1 -- 172.21.15.37:0/2717970541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219f5dd0 msgr2=0x55f4219fd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:13.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.226+0000 7fd567e00640 1 --2- 172.21.15.37:0/2717970541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219f5dd0 0x55f4219fd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.226+0000 7fd567e00640 1 -- 172.21.15.37:0/2717970541 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f4219fe460 msgr2=0x55f4219fdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:13.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.226+0000 7fd567e00640 1 --2- 172.21.15.37:0/2717970541 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f4219fe460 0x55f4219fdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.226+0000 7fd567e00640 1 -- 172.21.15.37:0/2717970541 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f42162d0f0 con 0x55f421a15ce0 2024-03-28T18:13:13.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.226+0000 7fd567e00640 1 --2- 172.21.15.37:0/2717970541 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f421a15ce0 0x55f4219ff850 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd56000fa90 tx=0x7fd56001c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5f7fa9f3ef17c93 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:13.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.227+0000 7fd553e00640 1 -- 172.21.15.37:0/2717970541 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd560022040 con 0x55f421a15ce0 2024-03-28T18:13:13.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.227+0000 7fd553e00640 1 -- 172.21.15.37:0/2717970541 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd56002b070 con 0x55f421a15ce0 2024-03-28T18:13:13.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.227+0000 7fd553e00640 1 -- 172.21.15.37:0/2717970541 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56000e3b0 con 0x55f421a15ce0 2024-03-28T18:13:13.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.228+0000 7fd5715c8740 1 -- 172.21.15.37:0/2717970541 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f421a15ce0 msgr2=0x55f4219ff850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:13.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.228+0000 7fd5715c8740 1 --2- 172.21.15.37:0/2717970541 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f421a15ce0 0x55f4219ff850 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd56000fa90 tx=0x7fd56001c000 comp rx=0 tx=0).stop 2024-03-28T18:13:13.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.228+0000 7fd5715c8740 1 -- 172.21.15.37:0/2717970541 shutdown_connections 2024-03-28T18:13:13.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.228+0000 7fd5715c8740 1 --2- 172.21.15.37:0/2717970541 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f421a15ce0 0x55f4219ff850 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.228+0000 7fd5715c8740 1 --2- 172.21.15.37:0/2717970541 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f4219fe460 0x55f4219fdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.228+0000 7fd5715c8740 1 --2- 172.21.15.37:0/2717970541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219f5dd0 0x55f4219fd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.228+0000 7fd5715c8740 1 -- 172.21.15.37:0/2717970541 >> 172.21.15.37:0/2717970541 conn(0x55f4219bc200 msgr2=0x55f4219eebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:13.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.228+0000 7fd5715c8740 1 -- 172.21.15.37:0/2717970541 shutdown_connections 2024-03-28T18:13:13.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.228+0000 7fd5715c8740 1 -- 172.21.15.37:0/2717970541 wait complete. 2024-03-28T18:13:13.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.229+0000 7fd5715c8740 1 -- start start 2024-03-28T18:13:13.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.229+0000 7fd5715c8740 1 Processor -- start 2024-03-28T18:13:13.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.230+0000 7fd5715c8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4219f5dd0 0x55f421a01390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:13.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.230+0000 7fd5715c8740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219fe460 0x55f421a018d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:13.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.230+0000 7fd5715c8740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f421a15ce0 0x55f421a01e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:13.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.230+0000 7fd5715c8740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5681b1df0 con 0x55f421a15ce0 2024-03-28T18:13:13.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.230+0000 7fd5715c8740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5681b30d0 con 0x55f4219fe460 2024-03-28T18:13:13.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.230+0000 7fd5715c8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5681b2410 con 0x55f4219f5dd0 2024-03-28T18:13:13.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.230+0000 7fd5715c8740 10 client.0 _ll_register_callbacks cb 0x7fd558009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:13:13.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.230+0000 7fd566a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219fe460 0x55f421a018d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:13.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.230+0000 7fd567400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4219f5dd0 0x55f421a01390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:13.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.230+0000 7fd566a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219fe460 0x55f421a018d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:35224/0 (socket says 172.21.15.37:35224) 2024-03-28T18:13:13.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.230+0000 7fd566a00640 1 -- 172.21.15.37:0/1798953761 learned_addr learned my addr 172.21.15.37:0/1798953761 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:13.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.230+0000 7fd567e00640 1 --2- 172.21.15.37:0/1798953761 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f421a15ce0 0x55f421a01e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:13.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.230+0000 7fd566a00640 1 -- 172.21.15.37:0/1798953761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4219f5dd0 msgr2=0x55f421a01390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:13.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.230+0000 7fd566a00640 1 --2- 172.21.15.37:0/1798953761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4219f5dd0 0x55f421a01390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.230+0000 7fd566a00640 1 -- 172.21.15.37:0/1798953761 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f421a15ce0 msgr2=0x55f421a01e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:13.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.230+0000 7fd566a00640 1 --2- 172.21.15.37:0/1798953761 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f421a15ce0 0x55f421a01e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.231+0000 7fd566a00640 1 -- 172.21.15.37:0/1798953761 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f421a0a220 con 0x55f4219fe460 2024-03-28T18:13:13.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.231+0000 7fd567e00640 1 --2- 172.21.15.37:0/1798953761 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f421a15ce0 0x55f421a01e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:13:13.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.231+0000 7fd567400640 1 --2- 172.21.15.37:0/1798953761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4219f5dd0 0x55f421a01390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:13:13.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.231+0000 7fd566a00640 1 --2- 172.21.15.37:0/1798953761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219fe460 0x55f421a018d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd5540b96e0 tx=0x7fd554005460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2c7775cdfc02e85 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:13.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.231+0000 7fd551600640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-28T18:13:13.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.231+0000 7fd551600640 1 -- 172.21.15.37:0/1798953761 <== mon.2 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5540c13d0 con 0x55f4219fe460 2024-03-28T18:13:13.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.231+0000 7fd551600640 1 -- 172.21.15.37:0/1798953761 <== mon.2 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd5540bb580 con 0x55f4219fe460 2024-03-28T18:13:13.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.231+0000 7fd5715c8740 1 -- 172.21.15.37:0/1798953761 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd56002b070 con 0x55f4219fe460 2024-03-28T18:13:13.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.231+0000 7fd551600640 1 -- 172.21.15.37:0/1798953761 <== mon.2 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5540dd0c0 con 0x55f4219fe460 2024-03-28T18:13:13.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.231+0000 7fd5715c8740 10 client.25886 fetch_fsmap learned FSMap version 24 2024-03-28T18:13:13.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.232+0000 7fd5715c8740 1 -- 172.21.15.37:0/1798953761 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55f421a6a990 con 0x55f4219fe460 2024-03-28T18:13:13.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.232+0000 7fd551600640 1 -- 172.21.15.37:0/1798953761 <== mon.2 v2:172.21.15.59:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd56002b070 con 0x55f4219fe460 2024-03-28T18:13:13.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.232+0000 7fd551600640 1 -- 172.21.15.37:0/1798953761 <== mon.2 v2:172.21.15.59:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd5540b98f0 con 0x55f4219fe460 2024-03-28T18:13:13.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.232+0000 7fd5715c8740 10 client.25886 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T18:13:13.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.232+0000 7fd5715c8740 10 client.25886 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:13:13.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.232+0000 7fd5715c8740 20 client.25886 populate_metadata read hostname 'smithi037' 2024-03-28T18:13:13.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.232+0000 7fd5715c8740 1 --2- 172.21.15.37:0/1798953761 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55f421a6ae60 0x55f421a6d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:13.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.232+0000 7fd5715c8740 4 client.25886 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:13:13.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.232+0000 7fd5715c8740 1 -- 172.21.15.37:0/1798953761 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f421960ae0 con 0x55f421a6ae60 2024-03-28T18:13:13.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.233+0000 7fd567400640 1 --2- 172.21.15.37:0/1798953761 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55f421a6ae60 0x55f421a6d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:13.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.234+0000 7fd567400640 1 --2- 172.21.15.37:0/1798953761 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55f421a6ae60 0x55f421a6d280 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d780ac51a363333f server_cookie=a97680731ffeb542 in_seq=0 out_seq=0 2024-03-28T18:13:13.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.234+0000 7fd551600640 10 client.25886 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:13:13.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.235+0000 7fd551600640 1 -- 172.21.15.37:0/1798953761 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+267 (crc 0 0 0) 0x55f421960ae0 con 0x55f421a6ae60 2024-03-28T18:13:13.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.235+0000 7fd551600640 10 client.25886 handle_command_reply: tid=0 2024-03-28T18:13:13.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.235+0000 7fd5715c8740 1 client.25886 shutdown 2024-03-28T18:13:13.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.236+0000 7fd5715c8740 1 -- 172.21.15.37:0/1798953761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219fe460 msgr2=0x55f421a018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:13.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.236+0000 7fd5715c8740 1 --2- 172.21.15.37:0/1798953761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219fe460 0x55f421a018d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd5540b96e0 tx=0x7fd554005460 comp rx=0 tx=0).stop 2024-03-28T18:13:13.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.236+0000 7fd5715c8740 1 -- 172.21.15.37:0/1798953761 shutdown_connections 2024-03-28T18:13:13.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.236+0000 7fd5715c8740 1 --2- 172.21.15.37:0/1798953761 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x55f421a6ae60 0x55f421a6d280 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.236+0000 7fd5715c8740 1 --2- 172.21.15.37:0/1798953761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f4219fe460 0x55f421a018d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.236+0000 7fd5715c8740 1 --2- 172.21.15.37:0/1798953761 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55f421a15ce0 0x55f421a01e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.236+0000 7fd5715c8740 1 --2- 172.21.15.37:0/1798953761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4219f5dd0 0x55f421a01390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.236+0000 7fd5715c8740 1 -- 172.21.15.37:0/1798953761 >> 172.21.15.37:0/1798953761 conn(0x55f4219bc200 msgr2=0x55f421a14cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:13.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.236+0000 7fd551600640 0 client.25886 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:13:13.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.236+0000 7fd5715c8740 1 -- 172.21.15.37:0/1798953761 shutdown_connections 2024-03-28T18:13:13.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.236+0000 7fd5715c8740 1 -- 172.21.15.37:0/1798953761 wait complete. 2024-03-28T18:13:13.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.236+0000 7fd5715c8740 20 client.25886 trim_cache size 0 max 16384 2024-03-28T18:13:13.240 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:13:13.240 INFO:teuthology.orchestra.run.smithi037.stdout: "status": "scrub active (893 inodes in the stack)", 2024-03-28T18:13:13.240 INFO:teuthology.orchestra.run.smithi037.stdout: "scrubs": { 2024-03-28T18:13:13.240 INFO:teuthology.orchestra.run.smithi037.stdout: "3d49dbce-f1b2-47ce-aaff-eb788d7d2af7": { 2024-03-28T18:13:13.240 INFO:teuthology.orchestra.run.smithi037.stdout: "path": "/", 2024-03-28T18:13:13.240 INFO:teuthology.orchestra.run.smithi037.stdout: "tag": "3d49dbce-f1b2-47ce-aaff-eb788d7d2af7", 2024-03-28T18:13:13.240 INFO:teuthology.orchestra.run.smithi037.stdout: "options": "recursive,force" 2024-03-28T18:13:13.241 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-03-28T18:13:13.241 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-03-28T18:13:13.241 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:13:13.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.237+0000 7fd56d200640 1 -- 172.21.15.37:0/445213875 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fd54c0a29f0 msgr2=0x7fd54c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:13.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.237+0000 7fd56d200640 1 --2- 172.21.15.37:0/445213875 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fd54c0a29f0 0x7fd54c0a4eb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd5600099a0 tx=0x7fd560009500 comp rx=0 tx=0).stop 2024-03-28T18:13:13.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.237+0000 7fd56d200640 1 -- 172.21.15.37:0/445213875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5681040e0 msgr2=0x7fd56819a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:13.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.237+0000 7fd56d200640 1 --2- 172.21.15.37:0/445213875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5681040e0 0x7fd56819a5d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd554036040 tx=0x7fd554004250 comp rx=0 tx=0).stop 2024-03-28T18:13:13.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.238+0000 7fd56d200640 1 -- 172.21.15.37:0/445213875 shutdown_connections 2024-03-28T18:13:13.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.238+0000 7fd56d200640 1 --2- 172.21.15.37:0/445213875 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fd54c0a29f0 0x7fd54c0a4eb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.238+0000 7fd56d200640 1 --2- 172.21.15.37:0/445213875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5681040e0 0x7fd56819a5d0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.238+0000 7fd56d200640 1 --2- 172.21.15.37:0/445213875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd568109730 0x7fd56819e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.238+0000 7fd56d200640 1 --2- 172.21.15.37:0/445213875 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd568103720 0x7fd56819a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:13.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.238+0000 7fd56d200640 1 -- 172.21.15.37:0/445213875 >> 172.21.15.37:0/445213875 conn(0x7fd5680ff1b0 msgr2=0x7fd5680ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:13.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.238+0000 7fd56d200640 1 -- 172.21.15.37:0/445213875 shutdown_connections 2024-03-28T18:13:13.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:13.238+0000 7fd56d200640 1 -- 172.21.15.37:0/445213875 wait complete. 2024-03-28T18:13:13.261 INFO:tasks.cephfs.filesystem:scrub status for tag:3d49dbce-f1b2-47ce-aaff-eb788d7d2af7 - {'path': '/', 'tag': '3d49dbce-f1b2-47ce-aaff-eb788d7d2af7', 'options': 'recursive,force'} 2024-03-28T18:13:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:13 smithi059 ceph-mon[36190]: pgmap v64: 97 pgs: 97 active+clean; 18 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 168 op/s 2024-03-28T18:13:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:13 smithi059 ceph-mon[36190]: scrub summary: active paths [/] 2024-03-28T18:13:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:13 smithi059 ceph-mon[36190]: scrub summary: idle+waiting paths [/] 2024-03-28T18:13:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:13 smithi059 ceph-mon[36190]: scrub summary: active paths [/] 2024-03-28T18:13:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:13 smithi059 ceph-mon[36190]: scrub summary: idle+waiting paths [/] 2024-03-28T18:13:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:13 smithi059 ceph-mon[36190]: scrub summary: active paths [/] 2024-03-28T18:13:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:13 smithi059 ceph-mon[36190]: scrub summary: idle+waiting paths [/] 2024-03-28T18:13:13.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:13 smithi170 ceph-mon[36280]: pgmap v64: 97 pgs: 97 active+clean; 18 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 168 op/s 2024-03-28T18:13:13.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:13 smithi170 ceph-mon[36280]: scrub summary: active paths [/] 2024-03-28T18:13:13.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:13 smithi170 ceph-mon[36280]: scrub summary: idle+waiting paths [/] 2024-03-28T18:13:13.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:13 smithi170 ceph-mon[36280]: scrub summary: active paths [/] 2024-03-28T18:13:13.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:13 smithi170 ceph-mon[36280]: scrub summary: idle+waiting paths [/] 2024-03-28T18:13:13.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:13 smithi170 ceph-mon[36280]: scrub summary: active paths [/] 2024-03-28T18:13:13.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:13 smithi170 ceph-mon[36280]: scrub summary: idle+waiting paths [/] 2024-03-28T18:13:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:13 smithi037 ceph-mon[31460]: pgmap v64: 97 pgs: 97 active+clean; 18 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 168 op/s 2024-03-28T18:13:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:13 smithi037 ceph-mon[31460]: scrub summary: active paths [/] 2024-03-28T18:13:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:13 smithi037 ceph-mon[31460]: scrub summary: idle+waiting paths [/] 2024-03-28T18:13:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:13 smithi037 ceph-mon[31460]: scrub summary: active paths [/] 2024-03-28T18:13:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:13 smithi037 ceph-mon[31460]: scrub summary: idle+waiting paths [/] 2024-03-28T18:13:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:13 smithi037 ceph-mon[31460]: scrub summary: active paths [/] 2024-03-28T18:13:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:13 smithi037 ceph-mon[31460]: scrub summary: idle+waiting paths [/] 2024-03-28T18:13:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:14 smithi059 ceph-mon[36190]: scrub queued for path: / 2024-03-28T18:13:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:14 smithi059 ceph-mon[36190]: scrub summary: idle+waiting paths [/] 2024-03-28T18:13:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:14 smithi059 ceph-mon[36190]: scrub summary: active paths [/] 2024-03-28T18:13:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:14 smithi059 ceph-mon[36190]: scrub summary: active paths [/] 2024-03-28T18:13:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:14 smithi059 ceph-mon[36190]: scrub summary: idle+waiting paths [/] 2024-03-28T18:13:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:14 smithi059 ceph-mon[36190]: scrub summary: idle+waiting paths [/] 2024-03-28T18:13:14.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:14 smithi170 ceph-mon[36280]: scrub queued for path: / 2024-03-28T18:13:14.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:14 smithi170 ceph-mon[36280]: scrub summary: idle+waiting paths [/] 2024-03-28T18:13:14.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:14 smithi170 ceph-mon[36280]: scrub summary: active paths [/] 2024-03-28T18:13:14.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:14 smithi170 ceph-mon[36280]: scrub summary: active paths [/] 2024-03-28T18:13:14.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:14 smithi170 ceph-mon[36280]: scrub summary: idle+waiting paths [/] 2024-03-28T18:13:14.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:14 smithi170 ceph-mon[36280]: scrub summary: idle+waiting paths [/] 2024-03-28T18:13:14.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:14 smithi037 ceph-mon[31460]: scrub queued for path: / 2024-03-28T18:13:14.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:14 smithi037 ceph-mon[31460]: scrub summary: idle+waiting paths [/] 2024-03-28T18:13:14.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:14 smithi037 ceph-mon[31460]: scrub summary: active paths [/] 2024-03-28T18:13:14.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:14 smithi037 ceph-mon[31460]: scrub summary: active paths [/] 2024-03-28T18:13:14.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:14 smithi037 ceph-mon[31460]: scrub summary: idle+waiting paths [/] 2024-03-28T18:13:14.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:14 smithi037 ceph-mon[31460]: scrub summary: idle+waiting paths [/] 2024-03-28T18:13:15.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:15 smithi059 ceph-mon[36190]: pgmap v65: 97 pgs: 97 active+clean; 18 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 411 KiB/s wr, 125 op/s 2024-03-28T18:13:15.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:15 smithi170 ceph-mon[36280]: pgmap v65: 97 pgs: 97 active+clean; 18 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 411 KiB/s wr, 125 op/s 2024-03-28T18:13:15.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:15 smithi037 ceph-mon[31460]: pgmap v65: 97 pgs: 97 active+clean; 18 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 411 KiB/s wr, 125 op/s 2024-03-28T18:13:17.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:17 smithi059 ceph-mon[36190]: pgmap v66: 97 pgs: 97 active+clean; 24 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 1.1 MiB/s wr, 311 op/s 2024-03-28T18:13:17.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:17 smithi170 ceph-mon[36280]: pgmap v66: 97 pgs: 97 active+clean; 24 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 1.1 MiB/s wr, 311 op/s 2024-03-28T18:13:17.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:17 smithi037 ceph-mon[31460]: pgmap v66: 97 pgs: 97 active+clean; 24 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 1.1 MiB/s wr, 311 op/s 2024-03-28T18:13:19.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:19 smithi037 ceph-mon[31460]: pgmap v67: 97 pgs: 97 active+clean; 24 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 923 KiB/s wr, 253 op/s 2024-03-28T18:13:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:19 smithi059 ceph-mon[36190]: pgmap v67: 97 pgs: 97 active+clean; 24 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 923 KiB/s wr, 253 op/s 2024-03-28T18:13:20.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:19 smithi170 ceph-mon[36280]: pgmap v67: 97 pgs: 97 active+clean; 24 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 923 KiB/s wr, 253 op/s 2024-03-28T18:13:21.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:21 smithi037 ceph-mon[31460]: pgmap v68: 97 pgs: 97 active+clean; 24 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 923 KiB/s wr, 253 op/s 2024-03-28T18:13:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:21 smithi037 ceph-mon[31460]: scrub summary: idle 2024-03-28T18:13:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:21 smithi037 ceph-mon[31460]: scrub summary: idle 2024-03-28T18:13:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:21 smithi037 ceph-mon[31460]: scrub summary: idle 2024-03-28T18:13:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:21 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:13:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:21 smithi059 ceph-mon[36190]: pgmap v68: 97 pgs: 97 active+clean; 24 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 923 KiB/s wr, 253 op/s 2024-03-28T18:13:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:21 smithi059 ceph-mon[36190]: scrub summary: idle 2024-03-28T18:13:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:21 smithi059 ceph-mon[36190]: scrub summary: idle 2024-03-28T18:13:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:21 smithi059 ceph-mon[36190]: scrub summary: idle 2024-03-28T18:13:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:21 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:13:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:21 smithi170 ceph-mon[36280]: pgmap v68: 97 pgs: 97 active+clean; 24 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 923 KiB/s wr, 253 op/s 2024-03-28T18:13:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:21 smithi170 ceph-mon[36280]: scrub summary: idle 2024-03-28T18:13:22.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:21 smithi170 ceph-mon[36280]: scrub summary: idle 2024-03-28T18:13:22.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:21 smithi170 ceph-mon[36280]: scrub summary: idle 2024-03-28T18:13:22.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:21 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:13:23.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:23 smithi037 ceph-mon[31460]: pgmap v69: 97 pgs: 97 active+clean; 31 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 1.6 MiB/s wr, 380 op/s 2024-03-28T18:13:23.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:23 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:13:23.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:23 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:13:23.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:23 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:13:24.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:23 smithi059 ceph-mon[36190]: pgmap v69: 97 pgs: 97 active+clean; 31 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 1.6 MiB/s wr, 380 op/s 2024-03-28T18:13:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:23 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:13:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:23 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:13:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:23 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:13:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:23 smithi170 ceph-mon[36280]: pgmap v69: 97 pgs: 97 active+clean; 31 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 1.6 MiB/s wr, 380 op/s 2024-03-28T18:13:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:23 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:13:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:23 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:13:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:23 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:13:25.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:25 smithi037 ceph-mon[31460]: pgmap v70: 97 pgs: 97 active+clean; 31 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.4 MiB/s wr, 313 op/s 2024-03-28T18:13:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:25 smithi059 ceph-mon[36190]: pgmap v70: 97 pgs: 97 active+clean; 31 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.4 MiB/s wr, 313 op/s 2024-03-28T18:13:26.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:25 smithi170 ceph-mon[36280]: pgmap v70: 97 pgs: 97 active+clean; 31 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.4 MiB/s wr, 313 op/s 2024-03-28T18:13:27.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:27 smithi037 ceph-mon[31460]: pgmap v71: 97 pgs: 97 active+clean; 34 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 1.7 MiB/s wr, 376 op/s 2024-03-28T18:13:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:27 smithi059 ceph-mon[36190]: pgmap v71: 97 pgs: 97 active+clean; 34 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 1.7 MiB/s wr, 376 op/s 2024-03-28T18:13:28.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:27 smithi170 ceph-mon[36280]: pgmap v71: 97 pgs: 97 active+clean; 34 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 1.7 MiB/s wr, 376 op/s 2024-03-28T18:13:29.837 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:29 smithi170 ceph-mon[36280]: pgmap v72: 97 pgs: 97 active+clean; 34 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 1.0 MiB/s wr, 189 op/s 2024-03-28T18:13:29.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:29 smithi037 ceph-mon[31460]: pgmap v72: 97 pgs: 97 active+clean; 34 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 1.0 MiB/s wr, 189 op/s 2024-03-28T18:13:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:29 smithi059 ceph-mon[36190]: pgmap v72: 97 pgs: 97 active+clean; 34 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 1.0 MiB/s wr, 189 op/s 2024-03-28T18:13:31.837 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:31 smithi059 ceph-mon[36190]: pgmap v73: 97 pgs: 97 active+clean; 34 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 1.0 MiB/s wr, 189 op/s 2024-03-28T18:13:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:31 smithi037 ceph-mon[31460]: pgmap v73: 97 pgs: 97 active+clean; 34 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 1.0 MiB/s wr, 189 op/s 2024-03-28T18:13:32.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:31 smithi170 ceph-mon[36280]: pgmap v73: 97 pgs: 97 active+clean; 34 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 1.0 MiB/s wr, 189 op/s 2024-03-28T18:13:33.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:33 smithi037 ceph-mon[31460]: pgmap v74: 97 pgs: 97 active+clean; 35 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 1.2 MiB/s wr, 238 op/s 2024-03-28T18:13:34.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:33 smithi059 ceph-mon[36190]: pgmap v74: 97 pgs: 97 active+clean; 35 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 1.2 MiB/s wr, 238 op/s 2024-03-28T18:13:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:33 smithi170 ceph-mon[36280]: pgmap v74: 97 pgs: 97 active+clean; 35 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 1.2 MiB/s wr, 238 op/s 2024-03-28T18:13:35.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:35 smithi037 ceph-mon[31460]: pgmap v75: 97 pgs: 97 active+clean; 35 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 495 KiB/s wr, 111 op/s 2024-03-28T18:13:35.880 INFO:tasks.workunit.client.0.smithi037.stderr:Updating files: 2% (120/4277) Updating files: 3% (129/4277) Updating files: 4% (172/4277) Updating files: 4% (195/4277) Updating files: 5% (214/4277) Updating files: 6% (257/4277) Updating files: 7% (300/4277) Updating files: 7% (319/4277) Updating files: 8% (343/4277) Updating files: 9% (385/4277) Updating files: 10% (428/4277) Updating files: 10% (434/4277) Updating files: 11% (471/4277) Updating files: 12% (514/4277) Updating files: 12% (548/4277) Updating files: 13% (557/4277) Updating files: 14% (599/4277) Updating files: 15% (642/4277) Updating files: 15% (668/4277) Updating files: 16% (685/4277) Updating files: 17% (728/4277) Updating files: 18% (770/4277) Updating files: 18% (787/4277) Updating files: 19% (813/4277) Updating files: 20% (856/4277) Updating files: 21% (899/4277) Updating files: 21% (902/4277) Updating files: 22% (941/4277) Updating files: 23% (984/4277) Updating files: 23% (1023/4277) Updating files: 24% (1027/4277) Updating files: 25% (1070/4277) Updating files: 26% (1113/4277) Updating files: 26% (1145/4277) Updating files: 27% (1155/4277) Updating files: 28% (1198/4277) Updating files: 29% (1241/4277) Updating files: 29% (1276/4277) Updating files: 30% (1284/4277) Updating files: 31% (1326/4277) Updating files: 32% (1369/4277) Updating files: 33% (1412/4277) Updating files: 34% (1455/4277) Updating files: 34% (1476/4277) Updating files: 35% (1497/4277) Updating files: 36% (1540/4277) Updating files: 37% (1583/4277) Updating files: 38% (1626/4277) Updating files: 38% (1659/4277) Updating files: 39% (1669/4277) Updating files: 40% (1711/4277) Updating files: 41% (1754/4277) Updating files: 42% (1797/4277) Updating files: 42% (1803/4277) Updating files: 43% (1840/4277) Updating files: 44% (1882/4277) Updating files: 45% (1925/4277) Updating files: 46% (1968/4277) Updating files: 46% (1997/4277) Updating files: 47% (2011/4277) Updating files: 48% (2053/4277) Updating files: 49% (2096/4277) Updating files: 49% (2131/4277) Updating files: 50% (2139/4277) Updating files: 51% (2182/4277) Updating files: 52% (2225/4277) Updating files: 52% (2258/4277) Updating files: 53% (2267/4277) Updating files: 53% (2270/4277) Updating files: 54% (2310/4277) Updating files: 55% (2353/4277) Updating files: 56% (2396/4277) Updating files: 56% (2428/4277) Updating files: 57% (2438/4277) Updating files: 58% (2481/4277) Updating files: 59% (2524/4277) Updating files: 59% (2555/4277) Updating files: 60% (2567/4277) Updating files: 61% (2609/4277) Updating files: 61% (2633/4277) Updating files: 62% (2652/4277) Updating files: 62% (2666/4277) Updating files: 63% (2695/4277) Updating files: 64% (2738/4277) Updating files: 65% (2781/4277) Updating files: 66% (2823/4277) Updating files: 66% (2862/4277) Updating files: 67% (2866/4277) Updating files: 68% (2909/4277) Updating files: 69% (2952/4277) Updating files: 69% (2989/4277) Updating files: 70% (2994/4277) Updating files: 71% (3037/4277) Updating files: 72% (3080/4277) Updating files: 72% (3102/4277) Updating files: 73% (3123/4277) Updating files: 74% (3165/4277) Updating files: 75% (3208/4277) Updating files: 75% (3213/4277) Updating files: 76% (3251/4277) Updating files: 77% (3294/4277) Updating files: 77% (3328/4277) Updating files: 78% (3337/4277) Updating files: 79% (3379/4277) Updating files: 80% (3422/4277) Updating files: 80% (3441/4277) Updating files: 81% (3465/4277) Updating files: 82% (3508/4277) Updating files: 83% (3550/4277) Updating files: 83% (3555/4277) Updating files: 84% (3593/4277) Updating files: 85% (3636/4277) Updating files: 85% (3662/4277) Updating files: 86% (3679/4277) Updating files: 87% (3721/4277) Updating files: 87% (3761/4277) Updating files: 88% (3764/4277) Updating files: 89% (3807/4277) Updating files: 90% (3850/4277) Updating files: 90% (3865/4277) Updating files: 91% (3893/4277) Updating files: 92% (3935/4277) Updating files: 93% (3978/4277) Updating files: 93% (3981/4277) Updating files: 94% (4021/4277) Updating files: 95% (4064/4277) Updating files: 95% (4099/4277) Updating files: 96% (4106/4277) Updating files: 97% (4149/4277) Updating files: 98% (4192/4277) Updating files: 98% (4198/4277) Updating files: 99% (4235/4277) Updating files: 100% (4277/4277) Updating files: 100% (4277/4277), done. 2024-03-28T18:13:35.897 INFO:tasks.workunit.client.0.smithi037.stderr:+ cd xfstests-dev 2024-03-28T18:13:35.898 INFO:tasks.workunit.client.0.smithi037.stderr:+ git checkout 12973fc04fd10d4af086901e10ffa8e48866b735 2024-03-28T18:13:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:35 smithi059 ceph-mon[36190]: pgmap v75: 97 pgs: 97 active+clean; 35 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 495 KiB/s wr, 111 op/s 2024-03-28T18:13:36.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:35 smithi170 ceph-mon[36280]: pgmap v75: 97 pgs: 97 active+clean; 35 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 495 KiB/s wr, 111 op/s 2024-03-28T18:13:37.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:37 smithi037 ceph-mon[31460]: pgmap v76: 97 pgs: 97 active+clean; 38 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 779 KiB/s wr, 171 op/s 2024-03-28T18:13:38.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:37 smithi059 ceph-mon[36190]: pgmap v76: 97 pgs: 97 active+clean; 38 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 779 KiB/s wr, 171 op/s 2024-03-28T18:13:38.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:37 smithi170 ceph-mon[36280]: pgmap v76: 97 pgs: 97 active+clean; 38 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 779 KiB/s wr, 171 op/s 2024-03-28T18:13:39.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:39 smithi037 ceph-mon[31460]: pgmap v77: 97 pgs: 97 active+clean; 38 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 430 KiB/s wr, 109 op/s 2024-03-28T18:13:40.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:39 smithi059 ceph-mon[36190]: pgmap v77: 97 pgs: 97 active+clean; 38 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 430 KiB/s wr, 109 op/s 2024-03-28T18:13:40.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:39 smithi170 ceph-mon[36280]: pgmap v77: 97 pgs: 97 active+clean; 38 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 430 KiB/s wr, 109 op/s 2024-03-28T18:13:40.440 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f03046bc700 time 2024-03-28T18:13:40.064404+0000 2024-03-28T18:13:40.441 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:13:40.441 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:13:40.441 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f030ece64a1] 2024-03-28T18:13:40.441 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f030ece6667] 2024-03-28T18:13:40.441 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5580f06f5223] 2024-03-28T18:13:40.441 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5580f056f5d6] 2024-03-28T18:13:40.441 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5580f056fda1] 2024-03-28T18:13:40.441 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5580f05722d5] 2024-03-28T18:13:40.441 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5580f05c2233] 2024-03-28T18:13:40.441 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5580f05c4197] 2024-03-28T18:13:40.441 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5580f04c311b] 2024-03-28T18:13:40.442 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5580f04c36bc] 2024-03-28T18:13:40.442 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5580f04ad1ff] 2024-03-28T18:13:40.442 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f030ef78488] 2024-03-28T18:13:40.442 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 13: (DispatchQueue::entry()+0x51f) [0x7f030ef7560f] 2024-03-28T18:13:40.442 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f030f048e81] 2024-03-28T18:13:40.442 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f030da631ca] 2024-03-28T18:13:40.442 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 16: clone() 2024-03-28T18:13:40.442 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2024-03-28T18:13:40.066+0000 7f03046bc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f03046bc700 time 2024-03-28T18:13:40.064404+0000 2024-03-28T18:13:40.442 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:13:40.442 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2024-03-28T18:13:40.442 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:13:40.442 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f030ece64a1] 2024-03-28T18:13:40.443 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f030ece6667] 2024-03-28T18:13:40.443 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5580f06f5223] 2024-03-28T18:13:40.443 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5580f056f5d6] 2024-03-28T18:13:40.443 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5580f056fda1] 2024-03-28T18:13:40.443 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5580f05722d5] 2024-03-28T18:13:40.443 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5580f05c2233] 2024-03-28T18:13:40.443 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5580f05c4197] 2024-03-28T18:13:40.443 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5580f04c311b] 2024-03-28T18:13:40.443 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5580f04c36bc] 2024-03-28T18:13:40.443 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5580f04ad1ff] 2024-03-28T18:13:40.443 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f030ef78488] 2024-03-28T18:13:40.443 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 13: (DispatchQueue::entry()+0x51f) [0x7f030ef7560f] 2024-03-28T18:13:40.443 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f030f048e81] 2024-03-28T18:13:40.443 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f030da631ca] 2024-03-28T18:13:40.444 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 16: clone() 2024-03-28T18:13:40.444 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2024-03-28T18:13:40.444 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: *** Caught signal (Aborted) ** 2024-03-28T18:13:40.444 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: in thread 7f03046bc700 thread_name:ms_dispatch 2024-03-28T18:13:40.444 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:13:40.444 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f030da6dd20] 2024-03-28T18:13:40.445 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2: gsignal() 2024-03-28T18:13:40.445 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 3: abort() 2024-03-28T18:13:40.445 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f030ece64fb] 2024-03-28T18:13:40.445 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f030ece6667] 2024-03-28T18:13:40.445 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5580f06f5223] 2024-03-28T18:13:40.445 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5580f056f5d6] 2024-03-28T18:13:40.445 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5580f056fda1] 2024-03-28T18:13:40.446 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5580f05722d5] 2024-03-28T18:13:40.446 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5580f05c2233] 2024-03-28T18:13:40.446 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5580f05c4197] 2024-03-28T18:13:40.446 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5580f04c311b] 2024-03-28T18:13:40.446 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5580f04c36bc] 2024-03-28T18:13:40.446 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5580f04ad1ff] 2024-03-28T18:13:40.446 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f030ef78488] 2024-03-28T18:13:40.446 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 16: (DispatchQueue::entry()+0x51f) [0x7f030ef7560f] 2024-03-28T18:13:40.446 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f030f048e81] 2024-03-28T18:13:40.446 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f030da631ca] 2024-03-28T18:13:40.446 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 19: clone() 2024-03-28T18:13:40.446 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2024-03-28T18:13:40.067+0000 7f03046bc700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:13:40.446 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: in thread 7f03046bc700 thread_name:ms_dispatch 2024-03-28T18:13:40.446 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2024-03-28T18:13:40.447 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:13:40.447 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f030da6dd20] 2024-03-28T18:13:40.447 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2: gsignal() 2024-03-28T18:13:40.447 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 3: abort() 2024-03-28T18:13:40.447 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f030ece64fb] 2024-03-28T18:13:40.447 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f030ece6667] 2024-03-28T18:13:40.447 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5580f06f5223] 2024-03-28T18:13:40.447 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5580f056f5d6] 2024-03-28T18:13:40.447 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5580f056fda1] 2024-03-28T18:13:40.447 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5580f05722d5] 2024-03-28T18:13:40.447 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5580f05c2233] 2024-03-28T18:13:40.447 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5580f05c4197] 2024-03-28T18:13:40.447 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5580f04c311b] 2024-03-28T18:13:40.447 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5580f04c36bc] 2024-03-28T18:13:40.448 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5580f04ad1ff] 2024-03-28T18:13:40.448 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f030ef78488] 2024-03-28T18:13:40.448 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 16: (DispatchQueue::entry()+0x51f) [0x7f030ef7560f] 2024-03-28T18:13:40.448 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f030f048e81] 2024-03-28T18:13:40.448 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f030da631ca] 2024-03-28T18:13:40.448 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 19: clone() 2024-03-28T18:13:40.448 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:13:40.448 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2024-03-28T18:13:40.448 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: -1> 2024-03-28T18:13:40.066+0000 7f03046bc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f03046bc700 time 2024-03-28T18:13:40.064404+0000 2024-03-28T18:13:40.448 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:13:40.448 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2024-03-28T18:13:40.448 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:13:40.448 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f030ece64a1] 2024-03-28T18:13:40.448 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f030ece6667] 2024-03-28T18:13:40.449 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5580f06f5223] 2024-03-28T18:13:40.449 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5580f056f5d6] 2024-03-28T18:13:40.449 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5580f056fda1] 2024-03-28T18:13:40.449 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5580f05722d5] 2024-03-28T18:13:40.449 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5580f05c2233] 2024-03-28T18:13:40.449 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5580f05c4197] 2024-03-28T18:13:40.449 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5580f04c311b] 2024-03-28T18:13:40.449 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5580f04c36bc] 2024-03-28T18:13:40.449 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5580f04ad1ff] 2024-03-28T18:13:40.449 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f030ef78488] 2024-03-28T18:13:40.449 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 13: (DispatchQueue::entry()+0x51f) [0x7f030ef7560f] 2024-03-28T18:13:40.449 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f030f048e81] 2024-03-28T18:13:40.449 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f030da631ca] 2024-03-28T18:13:40.449 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 16: clone() 2024-03-28T18:13:40.450 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2024-03-28T18:13:40.450 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 0> 2024-03-28T18:13:40.067+0000 7f03046bc700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:13:40.450 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: in thread 7f03046bc700 thread_name:ms_dispatch 2024-03-28T18:13:40.450 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2024-03-28T18:13:40.450 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:13:40.450 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f030da6dd20] 2024-03-28T18:13:40.450 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2: gsignal() 2024-03-28T18:13:40.450 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 3: abort() 2024-03-28T18:13:40.450 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f030ece64fb] 2024-03-28T18:13:40.450 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f030ece6667] 2024-03-28T18:13:40.450 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5580f06f5223] 2024-03-28T18:13:40.450 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5580f056f5d6] 2024-03-28T18:13:40.450 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5580f056fda1] 2024-03-28T18:13:40.450 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5580f05722d5] 2024-03-28T18:13:40.450 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5580f05c2233] 2024-03-28T18:13:40.451 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5580f05c4197] 2024-03-28T18:13:40.451 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5580f04c311b] 2024-03-28T18:13:40.451 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5580f04c36bc] 2024-03-28T18:13:40.451 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5580f04ad1ff] 2024-03-28T18:13:40.451 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f030ef78488] 2024-03-28T18:13:40.451 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 16: (DispatchQueue::entry()+0x51f) [0x7f030ef7560f] 2024-03-28T18:13:40.451 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f030f048e81] 2024-03-28T18:13:40.451 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f030da631ca] 2024-03-28T18:13:40.451 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 19: clone() 2024-03-28T18:13:40.451 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:13:40.451 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2024-03-28T18:13:40.451 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: -1> 2024-03-28T18:13:40.066+0000 7f03046bc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f03046bc700 time 2024-03-28T18:13:40.064404+0000 2024-03-28T18:13:40.451 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:13:40.451 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2024-03-28T18:13:40.451 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:13:40.451 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f030ece64a1] 2024-03-28T18:13:40.452 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f030ece6667] 2024-03-28T18:13:40.452 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5580f06f5223] 2024-03-28T18:13:40.452 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5580f056f5d6] 2024-03-28T18:13:40.452 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5580f056fda1] 2024-03-28T18:13:40.452 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5580f05722d5] 2024-03-28T18:13:40.452 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5580f05c2233] 2024-03-28T18:13:40.452 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5580f05c4197] 2024-03-28T18:13:40.452 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5580f04c311b] 2024-03-28T18:13:40.452 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5580f04c36bc] 2024-03-28T18:13:40.452 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5580f04ad1ff] 2024-03-28T18:13:40.452 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f030ef78488] 2024-03-28T18:13:40.452 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 13: (DispatchQueue::entry()+0x51f) [0x7f030ef7560f] 2024-03-28T18:13:40.452 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f030f048e81] 2024-03-28T18:13:40.452 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f030da631ca] 2024-03-28T18:13:40.452 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 16: clone() 2024-03-28T18:13:40.453 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2024-03-28T18:13:40.453 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 0> 2024-03-28T18:13:40.067+0000 7f03046bc700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:13:40.453 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: in thread 7f03046bc700 thread_name:ms_dispatch 2024-03-28T18:13:40.453 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2024-03-28T18:13:40.453 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:13:40.453 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f030da6dd20] 2024-03-28T18:13:40.453 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2: gsignal() 2024-03-28T18:13:40.453 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 3: abort() 2024-03-28T18:13:40.453 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f030ece64fb] 2024-03-28T18:13:40.453 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f030ece6667] 2024-03-28T18:13:40.453 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5580f06f5223] 2024-03-28T18:13:40.453 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5580f056f5d6] 2024-03-28T18:13:40.453 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5580f056fda1] 2024-03-28T18:13:40.453 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5580f05722d5] 2024-03-28T18:13:40.453 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5580f05c2233] 2024-03-28T18:13:40.454 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5580f05c4197] 2024-03-28T18:13:40.454 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5580f04c311b] 2024-03-28T18:13:40.454 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5580f04c36bc] 2024-03-28T18:13:40.454 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5580f04ad1ff] 2024-03-28T18:13:40.454 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f030ef78488] 2024-03-28T18:13:40.454 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 16: (DispatchQueue::entry()+0x51f) [0x7f030ef7560f] 2024-03-28T18:13:40.454 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f030f048e81] 2024-03-28T18:13:40.454 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f030da631ca] 2024-03-28T18:13:40.454 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 19: clone() 2024-03-28T18:13:40.454 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:13:40.454 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[66791]: 2024-03-28T18:13:40.454 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:40 smithi059 podman[70487]: 2024-03-28 18:13:40.128040369 +0000 UTC m=+0.019939604 container died 014effd69bd278b531ebf2a39e7be743f1fe0b1449dd873da7ebea38f558f280 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-28T18:13:41.345 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:41 smithi059 podman[70487]: 2024-03-28 18:13:41.093357089 +0000 UTC m=+0.985256332 container remove 014effd69bd278b531ebf2a39e7be743f1fe0b1449dd873da7ebea38f558f280 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-28T18:13:41.345 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:41 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:13:41.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:41 smithi059 ceph-mon[36190]: pgmap v78: 97 pgs: 97 active+clean; 38 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 430 KiB/s wr, 109 op/s 2024-03-28T18:13:41.785 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:41 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.b.service: Failed with result 'exit-code'. 2024-03-28T18:13:41.785 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:41 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.b.service: Consumed 2min 40.741s CPU time. 2024-03-28T18:13:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:41 smithi037 ceph-mon[31460]: pgmap v78: 97 pgs: 97 active+clean; 38 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 430 KiB/s wr, 109 op/s 2024-03-28T18:13:42.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:41 smithi170 ceph-mon[36280]: pgmap v78: 97 pgs: 97 active+clean; 38 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 430 KiB/s wr, 109 op/s 2024-03-28T18:13:43.262 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:13:43.263 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:13:43.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.505+0000 7f30c6e00640 1 -- 172.21.15.37:0/2626957978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30c01004c0 msgr2=0x7f30c01008a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:43.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.505+0000 7f30c6e00640 1 --2- 172.21.15.37:0/2626957978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30c01004c0 0x7f30c01008a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f30b4009900 tx=0x7f30b402bd10 comp rx=0 tx=0).stop 2024-03-28T18:13:43.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.505+0000 7f30c6e00640 1 -- 172.21.15.37:0/2626957978 shutdown_connections 2024-03-28T18:13:43.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.505+0000 7f30c6e00640 1 --2- 172.21.15.37:0/2626957978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30c0100de0 0x7f30c01052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.505+0000 7f30c6e00640 1 --2- 172.21.15.37:0/2626957978 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30c01058c0 0x7f30c0105d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.505+0000 7f30c6e00640 1 --2- 172.21.15.37:0/2626957978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30c01004c0 0x7f30c01008a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.505+0000 7f30c6e00640 1 -- 172.21.15.37:0/2626957978 >> 172.21.15.37:0/2626957978 conn(0x7f30c00fbef0 msgr2=0x7f30c00fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:43.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.505+0000 7f30c6e00640 1 -- 172.21.15.37:0/2626957978 shutdown_connections 2024-03-28T18:13:43.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.505+0000 7f30c6e00640 1 -- 172.21.15.37:0/2626957978 wait complete. 2024-03-28T18:13:43.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.506+0000 7f30c6e00640 1 Processor -- start 2024-03-28T18:13:43.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.507+0000 7f30c6e00640 1 -- start start 2024-03-28T18:13:43.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.507+0000 7f30c6e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30c01004c0 0x7f30c019e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.507+0000 7f30c6e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30c0100de0 0x7f30c019e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.507+0000 7f30c6e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30c01058c0 0x7f30c01a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.507+0000 7f30c6e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c0117f50 con 0x7f30c01058c0 2024-03-28T18:13:43.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.507+0000 7f30c6e00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c0117dd0 con 0x7f30c0100de0 2024-03-28T18:13:43.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.507+0000 7f30c6e00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c01180d0 con 0x7f30c01004c0 2024-03-28T18:13:43.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.507+0000 7f30c5a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30c01004c0 0x7f30c019e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.507+0000 7f30c5a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30c01004c0 0x7f30c019e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:59346/0 (socket says 172.21.15.37:59346) 2024-03-28T18:13:43.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.507+0000 7f30c5a00640 1 -- 172.21.15.37:0/2202079510 learned_addr learned my addr 172.21.15.37:0/2202079510 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:43.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.507+0000 7f30c6400640 1 --2- 172.21.15.37:0/2202079510 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30c01058c0 0x7f30c01a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.508+0000 7f30c5000640 1 --2- 172.21.15.37:0/2202079510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30c0100de0 0x7f30c019e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.508+0000 7f30c5a00640 1 -- 172.21.15.37:0/2202079510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30c0100de0 msgr2=0x7f30c019e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:43.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.508+0000 7f30c5a00640 1 --2- 172.21.15.37:0/2202079510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30c0100de0 0x7f30c019e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.508+0000 7f30c5a00640 1 -- 172.21.15.37:0/2202079510 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30c01058c0 msgr2=0x7f30c01a2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:43.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.508+0000 7f30c5a00640 1 --2- 172.21.15.37:0/2202079510 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30c01058c0 0x7f30c01a2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.508+0000 7f30c5a00640 1 -- 172.21.15.37:0/2202079510 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c01a33c0 con 0x7f30c01004c0 2024-03-28T18:13:43.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.508+0000 7f30c5000640 1 --2- 172.21.15.37:0/2202079510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30c0100de0 0x7f30c019e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:13:43.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.508+0000 7f30c5a00640 1 --2- 172.21.15.37:0/2202079510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30c01004c0 0x7f30c019e3d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f30b4000c00 tx=0x7f30b4004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:43.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.508+0000 7f30be000640 1 -- 172.21.15.37:0/2202079510 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b4042020 con 0x7f30c01004c0 2024-03-28T18:13:43.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.508+0000 7f30be000640 1 -- 172.21.15.37:0/2202079510 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30b4005000 con 0x7f30c01004c0 2024-03-28T18:13:43.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.509+0000 7f30be000640 1 -- 172.21.15.37:0/2202079510 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b4003b20 con 0x7f30c01004c0 2024-03-28T18:13:43.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.509+0000 7f30c6e00640 1 -- 172.21.15.37:0/2202079510 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c01a3650 con 0x7f30c01004c0 2024-03-28T18:13:43.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.509+0000 7f30c6e00640 1 -- 172.21.15.37:0/2202079510 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c01aaf40 con 0x7f30c01004c0 2024-03-28T18:13:43.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.512+0000 7f30be000640 1 -- 172.21.15.37:0/2202079510 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f30b4045020 con 0x7f30c01004c0 2024-03-28T18:13:43.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.512+0000 7f30cb009740 1 Processor -- start 2024-03-28T18:13:43.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.512+0000 7f30cb009740 1 -- start start 2024-03-28T18:13:43.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.512+0000 7f30cb009740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd04dd0 0x562e3bd03b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.512+0000 7f30cb009740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562e3bd040f0 0x562e3bd04570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.513+0000 7f30cb009740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd0d460 0x562e3bd0f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.513+0000 7f30cb009740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e3bc68250 con 0x562e3bd0d460 2024-03-28T18:13:43.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.513+0000 7f30cb009740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e3bc10a30 con 0x562e3bd040f0 2024-03-28T18:13:43.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.513+0000 7f30cb009740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e3bc642f0 con 0x562e3bd04dd0 2024-03-28T18:13:43.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.513+0000 7f30c6400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd04dd0 0x562e3bd03b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.513+0000 7f30c6400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd04dd0 0x562e3bd03b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:59360/0 (socket says 172.21.15.37:59360) 2024-03-28T18:13:43.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.513+0000 7f30c6400640 1 -- 172.21.15.37:0/285958787 learned_addr learned my addr 172.21.15.37:0/285958787 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:43.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.513+0000 7f30c5000640 1 --2- 172.21.15.37:0/285958787 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd0d460 0x562e3bd0f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.513+0000 7f30c5a00640 1 --2- 172.21.15.37:0/285958787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562e3bd040f0 0x562e3bd04570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.513+0000 7f30c6400640 1 -- 172.21.15.37:0/285958787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562e3bd040f0 msgr2=0x562e3bd04570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:43.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.513+0000 7f30c6400640 1 --2- 172.21.15.37:0/285958787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562e3bd040f0 0x562e3bd04570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.513+0000 7f30c6400640 1 -- 172.21.15.37:0/285958787 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd0d460 msgr2=0x562e3bd0f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:43.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.513+0000 7f30c6400640 1 --2- 172.21.15.37:0/285958787 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd0d460 0x562e3bd0f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.513+0000 7f30c6400640 1 -- 172.21.15.37:0/285958787 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562e3bbfb690 con 0x562e3bd04dd0 2024-03-28T18:13:43.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.514+0000 7f30c6400640 1 --2- 172.21.15.37:0/285958787 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd04dd0 0x562e3bd03b20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009f50 tx=0x7f30b80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e482ad9a6c9fb90b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:43.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.514+0000 7f30b2a00640 1 -- 172.21.15.37:0/285958787 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8004d20 con 0x562e3bd04dd0 2024-03-28T18:13:43.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.514+0000 7f30b2a00640 1 -- 172.21.15.37:0/285958787 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30b8004050 con 0x562e3bd04dd0 2024-03-28T18:13:43.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.514+0000 7f30b2a00640 1 -- 172.21.15.37:0/285958787 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8005280 con 0x562e3bd04dd0 2024-03-28T18:13:43.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.514+0000 7f30be000640 1 --2- 172.21.15.37:0/2202079510 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f30a80a29f0 0x7f30a80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.514+0000 7f30be000640 1 -- 172.21.15.37:0/2202079510 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f30b40f0960 con 0x7f30c01004c0 2024-03-28T18:13:43.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.514+0000 7f30c5a00640 1 --2- 172.21.15.37:0/2202079510 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f30a80a29f0 0x7f30a80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.514+0000 7f30cb009740 1 -- 172.21.15.37:0/285958787 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd04dd0 msgr2=0x562e3bd03b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:43.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.514+0000 7f30cb009740 1 --2- 172.21.15.37:0/285958787 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd04dd0 0x562e3bd03b20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009f50 tx=0x7f30b80118d0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.515+0000 7f30cb009740 1 -- 172.21.15.37:0/285958787 shutdown_connections 2024-03-28T18:13:43.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.515+0000 7f30cb009740 1 --2- 172.21.15.37:0/285958787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562e3bd040f0 0x562e3bd04570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.515+0000 7f30cb009740 1 --2- 172.21.15.37:0/285958787 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd0d460 0x562e3bd0f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.515+0000 7f30cb009740 1 --2- 172.21.15.37:0/285958787 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd04dd0 0x562e3bd03b20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.515+0000 7f30cb009740 1 -- 172.21.15.37:0/285958787 >> 172.21.15.37:0/285958787 conn(0x562e3bccb200 msgr2=0x562e3bcfeb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:43.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.515+0000 7f30cb009740 1 -- 172.21.15.37:0/285958787 shutdown_connections 2024-03-28T18:13:43.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.515+0000 7f30cb009740 1 -- 172.21.15.37:0/285958787 wait complete. 2024-03-28T18:13:43.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.515+0000 7f30c5a00640 1 --2- 172.21.15.37:0/2202079510 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f30a80a29f0 0x7f30a80a4eb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x562e3bd05320 tx=0x7f30b40dd040 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:43.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.515+0000 7f30cb009740 1 -- start start 2024-03-28T18:13:43.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.516+0000 7f30cb009740 1 Processor -- start 2024-03-28T18:13:43.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.516+0000 7f30cb009740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd040f0 0x562e3bd0cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.516+0000 7f30cb009740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd04dd0 0x562e3bd09f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.516+0000 7f30cb009740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562e3bd0d460 0x562e3bd0a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.516+0000 7f30cb009740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e3bd102d0 con 0x562e3bd04dd0 2024-03-28T18:13:43.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.516+0000 7f30cb009740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e3bc23e80 con 0x562e3bd0d460 2024-03-28T18:13:43.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.516+0000 7f30cb009740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e3bd10450 con 0x562e3bd040f0 2024-03-28T18:13:43.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.516+0000 7f30cb009740 10 client.0 _ll_register_callbacks cb 0x562e3bc36130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:13:43.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.516+0000 7f30c6400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd040f0 0x562e3bd0cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.516+0000 7f30c6400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd040f0 0x562e3bd0cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:59366/0 (socket says 172.21.15.37:59366) 2024-03-28T18:13:43.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.516+0000 7f30c6400640 1 -- 172.21.15.37:0/1733365849 learned_addr learned my addr 172.21.15.37:0/1733365849 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:43.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.516+0000 7f30c5000640 1 --2- 172.21.15.37:0/1733365849 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd04dd0 0x562e3bd09f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.517+0000 7f30c6400640 1 --2- 172.21.15.37:0/1733365849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562e3bd0d460 0x562e3bd0a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.517+0000 7f30c6400640 1 -- 172.21.15.37:0/1733365849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562e3bd0d460 msgr2=0x562e3bd0a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:43.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.517+0000 7f30c6400640 1 --2- 172.21.15.37:0/1733365849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562e3bd0d460 0x562e3bd0a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.517+0000 7f30c6400640 1 -- 172.21.15.37:0/1733365849 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd04dd0 msgr2=0x562e3bd09f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:43.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.517+0000 7f30c6400640 1 --2- 172.21.15.37:0/1733365849 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd04dd0 0x562e3bd09f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.517+0000 7f30c6400640 1 -- 172.21.15.37:0/1733365849 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562e3ba467d0 con 0x562e3bd040f0 2024-03-28T18:13:43.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.517+0000 7f30c6400640 1 --2- 172.21.15.37:0/1733365849 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd040f0 0x562e3bd0cb90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009250 tx=0x7f30b80096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a49037fb5e64216 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:43.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.517+0000 7f30a7400640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:13:43.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.517+0000 7f30a7400640 1 -- 172.21.15.37:0/1733365849 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b802b070 con 0x562e3bd040f0 2024-03-28T18:13:43.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.518+0000 7f30a7400640 1 -- 172.21.15.37:0/1733365849 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f30b8025450 con 0x562e3bd040f0 2024-03-28T18:13:43.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.518+0000 7f30cb009740 1 -- 172.21.15.37:0/1733365849 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f30b8004050 con 0x562e3bd040f0 2024-03-28T18:13:43.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.518+0000 7f30a7400640 1 -- 172.21.15.37:0/1733365849 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30b8003e40 con 0x562e3bd040f0 2024-03-28T18:13:43.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.518+0000 7f30cb009740 10 client.25912 fetch_fsmap learned FSMap version 24 2024-03-28T18:13:43.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.518+0000 7f30cb009740 1 -- 172.21.15.37:0/1733365849 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x562e3b97c3f0 con 0x562e3bd040f0 2024-03-28T18:13:43.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.518+0000 7f30a7400640 1 -- 172.21.15.37:0/1733365849 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30b8004050 con 0x562e3bd040f0 2024-03-28T18:13:43.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.518+0000 7f30a7400640 1 -- 172.21.15.37:0/1733365849 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f30b8027740 con 0x562e3bd040f0 2024-03-28T18:13:43.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.518+0000 7f30cb009740 10 client.25912 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T18:13:43.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.518+0000 7f30cb009740 10 client.25912 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:13:43.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.519+0000 7f30cb009740 20 client.25912 populate_metadata read hostname 'smithi037' 2024-03-28T18:13:43.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.519+0000 7f30cb009740 1 --2- 172.21.15.37:0/1733365849 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x562e3bd1d2c0 0x562e3bd1f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.519+0000 7f30cb009740 4 client.25912 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:13:43.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.519+0000 7f30cb009740 1 -- 172.21.15.37:0/1733365849 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562e3bc75790 con 0x562e3bd1d2c0 2024-03-28T18:13:43.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.519+0000 7f30c5000640 1 --2- 172.21.15.37:0/1733365849 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x562e3bd1d2c0 0x562e3bd1f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.519+0000 7f30c5000640 1 --2- 172.21.15.37:0/1733365849 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x562e3bd1d2c0 0x562e3bd1f6b0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d70c167159489f server_cookie=38dfe57a032680e7 in_seq=0 out_seq=0 2024-03-28T18:13:43.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.519+0000 7f30a7400640 10 client.25912 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:13:43.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.520+0000 7f30a7400640 1 -- 172.21.15.37:0/1733365849 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562e3bc75790 con 0x562e3bd1d2c0 2024-03-28T18:13:43.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.520+0000 7f30a7400640 10 client.25912 handle_command_reply: tid=0 2024-03-28T18:13:43.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.520+0000 7f30cb009740 1 client.25912 shutdown 2024-03-28T18:13:43.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.520+0000 7f30cb009740 1 -- 172.21.15.37:0/1733365849 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd040f0 msgr2=0x562e3bd0cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:43.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.520+0000 7f30cb009740 1 --2- 172.21.15.37:0/1733365849 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd040f0 0x562e3bd0cb90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009250 tx=0x7f30b80096c0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.521+0000 7f30cb009740 1 -- 172.21.15.37:0/1733365849 shutdown_connections 2024-03-28T18:13:43.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.521+0000 7f30cb009740 1 --2- 172.21.15.37:0/1733365849 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x562e3bd1d2c0 0x562e3bd1f6b0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.521+0000 7f30cb009740 1 --2- 172.21.15.37:0/1733365849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562e3bd0d460 0x562e3bd0a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.521+0000 7f30cb009740 1 --2- 172.21.15.37:0/1733365849 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd04dd0 0x562e3bd09f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.521+0000 7f30cb009740 1 --2- 172.21.15.37:0/1733365849 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd040f0 0x562e3bd0cb90 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.521+0000 7f30a7400640 0 client.25912 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:13:43.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.521+0000 7f30cb009740 1 -- 172.21.15.37:0/1733365849 >> 172.21.15.37:0/1733365849 conn(0x562e3bccb200 msgr2=0x562e3bd00890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:43.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.521+0000 7f30cb009740 1 -- 172.21.15.37:0/1733365849 shutdown_connections 2024-03-28T18:13:43.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.521+0000 7f30cb009740 1 -- 172.21.15.37:0/1733365849 wait complete. 2024-03-28T18:13:43.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.521+0000 7f30cb009740 20 client.25912 trim_cache size 0 max 16384 2024-03-28T18:13:43.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.542+0000 7f30cb009740 1 Processor -- start 2024-03-28T18:13:43.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.542+0000 7f30cb009740 1 -- start start 2024-03-28T18:13:43.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.542+0000 7f30cb009740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd040f0 0x562e3bd0c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.543+0000 7f30cb009740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562e3bd04dd0 0x562e3bd0cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.543+0000 7f30cb009740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd0d460 0x562e3bd18ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.543+0000 7f30cb009740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c01b5a80 con 0x562e3bd040f0 2024-03-28T18:13:43.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.543+0000 7f30cb009740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c01b3580 con 0x562e3bd04dd0 2024-03-28T18:13:43.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.543+0000 7f30cb009740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c01b1620 con 0x562e3bd0d460 2024-03-28T18:13:43.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.543+0000 7f30c6400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd040f0 0x562e3bd0c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.543+0000 7f30c6400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd040f0 0x562e3bd0c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56104/0 (socket says 172.21.15.37:56104) 2024-03-28T18:13:43.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.543+0000 7f30c6400640 1 -- 172.21.15.37:0/2704667809 learned_addr learned my addr 172.21.15.37:0/2704667809 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:43.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.543+0000 7f30c5000640 1 --2- 172.21.15.37:0/2704667809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562e3bd04dd0 0x562e3bd0cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.543+0000 7f30c6400640 1 --2- 172.21.15.37:0/2704667809 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd0d460 0x562e3bd18ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.543+0000 7f30c6400640 1 -- 172.21.15.37:0/2704667809 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd0d460 msgr2=0x562e3bd18ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:43.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.543+0000 7f30c6400640 1 --2- 172.21.15.37:0/2704667809 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd0d460 0x562e3bd18ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.543+0000 7f30c6400640 1 -- 172.21.15.37:0/2704667809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562e3bd04dd0 msgr2=0x562e3bd0cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:43.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.543+0000 7f30c6400640 1 --2- 172.21.15.37:0/2704667809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562e3bd04dd0 0x562e3bd0cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.543+0000 7f30c6400640 1 -- 172.21.15.37:0/2704667809 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562e3b93c0f0 con 0x562e3bd040f0 2024-03-28T18:13:43.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.544+0000 7f30c6400640 1 --2- 172.21.15.37:0/2704667809 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd040f0 0x562e3bd0c650 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009bf0 tx=0x7f30b8003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73cfae9440603377 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:43.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.544+0000 7f30b2000640 1 -- 172.21.15.37:0/2704667809 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b802b070 con 0x562e3bd040f0 2024-03-28T18:13:43.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.544+0000 7f30b2000640 1 -- 172.21.15.37:0/2704667809 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30b8027920 con 0x562e3bd040f0 2024-03-28T18:13:43.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.544+0000 7f30b2000640 1 -- 172.21.15.37:0/2704667809 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b800fb60 con 0x562e3bd040f0 2024-03-28T18:13:43.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.545+0000 7f30cb009740 1 -- 172.21.15.37:0/2704667809 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd040f0 msgr2=0x562e3bd0c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:43.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.545+0000 7f30cb009740 1 --2- 172.21.15.37:0/2704667809 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd040f0 0x562e3bd0c650 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009bf0 tx=0x7f30b8003970 comp rx=0 tx=0).stop 2024-03-28T18:13:43.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.545+0000 7f30cb009740 1 -- 172.21.15.37:0/2704667809 shutdown_connections 2024-03-28T18:13:43.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.545+0000 7f30cb009740 1 --2- 172.21.15.37:0/2704667809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562e3bd04dd0 0x562e3bd0cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.545+0000 7f30cb009740 1 --2- 172.21.15.37:0/2704667809 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd0d460 0x562e3bd18ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.545+0000 7f30cb009740 1 --2- 172.21.15.37:0/2704667809 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd040f0 0x562e3bd0c650 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.545+0000 7f30cb009740 1 -- 172.21.15.37:0/2704667809 >> 172.21.15.37:0/2704667809 conn(0x562e3bccb200 msgr2=0x562e3bcfe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:43.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.545+0000 7f30cb009740 1 -- 172.21.15.37:0/2704667809 shutdown_connections 2024-03-28T18:13:43.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.545+0000 7f30cb009740 1 -- 172.21.15.37:0/2704667809 wait complete. 2024-03-28T18:13:43.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.545+0000 7f30cb009740 1 -- start start 2024-03-28T18:13:43.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.546+0000 7f30cb009740 1 Processor -- start 2024-03-28T18:13:43.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.546+0000 7f30cb009740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd040f0 0x562e3bd0eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.546+0000 7f30cb009740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd04dd0 0x562e3bd071c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.546+0000 7f30cb009740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562e3bd0d460 0x562e3bd07700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.546+0000 7f30cb009740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c01b1c20 con 0x562e3bd040f0 2024-03-28T18:13:43.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.546+0000 7f30cb009740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c01b2f20 con 0x562e3bd0d460 2024-03-28T18:13:43.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.546+0000 7f30cb009740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c01b2260 con 0x562e3bd04dd0 2024-03-28T18:13:43.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.546+0000 7f30cb009740 10 client.0 _ll_register_callbacks cb 0x7f30c0069860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:13:43.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.546+0000 7f30c6400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd040f0 0x562e3bd0eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.546+0000 7f30c6400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd040f0 0x562e3bd0eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56116/0 (socket says 172.21.15.37:56116) 2024-03-28T18:13:43.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.546+0000 7f30c6400640 1 -- 172.21.15.37:0/1538344880 learned_addr learned my addr 172.21.15.37:0/1538344880 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:43.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.546+0000 7f30c5000640 1 --2- 172.21.15.37:0/1538344880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd04dd0 0x562e3bd071c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.547+0000 7f30c6400640 1 --2- 172.21.15.37:0/1538344880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562e3bd0d460 0x562e3bd07700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.547+0000 7f30c5000640 1 -- 172.21.15.37:0/1538344880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562e3bd0d460 msgr2=0x562e3bd07700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:43.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.547+0000 7f30c5000640 1 --2- 172.21.15.37:0/1538344880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562e3bd0d460 0x562e3bd07700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.547+0000 7f30c5000640 1 -- 172.21.15.37:0/1538344880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd040f0 msgr2=0x562e3bd0eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:43.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.547+0000 7f30c5000640 1 --2- 172.21.15.37:0/1538344880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd040f0 0x562e3bd0eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.547+0000 7f30c5000640 1 -- 172.21.15.37:0/1538344880 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562e3bd07e70 con 0x562e3bd04dd0 2024-03-28T18:13:43.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.547+0000 7f30c6400640 1 --2- 172.21.15.37:0/1538344880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd040f0 0x562e3bd0eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:13:43.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.547+0000 7f30c5000640 1 --2- 172.21.15.37:0/1538344880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd04dd0 0x562e3bd071c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x562e3bd1fd10 tx=0x7f30ac021dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f69f11272fc5fb07 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:43.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.547+0000 7f30a7400640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:13:43.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.547+0000 7f30a7400640 1 -- 172.21.15.37:0/1538344880 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ac012b80 con 0x562e3bd04dd0 2024-03-28T18:13:43.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.547+0000 7f30cb009740 1 -- 172.21.15.37:0/1538344880 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f30b8027920 con 0x562e3bd04dd0 2024-03-28T18:13:43.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.547+0000 7f30a7400640 1 -- 172.21.15.37:0/1538344880 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f30ac012d20 con 0x562e3bd04dd0 2024-03-28T18:13:43.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.548+0000 7f30a7400640 1 -- 172.21.15.37:0/1538344880 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30ac02a420 con 0x562e3bd04dd0 2024-03-28T18:13:43.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.548+0000 7f30cb009740 10 client.25918 fetch_fsmap learned FSMap version 24 2024-03-28T18:13:43.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.548+0000 7f30cb009740 1 -- 172.21.15.37:0/1538344880 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x562e3bd78990 con 0x562e3bd04dd0 2024-03-28T18:13:43.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.548+0000 7f30a7400640 1 -- 172.21.15.37:0/1538344880 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30b8027920 con 0x562e3bd04dd0 2024-03-28T18:13:43.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.548+0000 7f30a7400640 1 -- 172.21.15.37:0/1538344880 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f30ac033570 con 0x562e3bd04dd0 2024-03-28T18:13:43.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.548+0000 7f30cb009740 10 client.25918 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T18:13:43.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.548+0000 7f30cb009740 10 client.25918 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-28T18:13:43.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.548+0000 7f30cb009740 20 client.25918 populate_metadata read hostname 'smithi037' 2024-03-28T18:13:43.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.548+0000 7f30cb009740 1 --2- 172.21.15.37:0/1538344880 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x562e3bd790d0 0x562e3bd7b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.548+0000 7f30cb009740 4 client.25918 mds_command: new command op to 24580 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:13:43.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.548+0000 7f30cb009740 1 -- 172.21.15.37:0/1538344880 --> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562e3bc6fae0 con 0x562e3bd790d0 2024-03-28T18:13:43.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.549+0000 7f30c6400640 1 --2- 172.21.15.37:0/1538344880 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x562e3bd790d0 0x562e3bd7b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.549+0000 7f30c6400640 1 --2- 172.21.15.37:0/1538344880 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x562e3bd790d0 0x562e3bd7b4f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d29dd79da73285 server_cookie=79deb2432f99d744 in_seq=0 out_seq=0 2024-03-28T18:13:43.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.549+0000 7f30a7400640 10 client.25918 ms_handle_connect on v2:172.21.15.37:6840/1923263622 2024-03-28T18:13:43.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.550+0000 7f30a7400640 1 -- 172.21.15.37:0/1538344880 <== mds.0 v2:172.21.15.37:6840/1923263622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x562e3bc6fae0 con 0x562e3bd790d0 2024-03-28T18:13:43.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.550+0000 7f30a7400640 10 client.25918 handle_command_reply: tid=0 2024-03-28T18:13:43.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.550+0000 7f30cb009740 1 client.25918 shutdown 2024-03-28T18:13:43.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.550+0000 7f30cb009740 1 -- 172.21.15.37:0/1538344880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd04dd0 msgr2=0x562e3bd071c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:43.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.550+0000 7f30cb009740 1 --2- 172.21.15.37:0/1538344880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd04dd0 0x562e3bd071c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x562e3bd1fd10 tx=0x7f30ac021dd0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.550+0000 7f30cb009740 1 -- 172.21.15.37:0/1538344880 shutdown_connections 2024-03-28T18:13:43.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.550+0000 7f30cb009740 1 --2- 172.21.15.37:0/1538344880 >> [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] conn(0x562e3bd790d0 0x562e3bd7b4f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.550+0000 7f30cb009740 1 --2- 172.21.15.37:0/1538344880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x562e3bd0d460 0x562e3bd07700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.550+0000 7f30a7400640 0 client.25918 ms_handle_reset on v2:172.21.15.37:6840/1923263622 2024-03-28T18:13:43.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.550+0000 7f30cb009740 1 --2- 172.21.15.37:0/1538344880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e3bd04dd0 0x562e3bd071c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.550+0000 7f30cb009740 1 --2- 172.21.15.37:0/1538344880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x562e3bd040f0 0x562e3bd0eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.550+0000 7f30cb009740 1 -- 172.21.15.37:0/1538344880 >> 172.21.15.37:0/1538344880 conn(0x562e3bccb200 msgr2=0x562e3bd086e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:43.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.550+0000 7f30cb009740 1 -- 172.21.15.37:0/1538344880 shutdown_connections 2024-03-28T18:13:43.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.550+0000 7f30cb009740 1 -- 172.21.15.37:0/1538344880 wait complete. 2024-03-28T18:13:43.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.550+0000 7f30cb009740 20 client.25918 trim_cache size 0 max 16384 2024-03-28T18:13:43.554 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-03-28T18:13:43.554 INFO:teuthology.orchestra.run.smithi037.stdout: "status": "no active scrubs running", 2024-03-28T18:13:43.554 INFO:teuthology.orchestra.run.smithi037.stdout: "scrubs": {} 2024-03-28T18:13:43.554 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-03-28T18:13:43.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.551+0000 7f30c6e00640 1 -- 172.21.15.37:0/2202079510 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f30a80a29f0 msgr2=0x7f30a80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:43.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.551+0000 7f30c6e00640 1 --2- 172.21.15.37:0/2202079510 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f30a80a29f0 0x7f30a80a4eb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x562e3bd05320 tx=0x7f30b40dd040 comp rx=0 tx=0).stop 2024-03-28T18:13:43.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.551+0000 7f30c6e00640 1 -- 172.21.15.37:0/2202079510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30c01004c0 msgr2=0x7f30c019e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:43.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.551+0000 7f30c6e00640 1 --2- 172.21.15.37:0/2202079510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30c01004c0 0x7f30c019e3d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f30b4000c00 tx=0x7f30b4004360 comp rx=0 tx=0).stop 2024-03-28T18:13:43.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.551+0000 7f30c6e00640 1 -- 172.21.15.37:0/2202079510 shutdown_connections 2024-03-28T18:13:43.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.551+0000 7f30c6e00640 1 --2- 172.21.15.37:0/2202079510 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f30a80a29f0 0x7f30a80a4eb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.551+0000 7f30c6e00640 1 --2- 172.21.15.37:0/2202079510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30c0100de0 0x7f30c019e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.551+0000 7f30c6e00640 1 --2- 172.21.15.37:0/2202079510 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30c01058c0 0x7f30c01a2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.551+0000 7f30c6e00640 1 --2- 172.21.15.37:0/2202079510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30c01004c0 0x7f30c019e3d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.551+0000 7f30c6e00640 1 -- 172.21.15.37:0/2202079510 >> 172.21.15.37:0/2202079510 conn(0x7f30c00fbef0 msgr2=0x7f30c0103270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:43.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.551+0000 7f30c6e00640 1 -- 172.21.15.37:0/2202079510 shutdown_connections 2024-03-28T18:13:43.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.551+0000 7f30c6e00640 1 -- 172.21.15.37:0/2202079510 wait complete. 2024-03-28T18:13:43.568 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:13:43.568 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:13:43.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:43 smithi037 ceph-mon[31460]: pgmap v79: 97 pgs: 97 active+clean; 44 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 280 op/s 2024-03-28T18:13:43.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.795+0000 7fb8c3e00640 1 -- 172.21.15.37:0/2009795912 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8c4068800 msgr2=0x7fb8c4068c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:43.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.795+0000 7fb8c3e00640 1 --2- 172.21.15.37:0/2009795912 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8c4068800 0x7fb8c4068c80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0009900 tx=0x7fb8b002bc30 comp rx=0 tx=0).stop 2024-03-28T18:13:43.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.795+0000 7fb8c3e00640 1 -- 172.21.15.37:0/2009795912 shutdown_connections 2024-03-28T18:13:43.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.795+0000 7fb8c3e00640 1 --2- 172.21.15.37:0/2009795912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8c40691c0 0x7fb8c410e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.795+0000 7fb8c3e00640 1 --2- 172.21.15.37:0/2009795912 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8c4068800 0x7fb8c4068c80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.795+0000 7fb8c3e00640 1 --2- 172.21.15.37:0/2009795912 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8c4107590 0x7fb8c4107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.795+0000 7fb8c3e00640 1 -- 172.21.15.37:0/2009795912 >> 172.21.15.37:0/2009795912 conn(0x7fb8c4075fb0 msgr2=0x7fb8c40763c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:43.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.795+0000 7fb8c3e00640 1 -- 172.21.15.37:0/2009795912 shutdown_connections 2024-03-28T18:13:43.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.795+0000 7fb8c3e00640 1 -- 172.21.15.37:0/2009795912 wait complete. 2024-03-28T18:13:43.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.796+0000 7fb8c3e00640 1 Processor -- start 2024-03-28T18:13:43.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.796+0000 7fb8c3e00640 1 -- start start 2024-03-28T18:13:43.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.797+0000 7fb8c3e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8c4068800 0x7fb8c4197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.797+0000 7fb8c3e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8c40691c0 0x7fb8c4198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.797+0000 7fb8c3e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8c4107590 0x7fb8c419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.797+0000 7fb8c3e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c41118c0 con 0x7fb8c4068800 2024-03-28T18:13:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.797+0000 7fb8c3e00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c4111740 con 0x7fb8c40691c0 2024-03-28T18:13:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.797+0000 7fb8c3e00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c4111a40 con 0x7fb8c4107590 2024-03-28T18:13:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.797+0000 7fb8c3400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8c4107590 0x7fb8c419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.797+0000 7fb8c2a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8c4068800 0x7fb8c4197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.797+0000 7fb8c2000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8c40691c0 0x7fb8c4198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.797+0000 7fb8c3400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8c4107590 0x7fb8c419c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:59386/0 (socket says 172.21.15.37:59386) 2024-03-28T18:13:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.797+0000 7fb8c3400640 1 -- 172.21.15.37:0/261584990 learned_addr learned my addr 172.21.15.37:0/261584990 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.797+0000 7fb8c2000640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8c40691c0 0x7fb8c4198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.37:60326/0 (socket says 172.21.15.37:60326) 2024-03-28T18:13:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.798+0000 7fb8c3400640 1 -- 172.21.15.37:0/261584990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8c40691c0 msgr2=0x7fb8c4198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.798+0000 7fb8c3400640 1 --2- 172.21.15.37:0/261584990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8c40691c0 0x7fb8c4198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.798+0000 7fb8c3400640 1 -- 172.21.15.37:0/261584990 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8c4068800 msgr2=0x7fb8c4197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:43.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.798+0000 7fb8c3400640 1 --2- 172.21.15.37:0/261584990 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8c4068800 0x7fb8c4197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:43.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.798+0000 7fb8c3400640 1 -- 172.21.15.37:0/261584990 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c419ce30 con 0x7fb8c4107590 2024-03-28T18:13:43.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.798+0000 7fb8c3400640 1 --2- 172.21.15.37:0/261584990 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8c4107590 0x7fb8c419c700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc00efd0 tx=0x7fb8bc00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:43.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.798+0000 7fb8bb400640 1 -- 172.21.15.37:0/261584990 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc003b70 con 0x7fb8c4107590 2024-03-28T18:13:43.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.798+0000 7fb8bb400640 1 -- 172.21.15.37:0/261584990 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb8bc003d10 con 0x7fb8c4107590 2024-03-28T18:13:43.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.798+0000 7fb8c3e00640 1 -- 172.21.15.37:0/261584990 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c419d120 con 0x7fb8c4107590 2024-03-28T18:13:43.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.798+0000 7fb8bb400640 1 -- 172.21.15.37:0/261584990 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc0109f0 con 0x7fb8c4107590 2024-03-28T18:13:43.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.798+0000 7fb8c3e00640 1 -- 172.21.15.37:0/261584990 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c4100650 con 0x7fb8c4107590 2024-03-28T18:13:43.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.800+0000 7fb8c3e00640 1 -- 172.21.15.37:0/261584990 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c4102bb0 con 0x7fb8c4107590 2024-03-28T18:13:43.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.801+0000 7fb8bb400640 1 -- 172.21.15.37:0/261584990 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fb8bc004020 con 0x7fb8c4107590 2024-03-28T18:13:43.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.803+0000 7fb8bb400640 1 --2- 172.21.15.37:0/261584990 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fb8a80a2970 0x7fb8a80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:43.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.803+0000 7fb8bb400640 1 -- 172.21.15.37:0/261584990 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb8bc0cf310 con 0x7fb8c4107590 2024-03-28T18:13:43.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.803+0000 7fb8c2a00640 1 --2- 172.21.15.37:0/261584990 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fb8a80a2970 0x7fb8a80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:43.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.804+0000 7fb8c2a00640 1 --2- 172.21.15.37:0/261584990 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fb8a80a2970 0x7fb8a80a4e30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009b50 tx=0x7fb8b4005f90 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:43.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:43.807+0000 7fb8bb400640 1 -- 172.21.15.37:0/261584990 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb8bc09c040 con 0x7fb8c4107590 2024-03-28T18:13:44.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:43 smithi059 ceph-mon[36190]: pgmap v79: 97 pgs: 97 active+clean; 44 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 280 op/s 2024-03-28T18:13:44.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:43 smithi170 ceph-mon[36280]: pgmap v79: 97 pgs: 97 active+clean; 44 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 280 op/s 2024-03-28T18:13:44.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.077+0000 7fb8c3e00640 1 -- 172.21.15.37:0/261584990 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8c41099b0 con 0x7fb8c4107590 2024-03-28T18:13:44.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.078+0000 7fb8bb400640 1 -- 172.21.15.37:0/261584990 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11386 (secure 0 0 0) 0x7fb8bc09b940 con 0x7fb8c4107590 2024-03-28T18:13:44.079 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-28T18:13:44.079 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6835/967056034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":967056034},{"type":"v1","addr":"172.21.15.37:6835","nonce":967056034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1926270371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6838","nonce":1926270371},{"type":"v1","addr":"172.21.15.37:6839","nonce":1926270371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/16451159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":16451159},{"type":"v1","addr":"172.21.15.59:6833","nonce":16451159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/576965351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":576965351},{"type":"v1","addr":"172.21.15.170:6833","nonce":576965351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2253706559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2253706559},{"type":"v1","addr":"172.21.15.59:6835","nonce":2253706559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24487,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/3972375740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":3972375740},{"type":"v1","addr":"172.21.15.37:6837","nonce":3972375740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:08:19.968894+0000","modified":"2024-03-28T18:13:00.596304+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24580,"mds_1":24569,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6837/4034607523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":4034607523},{"type":"v1","addr":"172.21.15.59:6837","nonce":4034607523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/4133178144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4133178144},{"type":"v1","addr":"172.21.15.170:6835","nonce":4133178144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/1308157923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1308157923},{"type":"v1","addr":"172.21.15.170:6837","nonce":1308157923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6839/1575518279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":1575518279},{"type":"v1","addr":"172.21.15.170:6839","nonce":1575518279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6839/2091629078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":2091629078},{"type":"v1","addr":"172.21.15.59:6839","nonce":2091629078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24580":{"gid":24580,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.37:6841/1923263622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6840","nonce":1923263622},{"type":"v1","addr":"172.21.15.37:6841","nonce":1923263622}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24580,"qdb_cluster":[24569,24580,24550]},"id":1}]} 2024-03-28T18:13:44.079 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 24 2024-03-28T18:13:44.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.081+0000 7fb8c3e00640 1 -- 172.21.15.37:0/261584990 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fb8a80a2970 msgr2=0x7fb8a80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:44.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.081+0000 7fb8c3e00640 1 --2- 172.21.15.37:0/261584990 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fb8a80a2970 0x7fb8a80a4e30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009b50 tx=0x7fb8b4005f90 comp rx=0 tx=0).stop 2024-03-28T18:13:44.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.081+0000 7fb8c3e00640 1 -- 172.21.15.37:0/261584990 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8c4107590 msgr2=0x7fb8c419c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:44.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.081+0000 7fb8c3e00640 1 --2- 172.21.15.37:0/261584990 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8c4107590 0x7fb8c419c700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc00efd0 tx=0x7fb8bc00c6a0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.082+0000 7fb8c3e00640 1 -- 172.21.15.37:0/261584990 shutdown_connections 2024-03-28T18:13:44.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.082+0000 7fb8c3e00640 1 --2- 172.21.15.37:0/261584990 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7fb8a80a2970 0x7fb8a80a4e30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.082+0000 7fb8c3e00640 1 --2- 172.21.15.37:0/261584990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8c40691c0 0x7fb8c4198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.082+0000 7fb8c3e00640 1 --2- 172.21.15.37:0/261584990 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8c4107590 0x7fb8c419c700 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.082+0000 7fb8c3e00640 1 --2- 172.21.15.37:0/261584990 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8c4068800 0x7fb8c4197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.082+0000 7fb8c3e00640 1 -- 172.21.15.37:0/261584990 >> 172.21.15.37:0/261584990 conn(0x7fb8c4075fb0 msgr2=0x7fb8c40ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:44.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.082+0000 7fb8c3e00640 1 -- 172.21.15.37:0/261584990 shutdown_connections 2024-03-28T18:13:44.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.082+0000 7fb8c3e00640 1 -- 172.21.15.37:0/261584990 wait complete. 2024-03-28T18:13:44.103 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:13:44.103 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:13:44.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.320+0000 7f15b6e00640 1 -- 172.21.15.37:0/1939418622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b0109730 msgr2=0x7f15b0109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:44.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.320+0000 7f15b6e00640 1 --2- 172.21.15.37:0/1939418622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b0109730 0x7f15b0109b10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009930 tx=0x7f15a402bc20 comp rx=0 tx=0).stop 2024-03-28T18:13:44.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.320+0000 7f15b6e00640 1 -- 172.21.15.37:0/1939418622 shutdown_connections 2024-03-28T18:13:44.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.320+0000 7f15b6e00640 1 --2- 172.21.15.37:0/1939418622 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15b01040e0 0x7f15b01105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.320+0000 7f15b6e00640 1 --2- 172.21.15.37:0/1939418622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15b0103720 0x7f15b0103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.320+0000 7f15b6e00640 1 --2- 172.21.15.37:0/1939418622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b0109730 0x7f15b0109b10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.320+0000 7f15b6e00640 1 -- 172.21.15.37:0/1939418622 >> 172.21.15.37:0/1939418622 conn(0x7f15b00ff1d0 msgr2=0x7f15b01015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:44.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.320+0000 7f15b6e00640 1 -- 172.21.15.37:0/1939418622 shutdown_connections 2024-03-28T18:13:44.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.320+0000 7f15b6e00640 1 -- 172.21.15.37:0/1939418622 wait complete. 2024-03-28T18:13:44.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.321+0000 7f15b6e00640 1 Processor -- start 2024-03-28T18:13:44.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.321+0000 7f15b6e00640 1 -- start start 2024-03-28T18:13:44.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.322+0000 7f15b6e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b0103720 0x7f15b019a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.322+0000 7f15b6e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15b01040e0 0x7f15b019a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.322+0000 7f15b6e00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15b0109730 0x7f15b019e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.322+0000 7f15b6e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b0113b50 con 0x7f15b01040e0 2024-03-28T18:13:44.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.322+0000 7f15b6e00640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b01139d0 con 0x7f15b0109730 2024-03-28T18:13:44.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.322+0000 7f15b6e00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b0113cd0 con 0x7f15b0103720 2024-03-28T18:13:44.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.322+0000 7f15b5a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b0103720 0x7f15b019a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.322+0000 7f15b5a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b0103720 0x7f15b019a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:59396/0 (socket says 172.21.15.37:59396) 2024-03-28T18:13:44.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.322+0000 7f15b5a00640 1 -- 172.21.15.37:0/1501703994 learned_addr learned my addr 172.21.15.37:0/1501703994 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:44.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.322+0000 7f15b5000640 1 --2- 172.21.15.37:0/1501703994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15b01040e0 0x7f15b019a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.322+0000 7f15b6400640 1 --2- 172.21.15.37:0/1501703994 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15b0109730 0x7f15b019e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.323+0000 7f15b5a00640 1 -- 172.21.15.37:0/1501703994 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15b0109730 msgr2=0x7f15b019e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:44.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.323+0000 7f15b5a00640 1 --2- 172.21.15.37:0/1501703994 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15b0109730 0x7f15b019e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.323+0000 7f15b5a00640 1 -- 172.21.15.37:0/1501703994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15b01040e0 msgr2=0x7f15b019a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:44.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.323+0000 7f15b5a00640 1 --2- 172.21.15.37:0/1501703994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15b01040e0 0x7f15b019a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.323+0000 7f15b5a00640 1 -- 172.21.15.37:0/1501703994 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b019f000 con 0x7f15b0103720 2024-03-28T18:13:44.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.323+0000 7f15b5000640 1 --2- 172.21.15.37:0/1501703994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15b01040e0 0x7f15b019a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:13:44.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.323+0000 7f15b5a00640 1 --2- 172.21.15.37:0/1501703994 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b0103720 0x7f15b019a010 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f15a4005ab0 tx=0x7f15a4005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:44.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.323+0000 7f15ae000640 1 -- 172.21.15.37:0/1501703994 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a4043020 con 0x7f15b0103720 2024-03-28T18:13:44.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.323+0000 7f15ae000640 1 -- 172.21.15.37:0/1501703994 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15a4005000 con 0x7f15b0103720 2024-03-28T18:13:44.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.323+0000 7f15ae000640 1 -- 172.21.15.37:0/1501703994 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a4003a80 con 0x7f15b0103720 2024-03-28T18:13:44.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.323+0000 7f15b6e00640 1 -- 172.21.15.37:0/1501703994 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b019f290 con 0x7f15b0103720 2024-03-28T18:13:44.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.324+0000 7f15b6e00640 1 -- 172.21.15.37:0/1501703994 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b01a6b80 con 0x7f15b0103720 2024-03-28T18:13:44.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.328+0000 7f15bb184740 1 Processor -- start 2024-03-28T18:13:44.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.328+0000 7f15ae000640 1 -- 172.21.15.37:0/1501703994 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f15a4046020 con 0x7f15b0103720 2024-03-28T18:13:44.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.328+0000 7f15bb184740 1 -- start start 2024-03-28T18:13:44.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.328+0000 7f15bb184740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f358b0 0x5590f8f35c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.329+0000 7f15bb184740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f36260 0x5590f8f3e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.329+0000 7f15bb184740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f3ede0 0x5590f8f41230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.329+0000 7f15bb184740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590f8e992c0 con 0x5590f8f358b0 2024-03-28T18:13:44.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.329+0000 7f15bb184740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590f8e41c90 con 0x5590f8f3ede0 2024-03-28T18:13:44.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.329+0000 7f15bb184740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590f8e95360 con 0x5590f8f36260 2024-03-28T18:13:44.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.329+0000 7f15b5a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f36260 0x5590f8f3e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.329+0000 7f15b6400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f358b0 0x5590f8f35c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.329+0000 7f15b5a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f36260 0x5590f8f3e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:59398/0 (socket says 172.21.15.37:59398) 2024-03-28T18:13:44.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.329+0000 7f15b5a00640 1 -- 172.21.15.37:0/3151776062 learned_addr learned my addr 172.21.15.37:0/3151776062 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:44.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.329+0000 7f15b5000640 1 --2- 172.21.15.37:0/3151776062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f3ede0 0x5590f8f41230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.329+0000 7f15b5a00640 1 -- 172.21.15.37:0/3151776062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f3ede0 msgr2=0x5590f8f41230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:44.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.329+0000 7f15b5a00640 1 --2- 172.21.15.37:0/3151776062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f3ede0 0x5590f8f41230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.329+0000 7f15b5a00640 1 -- 172.21.15.37:0/3151776062 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f358b0 msgr2=0x5590f8f35c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:44.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.329+0000 7f15b5a00640 1 --2- 172.21.15.37:0/3151776062 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f358b0 0x5590f8f35c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.329+0000 7f15b5a00640 1 -- 172.21.15.37:0/3151776062 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5590f8e66080 con 0x5590f8f36260 2024-03-28T18:13:44.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.330+0000 7f15b6400640 1 --2- 172.21.15.37:0/3151776062 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f358b0 0x5590f8f35c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:13:44.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.330+0000 7f15b5000640 1 --2- 172.21.15.37:0/3151776062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f3ede0 0x5590f8f41230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:13:44.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.330+0000 7f15ae000640 1 --2- 172.21.15.37:0/1501703994 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f15980a29f0 0x7f15980a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.330+0000 7f15b5a00640 1 --2- 172.21.15.37:0/3151776062 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f36260 0x5590f8f3e810 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f15a40ee580 tx=0x7f15a40f4e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edd999979a82b267 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:44.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.330+0000 7f15ae000640 1 -- 172.21.15.37:0/1501703994 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f15a40f0d80 con 0x7f15b0103720 2024-03-28T18:13:44.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.330+0000 7f15b6400640 1 --2- 172.21.15.37:0/1501703994 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f15980a29f0 0x7f15980a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.330+0000 7f15a2a00640 1 -- 172.21.15.37:0/3151776062 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a40b97c0 con 0x5590f8f36260 2024-03-28T18:13:44.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.330+0000 7f15a2a00640 1 -- 172.21.15.37:0/3151776062 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15a40bdc20 con 0x5590f8f36260 2024-03-28T18:13:44.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.330+0000 7f15a2a00640 1 -- 172.21.15.37:0/3151776062 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a40bce80 con 0x5590f8f36260 2024-03-28T18:13:44.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.331+0000 7f15bb184740 1 -- 172.21.15.37:0/3151776062 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f36260 msgr2=0x5590f8f3e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:44.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.331+0000 7f15bb184740 1 --2- 172.21.15.37:0/3151776062 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f36260 0x5590f8f3e810 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f15a40ee580 tx=0x7f15a40f4e40 comp rx=0 tx=0).stop 2024-03-28T18:13:44.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.331+0000 7f15b6400640 1 --2- 172.21.15.37:0/1501703994 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f15980a29f0 0x7f15980a4eb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f15a8005fd0 tx=0x7f15a8009500 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:44.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.331+0000 7f15bb184740 1 -- 172.21.15.37:0/3151776062 shutdown_connections 2024-03-28T18:13:44.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.331+0000 7f15bb184740 1 --2- 172.21.15.37:0/3151776062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f3ede0 0x5590f8f41230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.331+0000 7f15bb184740 1 --2- 172.21.15.37:0/3151776062 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f36260 0x5590f8f3e810 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.331+0000 7f15bb184740 1 --2- 172.21.15.37:0/3151776062 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f358b0 0x5590f8f35c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.331+0000 7f15bb184740 1 -- 172.21.15.37:0/3151776062 >> 172.21.15.37:0/3151776062 conn(0x5590f8f2ff80 msgr2=0x5590f8f30360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:44.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.331+0000 7f15bb184740 1 -- 172.21.15.37:0/3151776062 shutdown_connections 2024-03-28T18:13:44.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.331+0000 7f15bb184740 1 -- 172.21.15.37:0/3151776062 wait complete. 2024-03-28T18:13:44.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.332+0000 7f15bb184740 1 -- start start 2024-03-28T18:13:44.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.332+0000 7f15bb184740 1 Processor -- start 2024-03-28T18:13:44.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.332+0000 7f15bb184740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f358b0 0x5590f8f3e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.332+0000 7f15bb184740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f36260 0x5590f8f3bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.332+0000 7f15bb184740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f3ede0 0x5590f8f3c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.332+0000 7f15bb184740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590f8f41b10 con 0x5590f8f36260 2024-03-28T18:13:44.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.332+0000 7f15bb184740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590f8e550d0 con 0x5590f8f3ede0 2024-03-28T18:13:44.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.332+0000 7f15bb184740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590f8f41c90 con 0x5590f8f358b0 2024-03-28T18:13:44.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.332+0000 7f15bb184740 10 client.0 _ll_register_callbacks cb 0x5590f8e59e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:13:44.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.333+0000 7f15b5a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f358b0 0x5590f8f3e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.333+0000 7f15b5a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f358b0 0x5590f8f3e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:59408/0 (socket says 172.21.15.37:59408) 2024-03-28T18:13:44.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.333+0000 7f15b5a00640 1 -- 172.21.15.37:0/3322773657 learned_addr learned my addr 172.21.15.37:0/3322773657 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:44.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.333+0000 7f15b5000640 1 --2- 172.21.15.37:0/3322773657 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f36260 0x5590f8f3bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.333+0000 7f15b6400640 1 --2- 172.21.15.37:0/3322773657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f3ede0 0x5590f8f3c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.333+0000 7f15b5a00640 1 -- 172.21.15.37:0/3322773657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f3ede0 msgr2=0x5590f8f3c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:44.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.333+0000 7f15b5a00640 1 --2- 172.21.15.37:0/3322773657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f3ede0 0x5590f8f3c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.333+0000 7f15b5a00640 1 -- 172.21.15.37:0/3322773657 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f36260 msgr2=0x5590f8f3bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:44.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.333+0000 7f15b5a00640 1 --2- 172.21.15.37:0/3322773657 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f36260 0x5590f8f3bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.333+0000 7f15b5a00640 1 -- 172.21.15.37:0/3322773657 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5590f8bad3f0 con 0x5590f8f358b0 2024-03-28T18:13:44.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.333+0000 7f15b6400640 1 --2- 172.21.15.37:0/3322773657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f3ede0 0x5590f8f3c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:13:44.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.333+0000 7f15b5000640 1 --2- 172.21.15.37:0/3322773657 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f36260 0x5590f8f3bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:13:44.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.333+0000 7f15b5a00640 1 --2- 172.21.15.37:0/3322773657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f358b0 0x5590f8f3e780 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f15a40092a0 tx=0x7f15a40b9e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8920af955d84742 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:44.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.333+0000 7f1597400640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:13:44.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.333+0000 7f1597400640 1 -- 172.21.15.37:0/3322773657 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a40bd060 con 0x5590f8f358b0 2024-03-28T18:13:44.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.334+0000 7f1597400640 1 -- 172.21.15.37:0/3322773657 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f15a40054e0 con 0x5590f8f358b0 2024-03-28T18:13:44.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.334+0000 7f15bb184740 1 -- 172.21.15.37:0/3322773657 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f15a40bdc20 con 0x5590f8f358b0 2024-03-28T18:13:44.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.334+0000 7f1597400640 1 -- 172.21.15.37:0/3322773657 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15a40bd2f0 con 0x5590f8f358b0 2024-03-28T18:13:44.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.334+0000 7f15bb184740 10 client.25936 fetch_fsmap learned FSMap version 24 2024-03-28T18:13:44.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.334+0000 7f15bb184740 1 -- 172.21.15.37:0/3322773657 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5590f8c777d0 con 0x5590f8f358b0 2024-03-28T18:13:44.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.334+0000 7f1597400640 1 -- 172.21.15.37:0/3322773657 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15a40bdc20 con 0x5590f8f358b0 2024-03-28T18:13:44.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.334+0000 7f1597400640 1 -- 172.21.15.37:0/3322773657 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f15a40f70d0 con 0x5590f8f358b0 2024-03-28T18:13:44.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.334+0000 7f15bb184740 10 client.25936 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T18:13:44.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.334+0000 7f15bb184740 10 client.25936 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-28T18:13:44.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.334+0000 7f15bb184740 20 client.25936 populate_metadata read hostname 'smithi037' 2024-03-28T18:13:44.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.334+0000 7f15bb184740 1 --2- 172.21.15.37:0/3322773657 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5590f8f4e810 0x5590f8f50c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.334+0000 7f15bb184740 4 client.25936 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:13:44.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.334+0000 7f15bb184740 1 -- 172.21.15.37:0/3322773657 --> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5590f8ea6800 con 0x5590f8f4e810 2024-03-28T18:13:44.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.335+0000 7f15b5000640 1 --2- 172.21.15.37:0/3322773657 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5590f8f4e810 0x5590f8f50c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.335+0000 7f15b5000640 1 --2- 172.21.15.37:0/3322773657 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5590f8f4e810 0x5590f8f50c00 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=687e60815febdd5d server_cookie=bbb456ca8a8d83c3 in_seq=0 out_seq=0 2024-03-28T18:13:44.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.335+0000 7f1597400640 10 client.25936 ms_handle_connect on v2:172.21.15.170:6838/1575518279 2024-03-28T18:13:44.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.336+0000 7f1597400640 1 -- 172.21.15.37:0/3322773657 <== mds.2 v2:172.21.15.170:6838/1575518279 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5590f8ea6800 con 0x5590f8f4e810 2024-03-28T18:13:44.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.336+0000 7f1597400640 10 client.25936 handle_command_reply: tid=0 2024-03-28T18:13:44.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.336+0000 7f15bb184740 1 client.25936 shutdown 2024-03-28T18:13:44.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.336+0000 7f15bb184740 1 -- 172.21.15.37:0/3322773657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f358b0 msgr2=0x5590f8f3e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:44.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.336+0000 7f15bb184740 1 --2- 172.21.15.37:0/3322773657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f358b0 0x5590f8f3e780 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f15a40092a0 tx=0x7f15a40b9e20 comp rx=0 tx=0).stop 2024-03-28T18:13:44.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.336+0000 7f15bb184740 1 -- 172.21.15.37:0/3322773657 shutdown_connections 2024-03-28T18:13:44.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.336+0000 7f15bb184740 1 --2- 172.21.15.37:0/3322773657 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5590f8f4e810 0x5590f8f50c00 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.336+0000 7f15bb184740 1 --2- 172.21.15.37:0/3322773657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f3ede0 0x5590f8f3c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.336+0000 7f15bb184740 1 --2- 172.21.15.37:0/3322773657 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f36260 0x5590f8f3bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.336+0000 7f15bb184740 1 --2- 172.21.15.37:0/3322773657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f358b0 0x5590f8f3e780 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.336+0000 7f1597400640 0 client.25936 ms_handle_reset on v2:172.21.15.170:6838/1575518279 2024-03-28T18:13:44.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.336+0000 7f15bb184740 1 -- 172.21.15.37:0/3322773657 >> 172.21.15.37:0/3322773657 conn(0x5590f8f2ff80 msgr2=0x5590f8f31cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:44.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.336+0000 7f15bb184740 1 -- 172.21.15.37:0/3322773657 shutdown_connections 2024-03-28T18:13:44.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.336+0000 7f15bb184740 1 -- 172.21.15.37:0/3322773657 wait complete. 2024-03-28T18:13:44.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.336+0000 7f15bb184740 20 client.25936 trim_cache size 0 max 16384 2024-03-28T18:13:44.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.350+0000 7f15bb184740 1 Processor -- start 2024-03-28T18:13:44.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.350+0000 7f15bb184740 1 -- start start 2024-03-28T18:13:44.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.350+0000 7f15bb184740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f358b0 0x5590f8f3e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.350+0000 7f15bb184740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f36260 0x5590f8f3e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.351+0000 7f15bb184740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f3ede0 0x5590f8f38db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.351+0000 7f15bb184740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b01b1740 con 0x5590f8f358b0 2024-03-28T18:13:44.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.351+0000 7f15bb184740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b01af240 con 0x5590f8f36260 2024-03-28T18:13:44.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.351+0000 7f15bb184740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b01ad2e0 con 0x5590f8f3ede0 2024-03-28T18:13:44.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.351+0000 7f15b6400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f3ede0 0x5590f8f38db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.351+0000 7f15b5a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f358b0 0x5590f8f3e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.351+0000 7f15b6400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f3ede0 0x5590f8f38db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:59424/0 (socket says 172.21.15.37:59424) 2024-03-28T18:13:44.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.351+0000 7f15b6400640 1 -- 172.21.15.37:0/2935600023 learned_addr learned my addr 172.21.15.37:0/2935600023 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:44.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.351+0000 7f15b5000640 1 --2- 172.21.15.37:0/2935600023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f36260 0x5590f8f3e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.351+0000 7f15b6400640 1 -- 172.21.15.37:0/2935600023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f36260 msgr2=0x5590f8f3e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:44.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.351+0000 7f15b6400640 1 --2- 172.21.15.37:0/2935600023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f36260 0x5590f8f3e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.351+0000 7f15b6400640 1 -- 172.21.15.37:0/2935600023 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f358b0 msgr2=0x5590f8f3e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:44.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.351+0000 7f15b6400640 1 --2- 172.21.15.37:0/2935600023 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f358b0 0x5590f8f3e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.351+0000 7f15b6400640 1 -- 172.21.15.37:0/2935600023 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5590f8f3a080 con 0x5590f8f3ede0 2024-03-28T18:13:44.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.352+0000 7f15b6400640 1 --2- 172.21.15.37:0/2935600023 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f3ede0 0x5590f8f38db0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f15a8009aa0 tx=0x7f15a8016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e971bc1e2e277c43 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:44.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.352+0000 7f15a2000640 1 -- 172.21.15.37:0/2935600023 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a802c070 con 0x5590f8f3ede0 2024-03-28T18:13:44.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.352+0000 7f15a2000640 1 -- 172.21.15.37:0/2935600023 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15a801e190 con 0x5590f8f3ede0 2024-03-28T18:13:44.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.352+0000 7f15a2000640 1 -- 172.21.15.37:0/2935600023 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a8027750 con 0x5590f8f3ede0 2024-03-28T18:13:44.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.352+0000 7f15bb184740 1 -- 172.21.15.37:0/2935600023 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f3ede0 msgr2=0x5590f8f38db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:44.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.352+0000 7f15bb184740 1 --2- 172.21.15.37:0/2935600023 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f3ede0 0x5590f8f38db0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f15a8009aa0 tx=0x7f15a8016000 comp rx=0 tx=0).stop 2024-03-28T18:13:44.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.352+0000 7f15bb184740 1 -- 172.21.15.37:0/2935600023 shutdown_connections 2024-03-28T18:13:44.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.353+0000 7f15bb184740 1 --2- 172.21.15.37:0/2935600023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f36260 0x5590f8f3e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.353+0000 7f15bb184740 1 --2- 172.21.15.37:0/2935600023 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f3ede0 0x5590f8f38db0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.353+0000 7f15bb184740 1 --2- 172.21.15.37:0/2935600023 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f358b0 0x5590f8f3e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.353+0000 7f15bb184740 1 -- 172.21.15.37:0/2935600023 >> 172.21.15.37:0/2935600023 conn(0x5590f8f2ff80 msgr2=0x5590f8f2f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:44.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.353+0000 7f15bb184740 1 -- 172.21.15.37:0/2935600023 shutdown_connections 2024-03-28T18:13:44.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.353+0000 7f15bb184740 1 -- 172.21.15.37:0/2935600023 wait complete. 2024-03-28T18:13:44.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.353+0000 7f15bb184740 1 -- start start 2024-03-28T18:13:44.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.353+0000 7f15bb184740 1 Processor -- start 2024-03-28T18:13:44.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.354+0000 7f15bb184740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f358b0 0x5590f8f45000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.354+0000 7f15bb184740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f36260 0x5590f8f45540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.354+0000 7f15bb184740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f3ede0 0x5590f8f45a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.354+0000 7f15bb184740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b01ad8e0 con 0x5590f8f36260 2024-03-28T18:13:44.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.354+0000 7f15bb184740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b01aebe0 con 0x5590f8f3ede0 2024-03-28T18:13:44.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.354+0000 7f15bb184740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b01adf20 con 0x5590f8f358b0 2024-03-28T18:13:44.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.354+0000 7f15bb184740 10 client.0 _ll_register_callbacks cb 0x7f15a80099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:13:44.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.354+0000 7f15b5a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f358b0 0x5590f8f45000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.354+0000 7f15b5a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f358b0 0x5590f8f45000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:59428/0 (socket says 172.21.15.37:59428) 2024-03-28T18:13:44.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.354+0000 7f15b5a00640 1 -- 172.21.15.37:0/4038519356 learned_addr learned my addr 172.21.15.37:0/4038519356 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:44.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.354+0000 7f15b6400640 1 --2- 172.21.15.37:0/4038519356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f3ede0 0x5590f8f45a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.354+0000 7f15b5000640 1 --2- 172.21.15.37:0/4038519356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f36260 0x5590f8f45540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.354+0000 7f15b5a00640 1 -- 172.21.15.37:0/4038519356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f3ede0 msgr2=0x5590f8f45a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:44.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.354+0000 7f15b5a00640 1 --2- 172.21.15.37:0/4038519356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f3ede0 0x5590f8f45a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.354+0000 7f15b5a00640 1 -- 172.21.15.37:0/4038519356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f36260 msgr2=0x5590f8f45540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:44.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.354+0000 7f15b5a00640 1 --2- 172.21.15.37:0/4038519356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f36260 0x5590f8f45540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.354+0000 7f15b5a00640 1 -- 172.21.15.37:0/4038519356 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5590f8fa9e10 con 0x5590f8f358b0 2024-03-28T18:13:44.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.354+0000 7f15b5000640 1 --2- 172.21.15.37:0/4038519356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f36260 0x5590f8f45540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:13:44.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.355+0000 7f15b5a00640 1 --2- 172.21.15.37:0/4038519356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f358b0 0x5590f8f45000 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f15a402c200 tx=0x7f15a40f4e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffeeaeaf9e666789 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:44.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.355+0000 7f1597400640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:13:44.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.355+0000 7f1597400640 1 -- 172.21.15.37:0/4038519356 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a40bd060 con 0x5590f8f358b0 2024-03-28T18:13:44.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.355+0000 7f1597400640 1 -- 172.21.15.37:0/4038519356 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f15a40ee6c0 con 0x5590f8f358b0 2024-03-28T18:13:44.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.355+0000 7f15bb184740 1 -- 172.21.15.37:0/4038519356 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f15a801e190 con 0x5590f8f358b0 2024-03-28T18:13:44.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.355+0000 7f1597400640 1 -- 172.21.15.37:0/4038519356 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15a40c5d10 con 0x5590f8f358b0 2024-03-28T18:13:44.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.355+0000 7f15bb184740 10 client.25942 fetch_fsmap learned FSMap version 24 2024-03-28T18:13:44.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.355+0000 7f15bb184740 1 -- 172.21.15.37:0/4038519356 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5590f8fa9e10 con 0x5590f8f358b0 2024-03-28T18:13:44.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.355+0000 7f1597400640 1 -- 172.21.15.37:0/4038519356 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15a801e190 con 0x5590f8f358b0 2024-03-28T18:13:44.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.355+0000 7f1597400640 1 -- 172.21.15.37:0/4038519356 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f15a40c5830 con 0x5590f8f358b0 2024-03-28T18:13:44.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.355+0000 7f15bb184740 10 client.25942 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T18:13:44.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.355+0000 7f15bb184740 10 client.25942 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-28T18:13:44.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.355+0000 7f15bb184740 20 client.25942 populate_metadata read hostname 'smithi037' 2024-03-28T18:13:44.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.355+0000 7f15bb184740 1 --2- 172.21.15.37:0/4038519356 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5590f8fab270 0x5590f8fad660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.355+0000 7f15bb184740 4 client.25942 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:13:44.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.355+0000 7f15bb184740 1 -- 172.21.15.37:0/4038519356 --> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5590f8ea0b50 con 0x5590f8fab270 2024-03-28T18:13:44.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.356+0000 7f15b5000640 1 --2- 172.21.15.37:0/4038519356 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5590f8fab270 0x5590f8fad660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.356+0000 7f15b5000640 1 --2- 172.21.15.37:0/4038519356 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5590f8fab270 0x5590f8fad660 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=13cc4a1142f4b385 server_cookie=91a26d6eb6f58e5e in_seq=0 out_seq=0 2024-03-28T18:13:44.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.356+0000 7f1597400640 10 client.25942 ms_handle_connect on v2:172.21.15.170:6838/1575518279 2024-03-28T18:13:44.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.356+0000 7f1597400640 1 -- 172.21.15.37:0/4038519356 <== mds.2 v2:172.21.15.170:6838/1575518279 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5590f8ea0b50 con 0x5590f8fab270 2024-03-28T18:13:44.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.356+0000 7f1597400640 10 client.25942 handle_command_reply: tid=0 2024-03-28T18:13:44.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.356+0000 7f15bb184740 1 client.25942 shutdown 2024-03-28T18:13:44.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.356+0000 7f15bb184740 1 -- 172.21.15.37:0/4038519356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f358b0 msgr2=0x5590f8f45000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:44.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.356+0000 7f15bb184740 1 --2- 172.21.15.37:0/4038519356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f358b0 0x5590f8f45000 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f15a402c200 tx=0x7f15a40f4e40 comp rx=0 tx=0).stop 2024-03-28T18:13:44.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.357+0000 7f15bb184740 1 -- 172.21.15.37:0/4038519356 shutdown_connections 2024-03-28T18:13:44.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.357+0000 7f15bb184740 1 --2- 172.21.15.37:0/4038519356 >> [v2:172.21.15.170:6838/1575518279,v1:172.21.15.170:6839/1575518279] conn(0x5590f8fab270 0x5590f8fad660 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.357+0000 7f15bb184740 1 --2- 172.21.15.37:0/4038519356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5590f8f3ede0 0x5590f8f45a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.357+0000 7f15bb184740 1 --2- 172.21.15.37:0/4038519356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5590f8f36260 0x5590f8f45540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.357+0000 7f1597400640 0 client.25942 ms_handle_reset on v2:172.21.15.170:6838/1575518279 2024-03-28T18:13:44.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.357+0000 7f15bb184740 1 --2- 172.21.15.37:0/4038519356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5590f8f358b0 0x5590f8f45000 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.357+0000 7f15bb184740 1 -- 172.21.15.37:0/4038519356 >> 172.21.15.37:0/4038519356 conn(0x5590f8f2ff80 msgr2=0x5590f8f36640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:44.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.357+0000 7f15bb184740 1 -- 172.21.15.37:0/4038519356 shutdown_connections 2024-03-28T18:13:44.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.357+0000 7f15bb184740 1 -- 172.21.15.37:0/4038519356 wait complete. 2024-03-28T18:13:44.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.357+0000 7f15bb184740 20 client.25942 trim_cache size 0 max 16384 2024-03-28T18:13:44.362 INFO:teuthology.orchestra.run.smithi037.stdout:[] 2024-03-28T18:13:44.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.357+0000 7f15b6e00640 1 -- 172.21.15.37:0/1501703994 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f15980a29f0 msgr2=0x7f15980a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:44.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.357+0000 7f15b6e00640 1 --2- 172.21.15.37:0/1501703994 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f15980a29f0 0x7f15980a4eb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f15a8005fd0 tx=0x7f15a8009500 comp rx=0 tx=0).stop 2024-03-28T18:13:44.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.357+0000 7f15b6e00640 1 -- 172.21.15.37:0/1501703994 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b0103720 msgr2=0x7f15b019a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:44.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.357+0000 7f15b6e00640 1 --2- 172.21.15.37:0/1501703994 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b0103720 0x7f15b019a010 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f15a4005ab0 tx=0x7f15a4005e80 comp rx=0 tx=0).stop 2024-03-28T18:13:44.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.358+0000 7f15b6e00640 1 -- 172.21.15.37:0/1501703994 shutdown_connections 2024-03-28T18:13:44.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.358+0000 7f15b6e00640 1 --2- 172.21.15.37:0/1501703994 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f15980a29f0 0x7f15980a4eb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.358+0000 7f15b6e00640 1 --2- 172.21.15.37:0/1501703994 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15b0109730 0x7f15b019e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.358+0000 7f15b6e00640 1 --2- 172.21.15.37:0/1501703994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15b01040e0 0x7f15b019a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.358+0000 7f15b6e00640 1 --2- 172.21.15.37:0/1501703994 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b0103720 0x7f15b019a010 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.358+0000 7f15b6e00640 1 -- 172.21.15.37:0/1501703994 >> 172.21.15.37:0/1501703994 conn(0x7f15b00ff1d0 msgr2=0x7f15b00ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:44.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.358+0000 7f15b6e00640 1 -- 172.21.15.37:0/1501703994 shutdown_connections 2024-03-28T18:13:44.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.358+0000 7f15b6e00640 1 -- 172.21.15.37:0/1501703994 wait complete. 2024-03-28T18:13:44.375 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:13:44.375 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:13:44.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.581+0000 7f4f89600640 1 -- 172.21.15.37:0/2187723836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f84109850 msgr2=0x7f4f84109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:44.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.581+0000 7f4f89600640 1 --2- 172.21.15.37:0/2187723836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f84109850 0x7f4f84109c30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4f74009930 tx=0x7f4f7402bc20 comp rx=0 tx=0).stop 2024-03-28T18:13:44.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.581+0000 7f4f89600640 1 -- 172.21.15.37:0/2187723836 shutdown_connections 2024-03-28T18:13:44.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.581+0000 7f4f89600640 1 --2- 172.21.15.37:0/2187723836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f84103840 0x7f4f84103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.581+0000 7f4f89600640 1 --2- 172.21.15.37:0/2187723836 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f84104200 0x7f4f84110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.581+0000 7f4f89600640 1 --2- 172.21.15.37:0/2187723836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f84109850 0x7f4f84109c30 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.581+0000 7f4f89600640 1 -- 172.21.15.37:0/2187723836 >> 172.21.15.37:0/2187723836 conn(0x7f4f840ff350 msgr2=0x7f4f84101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:44.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.582+0000 7f4f89600640 1 -- 172.21.15.37:0/2187723836 shutdown_connections 2024-03-28T18:13:44.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.582+0000 7f4f89600640 1 -- 172.21.15.37:0/2187723836 wait complete. 2024-03-28T18:13:44.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.583+0000 7f4f89600640 1 Processor -- start 2024-03-28T18:13:44.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.583+0000 7f4f89600640 1 -- start start 2024-03-28T18:13:44.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.583+0000 7f4f89600640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f84103840 0x7f4f8419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.583+0000 7f4f89600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f84104200 0x7f4f8419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.583+0000 7f4f89600640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f84109850 0x7f4f8419ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.583+0000 7f4f89600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f84113c70 con 0x7f4f84104200 2024-03-28T18:13:44.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.583+0000 7f4f89600640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f84113af0 con 0x7f4f84109850 2024-03-28T18:13:44.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.583+0000 7f4f89600640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f84113df0 con 0x7f4f84103840 2024-03-28T18:13:44.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.583+0000 7f4f82a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f84103840 0x7f4f8419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.583+0000 7f4f82a00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f84103840 0x7f4f8419a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.37:59448/0 (socket says 172.21.15.37:59448) 2024-03-28T18:13:44.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.584+0000 7f4f82a00640 1 -- 172.21.15.37:0/1519812982 learned_addr learned my addr 172.21.15.37:0/1519812982 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:44.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.584+0000 7f4f82000640 1 --2- 172.21.15.37:0/1519812982 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f84104200 0x7f4f8419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.584+0000 7f4f83400640 1 --2- 172.21.15.37:0/1519812982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f84109850 0x7f4f8419ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.584+0000 7f4f82a00640 1 -- 172.21.15.37:0/1519812982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f84109850 msgr2=0x7f4f8419ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:44.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.584+0000 7f4f82a00640 1 --2- 172.21.15.37:0/1519812982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f84109850 0x7f4f8419ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.584+0000 7f4f82a00640 1 -- 172.21.15.37:0/1519812982 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f84104200 msgr2=0x7f4f8419a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:44.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.584+0000 7f4f82a00640 1 --2- 172.21.15.37:0/1519812982 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f84104200 0x7f4f8419a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.584+0000 7f4f82a00640 1 -- 172.21.15.37:0/1519812982 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8419f190 con 0x7f4f84103840 2024-03-28T18:13:44.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.584+0000 7f4f82000640 1 --2- 172.21.15.37:0/1519812982 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f84104200 0x7f4f8419a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:13:44.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.584+0000 7f4f82a00640 1 --2- 172.21.15.37:0/1519812982 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f84103840 0x7f4f8419a1f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4f74005ab0 tx=0x7f4f740042e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:44.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.584+0000 7f4f7b400640 1 -- 172.21.15.37:0/1519812982 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f74043020 con 0x7f4f84103840 2024-03-28T18:13:44.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.585+0000 7f4f7b400640 1 -- 172.21.15.37:0/1519812982 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f740044d0 con 0x7f4f84103840 2024-03-28T18:13:44.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.585+0000 7f4f7b400640 1 -- 172.21.15.37:0/1519812982 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f74003b90 con 0x7f4f84103840 2024-03-28T18:13:44.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.585+0000 7f4f89600640 1 -- 172.21.15.37:0/1519812982 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f8419f420 con 0x7f4f84103840 2024-03-28T18:13:44.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.585+0000 7f4f89600640 1 -- 172.21.15.37:0/1519812982 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f841a6d10 con 0x7f4f84103840 2024-03-28T18:13:44.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.588+0000 7f4f7b400640 1 -- 172.21.15.37:0/1519812982 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f4f74046020 con 0x7f4f84103840 2024-03-28T18:13:44.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.588+0000 7f4f8c373740 1 Processor -- start 2024-03-28T18:13:44.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.588+0000 7f4f8c373740 1 -- start start 2024-03-28T18:13:44.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.589+0000 7f4f8c373740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56357a37e6d0 0x56357a37eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.589+0000 7f4f8c373740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56357a37f080 0x56357a387630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.589+0000 7f4f8c373740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56357a387c00 0x56357a38a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.589+0000 7f4f8c373740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357a2e22c0 con 0x56357a37f080 2024-03-28T18:13:44.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.589+0000 7f4f8c373740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357a28ac90 con 0x56357a37e6d0 2024-03-28T18:13:44.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.589+0000 7f4f8c373740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357a2de360 con 0x56357a387c00 2024-03-28T18:13:44.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.589+0000 7f4f82a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56357a37f080 0x56357a387630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.589+0000 7f4f82000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56357a387c00 0x56357a38a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.589+0000 7f4f82a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56357a37f080 0x56357a387630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56234/0 (socket says 172.21.15.37:56234) 2024-03-28T18:13:44.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.589+0000 7f4f82a00640 1 -- 172.21.15.37:0/1215343077 learned_addr learned my addr 172.21.15.37:0/1215343077 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:44.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.589+0000 7f4f83400640 1 --2- 172.21.15.37:0/1215343077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56357a37e6d0 0x56357a37eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.589+0000 7f4f82000640 1 -- 172.21.15.37:0/1215343077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56357a37e6d0 msgr2=0x56357a37eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:44.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.589+0000 7f4f82000640 1 --2- 172.21.15.37:0/1215343077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56357a37e6d0 0x56357a37eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.590+0000 7f4f82000640 1 -- 172.21.15.37:0/1215343077 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56357a37f080 msgr2=0x56357a387630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:44.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.590+0000 7f4f82000640 1 --2- 172.21.15.37:0/1215343077 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56357a37f080 0x56357a387630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.590+0000 7f4f82000640 1 -- 172.21.15.37:0/1215343077 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56357a2af080 con 0x56357a387c00 2024-03-28T18:13:44.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.590+0000 7f4f82000640 1 --2- 172.21.15.37:0/1215343077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56357a387c00 0x56357a38a050 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4f841019e0 tx=0x7f4f7000f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70c9a8f8cbc3feea server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:44.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.590+0000 7f4f7b400640 1 --2- 172.21.15.37:0/1519812982 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f4f680a29f0 0x7f4f680a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.590+0000 7f4f7b400640 1 -- 172.21.15.37:0/1519812982 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4f740f0dd0 con 0x7f4f84103840 2024-03-28T18:13:44.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.590+0000 7f4f6f400640 1 -- 172.21.15.37:0/1215343077 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f70015070 con 0x56357a387c00 2024-03-28T18:13:44.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.590+0000 7f4f6f400640 1 -- 172.21.15.37:0/1215343077 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f7001a940 con 0x56357a387c00 2024-03-28T18:13:44.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.590+0000 7f4f6f400640 1 -- 172.21.15.37:0/1215343077 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f70011980 con 0x56357a387c00 2024-03-28T18:13:44.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.590+0000 7f4f83400640 1 --2- 172.21.15.37:0/1519812982 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f4f680a29f0 0x7f4f680a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.591+0000 7f4f8c373740 1 -- 172.21.15.37:0/1215343077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56357a387c00 msgr2=0x56357a38a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:13:44.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.591+0000 7f4f8c373740 1 --2- 172.21.15.37:0/1215343077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56357a387c00 0x56357a38a050 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4f841019e0 tx=0x7f4f7000f730 comp rx=0 tx=0).stop 2024-03-28T18:13:44.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.591+0000 7f4f83400640 1 --2- 172.21.15.37:0/1519812982 >> [v2:172.21.15.37:6800/683867216,v1:172.21.15.37:6801/683867216] conn(0x7f4f680a29f0 0x7f4f680a4eb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c005fd0 tx=0x7f4f7c009500 comp rx=0 tx=0).ready entity=mgr.15207 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:44.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.591+0000 7f4f8c373740 1 -- 172.21.15.37:0/1215343077 shutdown_connections 2024-03-28T18:13:44.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.591+0000 7f4f8c373740 1 --2- 172.21.15.37:0/1215343077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56357a387c00 0x56357a38a050 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.591+0000 7f4f8c373740 1 --2- 172.21.15.37:0/1215343077 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56357a37f080 0x56357a387630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.591+0000 7f4f8c373740 1 --2- 172.21.15.37:0/1215343077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56357a37e6d0 0x56357a37eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.591+0000 7f4f8c373740 1 -- 172.21.15.37:0/1215343077 >> 172.21.15.37:0/1215343077 conn(0x56357a378da0 msgr2=0x56357a379180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:13:44.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.591+0000 7f4f8c373740 1 -- 172.21.15.37:0/1215343077 shutdown_connections 2024-03-28T18:13:44.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.591+0000 7f4f8c373740 1 -- 172.21.15.37:0/1215343077 wait complete. 2024-03-28T18:13:44.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.592+0000 7f4f8c373740 1 -- start start 2024-03-28T18:13:44.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.592+0000 7f4f8c373740 1 Processor -- start 2024-03-28T18:13:44.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.592+0000 7f4f8c373740 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56357a37e6d0 0x56357a3875b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.592+0000 7f4f8c373740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56357a37f080 0x56357a3849b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.592+0000 7f4f8c373740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56357a387c00 0x56357a3850f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.592+0000 7f4f8c373740 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357a38ab70 con 0x56357a37e6d0 2024-03-28T18:13:44.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.592+0000 7f4f8c373740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357a29e0d0 con 0x56357a37f080 2024-03-28T18:13:44.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.592+0000 7f4f8c373740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357a38acf0 con 0x56357a387c00 2024-03-28T18:13:44.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.593+0000 7f4f8c373740 10 client.0 _ll_register_callbacks cb 0x56357a2a2e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:13:44.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.593+0000 7f4f82a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56357a37e6d0 0x56357a3875b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.593+0000 7f4f83400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56357a387c00 0x56357a3850f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.593+0000 7f4f82a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56357a37e6d0 0x56357a3875b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56250/0 (socket says 172.21.15.37:56250) 2024-03-28T18:13:44.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.593+0000 7f4f82a00640 1 -- 172.21.15.37:0/2018652346 learned_addr learned my addr 172.21.15.37:0/2018652346 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-03-28T18:13:44.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.593+0000 7f4f82000640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56357a37f080 0x56357a3849b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:44.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.593+0000 7f4f83400640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56357a37f080 msgr2=0x56357a3849b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:44.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.593+0000 7f4f83400640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56357a37f080 0x56357a3849b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.593+0000 7f4f83400640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56357a37e6d0 msgr2=0x56357a3875b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:13:44.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.593+0000 7f4f83400640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x56357a37e6d0 0x56357a3875b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:13:44.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.593+0000 7f4f83400640 1 -- 172.21.15.37:0/2018652346 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563579ff63f0 con 0x56357a387c00 2024-03-28T18:13:44.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.593+0000 7f4f82000640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56357a37f080 0x56357a3849b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:13:44.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.594+0000 7f4f83400640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56357a387c00 0x56357a3850f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c01f6b0 tx=0x7f4f7c004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d724a269efb69dd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:13:44.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.594+0000 7f4f63e00640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T18:13:44.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.594+0000 7f4f63e00640 1 -- 172.21.15.37:0/2018652346 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c011810 con 0x56357a387c00 2024-03-28T18:13:44.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.594+0000 7f4f63e00640 1 -- 172.21.15.37:0/2018652346 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4f7c017030 con 0x56357a387c00 2024-03-28T18:13:44.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.594+0000 7f4f8c373740 1 -- 172.21.15.37:0/2018652346 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f7001a940 con 0x56357a387c00 2024-03-28T18:13:44.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.594+0000 7f4f63e00640 1 -- 172.21.15.37:0/2018652346 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f7c0119b0 con 0x56357a387c00 2024-03-28T18:13:44.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.594+0000 7f4f8c373740 10 client.25954 fetch_fsmap learned FSMap version 24 2024-03-28T18:13:44.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.594+0000 7f4f8c373740 1 -- 172.21.15.37:0/2018652346 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56357a0c07d0 con 0x56357a387c00 2024-03-28T18:13:44.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.594+0000 7f4f63e00640 1 -- 172.21.15.37:0/2018652346 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f7001a940 con 0x56357a387c00 2024-03-28T18:13:44.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.595+0000 7f4f63e00640 1 -- 172.21.15.37:0/2018652346 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f4f7c01c3f0 con 0x56357a387c00 2024-03-28T18:13:44.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.595+0000 7f4f8c373740 10 client.25954 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T18:13:44.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.595+0000 7f4f8c373740 10 client.25954 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-28T18:13:44.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.595+0000 7f4f8c373740 20 client.25954 populate_metadata read hostname 'smithi037' 2024-03-28T18:13:44.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.595+0000 7f4f8c373740 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:13:44.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.595+0000 7f4f8c373740 4 client.25954 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:13:44.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.595+0000 7f4f8c373740 1 -- 172.21.15.37:0/2018652346 --> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56357a2ef800 con 0x56357a397bd0 2024-03-28T18:13:44.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.595+0000 7f4f82a00640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 msgr2=0x56357a39a010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:6838/2091629078 2024-03-28T18:13:44.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.595+0000 7f4f63e00640 1 client.25954 ms_handle_refused on v2:172.21.15.59:6838/2091629078 2024-03-28T18:13:44.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.595+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:13:44.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:44 smithi037 ceph-mon[31460]: from='client.? 172.21.15.37:0/261584990' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:13:44.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.796+0000 7f4f82a00640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 msgr2=0x56357a39a010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:6838/2091629078 2024-03-28T18:13:44.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.796+0000 7f4f63e00640 1 client.25954 ms_handle_refused on v2:172.21.15.59:6838/2091629078 2024-03-28T18:13:44.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:44.796+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T18:13:45.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:44 smithi059 ceph-mon[36190]: from='client.? 172.21.15.37:0/261584990' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:13:45.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:44 smithi170 ceph-mon[36280]: from='client.? 172.21.15.37:0/261584990' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:13:45.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:45.197+0000 7f4f82a00640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 msgr2=0x56357a39a010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:6838/2091629078 2024-03-28T18:13:45.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:45.197+0000 7f4f63e00640 1 client.25954 ms_handle_refused on v2:172.21.15.59:6838/2091629078 2024-03-28T18:13:45.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:45.197+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T18:13:45.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:45 smithi037 ceph-mon[31460]: pgmap v80: 97 pgs: 97 active+clean; 44 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 941 KiB/s wr, 230 op/s 2024-03-28T18:13:45.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:45.998+0000 7f4f82a00640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 msgr2=0x56357a39a010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:6838/2091629078 2024-03-28T18:13:45.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:45.998+0000 7f4f63e00640 1 client.25954 ms_handle_refused on v2:172.21.15.59:6838/2091629078 2024-03-28T18:13:45.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:45.998+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T18:13:46.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:45 smithi059 ceph-mon[36190]: pgmap v80: 97 pgs: 97 active+clean; 44 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 941 KiB/s wr, 230 op/s 2024-03-28T18:13:46.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:45 smithi170 ceph-mon[36280]: pgmap v80: 97 pgs: 97 active+clean; 44 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 941 KiB/s wr, 230 op/s 2024-03-28T18:13:47.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:47.599+0000 7f4f82a00640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 msgr2=0x56357a39a010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:6838/2091629078 2024-03-28T18:13:47.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:47.599+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T18:13:47.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:47.599+0000 7f4f63e00640 1 client.25954 ms_handle_refused on v2:172.21.15.59:6838/2091629078 2024-03-28T18:13:47.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:47 smithi037 ceph-mon[31460]: pgmap v81: 97 pgs: 97 active+clean; 44 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1005 KiB/s wr, 248 op/s 2024-03-28T18:13:48.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:47 smithi059 ceph-mon[36190]: pgmap v81: 97 pgs: 97 active+clean; 44 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1005 KiB/s wr, 248 op/s 2024-03-28T18:13:48.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:47 smithi170 ceph-mon[36280]: pgmap v81: 97 pgs: 97 active+clean; 44 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1005 KiB/s wr, 248 op/s 2024-03-28T18:13:49.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:49 smithi037 ceph-mon[31460]: pgmap v82: 97 pgs: 97 active+clean; 44 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 722 KiB/s wr, 188 op/s 2024-03-28T18:13:50.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:49 smithi059 ceph-mon[36190]: pgmap v82: 97 pgs: 97 active+clean; 44 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 722 KiB/s wr, 188 op/s 2024-03-28T18:13:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:49 smithi170 ceph-mon[36280]: pgmap v82: 97 pgs: 97 active+clean; 44 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 722 KiB/s wr, 188 op/s 2024-03-28T18:13:50.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:50.802+0000 7f4f82a00640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 msgr2=0x56357a39a010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:6838/2091629078 2024-03-28T18:13:50.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:50.802+0000 7f4f63e00640 1 client.25954 ms_handle_refused on v2:172.21.15.59:6838/2091629078 2024-03-28T18:13:50.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:50.802+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-28T18:13:51.802 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:51 smithi059 ceph-mon[36190]: pgmap v83: 97 pgs: 97 active+clean; 44 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 722 KiB/s wr, 188 op/s 2024-03-28T18:13:51.802 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:51 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.b.service: Scheduled restart job, restart counter is at 1. 2024-03-28T18:13:51.802 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:51 smithi059 systemd[1]: Stopped Ceph mds.b for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:13:51.802 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:51 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.b.service: Consumed 2min 40.741s CPU time. 2024-03-28T18:13:51.802 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:51 smithi059 systemd[1]: Starting Ceph mds.b for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:13:51.803 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:51 smithi059 podman[70589]: 2024-03-28 18:13:51.678351769 +0000 UTC m=+0.018906321 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:13:51.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:51 smithi037 ceph-mon[31460]: pgmap v83: 97 pgs: 97 active+clean; 44 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 722 KiB/s wr, 188 op/s 2024-03-28T18:13:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:51 smithi170 ceph-mon[36280]: pgmap v83: 97 pgs: 97 active+clean; 44 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 722 KiB/s wr, 188 op/s 2024-03-28T18:13:52.055 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:51 smithi059 podman[70589]: 2024-03-28 18:13:51.803150027 +0000 UTC m=+0.143704564 container create b374db49781bd05ceda2da0610738b1620b638fa7b81b7e45d767be28f2394f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-28T18:13:52.535 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:52 smithi059 podman[70589]: 2024-03-28 18:13:52.056039335 +0000 UTC m=+0.396593878 container init b374db49781bd05ceda2da0610738b1620b638fa7b81b7e45d767be28f2394f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True) 2024-03-28T18:13:52.535 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:52 smithi059 podman[70589]: 2024-03-28 18:13:52.059477264 +0000 UTC m=+0.400031803 container start b374db49781bd05ceda2da0610738b1620b638fa7b81b7e45d767be28f2394f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-28T18:13:52.535 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:52 smithi059 ceph-mds[70608]: -- 172.21.15.59:0/4187187271 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5605a42b9a40 con 0x5605a428cc00 2024-03-28T18:13:52.535 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: starting mds.b at 2024-03-28T18:13:52.535 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:52 smithi059 bash[70589]: b374db49781bd05ceda2da0610738b1620b638fa7b81b7e45d767be28f2394f2 2024-03-28T18:13:52.535 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:13:52 smithi059 systemd[1]: Started Ceph mds.b for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:13:52.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:52 smithi037 ceph-mon[31460]: daemon mds.b restarted 2024-03-28T18:13:52.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:52 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:13:52.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:52 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:13:52.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:52 smithi037 ceph-mon[31460]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T18:13:52.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:52 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/362278957,v1:172.21.15.59:6839/362278957] up:boot 2024-03-28T18:13:52.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:52 smithi037 ceph-mon[31460]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-28T18:13:52.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:52 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:13:52.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:52 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:13:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:52 smithi059 ceph-mon[36190]: daemon mds.b restarted 2024-03-28T18:13:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:52 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:13:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:52 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:13:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:52 smithi059 ceph-mon[36190]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T18:13:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:52 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/362278957,v1:172.21.15.59:6839/362278957] up:boot 2024-03-28T18:13:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:52 smithi059 ceph-mon[36190]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-28T18:13:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:52 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:13:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:52 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:13:53.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:52 smithi170 ceph-mon[36280]: daemon mds.b restarted 2024-03-28T18:13:53.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:52 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:13:53.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:52 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:13:53.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:52 smithi170 ceph-mon[36280]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T18:13:53.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:52 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/362278957,v1:172.21.15.59:6839/362278957] up:boot 2024-03-28T18:13:53.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:52 smithi170 ceph-mon[36280]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-28T18:13:53.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:52 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:13:53.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:52 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:13:53.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:53 smithi037 ceph-mon[31460]: pgmap v85: 97 pgs: 97 active+clean; 44 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 79 KiB/s wr, 24 op/s 2024-03-28T18:13:54.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:53 smithi059 ceph-mon[36190]: pgmap v85: 97 pgs: 97 active+clean; 44 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 79 KiB/s wr, 24 op/s 2024-03-28T18:13:54.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:53 smithi170 ceph-mon[36280]: pgmap v85: 97 pgs: 97 active+clean; 44 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 79 KiB/s wr, 24 op/s 2024-03-28T18:13:55.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:54 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/4133178144,v1:172.21.15.170:6835/4133178144] up:resolve 2024-03-28T18:13:55.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:54 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:13:55.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:54 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/4133178144,v1:172.21.15.170:6835/4133178144] up:resolve 2024-03-28T18:13:55.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:54 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:13:55.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:54 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/4133178144,v1:172.21.15.170:6835/4133178144] up:resolve 2024-03-28T18:13:55.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:54 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:13:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:55 smithi059 ceph-mon[36190]: pgmap v86: 97 pgs: 97 active+clean; 44 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 79 KiB/s wr, 24 op/s 2024-03-28T18:13:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:55 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199995 2024-03-28T18:13:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:55 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/4133178144,v1:172.21.15.170:6835/4133178144] up:reconnect 2024-03-28T18:13:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:55 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:13:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:55 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0489989 2024-03-28T18:13:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:55 smithi170 ceph-mon[36280]: pgmap v86: 97 pgs: 97 active+clean; 44 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 79 KiB/s wr, 24 op/s 2024-03-28T18:13:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:55 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199995 2024-03-28T18:13:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:55 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/4133178144,v1:172.21.15.170:6835/4133178144] up:reconnect 2024-03-28T18:13:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:55 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:13:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:55 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0489989 2024-03-28T18:13:56.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:55 smithi037 ceph-mon[31460]: pgmap v86: 97 pgs: 97 active+clean; 44 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 79 KiB/s wr, 24 op/s 2024-03-28T18:13:56.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:55 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199995 2024-03-28T18:13:56.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:55 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/4133178144,v1:172.21.15.170:6835/4133178144] up:reconnect 2024-03-28T18:13:56.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:55 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:13:56.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:55 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0489989 2024-03-28T18:13:57.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:56 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/4133178144,v1:172.21.15.170:6835/4133178144] up:rejoin 2024-03-28T18:13:57.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:56 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:13:57.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:56 smithi059 ceph-mon[36190]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-28T18:13:57.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/4133178144,v1:172.21.15.170:6835/4133178144] up:rejoin 2024-03-28T18:13:57.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:13:57.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-mon[36280]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-28T18:13:57.036 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff121fec700 time 2024-03-28T18:13:56.736332+0000 2024-03-28T18:13:57.036 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:13:57.036 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:13:57.036 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff12c6164a1] 2024-03-28T18:13:57.037 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff12c616667] 2024-03-28T18:13:57.037 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55adfb401223] 2024-03-28T18:13:57.037 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55adfb27b5d6] 2024-03-28T18:13:57.037 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55adfb27bda1] 2024-03-28T18:13:57.037 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55adfb27e2d5] 2024-03-28T18:13:57.037 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55adfb2ce233] 2024-03-28T18:13:57.037 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55adfb2d0197] 2024-03-28T18:13:57.037 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55adfb1cf11b] 2024-03-28T18:13:57.037 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55adfb1cf6bc] 2024-03-28T18:13:57.037 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55adfb1b91ff] 2024-03-28T18:13:57.037 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff12c8a8488] 2024-03-28T18:13:57.037 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 13: (DispatchQueue::entry()+0x51f) [0x7ff12c8a560f] 2024-03-28T18:13:57.037 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff12c978e81] 2024-03-28T18:13:57.037 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff12b3931ca] 2024-03-28T18:13:57.037 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 16: clone() 2024-03-28T18:13:57.038 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2024-03-28T18:13:56.736+0000 7ff121fec700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff121fec700 time 2024-03-28T18:13:56.736332+0000 2024-03-28T18:13:57.038 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:13:57.038 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2024-03-28T18:13:57.038 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:13:57.038 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff12c6164a1] 2024-03-28T18:13:57.038 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff12c616667] 2024-03-28T18:13:57.038 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55adfb401223] 2024-03-28T18:13:57.038 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55adfb27b5d6] 2024-03-28T18:13:57.038 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55adfb27bda1] 2024-03-28T18:13:57.038 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55adfb27e2d5] 2024-03-28T18:13:57.038 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55adfb2ce233] 2024-03-28T18:13:57.038 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55adfb2d0197] 2024-03-28T18:13:57.038 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55adfb1cf11b] 2024-03-28T18:13:57.039 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55adfb1cf6bc] 2024-03-28T18:13:57.039 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55adfb1b91ff] 2024-03-28T18:13:57.039 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff12c8a8488] 2024-03-28T18:13:57.039 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 13: (DispatchQueue::entry()+0x51f) [0x7ff12c8a560f] 2024-03-28T18:13:57.039 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff12c978e81] 2024-03-28T18:13:57.039 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff12b3931ca] 2024-03-28T18:13:57.039 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 16: clone() 2024-03-28T18:13:57.039 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2024-03-28T18:13:57.039 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: *** Caught signal (Aborted) ** 2024-03-28T18:13:57.039 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: in thread 7ff121fec700 thread_name:ms_dispatch 2024-03-28T18:13:57.039 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:13:57.039 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff12b39dd20] 2024-03-28T18:13:57.040 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2: gsignal() 2024-03-28T18:13:57.040 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 3: abort() 2024-03-28T18:13:57.040 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff12c6164fb] 2024-03-28T18:13:57.041 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff12c616667] 2024-03-28T18:13:57.041 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55adfb401223] 2024-03-28T18:13:57.041 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55adfb27b5d6] 2024-03-28T18:13:57.041 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55adfb27bda1] 2024-03-28T18:13:57.041 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55adfb27e2d5] 2024-03-28T18:13:57.041 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55adfb2ce233] 2024-03-28T18:13:57.041 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55adfb2d0197] 2024-03-28T18:13:57.041 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55adfb1cf11b] 2024-03-28T18:13:57.041 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55adfb1cf6bc] 2024-03-28T18:13:57.041 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55adfb1b91ff] 2024-03-28T18:13:57.041 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff12c8a8488] 2024-03-28T18:13:57.041 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 16: (DispatchQueue::entry()+0x51f) [0x7ff12c8a560f] 2024-03-28T18:13:57.041 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff12c978e81] 2024-03-28T18:13:57.042 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff12b3931ca] 2024-03-28T18:13:57.042 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 19: clone() 2024-03-28T18:13:57.042 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2024-03-28T18:13:56.738+0000 7ff121fec700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:13:57.042 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: in thread 7ff121fec700 thread_name:ms_dispatch 2024-03-28T18:13:57.042 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2024-03-28T18:13:57.042 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:13:57.042 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff12b39dd20] 2024-03-28T18:13:57.042 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2: gsignal() 2024-03-28T18:13:57.042 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 3: abort() 2024-03-28T18:13:57.042 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff12c6164fb] 2024-03-28T18:13:57.042 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff12c616667] 2024-03-28T18:13:57.042 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55adfb401223] 2024-03-28T18:13:57.042 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55adfb27b5d6] 2024-03-28T18:13:57.042 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55adfb27bda1] 2024-03-28T18:13:57.043 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55adfb27e2d5] 2024-03-28T18:13:57.043 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55adfb2ce233] 2024-03-28T18:13:57.043 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55adfb2d0197] 2024-03-28T18:13:57.043 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55adfb1cf11b] 2024-03-28T18:13:57.043 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55adfb1cf6bc] 2024-03-28T18:13:57.043 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55adfb1b91ff] 2024-03-28T18:13:57.043 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff12c8a8488] 2024-03-28T18:13:57.043 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 16: (DispatchQueue::entry()+0x51f) [0x7ff12c8a560f] 2024-03-28T18:13:57.043 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff12c978e81] 2024-03-28T18:13:57.043 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff12b3931ca] 2024-03-28T18:13:57.043 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 19: clone() 2024-03-28T18:13:57.043 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:13:57.043 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2024-03-28T18:13:57.043 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: -1> 2024-03-28T18:13:56.736+0000 7ff121fec700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff121fec700 time 2024-03-28T18:13:56.736332+0000 2024-03-28T18:13:57.043 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:13:57.044 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2024-03-28T18:13:57.044 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:13:57.044 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff12c6164a1] 2024-03-28T18:13:57.044 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff12c616667] 2024-03-28T18:13:57.044 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55adfb401223] 2024-03-28T18:13:57.044 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55adfb27b5d6] 2024-03-28T18:13:57.044 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55adfb27bda1] 2024-03-28T18:13:57.044 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55adfb27e2d5] 2024-03-28T18:13:57.044 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55adfb2ce233] 2024-03-28T18:13:57.044 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55adfb2d0197] 2024-03-28T18:13:57.044 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55adfb1cf11b] 2024-03-28T18:13:57.044 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55adfb1cf6bc] 2024-03-28T18:13:57.044 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55adfb1b91ff] 2024-03-28T18:13:57.045 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff12c8a8488] 2024-03-28T18:13:57.045 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 13: (DispatchQueue::entry()+0x51f) [0x7ff12c8a560f] 2024-03-28T18:13:57.045 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff12c978e81] 2024-03-28T18:13:57.045 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff12b3931ca] 2024-03-28T18:13:57.045 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 16: clone() 2024-03-28T18:13:57.045 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2024-03-28T18:13:57.045 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 0> 2024-03-28T18:13:56.738+0000 7ff121fec700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:13:57.045 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: in thread 7ff121fec700 thread_name:ms_dispatch 2024-03-28T18:13:57.045 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2024-03-28T18:13:57.045 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:13:57.045 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff12b39dd20] 2024-03-28T18:13:57.045 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2: gsignal() 2024-03-28T18:13:57.045 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 3: abort() 2024-03-28T18:13:57.046 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff12c6164fb] 2024-03-28T18:13:57.046 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff12c616667] 2024-03-28T18:13:57.046 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55adfb401223] 2024-03-28T18:13:57.046 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55adfb27b5d6] 2024-03-28T18:13:57.046 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55adfb27bda1] 2024-03-28T18:13:57.046 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55adfb27e2d5] 2024-03-28T18:13:57.046 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55adfb2ce233] 2024-03-28T18:13:57.046 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55adfb2d0197] 2024-03-28T18:13:57.046 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55adfb1cf11b] 2024-03-28T18:13:57.046 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55adfb1cf6bc] 2024-03-28T18:13:57.046 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55adfb1b91ff] 2024-03-28T18:13:57.046 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff12c8a8488] 2024-03-28T18:13:57.046 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 16: (DispatchQueue::entry()+0x51f) [0x7ff12c8a560f] 2024-03-28T18:13:57.046 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff12c978e81] 2024-03-28T18:13:57.047 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff12b3931ca] 2024-03-28T18:13:57.047 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 19: clone() 2024-03-28T18:13:57.047 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:13:57.047 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2024-03-28T18:13:57.047 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: -1> 2024-03-28T18:13:56.736+0000 7ff121fec700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff121fec700 time 2024-03-28T18:13:56.736332+0000 2024-03-28T18:13:57.047 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:13:57.047 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2024-03-28T18:13:57.047 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:13:57.047 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff12c6164a1] 2024-03-28T18:13:57.047 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff12c616667] 2024-03-28T18:13:57.047 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55adfb401223] 2024-03-28T18:13:57.047 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55adfb27b5d6] 2024-03-28T18:13:57.047 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55adfb27bda1] 2024-03-28T18:13:57.047 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55adfb27e2d5] 2024-03-28T18:13:57.048 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55adfb2ce233] 2024-03-28T18:13:57.048 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55adfb2d0197] 2024-03-28T18:13:57.048 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55adfb1cf11b] 2024-03-28T18:13:57.048 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55adfb1cf6bc] 2024-03-28T18:13:57.048 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55adfb1b91ff] 2024-03-28T18:13:57.048 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff12c8a8488] 2024-03-28T18:13:57.048 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 13: (DispatchQueue::entry()+0x51f) [0x7ff12c8a560f] 2024-03-28T18:13:57.048 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff12c978e81] 2024-03-28T18:13:57.048 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff12b3931ca] 2024-03-28T18:13:57.048 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 16: clone() 2024-03-28T18:13:57.048 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2024-03-28T18:13:57.049 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 0> 2024-03-28T18:13:56.738+0000 7ff121fec700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:13:57.049 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: in thread 7ff121fec700 thread_name:ms_dispatch 2024-03-28T18:13:57.049 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2024-03-28T18:13:57.049 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:13:57.049 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff12b39dd20] 2024-03-28T18:13:57.049 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2: gsignal() 2024-03-28T18:13:57.049 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 3: abort() 2024-03-28T18:13:57.049 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff12c6164fb] 2024-03-28T18:13:57.049 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff12c616667] 2024-03-28T18:13:57.049 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55adfb401223] 2024-03-28T18:13:57.049 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55adfb27b5d6] 2024-03-28T18:13:57.049 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55adfb27bda1] 2024-03-28T18:13:57.049 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55adfb27e2d5] 2024-03-28T18:13:57.049 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55adfb2ce233] 2024-03-28T18:13:57.049 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55adfb2d0197] 2024-03-28T18:13:57.050 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55adfb1cf11b] 2024-03-28T18:13:57.050 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55adfb1cf6bc] 2024-03-28T18:13:57.050 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55adfb1b91ff] 2024-03-28T18:13:57.050 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff12c8a8488] 2024-03-28T18:13:57.050 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 16: (DispatchQueue::entry()+0x51f) [0x7ff12c8a560f] 2024-03-28T18:13:57.050 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff12c978e81] 2024-03-28T18:13:57.050 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff12b3931ca] 2024-03-28T18:13:57.050 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 19: clone() 2024-03-28T18:13:57.050 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:13:57.050 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[66035]: 2024-03-28T18:13:57.050 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:56 smithi170 podman[70417]: 2024-03-28 18:13:56.797665517 +0000 UTC m=+0.020319995 container died ac5c3af1d16c4f15ddb809bdc133f8174963bc5fd5ca9853d18e8a059b4c53b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-28T18:13:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:56 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/4133178144,v1:172.21.15.170:6835/4133178144] up:rejoin 2024-03-28T18:13:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:56 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:13:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:56 smithi037 ceph-mon[31460]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-28T18:13:57.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:57.209+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:13:57.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:57.210+0000 7f4f82a00640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 msgr2=0x56357a39a010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-28T18:13:57.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:57.210+0000 7f4f82a00640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 msgr2=0x56357a39a010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T18:13:57.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:57.210+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T18:13:57.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:13:57.210+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-28T18:13:58.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:57 smithi170 podman[70417]: 2024-03-28 18:13:57.784532414 +0000 UTC m=+1.007186890 container remove ac5c3af1d16c4f15ddb809bdc133f8174963bc5fd5ca9853d18e8a059b4c53b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:13:58.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:57 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:13:58.023 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:57 smithi170 ceph-mon[36280]: pgmap v87: 97 pgs: 97 active+clean; 45 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 7 op/s 2024-03-28T18:13:58.023 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:57 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:13:58.023 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:57 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:13:58.023 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:57 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/4133178144,v1:172.21.15.170:6835/4133178144] up:active 2024-03-28T18:13:58.023 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:57 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:13:58.023 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:57 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:13:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:57 smithi059 ceph-mon[36190]: pgmap v87: 97 pgs: 97 active+clean; 45 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 7 op/s 2024-03-28T18:13:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:57 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:13:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:57 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:13:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:57 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/4133178144,v1:172.21.15.170:6835/4133178144] up:active 2024-03-28T18:13:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:57 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:13:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:57 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:13:58.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:57 smithi037 ceph-mon[31460]: pgmap v87: 97 pgs: 97 active+clean; 45 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 7 op/s 2024-03-28T18:13:58.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:57 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:13:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:57 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:13:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:57 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/4133178144,v1:172.21.15.170:6835/4133178144] up:active 2024-03-28T18:13:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:57 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:13:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:57 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:13:58.285 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:58 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-28T18:13:58.286 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:13:58 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Consumed 1.206s CPU time. 2024-03-28T18:14:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:13:59 smithi059 ceph-mon[36190]: pgmap v88: 97 pgs: 97 active+clean; 45 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 7 op/s 2024-03-28T18:14:00.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:13:59 smithi170 ceph-mon[36280]: pgmap v88: 97 pgs: 97 active+clean; 45 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 7 op/s 2024-03-28T18:14:00.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:13:59 smithi037 ceph-mon[31460]: pgmap v88: 97 pgs: 97 active+clean; 45 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 7 op/s 2024-03-28T18:14:01.680 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:14:01 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:14:01.398+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:14:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:01 smithi059 ceph-mon[36190]: pgmap v89: 97 pgs: 97 active+clean; 45 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 7 op/s 2024-03-28T18:14:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:01 smithi170 ceph-mon[36280]: pgmap v89: 97 pgs: 97 active+clean; 45 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 7 op/s 2024-03-28T18:14:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:01 smithi037 ceph-mon[31460]: pgmap v89: 97 pgs: 97 active+clean; 45 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 7 op/s 2024-03-28T18:14:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:03 smithi059 ceph-mon[36190]: pgmap v90: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 58 KiB/s wr, 22 op/s 2024-03-28T18:14:04.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:03 smithi170 ceph-mon[36280]: pgmap v90: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 58 KiB/s wr, 22 op/s 2024-03-28T18:14:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:03 smithi037 ceph-mon[31460]: pgmap v90: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 58 KiB/s wr, 22 op/s 2024-03-28T18:14:06.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:05 smithi059 ceph-mon[36190]: pgmap v91: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 50 KiB/s wr, 19 op/s 2024-03-28T18:14:06.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:05 smithi170 ceph-mon[36280]: pgmap v91: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 50 KiB/s wr, 19 op/s 2024-03-28T18:14:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:05 smithi037 ceph-mon[31460]: pgmap v91: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 50 KiB/s wr, 19 op/s 2024-03-28T18:14:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:07 smithi059 ceph-mon[36190]: pgmap v92: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 50 KiB/s wr, 21 op/s 2024-03-28T18:14:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:07 smithi170 ceph-mon[36280]: pgmap v92: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 50 KiB/s wr, 21 op/s 2024-03-28T18:14:08.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:07 smithi037 ceph-mon[31460]: pgmap v92: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 50 KiB/s wr, 21 op/s 2024-03-28T18:14:08.545 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:08 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-03-28T18:14:08.545 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:08 smithi170 systemd[1]: Stopped Ceph mds.f for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:14:08.545 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:08 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Consumed 1.206s CPU time. 2024-03-28T18:14:08.545 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:08 smithi170 systemd[1]: Starting Ceph mds.f for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:14:08.545 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:08 smithi170 podman[70513]: 2024-03-28 18:14:08.418229512 +0000 UTC m=+0.018269795 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:14:08.807 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:08 smithi170 podman[70513]: 2024-03-28 18:14:08.545908613 +0000 UTC m=+0.145948888 container create 6e13622d45cec7e48417c1208fbfa533fe7fbfc640e0ea2e9164a690b8453ca3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-28T18:14:09.285 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:08 smithi170 podman[70513]: 2024-03-28 18:14:08.807810182 +0000 UTC m=+0.407850444 container init 6e13622d45cec7e48417c1208fbfa533fe7fbfc640e0ea2e9164a690b8453ca3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-28T18:14:09.286 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:08 smithi170 podman[70513]: 2024-03-28 18:14:08.810870143 +0000 UTC m=+0.410910418 container start 6e13622d45cec7e48417c1208fbfa533fe7fbfc640e0ea2e9164a690b8453ca3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-28T18:14:09.286 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:08 smithi170 ceph-mds[70532]: -- 172.21.15.170:0/536701237 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560dbaa0ba40 con 0x560dba9dec00 2024-03-28T18:14:09.286 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:08 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: starting mds.f at 2024-03-28T18:14:09.286 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:08 smithi170 bash[70513]: 6e13622d45cec7e48417c1208fbfa533fe7fbfc640e0ea2e9164a690b8453ca3 2024-03-28T18:14:09.286 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:08 smithi170 systemd[1]: Started Ceph mds.f for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:14:10.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:14:10.012+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:14:10.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:14:10.012+0000 7f4f82a00640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 msgr2=0x56357a39a010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-28T18:14:10.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:14:10.013+0000 7f4f82a00640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 msgr2=0x56357a39a010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T18:14:10.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:14:10.013+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T18:14:10.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:14:10.013+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-28T18:14:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:09 smithi059 ceph-mon[36190]: pgmap v93: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-03-28T18:14:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:09 smithi059 ceph-mon[36190]: daemon mds.f restarted 2024-03-28T18:14:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:09 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:14:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:09 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:14:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:09 smithi059 ceph-mon[36190]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T18:14:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:09 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/3853013762,v1:172.21.15.170:6835/3853013762] up:boot 2024-03-28T18:14:10.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:09 smithi059 ceph-mon[36190]: Standby daemon mds.b assigned to filesystem cephfs as rank 1 2024-03-28T18:14:10.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:09 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:14:10.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:09 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:10.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:09 smithi170 ceph-mon[36280]: pgmap v93: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-03-28T18:14:10.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:09 smithi170 ceph-mon[36280]: daemon mds.f restarted 2024-03-28T18:14:10.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:09 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:14:10.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:09 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:14:10.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:09 smithi170 ceph-mon[36280]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T18:14:10.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:09 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/3853013762,v1:172.21.15.170:6835/3853013762] up:boot 2024-03-28T18:14:10.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:09 smithi170 ceph-mon[36280]: Standby daemon mds.b assigned to filesystem cephfs as rank 1 2024-03-28T18:14:10.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:09 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:14:10.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:09 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:10.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:09 smithi037 ceph-mon[31460]: pgmap v93: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-03-28T18:14:10.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:09 smithi037 ceph-mon[31460]: daemon mds.f restarted 2024-03-28T18:14:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:09 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:14:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:09 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:14:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:09 smithi037 ceph-mon[31460]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T18:14:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:09 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/3853013762,v1:172.21.15.170:6835/3853013762] up:boot 2024-03-28T18:14:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:09 smithi037 ceph-mon[31460]: Standby daemon mds.b assigned to filesystem cephfs as rank 1 2024-03-28T18:14:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:09 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:14:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:09 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:11 smithi059 ceph-mon[36190]: pgmap v95: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-03-28T18:14:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:11 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/362278957,v1:172.21.15.59:6839/362278957] up:resolve 2024-03-28T18:14:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:11 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:11 smithi170 ceph-mon[36280]: pgmap v95: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-03-28T18:14:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:11 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/362278957,v1:172.21.15.59:6839/362278957] up:resolve 2024-03-28T18:14:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:11 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:11 smithi037 ceph-mon[31460]: pgmap v95: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-03-28T18:14:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:11 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/362278957,v1:172.21.15.59:6839/362278957] up:resolve 2024-03-28T18:14:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:11 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:13.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:12 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/362278957,v1:172.21.15.59:6839/362278957] up:reconnect 2024-03-28T18:14:13.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:12 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:13.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:12 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999988 2024-03-28T18:14:13.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:12 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00999988 2024-03-28T18:14:13.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:12 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/362278957,v1:172.21.15.59:6839/362278957] up:reconnect 2024-03-28T18:14:13.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:12 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:13.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:12 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999988 2024-03-28T18:14:13.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:12 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00999988 2024-03-28T18:14:13.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:12 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/362278957,v1:172.21.15.59:6839/362278957] up:reconnect 2024-03-28T18:14:13.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:12 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:13.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:12 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999988 2024-03-28T18:14:13.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:12 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00999988 2024-03-28T18:14:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:13 smithi037 ceph-mon[31460]: pgmap v96: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-03-28T18:14:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:13 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/362278957,v1:172.21.15.59:6839/362278957] up:rejoin 2024-03-28T18:14:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:13 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:13 smithi037 ceph-mon[31460]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-28T18:14:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:13 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:14:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:13 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:14:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:13 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/362278957,v1:172.21.15.59:6839/362278957] up:active 2024-03-28T18:14:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:13 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:14:13.997 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-mon[36190]: pgmap v96: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-03-28T18:14:13.997 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/362278957,v1:172.21.15.59:6839/362278957] up:rejoin 2024-03-28T18:14:13.997 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:13.998 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-mon[36190]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-28T18:14:13.998 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:14:13.998 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:14:13.998 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/362278957,v1:172.21.15.59:6839/362278957] up:active 2024-03-28T18:14:13.998 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:14:14.001 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f6a99def700 time 2024-03-28T18:14:13.602946+0000 2024-03-28T18:14:14.001 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:14:14.001 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:14.001 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6aa44194a1] 2024-03-28T18:14:14.001 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6aa4419667] 2024-03-28T18:14:14.002 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5605a2876223] 2024-03-28T18:14:14.002 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5605a26f05d6] 2024-03-28T18:14:14.002 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5605a26f0da1] 2024-03-28T18:14:14.002 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5605a26f32d5] 2024-03-28T18:14:14.002 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5605a2743233] 2024-03-28T18:14:14.002 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5605a2745197] 2024-03-28T18:14:14.002 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5605a264411b] 2024-03-28T18:14:14.002 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5605a26446bc] 2024-03-28T18:14:14.003 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5605a262e1ff] 2024-03-28T18:14:14.003 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6aa46ab488] 2024-03-28T18:14:14.003 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 13: (DispatchQueue::entry()+0x51f) [0x7f6aa46a860f] 2024-03-28T18:14:14.003 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6aa477be81] 2024-03-28T18:14:14.003 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6aa31961ca] 2024-03-28T18:14:14.003 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 16: clone() 2024-03-28T18:14:14.003 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2024-03-28T18:14:13.603+0000 7f6a99def700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f6a99def700 time 2024-03-28T18:14:13.602946+0000 2024-03-28T18:14:14.003 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:14:14.003 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2024-03-28T18:14:14.004 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:14.004 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6aa44194a1] 2024-03-28T18:14:14.004 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6aa4419667] 2024-03-28T18:14:14.004 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5605a2876223] 2024-03-28T18:14:14.004 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5605a26f05d6] 2024-03-28T18:14:14.004 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5605a26f0da1] 2024-03-28T18:14:14.004 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5605a26f32d5] 2024-03-28T18:14:14.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5605a2743233] 2024-03-28T18:14:14.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5605a2745197] 2024-03-28T18:14:14.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5605a264411b] 2024-03-28T18:14:14.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5605a26446bc] 2024-03-28T18:14:14.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5605a262e1ff] 2024-03-28T18:14:14.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6aa46ab488] 2024-03-28T18:14:14.006 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 13: (DispatchQueue::entry()+0x51f) [0x7f6aa46a860f] 2024-03-28T18:14:14.006 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6aa477be81] 2024-03-28T18:14:14.006 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6aa31961ca] 2024-03-28T18:14:14.006 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 16: clone() 2024-03-28T18:14:14.006 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2024-03-28T18:14:14.006 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: *** Caught signal (Aborted) ** 2024-03-28T18:14:14.006 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: in thread 7f6a99def700 thread_name:ms_dispatch 2024-03-28T18:14:14.007 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:14.007 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6aa31a0d20] 2024-03-28T18:14:14.007 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2: gsignal() 2024-03-28T18:14:14.007 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 3: abort() 2024-03-28T18:14:14.007 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6aa44194fb] 2024-03-28T18:14:14.007 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6aa4419667] 2024-03-28T18:14:14.007 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5605a2876223] 2024-03-28T18:14:14.007 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5605a26f05d6] 2024-03-28T18:14:14.007 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5605a26f0da1] 2024-03-28T18:14:14.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5605a26f32d5] 2024-03-28T18:14:14.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5605a2743233] 2024-03-28T18:14:14.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5605a2745197] 2024-03-28T18:14:14.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5605a264411b] 2024-03-28T18:14:14.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5605a26446bc] 2024-03-28T18:14:14.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5605a262e1ff] 2024-03-28T18:14:14.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6aa46ab488] 2024-03-28T18:14:14.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 16: (DispatchQueue::entry()+0x51f) [0x7f6aa46a860f] 2024-03-28T18:14:14.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6aa477be81] 2024-03-28T18:14:14.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6aa31961ca] 2024-03-28T18:14:14.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 19: clone() 2024-03-28T18:14:14.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2024-03-28T18:14:13.605+0000 7f6a99def700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:14:14.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: in thread 7f6a99def700 thread_name:ms_dispatch 2024-03-28T18:14:14.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2024-03-28T18:14:14.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:14.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6aa31a0d20] 2024-03-28T18:14:14.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2: gsignal() 2024-03-28T18:14:14.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 3: abort() 2024-03-28T18:14:14.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6aa44194fb] 2024-03-28T18:14:14.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6aa4419667] 2024-03-28T18:14:14.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5605a2876223] 2024-03-28T18:14:14.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5605a26f05d6] 2024-03-28T18:14:14.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5605a26f0da1] 2024-03-28T18:14:14.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5605a26f32d5] 2024-03-28T18:14:14.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5605a2743233] 2024-03-28T18:14:14.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5605a2745197] 2024-03-28T18:14:14.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5605a264411b] 2024-03-28T18:14:14.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5605a26446bc] 2024-03-28T18:14:14.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5605a262e1ff] 2024-03-28T18:14:14.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6aa46ab488] 2024-03-28T18:14:14.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 16: (DispatchQueue::entry()+0x51f) [0x7f6aa46a860f] 2024-03-28T18:14:14.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6aa477be81] 2024-03-28T18:14:14.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6aa31961ca] 2024-03-28T18:14:14.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 19: clone() 2024-03-28T18:14:14.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:14:14.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2024-03-28T18:14:14.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: -1> 2024-03-28T18:14:13.603+0000 7f6a99def700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f6a99def700 time 2024-03-28T18:14:13.602946+0000 2024-03-28T18:14:14.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:14:14.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2024-03-28T18:14:14.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:14.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6aa44194a1] 2024-03-28T18:14:14.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6aa4419667] 2024-03-28T18:14:14.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5605a2876223] 2024-03-28T18:14:14.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5605a26f05d6] 2024-03-28T18:14:14.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5605a26f0da1] 2024-03-28T18:14:14.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5605a26f32d5] 2024-03-28T18:14:14.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5605a2743233] 2024-03-28T18:14:14.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5605a2745197] 2024-03-28T18:14:14.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5605a264411b] 2024-03-28T18:14:14.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5605a26446bc] 2024-03-28T18:14:14.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5605a262e1ff] 2024-03-28T18:14:14.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6aa46ab488] 2024-03-28T18:14:14.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 13: (DispatchQueue::entry()+0x51f) [0x7f6aa46a860f] 2024-03-28T18:14:14.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6aa477be81] 2024-03-28T18:14:14.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6aa31961ca] 2024-03-28T18:14:14.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 16: clone() 2024-03-28T18:14:14.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2024-03-28T18:14:14.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 0> 2024-03-28T18:14:13.605+0000 7f6a99def700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:14:14.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: in thread 7f6a99def700 thread_name:ms_dispatch 2024-03-28T18:14:14.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2024-03-28T18:14:14.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:14.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6aa31a0d20] 2024-03-28T18:14:14.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2: gsignal() 2024-03-28T18:14:14.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 3: abort() 2024-03-28T18:14:14.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6aa44194fb] 2024-03-28T18:14:14.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6aa4419667] 2024-03-28T18:14:14.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5605a2876223] 2024-03-28T18:14:14.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5605a26f05d6] 2024-03-28T18:14:14.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5605a26f0da1] 2024-03-28T18:14:14.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5605a26f32d5] 2024-03-28T18:14:14.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5605a2743233] 2024-03-28T18:14:14.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5605a2745197] 2024-03-28T18:14:14.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5605a264411b] 2024-03-28T18:14:14.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5605a26446bc] 2024-03-28T18:14:14.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5605a262e1ff] 2024-03-28T18:14:14.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6aa46ab488] 2024-03-28T18:14:14.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 16: (DispatchQueue::entry()+0x51f) [0x7f6aa46a860f] 2024-03-28T18:14:14.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6aa477be81] 2024-03-28T18:14:14.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6aa31961ca] 2024-03-28T18:14:14.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 19: clone() 2024-03-28T18:14:14.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:14:14.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2024-03-28T18:14:14.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: -1> 2024-03-28T18:14:13.603+0000 7f6a99def700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f6a99def700 time 2024-03-28T18:14:13.602946+0000 2024-03-28T18:14:14.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:14:14.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2024-03-28T18:14:14.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:14.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6aa44194a1] 2024-03-28T18:14:14.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6aa4419667] 2024-03-28T18:14:14.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5605a2876223] 2024-03-28T18:14:14.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5605a26f05d6] 2024-03-28T18:14:14.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5605a26f0da1] 2024-03-28T18:14:14.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5605a26f32d5] 2024-03-28T18:14:14.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5605a2743233] 2024-03-28T18:14:14.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5605a2745197] 2024-03-28T18:14:14.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5605a264411b] 2024-03-28T18:14:14.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5605a26446bc] 2024-03-28T18:14:14.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5605a262e1ff] 2024-03-28T18:14:14.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6aa46ab488] 2024-03-28T18:14:14.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 13: (DispatchQueue::entry()+0x51f) [0x7f6aa46a860f] 2024-03-28T18:14:14.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6aa477be81] 2024-03-28T18:14:14.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6aa31961ca] 2024-03-28T18:14:14.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 16: clone() 2024-03-28T18:14:14.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2024-03-28T18:14:14.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 0> 2024-03-28T18:14:13.605+0000 7f6a99def700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:14:14.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: in thread 7f6a99def700 thread_name:ms_dispatch 2024-03-28T18:14:14.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2024-03-28T18:14:14.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:14.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6aa31a0d20] 2024-03-28T18:14:14.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2: gsignal() 2024-03-28T18:14:14.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 3: abort() 2024-03-28T18:14:14.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6aa44194fb] 2024-03-28T18:14:14.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6aa4419667] 2024-03-28T18:14:14.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5605a2876223] 2024-03-28T18:14:14.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5605a26f05d6] 2024-03-28T18:14:14.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5605a26f0da1] 2024-03-28T18:14:14.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5605a26f32d5] 2024-03-28T18:14:14.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5605a2743233] 2024-03-28T18:14:14.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5605a2745197] 2024-03-28T18:14:14.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5605a264411b] 2024-03-28T18:14:14.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5605a26446bc] 2024-03-28T18:14:14.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5605a262e1ff] 2024-03-28T18:14:14.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6aa46ab488] 2024-03-28T18:14:14.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 16: (DispatchQueue::entry()+0x51f) [0x7f6aa46a860f] 2024-03-28T18:14:14.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6aa477be81] 2024-03-28T18:14:14.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6aa31961ca] 2024-03-28T18:14:14.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 19: clone() 2024-03-28T18:14:14.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:14:14.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70604]: 2024-03-28T18:14:14.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:13 smithi059 podman[70644]: 2024-03-28 18:14:13.684324455 +0000 UTC m=+0.020102352 container died b374db49781bd05ceda2da0610738b1620b638fa7b81b7e45d767be28f2394f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD) 2024-03-28T18:14:14.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:13 smithi170 ceph-mon[36280]: pgmap v96: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-03-28T18:14:14.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:13 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/362278957,v1:172.21.15.59:6839/362278957] up:rejoin 2024-03-28T18:14:14.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:13 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:14.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:13 smithi170 ceph-mon[36280]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-28T18:14:14.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:13 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:14:14.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:13 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:14:14.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:13 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/362278957,v1:172.21.15.59:6839/362278957] up:active 2024-03-28T18:14:14.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:13 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:14:14.785 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:14 smithi059 podman[70644]: 2024-03-28 18:14:14.328115512 +0000 UTC m=+0.663893390 container remove b374db49781bd05ceda2da0610738b1620b638fa7b81b7e45d767be28f2394f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:14:14.785 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:14 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:14:14.785 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:14 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.b.service: Failed with result 'exit-code'. 2024-03-28T18:14:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:15 smithi059 ceph-mon[36190]: pgmap v97: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-03-28T18:14:16.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:15 smithi170 ceph-mon[36280]: pgmap v97: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-03-28T18:14:16.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:15 smithi037 ceph-mon[31460]: pgmap v97: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-03-28T18:14:16.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:14:16 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:14:16.399+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:14:18.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:17 smithi059 ceph-mon[36190]: pgmap v98: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 31 KiB/s wr, 34 op/s 2024-03-28T18:14:18.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:17 smithi170 ceph-mon[36280]: pgmap v98: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 31 KiB/s wr, 34 op/s 2024-03-28T18:14:18.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:17 smithi037 ceph-mon[31460]: pgmap v98: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 31 KiB/s wr, 34 op/s 2024-03-28T18:14:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:19 smithi059 ceph-mon[36190]: pgmap v99: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 31 KiB/s wr, 34 op/s 2024-03-28T18:14:20.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:19 smithi170 ceph-mon[36280]: pgmap v99: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 31 KiB/s wr, 34 op/s 2024-03-28T18:14:20.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:19 smithi037 ceph-mon[31460]: pgmap v99: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 31 KiB/s wr, 34 op/s 2024-03-28T18:14:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:21 smithi059 ceph-mon[36190]: pgmap v100: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 27 KiB/s wr, 30 op/s 2024-03-28T18:14:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:21 smithi170 ceph-mon[36280]: pgmap v100: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 27 KiB/s wr, 30 op/s 2024-03-28T18:14:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:21 smithi037 ceph-mon[31460]: pgmap v100: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 27 KiB/s wr, 30 op/s 2024-03-28T18:14:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:23 smithi059 ceph-mon[36190]: pgmap v101: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 27 KiB/s wr, 32 op/s 2024-03-28T18:14:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:23 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:14:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:23 smithi170 ceph-mon[36280]: pgmap v101: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 27 KiB/s wr, 32 op/s 2024-03-28T18:14:24.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:23 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:14:24.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:23 smithi037 ceph-mon[31460]: pgmap v101: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 27 KiB/s wr, 32 op/s 2024-03-28T18:14:24.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:23 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:14:25.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:14:25.014+0000 7f4f82a00640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 msgr2=0x56357a39a010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:6838/2091629078 2024-03-28T18:14:25.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:14:25.014+0000 7f4f63e00640 1 client.25954 ms_handle_refused on v2:172.21.15.59:6838/2091629078 2024-03-28T18:14:25.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:14:25.014+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-28T18:14:25.023 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:24 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:14:25.023 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:24 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:14:25.023 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:24 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:14:25.023 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:24 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.b.service: Scheduled restart job, restart counter is at 2. 2024-03-28T18:14:25.023 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:24 smithi059 systemd[1]: Stopped Ceph mds.b for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:14:25.023 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:24 smithi059 systemd[1]: Starting Ceph mds.b for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:14:25.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:24 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:14:25.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:24 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:14:25.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:24 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:14:25.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:24 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:14:25.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:24 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:14:25.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:24 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:14:25.280 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:25 smithi059 podman[70871]: 2024-03-28 18:14:24.927337904 +0000 UTC m=+0.018639338 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:14:25.281 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:25 smithi059 podman[70871]: 2024-03-28 18:14:25.045785515 +0000 UTC m=+0.137086940 container create f5299f67b8ee88539cc53cf8abf27dd22aaaba54cd6698aff089e3625da9a619 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-28T18:14:25.535 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:25 smithi059 podman[70871]: 2024-03-28 18:14:25.281536236 +0000 UTC m=+0.372837669 container init f5299f67b8ee88539cc53cf8abf27dd22aaaba54cd6698aff089e3625da9a619 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-28T18:14:25.535 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:25 smithi059 podman[70871]: 2024-03-28 18:14:25.284470637 +0000 UTC m=+0.375772074 container start f5299f67b8ee88539cc53cf8abf27dd22aaaba54cd6698aff089e3625da9a619 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-28T18:14:25.536 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:25 smithi059 ceph-mds[70890]: -- 172.21.15.59:0/1501266344 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557773819a40 con 0x5577737ec800 2024-03-28T18:14:25.536 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: starting mds.b at 2024-03-28T18:14:25.536 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:25 smithi059 bash[70871]: f5299f67b8ee88539cc53cf8abf27dd22aaaba54cd6698aff089e3625da9a619 2024-03-28T18:14:25.536 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:25 smithi059 systemd[1]: Started Ceph mds.b for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:14:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:25 smithi059 ceph-mon[36190]: pgmap v102: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-28T18:14:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:25 smithi059 ceph-mon[36190]: daemon mds.b restarted 2024-03-28T18:14:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:25 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:14:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:25 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:14:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:25 smithi059 ceph-mon[36190]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T18:14:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:25 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/2774256011,v1:172.21.15.59:6839/2774256011] up:boot 2024-03-28T18:14:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:25 smithi059 ceph-mon[36190]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-28T18:14:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:25 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:14:26.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:25 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:26.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:25 smithi170 ceph-mon[36280]: pgmap v102: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-28T18:14:26.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:25 smithi170 ceph-mon[36280]: daemon mds.b restarted 2024-03-28T18:14:26.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:25 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:14:26.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:25 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:14:26.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:25 smithi170 ceph-mon[36280]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T18:14:26.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:25 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/2774256011,v1:172.21.15.59:6839/2774256011] up:boot 2024-03-28T18:14:26.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:25 smithi170 ceph-mon[36280]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-28T18:14:26.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:25 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:14:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:25 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:25 smithi037 ceph-mon[31460]: pgmap v102: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-28T18:14:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:25 smithi037 ceph-mon[31460]: daemon mds.b restarted 2024-03-28T18:14:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:25 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:14:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:25 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:14:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:25 smithi037 ceph-mon[31460]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T18:14:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:25 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/2774256011,v1:172.21.15.59:6839/2774256011] up:boot 2024-03-28T18:14:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:25 smithi037 ceph-mon[31460]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-28T18:14:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:25 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:14:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:25 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:27.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:27 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/3853013762,v1:172.21.15.170:6835/3853013762] up:resolve 2024-03-28T18:14:27.627 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:27 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:27.627 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:27 smithi037 ceph-mon[31460]: pgmap v104: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-28T18:14:27.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:27 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/3853013762,v1:172.21.15.170:6835/3853013762] up:resolve 2024-03-28T18:14:27.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:27 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:27.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:27 smithi059 ceph-mon[36190]: pgmap v104: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-28T18:14:27.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:27 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/3853013762,v1:172.21.15.170:6835/3853013762] up:resolve 2024-03-28T18:14:27.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:27 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:27.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:27 smithi170 ceph-mon[36280]: pgmap v104: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-28T18:14:28.610 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199995 2024-03-28T18:14:28.610 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/3853013762,v1:172.21.15.170:6835/3853013762] up:reconnect 2024-03-28T18:14:28.610 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:28.610 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089998 2024-03-28T18:14:28.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:28 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199995 2024-03-28T18:14:28.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:28 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/3853013762,v1:172.21.15.170:6835/3853013762] up:reconnect 2024-03-28T18:14:28.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:28 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:28.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:28 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089998 2024-03-28T18:14:28.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:28 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199995 2024-03-28T18:14:28.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:28 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/3853013762,v1:172.21.15.170:6835/3853013762] up:reconnect 2024-03-28T18:14:28.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:28 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:28.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:28 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089998 2024-03-28T18:14:28.977 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa196403700 time 2024-03-28T18:14:28.611060+0000 2024-03-28T18:14:28.977 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:14:28.978 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:28.978 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa1a0a2d4a1] 2024-03-28T18:14:28.978 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa1a0a2d667] 2024-03-28T18:14:28.978 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560db8a01223] 2024-03-28T18:14:28.978 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560db887b5d6] 2024-03-28T18:14:28.978 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560db887bda1] 2024-03-28T18:14:28.978 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560db887e2d5] 2024-03-28T18:14:28.978 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560db88ce233] 2024-03-28T18:14:28.978 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560db88d0197] 2024-03-28T18:14:28.978 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560db87cf11b] 2024-03-28T18:14:28.979 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560db87cf6bc] 2024-03-28T18:14:28.979 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560db87b91ff] 2024-03-28T18:14:28.979 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa1a0cbf488] 2024-03-28T18:14:28.979 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 13: (DispatchQueue::entry()+0x51f) [0x7fa1a0cbc60f] 2024-03-28T18:14:28.979 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa1a0d8fe81] 2024-03-28T18:14:28.979 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa19f7aa1ca] 2024-03-28T18:14:28.979 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 16: clone() 2024-03-28T18:14:28.979 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: *** Caught signal (Aborted) ** 2024-03-28T18:14:28.979 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: in thread 7fa196403700 thread_name:ms_dispatch 2024-03-28T18:14:28.979 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2024-03-28T18:14:28.612+0000 7fa196403700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa196403700 time 2024-03-28T18:14:28.611060+0000 2024-03-28T18:14:28.979 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:14:28.980 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2024-03-28T18:14:28.980 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:28.980 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa1a0a2d4a1] 2024-03-28T18:14:28.980 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa1a0a2d667] 2024-03-28T18:14:28.980 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560db8a01223] 2024-03-28T18:14:28.980 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560db887b5d6] 2024-03-28T18:14:28.980 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560db887bda1] 2024-03-28T18:14:28.980 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560db887e2d5] 2024-03-28T18:14:28.980 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560db88ce233] 2024-03-28T18:14:28.980 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560db88d0197] 2024-03-28T18:14:28.980 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560db87cf11b] 2024-03-28T18:14:28.980 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560db87cf6bc] 2024-03-28T18:14:28.980 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560db87b91ff] 2024-03-28T18:14:28.980 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa1a0cbf488] 2024-03-28T18:14:28.981 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 13: (DispatchQueue::entry()+0x51f) [0x7fa1a0cbc60f] 2024-03-28T18:14:28.981 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa1a0d8fe81] 2024-03-28T18:14:28.981 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa19f7aa1ca] 2024-03-28T18:14:28.981 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 16: clone() 2024-03-28T18:14:28.981 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2024-03-28T18:14:28.981 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:28.981 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa19f7b4d20] 2024-03-28T18:14:28.982 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2: gsignal() 2024-03-28T18:14:28.982 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 3: abort() 2024-03-28T18:14:28.982 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa1a0a2d4fb] 2024-03-28T18:14:28.982 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa1a0a2d667] 2024-03-28T18:14:28.982 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560db8a01223] 2024-03-28T18:14:28.982 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560db887b5d6] 2024-03-28T18:14:28.983 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560db887bda1] 2024-03-28T18:14:28.983 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560db887e2d5] 2024-03-28T18:14:28.983 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560db88ce233] 2024-03-28T18:14:28.983 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560db88d0197] 2024-03-28T18:14:28.983 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560db87cf11b] 2024-03-28T18:14:28.983 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560db87cf6bc] 2024-03-28T18:14:28.983 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560db87b91ff] 2024-03-28T18:14:28.983 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa1a0cbf488] 2024-03-28T18:14:28.983 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 16: (DispatchQueue::entry()+0x51f) [0x7fa1a0cbc60f] 2024-03-28T18:14:28.983 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa1a0d8fe81] 2024-03-28T18:14:28.983 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa19f7aa1ca] 2024-03-28T18:14:28.983 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 19: clone() 2024-03-28T18:14:28.983 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2024-03-28T18:14:28.614+0000 7fa196403700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:14:28.984 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: in thread 7fa196403700 thread_name:ms_dispatch 2024-03-28T18:14:28.984 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2024-03-28T18:14:28.984 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:28.984 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa19f7b4d20] 2024-03-28T18:14:28.984 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2: gsignal() 2024-03-28T18:14:28.984 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 3: abort() 2024-03-28T18:14:28.984 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa1a0a2d4fb] 2024-03-28T18:14:28.984 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa1a0a2d667] 2024-03-28T18:14:28.984 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560db8a01223] 2024-03-28T18:14:28.984 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560db887b5d6] 2024-03-28T18:14:28.984 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560db887bda1] 2024-03-28T18:14:28.984 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560db887e2d5] 2024-03-28T18:14:28.984 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560db88ce233] 2024-03-28T18:14:28.985 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560db88d0197] 2024-03-28T18:14:28.985 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560db87cf11b] 2024-03-28T18:14:28.985 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560db87cf6bc] 2024-03-28T18:14:28.985 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560db87b91ff] 2024-03-28T18:14:28.985 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa1a0cbf488] 2024-03-28T18:14:28.985 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 16: (DispatchQueue::entry()+0x51f) [0x7fa1a0cbc60f] 2024-03-28T18:14:28.985 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa1a0d8fe81] 2024-03-28T18:14:28.985 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa19f7aa1ca] 2024-03-28T18:14:28.985 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 19: clone() 2024-03-28T18:14:28.985 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:14:28.985 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2024-03-28T18:14:28.985 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: -1> 2024-03-28T18:14:28.612+0000 7fa196403700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa196403700 time 2024-03-28T18:14:28.611060+0000 2024-03-28T18:14:28.985 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:14:28.985 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2024-03-28T18:14:28.985 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:28.986 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa1a0a2d4a1] 2024-03-28T18:14:28.986 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa1a0a2d667] 2024-03-28T18:14:28.986 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560db8a01223] 2024-03-28T18:14:28.986 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560db887b5d6] 2024-03-28T18:14:28.986 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560db887bda1] 2024-03-28T18:14:28.986 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560db887e2d5] 2024-03-28T18:14:28.986 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560db88ce233] 2024-03-28T18:14:28.986 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560db88d0197] 2024-03-28T18:14:28.986 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560db87cf11b] 2024-03-28T18:14:28.986 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560db87cf6bc] 2024-03-28T18:14:28.986 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560db87b91ff] 2024-03-28T18:14:28.986 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa1a0cbf488] 2024-03-28T18:14:28.986 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 13: (DispatchQueue::entry()+0x51f) [0x7fa1a0cbc60f] 2024-03-28T18:14:28.987 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa1a0d8fe81] 2024-03-28T18:14:28.987 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa19f7aa1ca] 2024-03-28T18:14:28.987 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 16: clone() 2024-03-28T18:14:28.987 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2024-03-28T18:14:28.987 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 0> 2024-03-28T18:14:28.614+0000 7fa196403700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:14:28.987 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: in thread 7fa196403700 thread_name:ms_dispatch 2024-03-28T18:14:28.987 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2024-03-28T18:14:28.987 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:28.987 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa19f7b4d20] 2024-03-28T18:14:28.987 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2: gsignal() 2024-03-28T18:14:28.987 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 3: abort() 2024-03-28T18:14:28.987 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa1a0a2d4fb] 2024-03-28T18:14:28.987 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa1a0a2d667] 2024-03-28T18:14:28.987 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560db8a01223] 2024-03-28T18:14:28.988 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560db887b5d6] 2024-03-28T18:14:28.988 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560db887bda1] 2024-03-28T18:14:28.988 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560db887e2d5] 2024-03-28T18:14:28.988 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560db88ce233] 2024-03-28T18:14:28.988 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560db88d0197] 2024-03-28T18:14:28.988 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560db87cf11b] 2024-03-28T18:14:28.988 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560db87cf6bc] 2024-03-28T18:14:28.988 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560db87b91ff] 2024-03-28T18:14:28.988 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa1a0cbf488] 2024-03-28T18:14:28.988 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 16: (DispatchQueue::entry()+0x51f) [0x7fa1a0cbc60f] 2024-03-28T18:14:28.988 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa1a0d8fe81] 2024-03-28T18:14:28.988 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa19f7aa1ca] 2024-03-28T18:14:28.988 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 19: clone() 2024-03-28T18:14:28.989 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:14:28.989 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2024-03-28T18:14:28.989 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: -1> 2024-03-28T18:14:28.612+0000 7fa196403700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa196403700 time 2024-03-28T18:14:28.611060+0000 2024-03-28T18:14:28.989 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:14:28.989 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2024-03-28T18:14:28.989 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:28.989 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa1a0a2d4a1] 2024-03-28T18:14:28.989 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa1a0a2d667] 2024-03-28T18:14:28.989 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560db8a01223] 2024-03-28T18:14:28.989 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560db887b5d6] 2024-03-28T18:14:28.989 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560db887bda1] 2024-03-28T18:14:28.989 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560db887e2d5] 2024-03-28T18:14:28.989 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560db88ce233] 2024-03-28T18:14:28.989 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560db88d0197] 2024-03-28T18:14:28.990 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560db87cf11b] 2024-03-28T18:14:28.990 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560db87cf6bc] 2024-03-28T18:14:28.990 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560db87b91ff] 2024-03-28T18:14:28.990 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa1a0cbf488] 2024-03-28T18:14:28.990 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 13: (DispatchQueue::entry()+0x51f) [0x7fa1a0cbc60f] 2024-03-28T18:14:28.990 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa1a0d8fe81] 2024-03-28T18:14:28.990 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa19f7aa1ca] 2024-03-28T18:14:28.990 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 16: clone() 2024-03-28T18:14:28.990 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2024-03-28T18:14:28.990 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 0> 2024-03-28T18:14:28.614+0000 7fa196403700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:14:28.990 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: in thread 7fa196403700 thread_name:ms_dispatch 2024-03-28T18:14:28.990 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2024-03-28T18:14:28.990 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:28.990 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa19f7b4d20] 2024-03-28T18:14:28.991 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2: gsignal() 2024-03-28T18:14:28.991 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 3: abort() 2024-03-28T18:14:28.991 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa1a0a2d4fb] 2024-03-28T18:14:28.991 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa1a0a2d667] 2024-03-28T18:14:28.991 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560db8a01223] 2024-03-28T18:14:28.991 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560db887b5d6] 2024-03-28T18:14:28.991 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560db887bda1] 2024-03-28T18:14:28.991 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560db887e2d5] 2024-03-28T18:14:28.991 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560db88ce233] 2024-03-28T18:14:28.991 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560db88d0197] 2024-03-28T18:14:28.991 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560db87cf11b] 2024-03-28T18:14:28.991 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560db87cf6bc] 2024-03-28T18:14:28.991 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560db87b91ff] 2024-03-28T18:14:28.991 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa1a0cbf488] 2024-03-28T18:14:28.992 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 16: (DispatchQueue::entry()+0x51f) [0x7fa1a0cbc60f] 2024-03-28T18:14:28.992 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa1a0d8fe81] 2024-03-28T18:14:28.992 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa19f7aa1ca] 2024-03-28T18:14:28.992 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 19: clone() 2024-03-28T18:14:28.992 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:14:28.992 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70528]: 2024-03-28T18:14:28.992 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:28 smithi170 podman[70701]: 2024-03-28 18:14:28.670532788 +0000 UTC m=+0.019816005 container died 6e13622d45cec7e48417c1208fbfa533fe7fbfc640e0ea2e9164a690b8453ca3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-28T18:14:29.684 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:29 smithi170 podman[70701]: 2024-03-28 18:14:29.454301065 +0000 UTC m=+0.803584284 container remove 6e13622d45cec7e48417c1208fbfa533fe7fbfc640e0ea2e9164a690b8453ca3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-28T18:14:29.684 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:29 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:14:29.685 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:29 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/3853013762,v1:172.21.15.170:6835/3853013762] up:rejoin 2024-03-28T18:14:29.685 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:29 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:29.685 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:29 smithi170 ceph-mon[36280]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-28T18:14:29.685 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:29 smithi170 ceph-mon[36280]: pgmap v105: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-28T18:14:29.685 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:29 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:14:29.685 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:29 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:14:29.685 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:29 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/3853013762,v1:172.21.15.170:6835/3853013762] up:active 2024-03-28T18:14:29.685 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:29 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:14:29.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:29 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/3853013762,v1:172.21.15.170:6835/3853013762] up:rejoin 2024-03-28T18:14:29.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:29 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:29.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:29 smithi059 ceph-mon[36190]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-28T18:14:29.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:29 smithi059 ceph-mon[36190]: pgmap v105: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-28T18:14:29.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:29 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:14:29.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:29 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:14:29.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:29 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/3853013762,v1:172.21.15.170:6835/3853013762] up:active 2024-03-28T18:14:29.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:29 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:14:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:29 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/3853013762,v1:172.21.15.170:6835/3853013762] up:rejoin 2024-03-28T18:14:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:29 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:29 smithi037 ceph-mon[31460]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-28T18:14:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:29 smithi037 ceph-mon[31460]: pgmap v105: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-28T18:14:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:29 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:14:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:29 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:14:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:29 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/3853013762,v1:172.21.15.170:6835/3853013762] up:active 2024-03-28T18:14:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:29 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:14:30.035 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:29 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-28T18:14:31.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:31 smithi170 ceph-mon[36280]: pgmap v106: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-28T18:14:31.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:31 smithi059 ceph-mon[36190]: pgmap v106: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-28T18:14:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:31 smithi037 ceph-mon[31460]: pgmap v106: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-28T18:14:31.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:14:31 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:14:31.400+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:14:33.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:33 smithi059 ceph-mon[36190]: pgmap v107: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.9 KiB/s wr, 18 op/s 2024-03-28T18:14:33.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:33 smithi170 ceph-mon[36280]: pgmap v107: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.9 KiB/s wr, 18 op/s 2024-03-28T18:14:33.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:33 smithi037 ceph-mon[31460]: pgmap v107: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.9 KiB/s wr, 18 op/s 2024-03-28T18:14:35.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:35 smithi059 ceph-mon[36190]: pgmap v108: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.9 KiB/s wr, 18 op/s 2024-03-28T18:14:35.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:35 smithi170 ceph-mon[36280]: pgmap v108: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.9 KiB/s wr, 18 op/s 2024-03-28T18:14:35.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:35 smithi037 ceph-mon[31460]: pgmap v108: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.9 KiB/s wr, 18 op/s 2024-03-28T18:14:37.776 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:37 smithi037 ceph-mon[31460]: pgmap v109: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 2.6 KiB/s wr, 19 op/s 2024-03-28T18:14:37.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:37 smithi059 ceph-mon[36190]: pgmap v109: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 2.6 KiB/s wr, 19 op/s 2024-03-28T18:14:37.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:37 smithi170 ceph-mon[36280]: pgmap v109: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 2.6 KiB/s wr, 19 op/s 2024-03-28T18:14:39.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:39 smithi059 ceph-mon[36190]: pgmap v110: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 2.4 KiB/s wr, 15 op/s 2024-03-28T18:14:39.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:39 smithi170 ceph-mon[36280]: pgmap v110: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 2.4 KiB/s wr, 15 op/s 2024-03-28T18:14:39.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:39 smithi037 ceph-mon[31460]: pgmap v110: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 2.4 KiB/s wr, 15 op/s 2024-03-28T18:14:40.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:14:40.017+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:14:40.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:14:40.018+0000 7f4f82a00640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 msgr2=0x56357a39a010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-28T18:14:40.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:14:40.018+0000 7f4f82a00640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 msgr2=0x56357a39a010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T18:14:40.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:14:40.018+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T18:14:40.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:14:40.018+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-28T18:14:40.104 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:39 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 2. 2024-03-28T18:14:40.104 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:39 smithi170 systemd[1]: Stopped Ceph mds.f for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:14:40.104 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:39 smithi170 systemd[1]: Starting Ceph mds.f for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:14:40.104 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:40 smithi170 podman[70798]: 2024-03-28 18:14:39.917107466 +0000 UTC m=+0.017887158 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:14:40.357 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:40 smithi170 podman[70798]: 2024-03-28 18:14:40.105221527 +0000 UTC m=+0.206001202 container create 9fb7624a517b742ba2f7ff202ea293ad36a6c015348ca43780cb17bb67e41cb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-28T18:14:40.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:40 smithi059 ceph-mon[36190]: daemon mds.f restarted 2024-03-28T18:14:40.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:40 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:14:40.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:40 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:14:40.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:40 smithi059 ceph-mon[36190]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T18:14:40.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:40 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/1393026034,v1:172.21.15.170:6835/1393026034] up:boot 2024-03-28T18:14:40.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:40 smithi059 ceph-mon[36190]: Standby daemon mds.b assigned to filesystem cephfs as rank 1 2024-03-28T18:14:40.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:40 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:14:40.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:40 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:40.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:40 smithi170 ceph-mon[36280]: daemon mds.f restarted 2024-03-28T18:14:40.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:40 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:14:40.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:40 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:14:40.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:40 smithi170 ceph-mon[36280]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T18:14:40.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:40 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/1393026034,v1:172.21.15.170:6835/1393026034] up:boot 2024-03-28T18:14:40.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:40 smithi170 ceph-mon[36280]: Standby daemon mds.b assigned to filesystem cephfs as rank 1 2024-03-28T18:14:40.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:40 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:14:40.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:40 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:40.787 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:40 smithi170 podman[70798]: 2024-03-28 18:14:40.358116018 +0000 UTC m=+0.458895692 container init 9fb7624a517b742ba2f7ff202ea293ad36a6c015348ca43780cb17bb67e41cb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-28T18:14:40.787 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:40 smithi170 podman[70798]: 2024-03-28 18:14:40.361817821 +0000 UTC m=+0.462597487 container start 9fb7624a517b742ba2f7ff202ea293ad36a6c015348ca43780cb17bb67e41cb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-28T18:14:40.787 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:40 smithi170 ceph-mds[70818]: -- 172.21.15.170:0/1300090112 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a12e307a40 con 0x55a12e2dac00 2024-03-28T18:14:40.787 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: starting mds.f at 2024-03-28T18:14:40.787 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:40 smithi170 bash[70798]: 9fb7624a517b742ba2f7ff202ea293ad36a6c015348ca43780cb17bb67e41cb2 2024-03-28T18:14:40.787 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:40 smithi170 systemd[1]: Started Ceph mds.f for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:14:40.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:40 smithi037 ceph-mon[31460]: daemon mds.f restarted 2024-03-28T18:14:40.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:40 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:14:40.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:40 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:14:40.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:40 smithi037 ceph-mon[31460]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T18:14:40.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:40 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/1393026034,v1:172.21.15.170:6835/1393026034] up:boot 2024-03-28T18:14:40.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:40 smithi037 ceph-mon[31460]: Standby daemon mds.b assigned to filesystem cephfs as rank 1 2024-03-28T18:14:40.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:40 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:14:40.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:40 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:41.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:41 smithi059 ceph-mon[36190]: pgmap v112: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 2.9 KiB/s wr, 18 op/s 2024-03-28T18:14:41.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:41 smithi170 ceph-mon[36280]: pgmap v112: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 2.9 KiB/s wr, 18 op/s 2024-03-28T18:14:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:41 smithi037 ceph-mon[31460]: pgmap v112: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 2.9 KiB/s wr, 18 op/s 2024-03-28T18:14:42.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:42 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/2774256011,v1:172.21.15.59:6839/2774256011] up:resolve 2024-03-28T18:14:42.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:42 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:43.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:42 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/2774256011,v1:172.21.15.59:6839/2774256011] up:resolve 2024-03-28T18:14:43.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:42 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:43.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:42 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/2774256011,v1:172.21.15.59:6839/2774256011] up:resolve 2024-03-28T18:14:43.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:42 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:43.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:43 smithi037 ceph-mon[31460]: pgmap v113: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 409 B/s wr, 6 op/s 2024-03-28T18:14:43.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:43 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199998 2024-03-28T18:14:43.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:43 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/2774256011,v1:172.21.15.59:6839/2774256011] up:reconnect 2024-03-28T18:14:43.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:43 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:43.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:43 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089999 2024-03-28T18:14:44.002 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-mon[36190]: pgmap v113: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 409 B/s wr, 6 op/s 2024-03-28T18:14:44.002 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199998 2024-03-28T18:14:44.002 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/2774256011,v1:172.21.15.59:6839/2774256011] up:reconnect 2024-03-28T18:14:44.002 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:44.002 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089999 2024-03-28T18:14:44.003 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f86c4fd3700 time 2024-03-28T18:14:43.630113+0000 2024-03-28T18:14:44.003 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:14:44.003 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:44.003 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f86cf5fd4a1] 2024-03-28T18:14:44.003 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f86cf5fd667] 2024-03-28T18:14:44.003 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557772b77223] 2024-03-28T18:14:44.003 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5577729f15d6] 2024-03-28T18:14:44.003 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5577729f1da1] 2024-03-28T18:14:44.003 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5577729f42d5] 2024-03-28T18:14:44.003 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557772a44233] 2024-03-28T18:14:44.004 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557772a46197] 2024-03-28T18:14:44.004 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55777294511b] 2024-03-28T18:14:44.004 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5577729456bc] 2024-03-28T18:14:44.004 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55777292f1ff] 2024-03-28T18:14:44.004 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f86cf88f488] 2024-03-28T18:14:44.004 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 13: (DispatchQueue::entry()+0x51f) [0x7f86cf88c60f] 2024-03-28T18:14:44.004 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f86cf95fe81] 2024-03-28T18:14:44.004 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f86ce37a1ca] 2024-03-28T18:14:44.004 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 16: clone() 2024-03-28T18:14:44.004 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: *** Caught signal (Aborted) ** 2024-03-28T18:14:44.004 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: in thread 7f86c4fd3700 thread_name:ms_dispatch 2024-03-28T18:14:44.004 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2024-03-28T18:14:43.630+0000 7f86c4fd3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f86c4fd3700 time 2024-03-28T18:14:43.630113+0000 2024-03-28T18:14:44.004 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:14:44.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2024-03-28T18:14:44.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:44.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f86cf5fd4a1] 2024-03-28T18:14:44.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f86cf5fd667] 2024-03-28T18:14:44.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557772b77223] 2024-03-28T18:14:44.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5577729f15d6] 2024-03-28T18:14:44.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5577729f1da1] 2024-03-28T18:14:44.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5577729f42d5] 2024-03-28T18:14:44.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557772a44233] 2024-03-28T18:14:44.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557772a46197] 2024-03-28T18:14:44.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55777294511b] 2024-03-28T18:14:44.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5577729456bc] 2024-03-28T18:14:44.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55777292f1ff] 2024-03-28T18:14:44.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f86cf88f488] 2024-03-28T18:14:44.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 13: (DispatchQueue::entry()+0x51f) [0x7f86cf88c60f] 2024-03-28T18:14:44.005 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f86cf95fe81] 2024-03-28T18:14:44.006 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f86ce37a1ca] 2024-03-28T18:14:44.006 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 16: clone() 2024-03-28T18:14:44.006 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2024-03-28T18:14:44.006 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:44.006 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f86ce384d20] 2024-03-28T18:14:44.007 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2: gsignal() 2024-03-28T18:14:44.007 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 3: abort() 2024-03-28T18:14:44.007 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f86cf5fd4fb] 2024-03-28T18:14:44.007 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f86cf5fd667] 2024-03-28T18:14:44.007 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557772b77223] 2024-03-28T18:14:44.007 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5577729f15d6] 2024-03-28T18:14:44.007 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5577729f1da1] 2024-03-28T18:14:44.007 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5577729f42d5] 2024-03-28T18:14:44.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557772a44233] 2024-03-28T18:14:44.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557772a46197] 2024-03-28T18:14:44.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55777294511b] 2024-03-28T18:14:44.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5577729456bc] 2024-03-28T18:14:44.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55777292f1ff] 2024-03-28T18:14:44.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f86cf88f488] 2024-03-28T18:14:44.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 16: (DispatchQueue::entry()+0x51f) [0x7f86cf88c60f] 2024-03-28T18:14:44.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f86cf95fe81] 2024-03-28T18:14:44.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f86ce37a1ca] 2024-03-28T18:14:44.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 19: clone() 2024-03-28T18:14:44.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2024-03-28T18:14:43.632+0000 7f86c4fd3700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:14:44.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: in thread 7f86c4fd3700 thread_name:ms_dispatch 2024-03-28T18:14:44.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2024-03-28T18:14:44.008 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:44.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f86ce384d20] 2024-03-28T18:14:44.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2: gsignal() 2024-03-28T18:14:44.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 3: abort() 2024-03-28T18:14:44.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f86cf5fd4fb] 2024-03-28T18:14:44.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f86cf5fd667] 2024-03-28T18:14:44.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557772b77223] 2024-03-28T18:14:44.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5577729f15d6] 2024-03-28T18:14:44.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5577729f1da1] 2024-03-28T18:14:44.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5577729f42d5] 2024-03-28T18:14:44.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557772a44233] 2024-03-28T18:14:44.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557772a46197] 2024-03-28T18:14:44.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55777294511b] 2024-03-28T18:14:44.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5577729456bc] 2024-03-28T18:14:44.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55777292f1ff] 2024-03-28T18:14:44.009 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f86cf88f488] 2024-03-28T18:14:44.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 16: (DispatchQueue::entry()+0x51f) [0x7f86cf88c60f] 2024-03-28T18:14:44.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f86cf95fe81] 2024-03-28T18:14:44.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f86ce37a1ca] 2024-03-28T18:14:44.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 19: clone() 2024-03-28T18:14:44.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:14:44.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2024-03-28T18:14:44.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: -1> 2024-03-28T18:14:43.630+0000 7f86c4fd3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f86c4fd3700 time 2024-03-28T18:14:43.630113+0000 2024-03-28T18:14:44.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:14:44.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2024-03-28T18:14:44.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:44.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f86cf5fd4a1] 2024-03-28T18:14:44.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f86cf5fd667] 2024-03-28T18:14:44.010 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557772b77223] 2024-03-28T18:14:44.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5577729f15d6] 2024-03-28T18:14:44.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5577729f1da1] 2024-03-28T18:14:44.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5577729f42d5] 2024-03-28T18:14:44.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557772a44233] 2024-03-28T18:14:44.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557772a46197] 2024-03-28T18:14:44.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55777294511b] 2024-03-28T18:14:44.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5577729456bc] 2024-03-28T18:14:44.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55777292f1ff] 2024-03-28T18:14:44.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f86cf88f488] 2024-03-28T18:14:44.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 13: (DispatchQueue::entry()+0x51f) [0x7f86cf88c60f] 2024-03-28T18:14:44.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f86cf95fe81] 2024-03-28T18:14:44.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f86ce37a1ca] 2024-03-28T18:14:44.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 16: clone() 2024-03-28T18:14:44.011 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2024-03-28T18:14:44.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 0> 2024-03-28T18:14:43.632+0000 7f86c4fd3700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:14:44.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: in thread 7f86c4fd3700 thread_name:ms_dispatch 2024-03-28T18:14:44.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2024-03-28T18:14:44.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:44.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f86ce384d20] 2024-03-28T18:14:44.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2: gsignal() 2024-03-28T18:14:44.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 3: abort() 2024-03-28T18:14:44.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f86cf5fd4fb] 2024-03-28T18:14:44.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f86cf5fd667] 2024-03-28T18:14:44.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557772b77223] 2024-03-28T18:14:44.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5577729f15d6] 2024-03-28T18:14:44.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5577729f1da1] 2024-03-28T18:14:44.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5577729f42d5] 2024-03-28T18:14:44.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557772a44233] 2024-03-28T18:14:44.012 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557772a46197] 2024-03-28T18:14:44.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55777294511b] 2024-03-28T18:14:44.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5577729456bc] 2024-03-28T18:14:44.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55777292f1ff] 2024-03-28T18:14:44.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f86cf88f488] 2024-03-28T18:14:44.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 16: (DispatchQueue::entry()+0x51f) [0x7f86cf88c60f] 2024-03-28T18:14:44.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f86cf95fe81] 2024-03-28T18:14:44.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f86ce37a1ca] 2024-03-28T18:14:44.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 19: clone() 2024-03-28T18:14:44.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:14:44.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2024-03-28T18:14:44.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: -1> 2024-03-28T18:14:43.630+0000 7f86c4fd3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f86c4fd3700 time 2024-03-28T18:14:43.630113+0000 2024-03-28T18:14:44.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:14:44.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2024-03-28T18:14:44.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:44.013 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f86cf5fd4a1] 2024-03-28T18:14:44.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f86cf5fd667] 2024-03-28T18:14:44.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557772b77223] 2024-03-28T18:14:44.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5577729f15d6] 2024-03-28T18:14:44.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5577729f1da1] 2024-03-28T18:14:44.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5577729f42d5] 2024-03-28T18:14:44.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557772a44233] 2024-03-28T18:14:44.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557772a46197] 2024-03-28T18:14:44.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55777294511b] 2024-03-28T18:14:44.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5577729456bc] 2024-03-28T18:14:44.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55777292f1ff] 2024-03-28T18:14:44.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f86cf88f488] 2024-03-28T18:14:44.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 13: (DispatchQueue::entry()+0x51f) [0x7f86cf88c60f] 2024-03-28T18:14:44.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f86cf95fe81] 2024-03-28T18:14:44.014 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f86ce37a1ca] 2024-03-28T18:14:44.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 16: clone() 2024-03-28T18:14:44.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2024-03-28T18:14:44.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 0> 2024-03-28T18:14:43.632+0000 7f86c4fd3700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:14:44.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: in thread 7f86c4fd3700 thread_name:ms_dispatch 2024-03-28T18:14:44.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2024-03-28T18:14:44.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:14:44.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f86ce384d20] 2024-03-28T18:14:44.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2: gsignal() 2024-03-28T18:14:44.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 3: abort() 2024-03-28T18:14:44.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f86cf5fd4fb] 2024-03-28T18:14:44.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f86cf5fd667] 2024-03-28T18:14:44.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557772b77223] 2024-03-28T18:14:44.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5577729f15d6] 2024-03-28T18:14:44.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5577729f1da1] 2024-03-28T18:14:44.015 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5577729f42d5] 2024-03-28T18:14:44.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557772a44233] 2024-03-28T18:14:44.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557772a46197] 2024-03-28T18:14:44.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55777294511b] 2024-03-28T18:14:44.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5577729456bc] 2024-03-28T18:14:44.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55777292f1ff] 2024-03-28T18:14:44.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f86cf88f488] 2024-03-28T18:14:44.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 16: (DispatchQueue::entry()+0x51f) [0x7f86cf88c60f] 2024-03-28T18:14:44.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f86cf95fe81] 2024-03-28T18:14:44.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f86ce37a1ca] 2024-03-28T18:14:44.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 19: clone() 2024-03-28T18:14:44.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:14:44.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[70886]: 2024-03-28T18:14:44.016 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:43 smithi059 podman[70938]: 2024-03-28 18:14:43.69963865 +0000 UTC m=+0.020165630 container died f5299f67b8ee88539cc53cf8abf27dd22aaaba54cd6698aff089e3625da9a619 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:14:44.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:43 smithi170 ceph-mon[36280]: pgmap v113: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 409 B/s wr, 6 op/s 2024-03-28T18:14:44.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:43 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199998 2024-03-28T18:14:44.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:43 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/2774256011,v1:172.21.15.59:6839/2774256011] up:reconnect 2024-03-28T18:14:44.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:43 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:44.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:43 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089999 2024-03-28T18:14:44.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:44 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/2774256011,v1:172.21.15.59:6839/2774256011] up:rejoin 2024-03-28T18:14:44.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:44 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:44.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:44 smithi059 ceph-mon[36190]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-28T18:14:44.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:44 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:14:44.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:44 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:14:44.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:44 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/2774256011,v1:172.21.15.59:6839/2774256011] up:active 2024-03-28T18:14:44.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:44 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:14:44.786 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:44 smithi059 podman[70938]: 2024-03-28 18:14:44.345826676 +0000 UTC m=+0.666353641 container remove f5299f67b8ee88539cc53cf8abf27dd22aaaba54cd6698aff089e3625da9a619 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:14:44.786 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:44 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:14:44.786 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:44 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.b.service: Failed with result 'exit-code'. 2024-03-28T18:14:44.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:44 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/2774256011,v1:172.21.15.59:6839/2774256011] up:rejoin 2024-03-28T18:14:44.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:44 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:44.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:44 smithi037 ceph-mon[31460]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-28T18:14:44.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:44 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:14:44.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:44 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:14:44.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:44 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/2774256011,v1:172.21.15.59:6839/2774256011] up:active 2024-03-28T18:14:44.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:44 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:14:45.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:44 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/2774256011,v1:172.21.15.59:6839/2774256011] up:rejoin 2024-03-28T18:14:45.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:44 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:45.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:44 smithi170 ceph-mon[36280]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-28T18:14:45.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:44 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:14:45.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:44 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:14:45.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:44 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/2774256011,v1:172.21.15.59:6839/2774256011] up:active 2024-03-28T18:14:45.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:44 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:14:45.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:45 smithi037 ceph-mon[31460]: pgmap v114: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 409 B/s wr, 6 op/s 2024-03-28T18:14:46.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:45 smithi059 ceph-mon[36190]: pgmap v114: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 409 B/s wr, 6 op/s 2024-03-28T18:14:46.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:45 smithi170 ceph-mon[36280]: pgmap v114: 97 pgs: 97 active+clean; 45 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 409 B/s wr, 6 op/s 2024-03-28T18:14:46.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:14:46 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:14:46.402+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:14:47.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:47 smithi037 ceph-mon[31460]: pgmap v115: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 4.4 KiB/s wr, 18 op/s 2024-03-28T18:14:48.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:47 smithi059 ceph-mon[36190]: pgmap v115: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 4.4 KiB/s wr, 18 op/s 2024-03-28T18:14:48.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:47 smithi170 ceph-mon[36280]: pgmap v115: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 4.4 KiB/s wr, 18 op/s 2024-03-28T18:14:49.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:49 smithi037 ceph-mon[31460]: pgmap v116: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 4.4 KiB/s wr, 18 op/s 2024-03-28T18:14:50.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:49 smithi059 ceph-mon[36190]: pgmap v116: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 4.4 KiB/s wr, 18 op/s 2024-03-28T18:14:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:49 smithi170 ceph-mon[36280]: pgmap v116: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 4.4 KiB/s wr, 18 op/s 2024-03-28T18:14:51.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:51 smithi037 ceph-mon[31460]: pgmap v117: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 4.4 KiB/s wr, 18 op/s 2024-03-28T18:14:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:51 smithi059 ceph-mon[36190]: pgmap v117: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 4.4 KiB/s wr, 18 op/s 2024-03-28T18:14:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:51 smithi170 ceph-mon[36280]: pgmap v117: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 4.4 KiB/s wr, 18 op/s 2024-03-28T18:14:53.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:53 smithi037 ceph-mon[31460]: pgmap v118: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 5.5 KiB/s wr, 18 op/s 2024-03-28T18:14:54.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:53 smithi059 ceph-mon[36190]: pgmap v118: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 5.5 KiB/s wr, 18 op/s 2024-03-28T18:14:54.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:53 smithi170 ceph-mon[36280]: pgmap v118: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 5.5 KiB/s wr, 18 op/s 2024-03-28T18:14:55.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:14:55.019+0000 7f4f82a00640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 msgr2=0x56357a39a010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:6838/2091629078 2024-03-28T18:14:55.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:14:55.019+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-28T18:14:55.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:14:55.019+0000 7f4f63e00640 1 client.25954 ms_handle_refused on v2:172.21.15.59:6838/2091629078 2024-03-28T18:14:55.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:54 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.b.service: Scheduled restart job, restart counter is at 3. 2024-03-28T18:14:55.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:54 smithi059 systemd[1]: Stopped Ceph mds.b for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:14:55.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:54 smithi059 systemd[1]: Starting Ceph mds.b for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:14:55.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:55 smithi059 podman[71037]: 2024-03-28 18:14:54.923941691 +0000 UTC m=+0.017887560 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:14:55.308 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:55 smithi059 podman[71037]: 2024-03-28 18:14:55.04868693 +0000 UTC m=+0.142632794 container create 748e83b002889882274eaf7dba5f2f27169ce6a1f337ddc31cfb9b78a7d4085c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-28T18:14:55.309 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:55 smithi059 podman[71037]: 2024-03-28 18:14:55.284758934 +0000 UTC m=+0.378704798 container init 748e83b002889882274eaf7dba5f2f27169ce6a1f337ddc31cfb9b78a7d4085c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-28T18:14:55.309 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:55 smithi059 podman[71037]: 2024-03-28 18:14:55.287639618 +0000 UTC m=+0.381585493 container start 748e83b002889882274eaf7dba5f2f27169ce6a1f337ddc31cfb9b78a7d4085c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-28T18:14:55.620 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:55 smithi059 ceph-mds[71057]: -- 172.21.15.59:0/22064682 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f4e134da40 con 0x55f4e1321000 2024-03-28T18:14:55.620 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:55 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: starting mds.b at 2024-03-28T18:14:55.620 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:55 smithi059 bash[71037]: 748e83b002889882274eaf7dba5f2f27169ce6a1f337ddc31cfb9b78a7d4085c 2024-03-28T18:14:55.620 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:14:55 smithi059 systemd[1]: Started Ceph mds.b for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:14:55.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:55 smithi037 ceph-mon[31460]: pgmap v119: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 5.2 KiB/s wr, 15 op/s 2024-03-28T18:14:55.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:55 smithi037 ceph-mon[31460]: daemon mds.b restarted 2024-03-28T18:14:55.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:55 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:14:55.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:55 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:14:55.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:55 smithi037 ceph-mon[31460]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T18:14:55.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:55 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/802545442,v1:172.21.15.59:6839/802545442] up:boot 2024-03-28T18:14:55.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:55 smithi037 ceph-mon[31460]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-28T18:14:55.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:55 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:14:55.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:55 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:55 smithi059 ceph-mon[36190]: pgmap v119: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 5.2 KiB/s wr, 15 op/s 2024-03-28T18:14:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:55 smithi059 ceph-mon[36190]: daemon mds.b restarted 2024-03-28T18:14:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:55 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:14:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:55 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:14:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:55 smithi059 ceph-mon[36190]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T18:14:56.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:55 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/802545442,v1:172.21.15.59:6839/802545442] up:boot 2024-03-28T18:14:56.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:55 smithi059 ceph-mon[36190]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-28T18:14:56.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:55 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:14:56.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:55 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:55 smithi170 ceph-mon[36280]: pgmap v119: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 5.2 KiB/s wr, 15 op/s 2024-03-28T18:14:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:55 smithi170 ceph-mon[36280]: daemon mds.b restarted 2024-03-28T18:14:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:55 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:14:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:55 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:14:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:55 smithi170 ceph-mon[36280]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T18:14:56.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:55 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/802545442,v1:172.21.15.59:6839/802545442] up:boot 2024-03-28T18:14:56.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:55 smithi170 ceph-mon[36280]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-28T18:14:56.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:55 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:14:56.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:55 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:57 smithi059 ceph-mon[36190]: pgmap v121: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-28T18:14:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:57 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/1393026034,v1:172.21.15.170:6835/1393026034] up:resolve 2024-03-28T18:14:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:57 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:58.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:57 smithi170 ceph-mon[36280]: pgmap v121: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-28T18:14:58.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:57 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/1393026034,v1:172.21.15.170:6835/1393026034] up:resolve 2024-03-28T18:14:58.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:57 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:58.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:57 smithi037 ceph-mon[31460]: pgmap v121: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-28T18:14:58.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:57 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/1393026034,v1:172.21.15.170:6835/1393026034] up:resolve 2024-03-28T18:14:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:57 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:58 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199995 2024-03-28T18:14:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:58 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/1393026034,v1:172.21.15.170:6835/1393026034] up:reconnect 2024-03-28T18:14:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:58 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:58 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899979 2024-03-28T18:14:59.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:58 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199995 2024-03-28T18:14:59.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:58 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/1393026034,v1:172.21.15.170:6835/1393026034] up:reconnect 2024-03-28T18:14:59.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:58 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:59.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:58 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899979 2024-03-28T18:14:59.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:58 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199995 2024-03-28T18:14:59.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:58 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/1393026034,v1:172.21.15.170:6835/1393026034] up:reconnect 2024-03-28T18:14:59.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:58 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:14:59.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:58 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899979 2024-03-28T18:15:00.009 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-mon[36280]: pgmap v122: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-28T18:15:00.009 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/1393026034,v1:172.21.15.170:6835/1393026034] up:rejoin 2024-03-28T18:15:00.009 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:00.009 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-mon[36280]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-28T18:15:00.010 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff202cd9700 time 2024-03-28T18:14:59.689475+0000 2024-03-28T18:15:00.010 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:15:00.010 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:00.010 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff20d3034a1] 2024-03-28T18:15:00.010 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff20d303667] 2024-03-28T18:15:00.010 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a12c5fb223] 2024-03-28T18:15:00.010 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a12c4755d6] 2024-03-28T18:15:00.010 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a12c475da1] 2024-03-28T18:15:00.010 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a12c4782d5] 2024-03-28T18:15:00.010 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a12c4c8233] 2024-03-28T18:15:00.010 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a12c4ca197] 2024-03-28T18:15:00.010 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a12c3c911b] 2024-03-28T18:15:00.011 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a12c3c96bc] 2024-03-28T18:15:00.011 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a12c3b31ff] 2024-03-28T18:15:00.011 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff20d595488] 2024-03-28T18:15:00.011 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 13: (DispatchQueue::entry()+0x51f) [0x7ff20d59260f] 2024-03-28T18:15:00.011 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff20d665e81] 2024-03-28T18:15:00.011 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff20c0801ca] 2024-03-28T18:15:00.011 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 16: clone() 2024-03-28T18:15:00.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2024-03-28T18:14:59.690+0000 7ff202cd9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff202cd9700 time 2024-03-28T18:14:59.689475+0000 2024-03-28T18:15:00.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:15:00.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2024-03-28T18:15:00.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:00.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff20d3034a1] 2024-03-28T18:15:00.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff20d303667] 2024-03-28T18:15:00.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a12c5fb223] 2024-03-28T18:15:00.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a12c4755d6] 2024-03-28T18:15:00.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a12c475da1] 2024-03-28T18:15:00.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a12c4782d5] 2024-03-28T18:15:00.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a12c4c8233] 2024-03-28T18:15:00.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a12c4ca197] 2024-03-28T18:15:00.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a12c3c911b] 2024-03-28T18:15:00.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a12c3c96bc] 2024-03-28T18:15:00.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a12c3b31ff] 2024-03-28T18:15:00.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff20d595488] 2024-03-28T18:15:00.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 13: (DispatchQueue::entry()+0x51f) [0x7ff20d59260f] 2024-03-28T18:15:00.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff20d665e81] 2024-03-28T18:15:00.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff20c0801ca] 2024-03-28T18:15:00.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 16: clone() 2024-03-28T18:15:00.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2024-03-28T18:15:00.014 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: *** Caught signal (Aborted) ** 2024-03-28T18:15:00.014 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: in thread 7ff202cd9700 thread_name:ms_dispatch 2024-03-28T18:15:00.014 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:00.014 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff20c08ad20] 2024-03-28T18:15:00.014 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2: gsignal() 2024-03-28T18:15:00.014 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 3: abort() 2024-03-28T18:15:00.014 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff20d3034fb] 2024-03-28T18:15:00.014 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff20d303667] 2024-03-28T18:15:00.014 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a12c5fb223] 2024-03-28T18:15:00.014 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a12c4755d6] 2024-03-28T18:15:00.014 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a12c475da1] 2024-03-28T18:15:00.014 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a12c4782d5] 2024-03-28T18:15:00.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a12c4c8233] 2024-03-28T18:15:00.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a12c4ca197] 2024-03-28T18:15:00.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a12c3c911b] 2024-03-28T18:15:00.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a12c3c96bc] 2024-03-28T18:15:00.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a12c3b31ff] 2024-03-28T18:15:00.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff20d595488] 2024-03-28T18:15:00.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 16: (DispatchQueue::entry()+0x51f) [0x7ff20d59260f] 2024-03-28T18:15:00.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff20d665e81] 2024-03-28T18:15:00.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff20c0801ca] 2024-03-28T18:15:00.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 19: clone() 2024-03-28T18:15:00.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2024-03-28T18:14:59.693+0000 7ff202cd9700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:15:00.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: in thread 7ff202cd9700 thread_name:ms_dispatch 2024-03-28T18:15:00.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2024-03-28T18:15:00.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:00.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff20c08ad20] 2024-03-28T18:15:00.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2: gsignal() 2024-03-28T18:15:00.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 3: abort() 2024-03-28T18:15:00.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff20d3034fb] 2024-03-28T18:15:00.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff20d303667] 2024-03-28T18:15:00.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a12c5fb223] 2024-03-28T18:15:00.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a12c4755d6] 2024-03-28T18:15:00.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a12c475da1] 2024-03-28T18:15:00.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a12c4782d5] 2024-03-28T18:15:00.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a12c4c8233] 2024-03-28T18:15:00.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a12c4ca197] 2024-03-28T18:15:00.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a12c3c911b] 2024-03-28T18:15:00.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a12c3c96bc] 2024-03-28T18:15:00.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a12c3b31ff] 2024-03-28T18:15:00.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff20d595488] 2024-03-28T18:15:00.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 16: (DispatchQueue::entry()+0x51f) [0x7ff20d59260f] 2024-03-28T18:15:00.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff20d665e81] 2024-03-28T18:15:00.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff20c0801ca] 2024-03-28T18:15:00.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 19: clone() 2024-03-28T18:15:00.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:15:00.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2024-03-28T18:15:00.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: -1> 2024-03-28T18:14:59.690+0000 7ff202cd9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff202cd9700 time 2024-03-28T18:14:59.689475+0000 2024-03-28T18:15:00.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:15:00.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2024-03-28T18:15:00.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:00.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff20d3034a1] 2024-03-28T18:15:00.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff20d303667] 2024-03-28T18:15:00.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a12c5fb223] 2024-03-28T18:15:00.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a12c4755d6] 2024-03-28T18:15:00.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a12c475da1] 2024-03-28T18:15:00.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a12c4782d5] 2024-03-28T18:15:00.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a12c4c8233] 2024-03-28T18:15:00.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a12c4ca197] 2024-03-28T18:15:00.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a12c3c911b] 2024-03-28T18:15:00.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a12c3c96bc] 2024-03-28T18:15:00.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a12c3b31ff] 2024-03-28T18:15:00.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff20d595488] 2024-03-28T18:15:00.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 13: (DispatchQueue::entry()+0x51f) [0x7ff20d59260f] 2024-03-28T18:15:00.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff20d665e81] 2024-03-28T18:15:00.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff20c0801ca] 2024-03-28T18:15:00.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 16: clone() 2024-03-28T18:15:00.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2024-03-28T18:15:00.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 0> 2024-03-28T18:14:59.693+0000 7ff202cd9700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:15:00.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: in thread 7ff202cd9700 thread_name:ms_dispatch 2024-03-28T18:15:00.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2024-03-28T18:15:00.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:00.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff20c08ad20] 2024-03-28T18:15:00.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2: gsignal() 2024-03-28T18:15:00.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 3: abort() 2024-03-28T18:15:00.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff20d3034fb] 2024-03-28T18:15:00.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff20d303667] 2024-03-28T18:15:00.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a12c5fb223] 2024-03-28T18:15:00.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a12c4755d6] 2024-03-28T18:15:00.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a12c475da1] 2024-03-28T18:15:00.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a12c4782d5] 2024-03-28T18:15:00.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a12c4c8233] 2024-03-28T18:15:00.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a12c4ca197] 2024-03-28T18:15:00.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a12c3c911b] 2024-03-28T18:15:00.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a12c3c96bc] 2024-03-28T18:15:00.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a12c3b31ff] 2024-03-28T18:15:00.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff20d595488] 2024-03-28T18:15:00.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 16: (DispatchQueue::entry()+0x51f) [0x7ff20d59260f] 2024-03-28T18:15:00.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff20d665e81] 2024-03-28T18:15:00.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff20c0801ca] 2024-03-28T18:15:00.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 19: clone() 2024-03-28T18:15:00.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:15:00.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2024-03-28T18:15:00.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: -1> 2024-03-28T18:14:59.690+0000 7ff202cd9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff202cd9700 time 2024-03-28T18:14:59.689475+0000 2024-03-28T18:15:00.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:15:00.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2024-03-28T18:15:00.021 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:00.021 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff20d3034a1] 2024-03-28T18:15:00.021 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff20d303667] 2024-03-28T18:15:00.021 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a12c5fb223] 2024-03-28T18:15:00.021 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a12c4755d6] 2024-03-28T18:15:00.021 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a12c475da1] 2024-03-28T18:15:00.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a12c4782d5] 2024-03-28T18:15:00.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a12c4c8233] 2024-03-28T18:15:00.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a12c4ca197] 2024-03-28T18:15:00.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a12c3c911b] 2024-03-28T18:15:00.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a12c3c96bc] 2024-03-28T18:15:00.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a12c3b31ff] 2024-03-28T18:15:00.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff20d595488] 2024-03-28T18:15:00.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 13: (DispatchQueue::entry()+0x51f) [0x7ff20d59260f] 2024-03-28T18:15:00.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff20d665e81] 2024-03-28T18:15:00.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff20c0801ca] 2024-03-28T18:15:00.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 16: clone() 2024-03-28T18:15:00.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2024-03-28T18:15:00.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 0> 2024-03-28T18:14:59.693+0000 7ff202cd9700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:15:00.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: in thread 7ff202cd9700 thread_name:ms_dispatch 2024-03-28T18:15:00.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2024-03-28T18:15:00.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:00.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff20c08ad20] 2024-03-28T18:15:00.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2: gsignal() 2024-03-28T18:15:00.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 3: abort() 2024-03-28T18:15:00.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff20d3034fb] 2024-03-28T18:15:00.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff20d303667] 2024-03-28T18:15:00.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a12c5fb223] 2024-03-28T18:15:00.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a12c4755d6] 2024-03-28T18:15:00.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a12c475da1] 2024-03-28T18:15:00.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a12c4782d5] 2024-03-28T18:15:00.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a12c4c8233] 2024-03-28T18:15:00.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a12c4ca197] 2024-03-28T18:15:00.024 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a12c3c911b] 2024-03-28T18:15:00.024 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a12c3c96bc] 2024-03-28T18:15:00.024 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a12c3b31ff] 2024-03-28T18:15:00.024 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff20d595488] 2024-03-28T18:15:00.024 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 16: (DispatchQueue::entry()+0x51f) [0x7ff20d59260f] 2024-03-28T18:15:00.024 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff20d665e81] 2024-03-28T18:15:00.024 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff20c0801ca] 2024-03-28T18:15:00.024 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 19: clone() 2024-03-28T18:15:00.025 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:15:00.025 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70814]: 2024-03-28T18:15:00.025 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:14:59 smithi170 podman[70856]: 2024-03-28 18:14:59.764138198 +0000 UTC m=+0.022380388 container died 9fb7624a517b742ba2f7ff202ea293ad36a6c015348ca43780cb17bb67e41cb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:15:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:59 smithi059 ceph-mon[36190]: pgmap v122: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-28T18:15:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:59 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/1393026034,v1:172.21.15.170:6835/1393026034] up:rejoin 2024-03-28T18:15:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:59 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:14:59 smithi059 ceph-mon[36190]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-28T18:15:00.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:59 smithi037 ceph-mon[31460]: pgmap v122: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-28T18:15:00.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:59 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/1393026034,v1:172.21.15.170:6835/1393026034] up:rejoin 2024-03-28T18:15:00.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:59 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:00.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:14:59 smithi037 ceph-mon[31460]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-28T18:15:00.671 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:00 smithi170 podman[70856]: 2024-03-28 18:15:00.420982696 +0000 UTC m=+0.679224881 container remove 9fb7624a517b742ba2f7ff202ea293ad36a6c015348ca43780cb17bb67e41cb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-28T18:15:00.671 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:00 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:15:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:00 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:15:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:00 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:15:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:00 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/1393026034,v1:172.21.15.170:6835/1393026034] up:active 2024-03-28T18:15:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:00 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:00 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:15:01.036 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:00 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-28T18:15:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:00 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:15:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:00 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:15:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:00 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/1393026034,v1:172.21.15.170:6835/1393026034] up:active 2024-03-28T18:15:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:00 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:00 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:15:01.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:00 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:15:01.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:00 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:15:01.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:00 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/1393026034,v1:172.21.15.170:6835/1393026034] up:active 2024-03-28T18:15:01.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:00 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:01.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:00 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:15:01.678 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:15:01 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:15:01.403+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:15:01.936 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:01 smithi059 ceph-mon[36190]: pgmap v123: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-28T18:15:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:01 smithi170 ceph-mon[36280]: pgmap v123: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-28T18:15:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:01 smithi037 ceph-mon[31460]: pgmap v123: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-28T18:15:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:03 smithi059 ceph-mon[36190]: pgmap v124: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.2 KiB/s wr, 19 op/s 2024-03-28T18:15:04.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:03 smithi170 ceph-mon[36280]: pgmap v124: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.2 KiB/s wr, 19 op/s 2024-03-28T18:15:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:03 smithi037 ceph-mon[31460]: pgmap v124: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.2 KiB/s wr, 19 op/s 2024-03-28T18:15:06.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:05 smithi059 ceph-mon[36190]: pgmap v125: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.2 KiB/s wr, 19 op/s 2024-03-28T18:15:06.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:05 smithi170 ceph-mon[36280]: pgmap v125: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.2 KiB/s wr, 19 op/s 2024-03-28T18:15:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:05 smithi037 ceph-mon[31460]: pgmap v125: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.2 KiB/s wr, 19 op/s 2024-03-28T18:15:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:07 smithi059 ceph-mon[36190]: pgmap v126: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 2.0 KiB/s wr, 19 op/s 2024-03-28T18:15:08.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:07 smithi170 ceph-mon[36280]: pgmap v126: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 2.0 KiB/s wr, 19 op/s 2024-03-28T18:15:08.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:07 smithi037 ceph-mon[31460]: pgmap v126: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 2.0 KiB/s wr, 19 op/s 2024-03-28T18:15:10.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:15:10.022+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:15:10.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:15:10.023+0000 7f4f82a00640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 msgr2=0x56357a39a010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-28T18:15:10.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:15:10.023+0000 7f4f82a00640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 msgr2=0x56357a39a010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T18:15:10.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:15:10.023+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T18:15:10.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:15:10.023+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-28T18:15:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:09 smithi059 ceph-mon[36190]: pgmap v127: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.7 KiB/s wr, 15 op/s 2024-03-28T18:15:10.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:09 smithi170 ceph-mon[36280]: pgmap v127: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.7 KiB/s wr, 15 op/s 2024-03-28T18:15:10.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:09 smithi037 ceph-mon[31460]: pgmap v127: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.7 KiB/s wr, 15 op/s 2024-03-28T18:15:11.045 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:10 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 3. 2024-03-28T18:15:11.045 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:10 smithi170 systemd[1]: Stopped Ceph mds.f for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:15:11.046 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:10 smithi170 systemd[1]: Starting Ceph mds.f for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:15:11.046 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:11 smithi170 podman[70946]: 2024-03-28 18:15:10.92189179 +0000 UTC m=+0.018171902 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:15:11.046 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:11 smithi170 podman[70946]: 2024-03-28 18:15:11.046535364 +0000 UTC m=+0.142815466 container create 33c69e75620daa17f39e04e61aab332cc7967842b53b3ef7573ff96542b2aadf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-03-28T18:15:11.700 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:11 smithi170 podman[70946]: 2024-03-28 18:15:11.374361689 +0000 UTC m=+0.470641802 container init 33c69e75620daa17f39e04e61aab332cc7967842b53b3ef7573ff96542b2aadf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-28T18:15:11.700 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:11 smithi170 podman[70946]: 2024-03-28 18:15:11.377362862 +0000 UTC m=+0.473642967 container start 33c69e75620daa17f39e04e61aab332cc7967842b53b3ef7573ff96542b2aadf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:15:11.700 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:11 smithi170 ceph-mds[70965]: -- 172.21.15.170:0/10606906 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f314815a40 con 0x55f3147e8800 2024-03-28T18:15:11.701 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:11 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: starting mds.f at 2024-03-28T18:15:11.701 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:11 smithi170 bash[70946]: 33c69e75620daa17f39e04e61aab332cc7967842b53b3ef7573ff96542b2aadf 2024-03-28T18:15:11.701 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:11 smithi170 systemd[1]: Started Ceph mds.f for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:15:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:11 smithi059 ceph-mon[36190]: pgmap v128: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.7 KiB/s wr, 15 op/s 2024-03-28T18:15:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:11 smithi059 ceph-mon[36190]: daemon mds.f restarted 2024-03-28T18:15:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:11 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:15:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:11 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:15:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:11 smithi059 ceph-mon[36190]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T18:15:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:11 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/1772048823,v1:172.21.15.170:6835/1772048823] up:boot 2024-03-28T18:15:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:11 smithi059 ceph-mon[36190]: Standby daemon mds.b assigned to filesystem cephfs as rank 1 2024-03-28T18:15:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:11 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:15:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:11 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:11 smithi170 ceph-mon[36280]: pgmap v128: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.7 KiB/s wr, 15 op/s 2024-03-28T18:15:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:11 smithi170 ceph-mon[36280]: daemon mds.f restarted 2024-03-28T18:15:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:11 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:15:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:11 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:15:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:11 smithi170 ceph-mon[36280]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T18:15:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:11 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/1772048823,v1:172.21.15.170:6835/1772048823] up:boot 2024-03-28T18:15:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:11 smithi170 ceph-mon[36280]: Standby daemon mds.b assigned to filesystem cephfs as rank 1 2024-03-28T18:15:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:11 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:15:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:11 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:11 smithi037 ceph-mon[31460]: pgmap v128: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.7 KiB/s wr, 15 op/s 2024-03-28T18:15:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:11 smithi037 ceph-mon[31460]: daemon mds.f restarted 2024-03-28T18:15:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:11 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:15:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:11 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:15:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:11 smithi037 ceph-mon[31460]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T18:15:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:11 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/1772048823,v1:172.21.15.170:6835/1772048823] up:boot 2024-03-28T18:15:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:11 smithi037 ceph-mon[31460]: Standby daemon mds.b assigned to filesystem cephfs as rank 1 2024-03-28T18:15:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:11 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:15:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:11 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:14.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:13 smithi059 ceph-mon[36190]: pgmap v130: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:15:14.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:13 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/802545442,v1:172.21.15.59:6839/802545442] up:resolve 2024-03-28T18:15:14.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:13 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:14.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:13 smithi170 ceph-mon[36280]: pgmap v130: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:15:14.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:13 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/802545442,v1:172.21.15.59:6839/802545442] up:resolve 2024-03-28T18:15:14.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:13 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:13 smithi037 ceph-mon[31460]: pgmap v130: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:15:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:13 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/802545442,v1:172.21.15.59:6839/802545442] up:resolve 2024-03-28T18:15:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:13 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:14 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/802545442,v1:172.21.15.59:6839/802545442] up:reconnect 2024-03-28T18:15:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:14 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:14 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00799991 2024-03-28T18:15:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:14 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.0089999 2024-03-28T18:15:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:14 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/802545442,v1:172.21.15.59:6839/802545442] up:reconnect 2024-03-28T18:15:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:14 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:14 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00799991 2024-03-28T18:15:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:14 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.0089999 2024-03-28T18:15:15.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:14 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/802545442,v1:172.21.15.59:6839/802545442] up:reconnect 2024-03-28T18:15:15.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:14 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:15.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:14 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00799991 2024-03-28T18:15:15.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:14 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.0089999 2024-03-28T18:15:16.038 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-mon[36190]: pgmap v131: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:15:16.038 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/802545442,v1:172.21.15.59:6839/802545442] up:rejoin 2024-03-28T18:15:16.038 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:16.038 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-mon[36190]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-28T18:15:16.040 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f709c470700 time 2024-03-28T18:15:15.781447+0000 2024-03-28T18:15:16.041 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:15:16.041 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:16.041 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f70a6a9a4a1] 2024-03-28T18:15:16.041 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f70a6a9a667] 2024-03-28T18:15:16.041 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f4dffc8223] 2024-03-28T18:15:16.041 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f4dfe425d6] 2024-03-28T18:15:16.042 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f4dfe42da1] 2024-03-28T18:15:16.042 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f4dfe452d5] 2024-03-28T18:15:16.042 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f4dfe95233] 2024-03-28T18:15:16.042 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f4dfe97197] 2024-03-28T18:15:16.043 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f4dfd9611b] 2024-03-28T18:15:16.043 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f4dfd966bc] 2024-03-28T18:15:16.043 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f4dfd801ff] 2024-03-28T18:15:16.043 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f70a6d2c488] 2024-03-28T18:15:16.043 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 13: (DispatchQueue::entry()+0x51f) [0x7f70a6d2960f] 2024-03-28T18:15:16.043 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f70a6dfce81] 2024-03-28T18:15:16.043 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f70a58171ca] 2024-03-28T18:15:16.043 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 16: clone() 2024-03-28T18:15:16.043 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2024-03-28T18:15:15.783+0000 7f709c470700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f709c470700 time 2024-03-28T18:15:15.781447+0000 2024-03-28T18:15:16.043 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:15:16.043 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2024-03-28T18:15:16.043 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:16.043 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f70a6a9a4a1] 2024-03-28T18:15:16.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f70a6a9a667] 2024-03-28T18:15:16.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f4dffc8223] 2024-03-28T18:15:16.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f4dfe425d6] 2024-03-28T18:15:16.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f4dfe42da1] 2024-03-28T18:15:16.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f4dfe452d5] 2024-03-28T18:15:16.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f4dfe95233] 2024-03-28T18:15:16.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f4dfe97197] 2024-03-28T18:15:16.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f4dfd9611b] 2024-03-28T18:15:16.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f4dfd966bc] 2024-03-28T18:15:16.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f4dfd801ff] 2024-03-28T18:15:16.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f70a6d2c488] 2024-03-28T18:15:16.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 13: (DispatchQueue::entry()+0x51f) [0x7f70a6d2960f] 2024-03-28T18:15:16.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f70a6dfce81] 2024-03-28T18:15:16.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f70a58171ca] 2024-03-28T18:15:16.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 16: clone() 2024-03-28T18:15:16.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2024-03-28T18:15:16.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: *** Caught signal (Aborted) ** 2024-03-28T18:15:16.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: in thread 7f709c470700 thread_name:ms_dispatch 2024-03-28T18:15:16.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:16.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f70a5821d20] 2024-03-28T18:15:16.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2: gsignal() 2024-03-28T18:15:16.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 3: abort() 2024-03-28T18:15:16.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f70a6a9a4fb] 2024-03-28T18:15:16.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f70a6a9a667] 2024-03-28T18:15:16.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f4dffc8223] 2024-03-28T18:15:16.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f4dfe425d6] 2024-03-28T18:15:16.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f4dfe42da1] 2024-03-28T18:15:16.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f4dfe452d5] 2024-03-28T18:15:16.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f4dfe95233] 2024-03-28T18:15:16.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f4dfe97197] 2024-03-28T18:15:16.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f4dfd9611b] 2024-03-28T18:15:16.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f4dfd966bc] 2024-03-28T18:15:16.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f4dfd801ff] 2024-03-28T18:15:16.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f70a6d2c488] 2024-03-28T18:15:16.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 16: (DispatchQueue::entry()+0x51f) [0x7f70a6d2960f] 2024-03-28T18:15:16.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f70a6dfce81] 2024-03-28T18:15:16.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f70a58171ca] 2024-03-28T18:15:16.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 19: clone() 2024-03-28T18:15:16.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2024-03-28T18:15:15.784+0000 7f709c470700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:15:16.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: in thread 7f709c470700 thread_name:ms_dispatch 2024-03-28T18:15:16.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2024-03-28T18:15:16.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:16.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f70a5821d20] 2024-03-28T18:15:16.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2: gsignal() 2024-03-28T18:15:16.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 3: abort() 2024-03-28T18:15:16.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f70a6a9a4fb] 2024-03-28T18:15:16.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f70a6a9a667] 2024-03-28T18:15:16.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f4dffc8223] 2024-03-28T18:15:16.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f4dfe425d6] 2024-03-28T18:15:16.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f4dfe42da1] 2024-03-28T18:15:16.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f4dfe452d5] 2024-03-28T18:15:16.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f4dfe95233] 2024-03-28T18:15:16.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f4dfe97197] 2024-03-28T18:15:16.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f4dfd9611b] 2024-03-28T18:15:16.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f4dfd966bc] 2024-03-28T18:15:16.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f4dfd801ff] 2024-03-28T18:15:16.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f70a6d2c488] 2024-03-28T18:15:16.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 16: (DispatchQueue::entry()+0x51f) [0x7f70a6d2960f] 2024-03-28T18:15:16.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f70a6dfce81] 2024-03-28T18:15:16.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f70a58171ca] 2024-03-28T18:15:16.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 19: clone() 2024-03-28T18:15:16.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:15:16.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2024-03-28T18:15:16.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: -1> 2024-03-28T18:15:15.783+0000 7f709c470700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f709c470700 time 2024-03-28T18:15:15.781447+0000 2024-03-28T18:15:16.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:15:16.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2024-03-28T18:15:16.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:16.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f70a6a9a4a1] 2024-03-28T18:15:16.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f70a6a9a667] 2024-03-28T18:15:16.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f4dffc8223] 2024-03-28T18:15:16.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f4dfe425d6] 2024-03-28T18:15:16.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f4dfe42da1] 2024-03-28T18:15:16.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f4dfe452d5] 2024-03-28T18:15:16.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f4dfe95233] 2024-03-28T18:15:16.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f4dfe97197] 2024-03-28T18:15:16.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f4dfd9611b] 2024-03-28T18:15:16.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f4dfd966bc] 2024-03-28T18:15:16.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f4dfd801ff] 2024-03-28T18:15:16.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f70a6d2c488] 2024-03-28T18:15:16.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 13: (DispatchQueue::entry()+0x51f) [0x7f70a6d2960f] 2024-03-28T18:15:16.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f70a6dfce81] 2024-03-28T18:15:16.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f70a58171ca] 2024-03-28T18:15:16.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 16: clone() 2024-03-28T18:15:16.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2024-03-28T18:15:16.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 0> 2024-03-28T18:15:15.784+0000 7f709c470700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:15:16.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: in thread 7f709c470700 thread_name:ms_dispatch 2024-03-28T18:15:16.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2024-03-28T18:15:16.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:16.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f70a5821d20] 2024-03-28T18:15:16.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2: gsignal() 2024-03-28T18:15:16.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 3: abort() 2024-03-28T18:15:16.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f70a6a9a4fb] 2024-03-28T18:15:16.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f70a6a9a667] 2024-03-28T18:15:16.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f4dffc8223] 2024-03-28T18:15:16.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f4dfe425d6] 2024-03-28T18:15:16.051 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f4dfe42da1] 2024-03-28T18:15:16.051 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f4dfe452d5] 2024-03-28T18:15:16.051 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f4dfe95233] 2024-03-28T18:15:16.051 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f4dfe97197] 2024-03-28T18:15:16.051 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f4dfd9611b] 2024-03-28T18:15:16.051 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f4dfd966bc] 2024-03-28T18:15:16.051 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f4dfd801ff] 2024-03-28T18:15:16.051 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f70a6d2c488] 2024-03-28T18:15:16.051 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 16: (DispatchQueue::entry()+0x51f) [0x7f70a6d2960f] 2024-03-28T18:15:16.051 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f70a6dfce81] 2024-03-28T18:15:16.051 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f70a58171ca] 2024-03-28T18:15:16.051 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 19: clone() 2024-03-28T18:15:16.051 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:15:16.051 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2024-03-28T18:15:16.052 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: -1> 2024-03-28T18:15:15.783+0000 7f709c470700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f709c470700 time 2024-03-28T18:15:15.781447+0000 2024-03-28T18:15:16.052 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:15:16.052 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2024-03-28T18:15:16.052 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:16.052 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f70a6a9a4a1] 2024-03-28T18:15:16.052 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f70a6a9a667] 2024-03-28T18:15:16.052 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f4dffc8223] 2024-03-28T18:15:16.052 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f4dfe425d6] 2024-03-28T18:15:16.052 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f4dfe42da1] 2024-03-28T18:15:16.052 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f4dfe452d5] 2024-03-28T18:15:16.052 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f4dfe95233] 2024-03-28T18:15:16.052 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f4dfe97197] 2024-03-28T18:15:16.052 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f4dfd9611b] 2024-03-28T18:15:16.052 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f4dfd966bc] 2024-03-28T18:15:16.052 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f4dfd801ff] 2024-03-28T18:15:16.052 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f70a6d2c488] 2024-03-28T18:15:16.053 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 13: (DispatchQueue::entry()+0x51f) [0x7f70a6d2960f] 2024-03-28T18:15:16.053 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f70a6dfce81] 2024-03-28T18:15:16.053 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f70a58171ca] 2024-03-28T18:15:16.053 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 16: clone() 2024-03-28T18:15:16.053 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2024-03-28T18:15:16.053 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 0> 2024-03-28T18:15:15.784+0000 7f709c470700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:15:16.053 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: in thread 7f709c470700 thread_name:ms_dispatch 2024-03-28T18:15:16.053 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2024-03-28T18:15:16.053 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:16.053 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f70a5821d20] 2024-03-28T18:15:16.053 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2: gsignal() 2024-03-28T18:15:16.053 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 3: abort() 2024-03-28T18:15:16.053 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f70a6a9a4fb] 2024-03-28T18:15:16.053 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f70a6a9a667] 2024-03-28T18:15:16.053 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f4dffc8223] 2024-03-28T18:15:16.054 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f4dfe425d6] 2024-03-28T18:15:16.054 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f4dfe42da1] 2024-03-28T18:15:16.054 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f4dfe452d5] 2024-03-28T18:15:16.054 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f4dfe95233] 2024-03-28T18:15:16.054 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f4dfe97197] 2024-03-28T18:15:16.054 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f4dfd9611b] 2024-03-28T18:15:16.054 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f4dfd966bc] 2024-03-28T18:15:16.054 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f4dfd801ff] 2024-03-28T18:15:16.054 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f70a6d2c488] 2024-03-28T18:15:16.054 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 16: (DispatchQueue::entry()+0x51f) [0x7f70a6d2960f] 2024-03-28T18:15:16.054 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f70a6dfce81] 2024-03-28T18:15:16.054 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f70a58171ca] 2024-03-28T18:15:16.054 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 19: clone() 2024-03-28T18:15:16.054 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:15:16.054 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71053]: 2024-03-28T18:15:16.055 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:15 smithi059 podman[71092]: 2024-03-28 18:15:15.8498055 +0000 UTC m=+0.019900601 container died 748e83b002889882274eaf7dba5f2f27169ce6a1f337ddc31cfb9b78a7d4085c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-28T18:15:16.055 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:15 smithi170 ceph-mon[36280]: pgmap v131: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:15:16.055 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:15 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/802545442,v1:172.21.15.59:6839/802545442] up:rejoin 2024-03-28T18:15:16.055 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:15 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:16.055 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:15 smithi170 ceph-mon[36280]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-28T18:15:16.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:15 smithi037 ceph-mon[31460]: pgmap v131: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:15:16.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:15 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/802545442,v1:172.21.15.59:6839/802545442] up:rejoin 2024-03-28T18:15:16.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:15 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:16.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:15 smithi037 ceph-mon[31460]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-28T18:15:16.734 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:15:16 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:15:16.404+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:15:16.857 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:16 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:15:16.857 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:16 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:15:16.857 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:16 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/802545442,v1:172.21.15.59:6839/802545442] up:active 2024-03-28T18:15:16.857 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:16 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:16.857 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:16 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:15:16.857 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:16 smithi059 podman[71092]: 2024-03-28 18:15:16.607788807 +0000 UTC m=+0.777883931 container remove 748e83b002889882274eaf7dba5f2f27169ce6a1f337ddc31cfb9b78a7d4085c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-28T18:15:16.858 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:16 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:15:17.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:16 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:15:17.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:16 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:15:17.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:16 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/802545442,v1:172.21.15.59:6839/802545442] up:active 2024-03-28T18:15:17.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:16 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:17.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:16 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:15:17.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:16 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:15:17.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:16 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:15:17.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:16 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/802545442,v1:172.21.15.59:6839/802545442] up:active 2024-03-28T18:15:17.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:16 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:17.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:16 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:15:17.285 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:16 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.b.service: Failed with result 'exit-code'. 2024-03-28T18:15:18.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:17 smithi059 ceph-mon[36190]: pgmap v132: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 28 KiB/s wr, 34 op/s 2024-03-28T18:15:18.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:17 smithi170 ceph-mon[36280]: pgmap v132: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 28 KiB/s wr, 34 op/s 2024-03-28T18:15:18.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:17 smithi037 ceph-mon[31460]: pgmap v132: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 28 KiB/s wr, 34 op/s 2024-03-28T18:15:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:19 smithi059 ceph-mon[36190]: pgmap v133: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 28 KiB/s wr, 34 op/s 2024-03-28T18:15:20.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:19 smithi170 ceph-mon[36280]: pgmap v133: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 28 KiB/s wr, 34 op/s 2024-03-28T18:15:20.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:19 smithi037 ceph-mon[31460]: pgmap v133: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 28 KiB/s wr, 34 op/s 2024-03-28T18:15:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:21 smithi059 ceph-mon[36190]: pgmap v134: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 28 KiB/s wr, 34 op/s 2024-03-28T18:15:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:21 smithi170 ceph-mon[36280]: pgmap v134: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 28 KiB/s wr, 34 op/s 2024-03-28T18:15:22.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:21 smithi037 ceph-mon[31460]: pgmap v134: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 28 KiB/s wr, 34 op/s 2024-03-28T18:15:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:23 smithi059 ceph-mon[36190]: pgmap v135: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-03-28T18:15:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:23 smithi170 ceph-mon[36280]: pgmap v135: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-03-28T18:15:24.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:23 smithi037 ceph-mon[31460]: pgmap v135: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-03-28T18:15:25.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:15:25.031+0000 7f4f82a00640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 msgr2=0x56357a39a010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:6838/2091629078 2024-03-28T18:15:25.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:15:25.031+0000 7f4f63e00640 1 client.25954 ms_handle_refused on v2:172.21.15.59:6838/2091629078 2024-03-28T18:15:25.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:15:25.031+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-28T18:15:25.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:24 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:15:25.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:24 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:15:25.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:24 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:15:26.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:25 smithi037 ceph-mon[31460]: pgmap v136: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-28T18:15:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:25 smithi059 ceph-mon[36190]: pgmap v136: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-28T18:15:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:25 smithi170 ceph-mon[36280]: pgmap v136: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-28T18:15:27.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:26 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:15:27.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:26 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:15:27.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:26 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:15:27.276 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:26 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:15:27.276 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:26 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:15:27.277 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:26 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:15:27.277 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:27 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.b.service: Scheduled restart job, restart counter is at 4. 2024-03-28T18:15:27.277 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:27 smithi059 systemd[1]: Stopped Ceph mds.b for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:15:27.277 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:27 smithi059 systemd[1]: Starting Ceph mds.b for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:15:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:26 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:15:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:26 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:15:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:26 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:15:27.535 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:27 smithi059 podman[71320]: 2024-03-28 18:15:27.180130835 +0000 UTC m=+0.018734545 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:15:27.535 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:27 smithi059 podman[71320]: 2024-03-28 18:15:27.308921979 +0000 UTC m=+0.147525719 container create 63fb096776982ebf94d56399bf83ac6c649c6b706a19202b01d87892aecf7c25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:15:27.832 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:27 smithi059 podman[71320]: 2024-03-28 18:15:27.653349953 +0000 UTC m=+0.491953653 container init 63fb096776982ebf94d56399bf83ac6c649c6b706a19202b01d87892aecf7c25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:15:27.833 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:27 smithi059 podman[71320]: 2024-03-28 18:15:27.656027377 +0000 UTC m=+0.494631075 container start 63fb096776982ebf94d56399bf83ac6c649c6b706a19202b01d87892aecf7c25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:15:27.833 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:27 smithi059 ceph-mds[71340]: -- 172.21.15.59:0/838844385 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a1f72fda40 con 0x55a1f72d0c00 2024-03-28T18:15:27.833 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:27 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: starting mds.b at 2024-03-28T18:15:27.833 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:27 smithi059 bash[71320]: 63fb096776982ebf94d56399bf83ac6c649c6b706a19202b01d87892aecf7c25 2024-03-28T18:15:27.833 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:27 smithi059 systemd[1]: Started Ceph mds.b for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:15:28.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:27 smithi037 ceph-mon[31460]: pgmap v137: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 24 KiB/s wr, 31 op/s 2024-03-28T18:15:28.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:27 smithi037 ceph-mon[31460]: daemon mds.b restarted 2024-03-28T18:15:28.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:27 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:15:28.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:27 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:15:28.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:27 smithi037 ceph-mon[31460]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T18:15:28.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:27 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/3215592665,v1:172.21.15.59:6839/3215592665] up:boot 2024-03-28T18:15:28.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:27 smithi037 ceph-mon[31460]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-28T18:15:28.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:27 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:15:28.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:27 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:27 smithi059 ceph-mon[36190]: pgmap v137: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 24 KiB/s wr, 31 op/s 2024-03-28T18:15:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:27 smithi059 ceph-mon[36190]: daemon mds.b restarted 2024-03-28T18:15:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:27 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:15:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:27 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:15:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:27 smithi059 ceph-mon[36190]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T18:15:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:27 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/3215592665,v1:172.21.15.59:6839/3215592665] up:boot 2024-03-28T18:15:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:27 smithi059 ceph-mon[36190]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-28T18:15:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:27 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:15:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:27 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:27 smithi170 ceph-mon[36280]: pgmap v137: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 24 KiB/s wr, 31 op/s 2024-03-28T18:15:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:27 smithi170 ceph-mon[36280]: daemon mds.b restarted 2024-03-28T18:15:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:27 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:15:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:27 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:15:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:27 smithi170 ceph-mon[36280]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T18:15:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:27 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/3215592665,v1:172.21.15.59:6839/3215592665] up:boot 2024-03-28T18:15:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:27 smithi170 ceph-mon[36280]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-28T18:15:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:27 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:15:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:27 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:30.088 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:29 smithi170 ceph-mon[36280]: pgmap v139: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:15:30.088 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:29 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/1772048823,v1:172.21.15.170:6835/1772048823] up:resolve 2024-03-28T18:15:30.088 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:29 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:30.094 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:29 smithi059 ceph-mon[36190]: pgmap v139: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:15:30.094 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:29 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/1772048823,v1:172.21.15.170:6835/1772048823] up:resolve 2024-03-28T18:15:30.094 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:29 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:30.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:29 smithi037 ceph-mon[31460]: pgmap v139: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:15:30.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:29 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/1772048823,v1:172.21.15.170:6835/1772048823] up:resolve 2024-03-28T18:15:30.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:29 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:31.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:30 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.000999977 2024-03-28T18:15:31.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:30 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/1772048823,v1:172.21.15.170:6835/1772048823] up:reconnect 2024-03-28T18:15:31.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:30 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:31.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:30 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999978 2024-03-28T18:15:31.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:30 smithi037 ceph-mon[31460]: pgmap v140: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:15:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:30 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.000999977 2024-03-28T18:15:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:30 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/1772048823,v1:172.21.15.170:6835/1772048823] up:reconnect 2024-03-28T18:15:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:30 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:30 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999978 2024-03-28T18:15:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:30 smithi059 ceph-mon[36190]: pgmap v140: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:15:31.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:30 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.000999977 2024-03-28T18:15:31.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:30 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/1772048823,v1:172.21.15.170:6835/1772048823] up:reconnect 2024-03-28T18:15:31.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:30 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:31.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:30 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999978 2024-03-28T18:15:31.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:30 smithi170 ceph-mon[36280]: pgmap v140: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:15:32.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:31 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/1772048823,v1:172.21.15.170:6835/1772048823] up:rejoin 2024-03-28T18:15:32.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:31 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:32.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:31 smithi037 ceph-mon[31460]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-28T18:15:32.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:31 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/1772048823,v1:172.21.15.170:6835/1772048823] up:rejoin 2024-03-28T18:15:32.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:31 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:32.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:31 smithi059 ceph-mon[36190]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-28T18:15:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/1772048823,v1:172.21.15.170:6835/1772048823] up:rejoin 2024-03-28T18:15:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-mon[36280]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-28T18:15:32.286 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f29e5441700 time 2024-03-28T18:15:31.902332+0000 2024-03-28T18:15:32.286 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:15:32.286 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:32.286 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f29efa6b4a1] 2024-03-28T18:15:32.286 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f29efa6b667] 2024-03-28T18:15:32.286 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f3136ab223] 2024-03-28T18:15:32.287 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f3135255d6] 2024-03-28T18:15:32.287 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f313525da1] 2024-03-28T18:15:32.287 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f3135282d5] 2024-03-28T18:15:32.287 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f313578233] 2024-03-28T18:15:32.287 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f31357a197] 2024-03-28T18:15:32.287 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f31347911b] 2024-03-28T18:15:32.287 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f3134796bc] 2024-03-28T18:15:32.287 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f3134631ff] 2024-03-28T18:15:32.287 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f29efcfd488] 2024-03-28T18:15:32.287 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 13: (DispatchQueue::entry()+0x51f) [0x7f29efcfa60f] 2024-03-28T18:15:32.287 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f29efdcde81] 2024-03-28T18:15:32.287 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f29ee7e81ca] 2024-03-28T18:15:32.287 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 16: clone() 2024-03-28T18:15:32.288 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2024-03-28T18:15:31.902+0000 7f29e5441700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f29e5441700 time 2024-03-28T18:15:31.902332+0000 2024-03-28T18:15:32.288 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:15:32.288 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2024-03-28T18:15:32.288 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:32.288 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f29efa6b4a1] 2024-03-28T18:15:32.288 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f29efa6b667] 2024-03-28T18:15:32.288 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f3136ab223] 2024-03-28T18:15:32.289 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f3135255d6] 2024-03-28T18:15:32.289 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f313525da1] 2024-03-28T18:15:32.289 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f3135282d5] 2024-03-28T18:15:32.289 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f313578233] 2024-03-28T18:15:32.289 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f31357a197] 2024-03-28T18:15:32.289 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f31347911b] 2024-03-28T18:15:32.289 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f3134796bc] 2024-03-28T18:15:32.289 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f3134631ff] 2024-03-28T18:15:32.289 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f29efcfd488] 2024-03-28T18:15:32.289 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 13: (DispatchQueue::entry()+0x51f) [0x7f29efcfa60f] 2024-03-28T18:15:32.289 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f29efdcde81] 2024-03-28T18:15:32.289 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f29ee7e81ca] 2024-03-28T18:15:32.289 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 16: clone() 2024-03-28T18:15:32.289 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2024-03-28T18:15:32.290 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: *** Caught signal (Aborted) ** 2024-03-28T18:15:32.290 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: in thread 7f29e5441700 thread_name:ms_dispatch 2024-03-28T18:15:32.290 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:32.290 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f29ee7f2d20] 2024-03-28T18:15:32.290 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2: gsignal() 2024-03-28T18:15:32.290 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 3: abort() 2024-03-28T18:15:32.290 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f29efa6b4fb] 2024-03-28T18:15:32.290 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f29efa6b667] 2024-03-28T18:15:32.290 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f3136ab223] 2024-03-28T18:15:32.290 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f3135255d6] 2024-03-28T18:15:32.290 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f313525da1] 2024-03-28T18:15:32.290 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f3135282d5] 2024-03-28T18:15:32.290 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f313578233] 2024-03-28T18:15:32.290 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f31357a197] 2024-03-28T18:15:32.291 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f31347911b] 2024-03-28T18:15:32.291 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f3134796bc] 2024-03-28T18:15:32.291 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f3134631ff] 2024-03-28T18:15:32.291 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f29efcfd488] 2024-03-28T18:15:32.291 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 16: (DispatchQueue::entry()+0x51f) [0x7f29efcfa60f] 2024-03-28T18:15:32.291 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f29efdcde81] 2024-03-28T18:15:32.291 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f29ee7e81ca] 2024-03-28T18:15:32.291 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 19: clone() 2024-03-28T18:15:32.291 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2024-03-28T18:15:31.905+0000 7f29e5441700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:15:32.291 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: in thread 7f29e5441700 thread_name:ms_dispatch 2024-03-28T18:15:32.291 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2024-03-28T18:15:32.291 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:32.291 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f29ee7f2d20] 2024-03-28T18:15:32.291 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2: gsignal() 2024-03-28T18:15:32.292 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 3: abort() 2024-03-28T18:15:32.292 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f29efa6b4fb] 2024-03-28T18:15:32.292 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f29efa6b667] 2024-03-28T18:15:32.292 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f3136ab223] 2024-03-28T18:15:32.293 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f3135255d6] 2024-03-28T18:15:32.293 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f313525da1] 2024-03-28T18:15:32.293 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f3135282d5] 2024-03-28T18:15:32.293 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f313578233] 2024-03-28T18:15:32.293 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f31357a197] 2024-03-28T18:15:32.293 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f31347911b] 2024-03-28T18:15:32.293 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f3134796bc] 2024-03-28T18:15:32.293 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f3134631ff] 2024-03-28T18:15:32.293 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f29efcfd488] 2024-03-28T18:15:32.293 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 16: (DispatchQueue::entry()+0x51f) [0x7f29efcfa60f] 2024-03-28T18:15:32.293 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f29efdcde81] 2024-03-28T18:15:32.293 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f29ee7e81ca] 2024-03-28T18:15:32.293 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 19: clone() 2024-03-28T18:15:32.293 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:15:32.294 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2024-03-28T18:15:32.294 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: -1> 2024-03-28T18:15:31.902+0000 7f29e5441700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f29e5441700 time 2024-03-28T18:15:31.902332+0000 2024-03-28T18:15:32.294 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:15:32.294 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2024-03-28T18:15:32.294 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:32.294 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f29efa6b4a1] 2024-03-28T18:15:32.294 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f29efa6b667] 2024-03-28T18:15:32.294 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f3136ab223] 2024-03-28T18:15:32.294 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f3135255d6] 2024-03-28T18:15:32.294 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f313525da1] 2024-03-28T18:15:32.294 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f3135282d5] 2024-03-28T18:15:32.294 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f313578233] 2024-03-28T18:15:32.294 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f31357a197] 2024-03-28T18:15:32.294 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f31347911b] 2024-03-28T18:15:32.294 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f3134796bc] 2024-03-28T18:15:32.295 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f3134631ff] 2024-03-28T18:15:32.295 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f29efcfd488] 2024-03-28T18:15:32.295 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 13: (DispatchQueue::entry()+0x51f) [0x7f29efcfa60f] 2024-03-28T18:15:32.295 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f29efdcde81] 2024-03-28T18:15:32.295 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f29ee7e81ca] 2024-03-28T18:15:32.295 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 16: clone() 2024-03-28T18:15:32.295 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2024-03-28T18:15:32.295 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 0> 2024-03-28T18:15:31.905+0000 7f29e5441700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:15:32.295 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: in thread 7f29e5441700 thread_name:ms_dispatch 2024-03-28T18:15:32.295 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2024-03-28T18:15:32.295 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:32.295 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f29ee7f2d20] 2024-03-28T18:15:32.295 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2: gsignal() 2024-03-28T18:15:32.295 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 3: abort() 2024-03-28T18:15:32.295 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f29efa6b4fb] 2024-03-28T18:15:32.295 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f29efa6b667] 2024-03-28T18:15:32.296 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f3136ab223] 2024-03-28T18:15:32.296 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f3135255d6] 2024-03-28T18:15:32.296 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f313525da1] 2024-03-28T18:15:32.296 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f3135282d5] 2024-03-28T18:15:32.296 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f313578233] 2024-03-28T18:15:32.296 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f31357a197] 2024-03-28T18:15:32.296 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f31347911b] 2024-03-28T18:15:32.296 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f3134796bc] 2024-03-28T18:15:32.296 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f3134631ff] 2024-03-28T18:15:32.296 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f29efcfd488] 2024-03-28T18:15:32.296 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 16: (DispatchQueue::entry()+0x51f) [0x7f29efcfa60f] 2024-03-28T18:15:32.296 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f29efdcde81] 2024-03-28T18:15:32.296 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f29ee7e81ca] 2024-03-28T18:15:32.296 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 19: clone() 2024-03-28T18:15:32.296 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:15:32.297 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2024-03-28T18:15:32.297 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: -1> 2024-03-28T18:15:31.902+0000 7f29e5441700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f29e5441700 time 2024-03-28T18:15:31.902332+0000 2024-03-28T18:15:32.297 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:15:32.297 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2024-03-28T18:15:32.297 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:32.297 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f29efa6b4a1] 2024-03-28T18:15:32.297 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f29efa6b667] 2024-03-28T18:15:32.297 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f3136ab223] 2024-03-28T18:15:32.297 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f3135255d6] 2024-03-28T18:15:32.297 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f313525da1] 2024-03-28T18:15:32.297 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f3135282d5] 2024-03-28T18:15:32.297 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f313578233] 2024-03-28T18:15:32.297 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f31357a197] 2024-03-28T18:15:32.297 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f31347911b] 2024-03-28T18:15:32.297 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f3134796bc] 2024-03-28T18:15:32.297 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f3134631ff] 2024-03-28T18:15:32.298 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f29efcfd488] 2024-03-28T18:15:32.298 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 13: (DispatchQueue::entry()+0x51f) [0x7f29efcfa60f] 2024-03-28T18:15:32.298 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f29efdcde81] 2024-03-28T18:15:32.298 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f29ee7e81ca] 2024-03-28T18:15:32.298 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 16: clone() 2024-03-28T18:15:32.298 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2024-03-28T18:15:32.298 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 0> 2024-03-28T18:15:31.905+0000 7f29e5441700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:15:32.298 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: in thread 7f29e5441700 thread_name:ms_dispatch 2024-03-28T18:15:32.298 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2024-03-28T18:15:32.298 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:32.298 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f29ee7f2d20] 2024-03-28T18:15:32.298 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2: gsignal() 2024-03-28T18:15:32.298 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 3: abort() 2024-03-28T18:15:32.298 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f29efa6b4fb] 2024-03-28T18:15:32.299 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f29efa6b667] 2024-03-28T18:15:32.299 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f3136ab223] 2024-03-28T18:15:32.299 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f3135255d6] 2024-03-28T18:15:32.299 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f313525da1] 2024-03-28T18:15:32.299 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f3135282d5] 2024-03-28T18:15:32.299 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f313578233] 2024-03-28T18:15:32.299 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f31357a197] 2024-03-28T18:15:32.299 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f31347911b] 2024-03-28T18:15:32.299 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f3134796bc] 2024-03-28T18:15:32.299 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f3134631ff] 2024-03-28T18:15:32.299 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f29efcfd488] 2024-03-28T18:15:32.299 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 16: (DispatchQueue::entry()+0x51f) [0x7f29efcfa60f] 2024-03-28T18:15:32.299 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f29efdcde81] 2024-03-28T18:15:32.299 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f29ee7e81ca] 2024-03-28T18:15:32.299 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 19: clone() 2024-03-28T18:15:32.299 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:15:32.300 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[70961]: 2024-03-28T18:15:32.300 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:31 smithi170 podman[71135]: 2024-03-28 18:15:31.98349669 +0000 UTC m=+0.018822174 container died 33c69e75620daa17f39e04e61aab332cc7967842b53b3ef7573ff96542b2aadf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:15:32.918 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:32 smithi170 podman[71135]: 2024-03-28 18:15:32.662601496 +0000 UTC m=+0.697926968 container remove 33c69e75620daa17f39e04e61aab332cc7967842b53b3ef7573ff96542b2aadf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:15:32.919 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:32 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:15:32.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:32 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:15:32.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:32 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:15:32.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:32 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/1772048823,v1:172.21.15.170:6835/1772048823] up:active 2024-03-28T18:15:32.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:32 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:32.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:32 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:15:32.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:32 smithi170 ceph-mon[36280]: pgmap v141: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-28T18:15:33.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:32 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:15:33.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:32 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:15:33.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:32 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/1772048823,v1:172.21.15.170:6835/1772048823] up:active 2024-03-28T18:15:33.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:32 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:33.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:32 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:15:33.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:32 smithi037 ceph-mon[31460]: pgmap v141: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-28T18:15:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:32 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:15:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:32 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:15:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:32 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/1772048823,v1:172.21.15.170:6835/1772048823] up:active 2024-03-28T18:15:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:32 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:32 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:15:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:32 smithi059 ceph-mon[36190]: pgmap v141: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-28T18:15:33.286 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:32 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-28T18:15:35.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:35 smithi059 ceph-mon[36190]: pgmap v142: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-28T18:15:35.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:35 smithi170 ceph-mon[36280]: pgmap v142: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-28T18:15:35.806 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:35 smithi037 ceph-mon[31460]: pgmap v142: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-28T18:15:36.877 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:15:36 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:15:36.406+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:15:37.776 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:37 smithi037 ceph-mon[31460]: pgmap v143: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 1.1 KiB/s wr, 17 op/s 2024-03-28T18:15:37.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:37 smithi170 ceph-mon[36280]: pgmap v143: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 1.1 KiB/s wr, 17 op/s 2024-03-28T18:15:38.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:37 smithi059 ceph-mon[36190]: pgmap v143: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 1.1 KiB/s wr, 17 op/s 2024-03-28T18:15:39.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:39 smithi037 ceph-mon[31460]: pgmap v144: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 1.0 KiB/s wr, 16 op/s 2024-03-28T18:15:40.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:15:40.033+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:15:40.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:15:40.034+0000 7f4f82a00640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 msgr2=0x56357a39a010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-28T18:15:40.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:15:40.034+0000 7f4f82a00640 1 -- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 msgr2=0x56357a39a010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T18:15:40.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:15:40.034+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T18:15:40.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-28T18:15:40.034+0000 7f4f82a00640 1 --2- 172.21.15.37:0/2018652346 >> [v2:172.21.15.59:6838/2091629078,v1:172.21.15.59:6839/2091629078] conn(0x56357a397bd0 0x56357a39a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-28T18:15:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:39 smithi059 ceph-mon[36190]: pgmap v144: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 1.0 KiB/s wr, 16 op/s 2024-03-28T18:15:40.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:39 smithi170 ceph-mon[36280]: pgmap v144: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 1.0 KiB/s wr, 16 op/s 2024-03-28T18:15:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:41 smithi037 ceph-mon[31460]: pgmap v145: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 938 B/s wr, 14 op/s 2024-03-28T18:15:42.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:41 smithi059 ceph-mon[36190]: pgmap v145: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 938 B/s wr, 14 op/s 2024-03-28T18:15:42.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:41 smithi170 ceph-mon[36280]: pgmap v145: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 938 B/s wr, 14 op/s 2024-03-28T18:15:43.303 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:43 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 4. 2024-03-28T18:15:43.303 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:43 smithi170 systemd[1]: Stopped Ceph mds.f for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:15:43.304 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:43 smithi170 systemd[1]: Starting Ceph mds.f for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:15:43.304 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:43 smithi170 podman[71233]: 2024-03-28 18:15:43.168588031 +0000 UTC m=+0.019896614 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:15:43.615 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:43 smithi170 ceph-mon[36280]: pgmap v146: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 938 B/s wr, 16 op/s 2024-03-28T18:15:43.615 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:43 smithi170 podman[71233]: 2024-03-28 18:15:43.304620094 +0000 UTC m=+0.155928662 container create 79ade11bcebb55abfddbc639e073d773370985d599378a3221764b653093f84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:15:43.676 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:43 smithi037 ceph-mon[31460]: pgmap v146: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 938 B/s wr, 16 op/s 2024-03-28T18:15:44.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:43 smithi059 ceph-mon[36190]: pgmap v146: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 938 B/s wr, 16 op/s 2024-03-28T18:15:44.035 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:43 smithi170 podman[71233]: 2024-03-28 18:15:43.616330048 +0000 UTC m=+0.467638628 container init 79ade11bcebb55abfddbc639e073d773370985d599378a3221764b653093f84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:15:44.035 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:43 smithi170 podman[71233]: 2024-03-28 18:15:43.620268601 +0000 UTC m=+0.471577161 container start 79ade11bcebb55abfddbc639e073d773370985d599378a3221764b653093f84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:15:44.035 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:43 smithi170 ceph-mds[71251]: -- 172.21.15.170:0/1814772637 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d36507fa40 con 0x55d365053000 2024-03-28T18:15:44.035 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:43 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: starting mds.f at 2024-03-28T18:15:44.035 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:43 smithi170 bash[71233]: 79ade11bcebb55abfddbc639e073d773370985d599378a3221764b653093f84b 2024-03-28T18:15:44.036 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:15:43 smithi170 systemd[1]: Started Ceph mds.f for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:15:44.383 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() socket.timeout 2024-03-28T18:15:44.473 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-28T18:15:44.474 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/fwd_scrub.py", line 79, in _scrub self._check_damage() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/fwd_scrub.py", line 82, in _check_damage rdmg = self.fs.get_damage() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 1777, in get_damage result[rank] = self.get_damage(rank=rank) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 1780, in get_damage return self.rank_tell(['damage', 'ls'], rank=rank) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 1319, in rank_tell out = self.get_ceph_cmd_stdout("tell", f"mds.{self.id}:{rank}", *command, timeout=timeout) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() socket.timeout 2024-03-28T18:15:44.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:44 smithi037 ceph-mon[31460]: daemon mds.f restarted 2024-03-28T18:15:44.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:44 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:15:44.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:44 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:15:44.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:44 smithi037 ceph-mon[31460]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T18:15:44.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:44 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/2749107471,v1:172.21.15.170:6835/2749107471] up:boot 2024-03-28T18:15:44.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:44 smithi037 ceph-mon[31460]: Standby daemon mds.b assigned to filesystem cephfs as rank 1 2024-03-28T18:15:44.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:44 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:15:44.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:44 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:45.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:44 smithi059 ceph-mon[36190]: daemon mds.f restarted 2024-03-28T18:15:45.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:44 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:15:45.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:44 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:15:45.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:44 smithi059 ceph-mon[36190]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T18:15:45.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:44 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/2749107471,v1:172.21.15.170:6835/2749107471] up:boot 2024-03-28T18:15:45.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:44 smithi059 ceph-mon[36190]: Standby daemon mds.b assigned to filesystem cephfs as rank 1 2024-03-28T18:15:45.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:44 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:15:45.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:44 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:45.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:44 smithi170 ceph-mon[36280]: daemon mds.f restarted 2024-03-28T18:15:45.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:44 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:15:45.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:44 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:15:45.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:44 smithi170 ceph-mon[36280]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T18:15:45.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:44 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/2749107471,v1:172.21.15.170:6835/2749107471] up:boot 2024-03-28T18:15:45.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:44 smithi170 ceph-mon[36280]: Standby daemon mds.b assigned to filesystem cephfs as rank 1 2024-03-28T18:15:45.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:44 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T18:15:45.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:44 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:46.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:45 smithi059 ceph-mon[36190]: pgmap v148: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 16 op/s 2024-03-28T18:15:46.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:45 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/3215592665,v1:172.21.15.59:6839/3215592665] up:resolve 2024-03-28T18:15:46.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:45 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:46.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:45 smithi170 ceph-mon[36280]: pgmap v148: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 16 op/s 2024-03-28T18:15:46.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:45 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/3215592665,v1:172.21.15.59:6839/3215592665] up:resolve 2024-03-28T18:15:46.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:45 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:46.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:45 smithi037 ceph-mon[31460]: pgmap v148: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 16 op/s 2024-03-28T18:15:46.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:45 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/3215592665,v1:172.21.15.59:6839/3215592665] up:resolve 2024-03-28T18:15:46.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:45 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:47.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:46 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/3215592665,v1:172.21.15.59:6839/3215592665] up:reconnect 2024-03-28T18:15:47.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:46 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:47.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:46 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00999989 2024-03-28T18:15:47.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:46 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0109999 2024-03-28T18:15:47.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:46 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/3215592665,v1:172.21.15.59:6839/3215592665] up:reconnect 2024-03-28T18:15:47.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:46 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:47.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:46 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00999989 2024-03-28T18:15:47.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:46 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0109999 2024-03-28T18:15:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:46 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/3215592665,v1:172.21.15.59:6839/3215592665] up:reconnect 2024-03-28T18:15:47.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:46 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:47.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:46 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00999989 2024-03-28T18:15:47.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:46 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0109999 2024-03-28T18:15:48.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-mon[36190]: pgmap v149: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-28T18:15:48.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/3215592665,v1:172.21.15.59:6839/3215592665] up:rejoin 2024-03-28T18:15:48.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:48.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-mon[36190]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-28T18:15:48.036 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7efe1a335700 time 2024-03-28T18:15:47.747791+0000 2024-03-28T18:15:48.036 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:15:48.036 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:48.036 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7efe2495f4a1] 2024-03-28T18:15:48.036 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7efe2495f667] 2024-03-28T18:15:48.036 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a1f5bf4223] 2024-03-28T18:15:48.037 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a1f5a6e5d6] 2024-03-28T18:15:48.037 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a1f5a6eda1] 2024-03-28T18:15:48.037 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a1f5a712d5] 2024-03-28T18:15:48.037 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a1f5ac1233] 2024-03-28T18:15:48.037 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a1f5ac3197] 2024-03-28T18:15:48.037 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a1f59c211b] 2024-03-28T18:15:48.037 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a1f59c26bc] 2024-03-28T18:15:48.037 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a1f59ac1ff] 2024-03-28T18:15:48.037 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7efe24bf1488] 2024-03-28T18:15:48.037 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 13: (DispatchQueue::entry()+0x51f) [0x7efe24bee60f] 2024-03-28T18:15:48.037 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efe24cc1e81] 2024-03-28T18:15:48.037 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7efe236dc1ca] 2024-03-28T18:15:48.037 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 16: clone() 2024-03-28T18:15:48.038 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2024-03-28T18:15:47.748+0000 7efe1a335700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7efe1a335700 time 2024-03-28T18:15:47.747791+0000 2024-03-28T18:15:48.038 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:15:48.038 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2024-03-28T18:15:48.038 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:48.038 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7efe2495f4a1] 2024-03-28T18:15:48.038 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7efe2495f667] 2024-03-28T18:15:48.038 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a1f5bf4223] 2024-03-28T18:15:48.038 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a1f5a6e5d6] 2024-03-28T18:15:48.038 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a1f5a6eda1] 2024-03-28T18:15:48.038 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a1f5a712d5] 2024-03-28T18:15:48.038 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a1f5ac1233] 2024-03-28T18:15:48.038 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a1f5ac3197] 2024-03-28T18:15:48.038 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a1f59c211b] 2024-03-28T18:15:48.039 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a1f59c26bc] 2024-03-28T18:15:48.039 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a1f59ac1ff] 2024-03-28T18:15:48.039 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7efe24bf1488] 2024-03-28T18:15:48.039 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 13: (DispatchQueue::entry()+0x51f) [0x7efe24bee60f] 2024-03-28T18:15:48.039 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efe24cc1e81] 2024-03-28T18:15:48.039 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7efe236dc1ca] 2024-03-28T18:15:48.039 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 16: clone() 2024-03-28T18:15:48.039 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2024-03-28T18:15:48.039 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: *** Caught signal (Aborted) ** 2024-03-28T18:15:48.039 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: in thread 7efe1a335700 thread_name:ms_dispatch 2024-03-28T18:15:48.039 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:48.039 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efe236e6d20] 2024-03-28T18:15:48.039 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2: gsignal() 2024-03-28T18:15:48.039 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 3: abort() 2024-03-28T18:15:48.040 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7efe2495f4fb] 2024-03-28T18:15:48.040 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7efe2495f667] 2024-03-28T18:15:48.040 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a1f5bf4223] 2024-03-28T18:15:48.040 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a1f5a6e5d6] 2024-03-28T18:15:48.040 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a1f5a6eda1] 2024-03-28T18:15:48.040 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a1f5a712d5] 2024-03-28T18:15:48.040 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a1f5ac1233] 2024-03-28T18:15:48.040 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a1f5ac3197] 2024-03-28T18:15:48.040 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a1f59c211b] 2024-03-28T18:15:48.040 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a1f59c26bc] 2024-03-28T18:15:48.040 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a1f59ac1ff] 2024-03-28T18:15:48.041 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7efe24bf1488] 2024-03-28T18:15:48.041 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 16: (DispatchQueue::entry()+0x51f) [0x7efe24bee60f] 2024-03-28T18:15:48.041 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efe24cc1e81] 2024-03-28T18:15:48.041 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7efe236dc1ca] 2024-03-28T18:15:48.041 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 19: clone() 2024-03-28T18:15:48.041 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2024-03-28T18:15:47.751+0000 7efe1a335700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:15:48.041 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: in thread 7efe1a335700 thread_name:ms_dispatch 2024-03-28T18:15:48.041 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2024-03-28T18:15:48.041 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:48.041 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efe236e6d20] 2024-03-28T18:15:48.041 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2: gsignal() 2024-03-28T18:15:48.041 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 3: abort() 2024-03-28T18:15:48.041 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7efe2495f4fb] 2024-03-28T18:15:48.041 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7efe2495f667] 2024-03-28T18:15:48.042 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:47 smithi170 ceph-mon[36280]: pgmap v149: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-28T18:15:48.042 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:47 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/3215592665,v1:172.21.15.59:6839/3215592665] up:rejoin 2024-03-28T18:15:48.042 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:47 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:48.042 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:47 smithi170 ceph-mon[36280]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-28T18:15:48.043 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a1f5bf4223] 2024-03-28T18:15:48.043 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a1f5a6e5d6] 2024-03-28T18:15:48.043 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a1f5a6eda1] 2024-03-28T18:15:48.043 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a1f5a712d5] 2024-03-28T18:15:48.043 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a1f5ac1233] 2024-03-28T18:15:48.043 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a1f5ac3197] 2024-03-28T18:15:48.043 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a1f59c211b] 2024-03-28T18:15:48.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a1f59c26bc] 2024-03-28T18:15:48.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a1f59ac1ff] 2024-03-28T18:15:48.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7efe24bf1488] 2024-03-28T18:15:48.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 16: (DispatchQueue::entry()+0x51f) [0x7efe24bee60f] 2024-03-28T18:15:48.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efe24cc1e81] 2024-03-28T18:15:48.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7efe236dc1ca] 2024-03-28T18:15:48.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 19: clone() 2024-03-28T18:15:48.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:15:48.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2024-03-28T18:15:48.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: -1> 2024-03-28T18:15:47.748+0000 7efe1a335700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7efe1a335700 time 2024-03-28T18:15:47.747791+0000 2024-03-28T18:15:48.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:15:48.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2024-03-28T18:15:48.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:48.044 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7efe2495f4a1] 2024-03-28T18:15:48.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7efe2495f667] 2024-03-28T18:15:48.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a1f5bf4223] 2024-03-28T18:15:48.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a1f5a6e5d6] 2024-03-28T18:15:48.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a1f5a6eda1] 2024-03-28T18:15:48.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a1f5a712d5] 2024-03-28T18:15:48.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a1f5ac1233] 2024-03-28T18:15:48.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a1f5ac3197] 2024-03-28T18:15:48.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a1f59c211b] 2024-03-28T18:15:48.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a1f59c26bc] 2024-03-28T18:15:48.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a1f59ac1ff] 2024-03-28T18:15:48.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7efe24bf1488] 2024-03-28T18:15:48.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 13: (DispatchQueue::entry()+0x51f) [0x7efe24bee60f] 2024-03-28T18:15:48.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efe24cc1e81] 2024-03-28T18:15:48.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7efe236dc1ca] 2024-03-28T18:15:48.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 16: clone() 2024-03-28T18:15:48.045 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2024-03-28T18:15:48.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 0> 2024-03-28T18:15:47.751+0000 7efe1a335700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:15:48.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: in thread 7efe1a335700 thread_name:ms_dispatch 2024-03-28T18:15:48.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2024-03-28T18:15:48.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:48.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efe236e6d20] 2024-03-28T18:15:48.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2: gsignal() 2024-03-28T18:15:48.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 3: abort() 2024-03-28T18:15:48.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7efe2495f4fb] 2024-03-28T18:15:48.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7efe2495f667] 2024-03-28T18:15:48.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a1f5bf4223] 2024-03-28T18:15:48.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a1f5a6e5d6] 2024-03-28T18:15:48.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a1f5a6eda1] 2024-03-28T18:15:48.046 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a1f5a712d5] 2024-03-28T18:15:48.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a1f5ac1233] 2024-03-28T18:15:48.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a1f5ac3197] 2024-03-28T18:15:48.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a1f59c211b] 2024-03-28T18:15:48.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a1f59c26bc] 2024-03-28T18:15:48.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a1f59ac1ff] 2024-03-28T18:15:48.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7efe24bf1488] 2024-03-28T18:15:48.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 16: (DispatchQueue::entry()+0x51f) [0x7efe24bee60f] 2024-03-28T18:15:48.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efe24cc1e81] 2024-03-28T18:15:48.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7efe236dc1ca] 2024-03-28T18:15:48.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 19: clone() 2024-03-28T18:15:48.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:15:48.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2024-03-28T18:15:48.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: -1> 2024-03-28T18:15:47.748+0000 7efe1a335700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7efe1a335700 time 2024-03-28T18:15:47.747791+0000 2024-03-28T18:15:48.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:15:48.047 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2024-03-28T18:15:48.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:48.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7efe2495f4a1] 2024-03-28T18:15:48.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7efe2495f667] 2024-03-28T18:15:48.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a1f5bf4223] 2024-03-28T18:15:48.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a1f5a6e5d6] 2024-03-28T18:15:48.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a1f5a6eda1] 2024-03-28T18:15:48.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a1f5a712d5] 2024-03-28T18:15:48.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a1f5ac1233] 2024-03-28T18:15:48.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a1f5ac3197] 2024-03-28T18:15:48.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a1f59c211b] 2024-03-28T18:15:48.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a1f59c26bc] 2024-03-28T18:15:48.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a1f59ac1ff] 2024-03-28T18:15:48.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7efe24bf1488] 2024-03-28T18:15:48.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 13: (DispatchQueue::entry()+0x51f) [0x7efe24bee60f] 2024-03-28T18:15:48.048 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efe24cc1e81] 2024-03-28T18:15:48.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7efe236dc1ca] 2024-03-28T18:15:48.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 16: clone() 2024-03-28T18:15:48.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2024-03-28T18:15:48.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 0> 2024-03-28T18:15:47.751+0000 7efe1a335700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:15:48.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: in thread 7efe1a335700 thread_name:ms_dispatch 2024-03-28T18:15:48.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2024-03-28T18:15:48.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:15:48.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efe236e6d20] 2024-03-28T18:15:48.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2: gsignal() 2024-03-28T18:15:48.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 3: abort() 2024-03-28T18:15:48.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7efe2495f4fb] 2024-03-28T18:15:48.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7efe2495f667] 2024-03-28T18:15:48.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a1f5bf4223] 2024-03-28T18:15:48.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a1f5a6e5d6] 2024-03-28T18:15:48.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a1f5a6eda1] 2024-03-28T18:15:48.049 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a1f5a712d5] 2024-03-28T18:15:48.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a1f5ac1233] 2024-03-28T18:15:48.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a1f5ac3197] 2024-03-28T18:15:48.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a1f59c211b] 2024-03-28T18:15:48.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a1f59c26bc] 2024-03-28T18:15:48.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a1f59ac1ff] 2024-03-28T18:15:48.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7efe24bf1488] 2024-03-28T18:15:48.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 16: (DispatchQueue::entry()+0x51f) [0x7efe24bee60f] 2024-03-28T18:15:48.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efe24cc1e81] 2024-03-28T18:15:48.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7efe236dc1ca] 2024-03-28T18:15:48.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 19: clone() 2024-03-28T18:15:48.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:15:48.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b[71336]: 2024-03-28T18:15:48.050 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:47 smithi059 podman[71386]: 2024-03-28 18:15:47.829078719 +0000 UTC m=+0.020192586 container died 63fb096776982ebf94d56399bf83ac6c649c6b706a19202b01d87892aecf7c25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-28T18:15:48.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:47 smithi037 ceph-mon[31460]: pgmap v149: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-28T18:15:48.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:47 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/3215592665,v1:172.21.15.59:6839/3215592665] up:rejoin 2024-03-28T18:15:48.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:47 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=b=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:48.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:47 smithi037 ceph-mon[31460]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-28T18:15:48.812 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:48 smithi059 podman[71386]: 2024-03-28 18:15:48.558911218 +0000 UTC m=+0.750025073 container remove 63fb096776982ebf94d56399bf83ac6c649c6b706a19202b01d87892aecf7c25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-28T18:15:48.812 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:48 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:15:48.812 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:48 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:15:48.812 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:48 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:15:48.812 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:48 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6838/3215592665,v1:172.21.15.59:6839/3215592665] up:active 2024-03-28T18:15:48.812 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:48 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:48.812 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:48 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:15:49.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:48 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:15:49.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:48 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:15:49.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:48 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6838/3215592665,v1:172.21.15.59:6839/3215592665] up:active 2024-03-28T18:15:49.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:48 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:49.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:48 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:15:49.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:48 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:15:49.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:48 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:15:49.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:48 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6838/3215592665,v1:172.21.15.59:6839/3215592665] up:active 2024-03-28T18:15:49.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:48 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T18:15:49.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:48 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T18:15:49.285 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:48 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.b.service: Failed with result 'exit-code'. 2024-03-28T18:15:50.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:49 smithi059 ceph-mon[36190]: pgmap v150: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-28T18:15:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:49 smithi170 ceph-mon[36280]: pgmap v150: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-28T18:15:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:49 smithi037 ceph-mon[31460]: pgmap v150: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-28T18:15:51.678 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:15:51 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:15:51.407+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:15:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:51 smithi059 ceph-mon[36190]: pgmap v151: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-28T18:15:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:51 smithi170 ceph-mon[36280]: pgmap v151: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-28T18:15:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:51 smithi037 ceph-mon[31460]: pgmap v151: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-28T18:15:54.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:53 smithi059 ceph-mon[36190]: pgmap v152: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 204 B/s wr, 17 op/s 2024-03-28T18:15:54.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:53 smithi170 ceph-mon[36280]: pgmap v152: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 204 B/s wr, 17 op/s 2024-03-28T18:15:54.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:53 smithi037 ceph-mon[31460]: pgmap v152: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 204 B/s wr, 17 op/s 2024-03-28T18:15:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:55 smithi059 ceph-mon[36190]: pgmap v153: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 189 B/s wr, 16 op/s 2024-03-28T18:15:56.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:55 smithi170 ceph-mon[36280]: pgmap v153: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 189 B/s wr, 16 op/s 2024-03-28T18:15:56.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:55 smithi037 ceph-mon[31460]: pgmap v153: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 189 B/s wr, 16 op/s 2024-03-28T18:15:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:57 smithi059 ceph-mon[36190]: pgmap v154: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 17 op/s 2024-03-28T18:15:58.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:57 smithi170 ceph-mon[36280]: pgmap v154: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 17 op/s 2024-03-28T18:15:58.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:57 smithi037 ceph-mon[31460]: pgmap v154: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 17 op/s 2024-03-28T18:15:59.535 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:59 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.b.service: Scheduled restart job, restart counter is at 5. 2024-03-28T18:15:59.535 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:59 smithi059 systemd[1]: Stopped Ceph mds.b for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:15:59.535 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:59 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.b.service: Start request repeated too quickly. 2024-03-28T18:15:59.535 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:59 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.b.service: Failed with result 'exit-code'. 2024-03-28T18:15:59.535 INFO:journalctl@ceph.mds.b.smithi059.stdout:Mar 28 18:15:59 smithi059 systemd[1]: Failed to start Ceph mds.b for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:16:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:15:59 smithi059 ceph-mon[36190]: pgmap v155: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:16:00.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:15:59 smithi170 ceph-mon[36280]: pgmap v155: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:16:00.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:15:59 smithi037 ceph-mon[31460]: pgmap v155: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:16:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:01 smithi059 ceph-mon[36190]: pgmap v156: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:16:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:01 smithi170 ceph-mon[36280]: pgmap v156: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:16:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:01 smithi037 ceph-mon[31460]: pgmap v156: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:16:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:03 smithi059 ceph-mon[36190]: pgmap v157: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 18 op/s 2024-03-28T18:16:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:03 smithi059 ceph-mon[36190]: Replacing daemon mds.b as rank 1 with standby daemon mds.f 2024-03-28T18:16:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:03 smithi059 ceph-mon[36190]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-03-28T18:16:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:03 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:16:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:03 smithi059 ceph-mon[36190]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T18:16:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:03 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:replay,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:04.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:03 smithi170 ceph-mon[36280]: pgmap v157: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 18 op/s 2024-03-28T18:16:04.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:03 smithi170 ceph-mon[36280]: Replacing daemon mds.b as rank 1 with standby daemon mds.f 2024-03-28T18:16:04.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:03 smithi170 ceph-mon[36280]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-03-28T18:16:04.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:03 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:16:04.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:03 smithi170 ceph-mon[36280]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T18:16:04.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:03 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:replay,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:03 smithi037 ceph-mon[31460]: pgmap v157: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 18 op/s 2024-03-28T18:16:04.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:03 smithi037 ceph-mon[31460]: Replacing daemon mds.b as rank 1 with standby daemon mds.f 2024-03-28T18:16:04.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:03 smithi037 ceph-mon[31460]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-03-28T18:16:04.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:03 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:16:04.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:03 smithi037 ceph-mon[31460]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T18:16:04.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:03 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:replay,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:06.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:05 smithi059 ceph-mon[36190]: pgmap v159: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:16:06.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:05 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/2749107471,v1:172.21.15.170:6835/2749107471] up:resolve 2024-03-28T18:16:06.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:05 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:resolve,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:06.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:05 smithi170 ceph-mon[36280]: pgmap v159: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:16:06.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:05 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/2749107471,v1:172.21.15.170:6835/2749107471] up:resolve 2024-03-28T18:16:06.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:05 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:resolve,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:05 smithi037 ceph-mon[31460]: pgmap v159: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:16:06.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:05 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/2749107471,v1:172.21.15.170:6835/2749107471] up:resolve 2024-03-28T18:16:06.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:05 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:resolve,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:06 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199996 2024-03-28T18:16:07.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:06 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/2749107471,v1:172.21.15.170:6835/2749107471] up:reconnect 2024-03-28T18:16:07.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:06 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:reconnect,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:07.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:06 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089998 2024-03-28T18:16:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:06 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199996 2024-03-28T18:16:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:06 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/2749107471,v1:172.21.15.170:6835/2749107471] up:reconnect 2024-03-28T18:16:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:06 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:reconnect,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:06 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089998 2024-03-28T18:16:07.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:06 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199996 2024-03-28T18:16:07.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:06 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/2749107471,v1:172.21.15.170:6835/2749107471] up:reconnect 2024-03-28T18:16:07.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:06 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:reconnect,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:07.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:06 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089998 2024-03-28T18:16:08.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:07 smithi037 ceph-mon[31460]: pgmap v160: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:16:08.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:07 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/2749107471,v1:172.21.15.170:6835/2749107471] up:rejoin 2024-03-28T18:16:08.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:07 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:rejoin,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:08.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:07 smithi037 ceph-mon[31460]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-28T18:16:08.193 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-mon[36280]: pgmap v160: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:16:08.194 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/2749107471,v1:172.21.15.170:6835/2749107471] up:rejoin 2024-03-28T18:16:08.194 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:rejoin,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:08.194 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-mon[36280]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-28T18:16:08.195 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fea71e6b700 time 2024-03-28T18:16:07.827816+0000 2024-03-28T18:16:08.195 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:16:08.195 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:08.195 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fea7c4954a1] 2024-03-28T18:16:08.195 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fea7c495667] 2024-03-28T18:16:08.195 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d363d4f223] 2024-03-28T18:16:08.195 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d363bc95d6] 2024-03-28T18:16:08.195 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d363bc9da1] 2024-03-28T18:16:08.195 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d363bcc2d5] 2024-03-28T18:16:08.195 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d363c1c233] 2024-03-28T18:16:08.196 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d363c1e197] 2024-03-28T18:16:08.196 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d363b1d11b] 2024-03-28T18:16:08.196 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d363b1d6bc] 2024-03-28T18:16:08.196 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d363b071ff] 2024-03-28T18:16:08.196 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fea7c727488] 2024-03-28T18:16:08.196 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 13: (DispatchQueue::entry()+0x51f) [0x7fea7c72460f] 2024-03-28T18:16:08.196 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fea7c7f7e81] 2024-03-28T18:16:08.196 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fea7b2121ca] 2024-03-28T18:16:08.196 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 16: clone() 2024-03-28T18:16:08.196 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2024-03-28T18:16:07.828+0000 7fea71e6b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fea71e6b700 time 2024-03-28T18:16:07.827816+0000 2024-03-28T18:16:08.196 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:16:08.196 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2024-03-28T18:16:08.196 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:08.196 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fea7c4954a1] 2024-03-28T18:16:08.196 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fea7c495667] 2024-03-28T18:16:08.197 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d363d4f223] 2024-03-28T18:16:08.197 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d363bc95d6] 2024-03-28T18:16:08.197 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d363bc9da1] 2024-03-28T18:16:08.197 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d363bcc2d5] 2024-03-28T18:16:08.197 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d363c1c233] 2024-03-28T18:16:08.197 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d363c1e197] 2024-03-28T18:16:08.197 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d363b1d11b] 2024-03-28T18:16:08.197 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d363b1d6bc] 2024-03-28T18:16:08.197 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d363b071ff] 2024-03-28T18:16:08.197 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fea7c727488] 2024-03-28T18:16:08.197 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 13: (DispatchQueue::entry()+0x51f) [0x7fea7c72460f] 2024-03-28T18:16:08.197 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fea7c7f7e81] 2024-03-28T18:16:08.197 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fea7b2121ca] 2024-03-28T18:16:08.197 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 16: clone() 2024-03-28T18:16:08.197 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2024-03-28T18:16:08.198 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: *** Caught signal (Aborted) ** 2024-03-28T18:16:08.198 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: in thread 7fea71e6b700 thread_name:ms_dispatch 2024-03-28T18:16:08.198 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:08.198 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fea7b21cd20] 2024-03-28T18:16:08.198 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2: gsignal() 2024-03-28T18:16:08.198 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 3: abort() 2024-03-28T18:16:08.198 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fea7c4954fb] 2024-03-28T18:16:08.198 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fea7c495667] 2024-03-28T18:16:08.198 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d363d4f223] 2024-03-28T18:16:08.198 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d363bc95d6] 2024-03-28T18:16:08.198 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d363bc9da1] 2024-03-28T18:16:08.198 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d363bcc2d5] 2024-03-28T18:16:08.198 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d363c1c233] 2024-03-28T18:16:08.198 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d363c1e197] 2024-03-28T18:16:08.199 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d363b1d11b] 2024-03-28T18:16:08.199 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d363b1d6bc] 2024-03-28T18:16:08.199 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d363b071ff] 2024-03-28T18:16:08.199 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fea7c727488] 2024-03-28T18:16:08.199 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 16: (DispatchQueue::entry()+0x51f) [0x7fea7c72460f] 2024-03-28T18:16:08.199 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fea7c7f7e81] 2024-03-28T18:16:08.199 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fea7b2121ca] 2024-03-28T18:16:08.199 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 19: clone() 2024-03-28T18:16:08.199 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2024-03-28T18:16:07.830+0000 7fea71e6b700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:16:08.199 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: in thread 7fea71e6b700 thread_name:ms_dispatch 2024-03-28T18:16:08.199 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2024-03-28T18:16:08.199 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:08.199 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fea7b21cd20] 2024-03-28T18:16:08.199 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2: gsignal() 2024-03-28T18:16:08.199 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 3: abort() 2024-03-28T18:16:08.200 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fea7c4954fb] 2024-03-28T18:16:08.200 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fea7c495667] 2024-03-28T18:16:08.200 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d363d4f223] 2024-03-28T18:16:08.200 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d363bc95d6] 2024-03-28T18:16:08.200 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d363bc9da1] 2024-03-28T18:16:08.200 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d363bcc2d5] 2024-03-28T18:16:08.200 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d363c1c233] 2024-03-28T18:16:08.200 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d363c1e197] 2024-03-28T18:16:08.200 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d363b1d11b] 2024-03-28T18:16:08.200 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d363b1d6bc] 2024-03-28T18:16:08.200 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d363b071ff] 2024-03-28T18:16:08.200 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fea7c727488] 2024-03-28T18:16:08.200 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 16: (DispatchQueue::entry()+0x51f) [0x7fea7c72460f] 2024-03-28T18:16:08.200 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fea7c7f7e81] 2024-03-28T18:16:08.200 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fea7b2121ca] 2024-03-28T18:16:08.201 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 19: clone() 2024-03-28T18:16:08.201 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:16:08.201 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2024-03-28T18:16:08.201 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: -1> 2024-03-28T18:16:07.828+0000 7fea71e6b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fea71e6b700 time 2024-03-28T18:16:07.827816+0000 2024-03-28T18:16:08.201 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:16:08.201 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2024-03-28T18:16:08.201 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:08.201 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fea7c4954a1] 2024-03-28T18:16:08.201 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fea7c495667] 2024-03-28T18:16:08.202 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d363d4f223] 2024-03-28T18:16:08.202 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d363bc95d6] 2024-03-28T18:16:08.202 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d363bc9da1] 2024-03-28T18:16:08.202 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d363bcc2d5] 2024-03-28T18:16:08.202 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d363c1c233] 2024-03-28T18:16:08.202 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d363c1e197] 2024-03-28T18:16:08.202 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d363b1d11b] 2024-03-28T18:16:08.202 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d363b1d6bc] 2024-03-28T18:16:08.202 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d363b071ff] 2024-03-28T18:16:08.203 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fea7c727488] 2024-03-28T18:16:08.203 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 13: (DispatchQueue::entry()+0x51f) [0x7fea7c72460f] 2024-03-28T18:16:08.203 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fea7c7f7e81] 2024-03-28T18:16:08.203 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fea7b2121ca] 2024-03-28T18:16:08.203 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 16: clone() 2024-03-28T18:16:08.203 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2024-03-28T18:16:08.203 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 0> 2024-03-28T18:16:07.830+0000 7fea71e6b700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:16:08.203 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: in thread 7fea71e6b700 thread_name:ms_dispatch 2024-03-28T18:16:08.203 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2024-03-28T18:16:08.203 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:08.203 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fea7b21cd20] 2024-03-28T18:16:08.203 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2: gsignal() 2024-03-28T18:16:08.203 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 3: abort() 2024-03-28T18:16:08.203 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fea7c4954fb] 2024-03-28T18:16:08.203 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fea7c495667] 2024-03-28T18:16:08.204 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d363d4f223] 2024-03-28T18:16:08.204 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d363bc95d6] 2024-03-28T18:16:08.204 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d363bc9da1] 2024-03-28T18:16:08.204 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d363bcc2d5] 2024-03-28T18:16:08.204 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d363c1c233] 2024-03-28T18:16:08.204 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d363c1e197] 2024-03-28T18:16:08.204 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d363b1d11b] 2024-03-28T18:16:08.204 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d363b1d6bc] 2024-03-28T18:16:08.204 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d363b071ff] 2024-03-28T18:16:08.204 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fea7c727488] 2024-03-28T18:16:08.204 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 16: (DispatchQueue::entry()+0x51f) [0x7fea7c72460f] 2024-03-28T18:16:08.204 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fea7c7f7e81] 2024-03-28T18:16:08.204 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fea7b2121ca] 2024-03-28T18:16:08.204 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 19: clone() 2024-03-28T18:16:08.204 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:16:08.205 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2024-03-28T18:16:08.205 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: -1> 2024-03-28T18:16:07.828+0000 7fea71e6b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fea71e6b700 time 2024-03-28T18:16:07.827816+0000 2024-03-28T18:16:08.205 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:16:08.205 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2024-03-28T18:16:08.205 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:08.205 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fea7c4954a1] 2024-03-28T18:16:08.205 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fea7c495667] 2024-03-28T18:16:08.205 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d363d4f223] 2024-03-28T18:16:08.205 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d363bc95d6] 2024-03-28T18:16:08.205 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d363bc9da1] 2024-03-28T18:16:08.205 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d363bcc2d5] 2024-03-28T18:16:08.205 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d363c1c233] 2024-03-28T18:16:08.205 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d363c1e197] 2024-03-28T18:16:08.205 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d363b1d11b] 2024-03-28T18:16:08.206 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d363b1d6bc] 2024-03-28T18:16:08.206 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d363b071ff] 2024-03-28T18:16:08.206 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fea7c727488] 2024-03-28T18:16:08.206 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 13: (DispatchQueue::entry()+0x51f) [0x7fea7c72460f] 2024-03-28T18:16:08.206 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fea7c7f7e81] 2024-03-28T18:16:08.206 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fea7b2121ca] 2024-03-28T18:16:08.206 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 16: clone() 2024-03-28T18:16:08.206 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2024-03-28T18:16:08.206 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 0> 2024-03-28T18:16:07.830+0000 7fea71e6b700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:16:08.206 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: in thread 7fea71e6b700 thread_name:ms_dispatch 2024-03-28T18:16:08.206 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2024-03-28T18:16:08.206 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:08.206 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fea7b21cd20] 2024-03-28T18:16:08.206 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2: gsignal() 2024-03-28T18:16:08.206 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 3: abort() 2024-03-28T18:16:08.207 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fea7c4954fb] 2024-03-28T18:16:08.207 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fea7c495667] 2024-03-28T18:16:08.207 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d363d4f223] 2024-03-28T18:16:08.207 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d363bc95d6] 2024-03-28T18:16:08.207 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d363bc9da1] 2024-03-28T18:16:08.207 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d363bcc2d5] 2024-03-28T18:16:08.207 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d363c1c233] 2024-03-28T18:16:08.207 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d363c1e197] 2024-03-28T18:16:08.207 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d363b1d11b] 2024-03-28T18:16:08.207 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d363b1d6bc] 2024-03-28T18:16:08.207 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d363b071ff] 2024-03-28T18:16:08.207 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fea7c727488] 2024-03-28T18:16:08.207 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 16: (DispatchQueue::entry()+0x51f) [0x7fea7c72460f] 2024-03-28T18:16:08.207 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fea7c7f7e81] 2024-03-28T18:16:08.207 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fea7b2121ca] 2024-03-28T18:16:08.208 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 19: clone() 2024-03-28T18:16:08.208 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:16:08.208 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71247]: 2024-03-28T18:16:08.208 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:07 smithi170 podman[71286]: 2024-03-28 18:16:07.897015707 +0000 UTC m=+0.019767282 container died 79ade11bcebb55abfddbc639e073d773370985d599378a3221764b653093f84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:16:08.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:07 smithi059 ceph-mon[36190]: pgmap v160: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:16:08.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:07 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/2749107471,v1:172.21.15.170:6835/2749107471] up:rejoin 2024-03-28T18:16:08.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:07 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:rejoin,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:08.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:07 smithi059 ceph-mon[36190]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-28T18:16:08.813 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:08 smithi170 podman[71286]: 2024-03-28 18:16:08.537299654 +0000 UTC m=+0.660051205 container remove 79ade11bcebb55abfddbc639e073d773370985d599378a3221764b653093f84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-28T18:16:08.813 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:08 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:16:08.813 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:08 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-28T18:16:08.837 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:08 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:16:08.837 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:08 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:16:08.837 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:08 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/2749107471,v1:172.21.15.170:6835/2749107471] up:active 2024-03-28T18:16:08.837 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:08 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:08.837 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:08 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T18:16:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:08 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:16:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:08 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:16:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:08 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/2749107471,v1:172.21.15.170:6835/2749107471] up:active 2024-03-28T18:16:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:08 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:08 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T18:16:09.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:08 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:16:09.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:08 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:16:09.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:08 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/2749107471,v1:172.21.15.170:6835/2749107471] up:active 2024-03-28T18:16:09.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:08 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:09.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:08 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T18:16:10.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:09 smithi037 ceph-mon[31460]: pgmap v161: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:16:10.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:09 smithi059 ceph-mon[36190]: pgmap v161: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:16:10.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:09 smithi170 ceph-mon[36280]: pgmap v161: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:16:11.829 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:16:11 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:16:11.408+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:16:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:11 smithi037 ceph-mon[31460]: pgmap v162: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:16:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:11 smithi059 ceph-mon[36190]: pgmap v162: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:16:12.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:11 smithi170 ceph-mon[36280]: pgmap v162: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:16:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:13 smithi037 ceph-mon[31460]: pgmap v163: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:16:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:13 smithi059 ceph-mon[36190]: pgmap v163: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:16:14.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:13 smithi170 ceph-mon[36280]: pgmap v163: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:16:16.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:15 smithi037 ceph-mon[31460]: pgmap v164: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 25 KiB/s wr, 30 op/s 2024-03-28T18:16:16.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:15 smithi059 ceph-mon[36190]: pgmap v164: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 25 KiB/s wr, 30 op/s 2024-03-28T18:16:16.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:15 smithi170 ceph-mon[36280]: pgmap v164: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 25 KiB/s wr, 30 op/s 2024-03-28T18:16:18.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:17 smithi037 ceph-mon[31460]: pgmap v165: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 23 KiB/s wr, 30 op/s 2024-03-28T18:16:18.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:17 smithi059 ceph-mon[36190]: pgmap v165: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 23 KiB/s wr, 30 op/s 2024-03-28T18:16:18.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:17 smithi170 ceph-mon[36280]: pgmap v165: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 23 KiB/s wr, 30 op/s 2024-03-28T18:16:19.062 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:18 smithi170 ceph-mon[36280]: pgmap v166: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-28T18:16:19.062 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:18 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 5. 2024-03-28T18:16:19.063 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:18 smithi170 systemd[1]: Stopped Ceph mds.f for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:16:19.063 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:18 smithi170 systemd[1]: Starting Ceph mds.f for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:16:19.063 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:19 smithi170 podman[71383]: 2024-03-28 18:16:18.91628762 +0000 UTC m=+0.018228135 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:16:19.063 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:19 smithi170 podman[71383]: 2024-03-28 18:16:19.062909376 +0000 UTC m=+0.164849878 container create 7bfaedb0cfb493dd53151e8a7c5c6503fbf40c4c16b3fd315903713e89b2880b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-28T18:16:19.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:18 smithi059 ceph-mon[36190]: pgmap v166: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-28T18:16:19.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:18 smithi037 ceph-mon[31460]: pgmap v166: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-28T18:16:19.785 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:19 smithi170 podman[71383]: 2024-03-28 18:16:19.315715624 +0000 UTC m=+0.417656121 container init 7bfaedb0cfb493dd53151e8a7c5c6503fbf40c4c16b3fd315903713e89b2880b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:16:19.786 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:19 smithi170 podman[71383]: 2024-03-28 18:16:19.318489724 +0000 UTC m=+0.420430221 container start 7bfaedb0cfb493dd53151e8a7c5c6503fbf40c4c16b3fd315903713e89b2880b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-03-28T18:16:19.786 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:19 smithi170 ceph-mds[71401]: -- 172.21.15.170:0/2254078901 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55840e5ffa40 con 0x55840e5d2800 2024-03-28T18:16:19.786 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:19 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: starting mds.f at 2024-03-28T18:16:19.786 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:19 smithi170 bash[71383]: 7bfaedb0cfb493dd53151e8a7c5c6503fbf40c4c16b3fd315903713e89b2880b 2024-03-28T18:16:19.786 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:19 smithi170 systemd[1]: Started Ceph mds.f for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:16:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:19 smithi059 ceph-mon[36190]: daemon mds.f restarted 2024-03-28T18:16:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:19 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:16:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:19 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:16:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:19 smithi059 ceph-mon[36190]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T18:16:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:19 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/1475489176,v1:172.21.15.170:6835/1475489176] up:boot 2024-03-28T18:16:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:19 smithi059 ceph-mon[36190]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-28T18:16:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:19 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-03-28T18:16:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:19 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:20.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:19 smithi170 ceph-mon[36280]: daemon mds.f restarted 2024-03-28T18:16:20.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:19 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:16:20.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:19 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:16:20.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:19 smithi170 ceph-mon[36280]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T18:16:20.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:19 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/1475489176,v1:172.21.15.170:6835/1475489176] up:boot 2024-03-28T18:16:20.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:19 smithi170 ceph-mon[36280]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-28T18:16:20.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:19 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-03-28T18:16:20.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:19 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:20.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:19 smithi037 ceph-mon[31460]: daemon mds.f restarted 2024-03-28T18:16:20.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:19 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:16:20.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:19 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:16:20.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:19 smithi037 ceph-mon[31460]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T18:16:20.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:19 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/1475489176,v1:172.21.15.170:6835/1475489176] up:boot 2024-03-28T18:16:20.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:19 smithi037 ceph-mon[31460]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-28T18:16:20.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:19 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-03-28T18:16:20.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:19 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:20 smithi059 ceph-mon[36190]: pgmap v168: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:16:21.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:20 smithi170 ceph-mon[36280]: pgmap v168: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:16:21.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:20 smithi037 ceph-mon[31460]: pgmap v168: 97 pgs: 97 active+clean; 45 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:16:22.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:21 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3972375740,v1:172.21.15.37:6837/3972375740] up:resolve 2024-03-28T18:16:22.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:21 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:21 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3972375740,v1:172.21.15.37:6837/3972375740] up:resolve 2024-03-28T18:16:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:21 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:22.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:21 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3972375740,v1:172.21.15.37:6837/3972375740] up:resolve 2024-03-28T18:16:22.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:21 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:22 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:16:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:22 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3972375740,v1:172.21.15.37:6837/3972375740] up:reconnect 2024-03-28T18:16:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:22 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:22 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0109999 2024-03-28T18:16:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:22 smithi059 ceph-mon[36190]: pgmap v169: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:16:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:22 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:16:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:22 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3972375740,v1:172.21.15.37:6837/3972375740] up:reconnect 2024-03-28T18:16:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:22 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:22 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0109999 2024-03-28T18:16:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:22 smithi170 ceph-mon[36280]: pgmap v169: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:16:23.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:22 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:16:23.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:22 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3972375740,v1:172.21.15.37:6837/3972375740] up:reconnect 2024-03-28T18:16:23.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:22 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:22 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0109999 2024-03-28T18:16:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:22 smithi037 ceph-mon[31460]: pgmap v169: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:16:23.943 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f41e3000700 time 2024-03-28T18:16:23.713804+0000 2024-03-28T18:16:23.943 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:16:23.943 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:23.943 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f41ee8d24a1] 2024-03-28T18:16:23.943 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f41ee8d2667] 2024-03-28T18:16:23.944 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5645b67b7223] 2024-03-28T18:16:23.944 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5645b66315d6] 2024-03-28T18:16:23.944 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5645b6631da1] 2024-03-28T18:16:23.944 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5645b66342d5] 2024-03-28T18:16:23.944 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5645b6684233] 2024-03-28T18:16:23.944 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5645b6686197] 2024-03-28T18:16:23.944 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5645b658511b] 2024-03-28T18:16:23.944 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5645b65856bc] 2024-03-28T18:16:23.944 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5645b656f1ff] 2024-03-28T18:16:23.944 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f41eeb64488] 2024-03-28T18:16:23.944 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 13: (DispatchQueue::entry()+0x51f) [0x7f41eeb6160f] 2024-03-28T18:16:23.944 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f41eec34e81] 2024-03-28T18:16:23.944 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f41ed64f1ca] 2024-03-28T18:16:23.944 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 16: clone() 2024-03-28T18:16:23.946 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2024-03-28T18:16:23.715+0000 7f41e3000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f41e3000700 time 2024-03-28T18:16:23.713804+0000 2024-03-28T18:16:23.947 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:16:23.947 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2024-03-28T18:16:23.947 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:23.947 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f41ee8d24a1] 2024-03-28T18:16:23.947 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f41ee8d2667] 2024-03-28T18:16:23.947 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5645b67b7223] 2024-03-28T18:16:23.947 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5645b66315d6] 2024-03-28T18:16:23.947 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5645b6631da1] 2024-03-28T18:16:23.947 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5645b66342d5] 2024-03-28T18:16:23.947 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5645b6684233] 2024-03-28T18:16:23.947 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5645b6686197] 2024-03-28T18:16:23.947 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5645b658511b] 2024-03-28T18:16:23.947 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5645b65856bc] 2024-03-28T18:16:23.948 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5645b656f1ff] 2024-03-28T18:16:23.948 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f41eeb64488] 2024-03-28T18:16:23.948 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 13: (DispatchQueue::entry()+0x51f) [0x7f41eeb6160f] 2024-03-28T18:16:23.948 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f41eec34e81] 2024-03-28T18:16:23.948 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f41ed64f1ca] 2024-03-28T18:16:23.948 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 16: clone() 2024-03-28T18:16:23.948 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2024-03-28T18:16:23.948 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: *** Caught signal (Aborted) ** 2024-03-28T18:16:23.948 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: in thread 7f41e3000700 thread_name:ms_dispatch 2024-03-28T18:16:23.948 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:23.948 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f41ed659d20] 2024-03-28T18:16:23.949 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2: gsignal() 2024-03-28T18:16:23.949 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 3: abort() 2024-03-28T18:16:23.949 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f41ee8d24fb] 2024-03-28T18:16:23.949 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f41ee8d2667] 2024-03-28T18:16:23.949 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5645b67b7223] 2024-03-28T18:16:23.949 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5645b66315d6] 2024-03-28T18:16:23.949 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5645b6631da1] 2024-03-28T18:16:23.949 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5645b66342d5] 2024-03-28T18:16:23.949 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5645b6684233] 2024-03-28T18:16:23.949 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5645b6686197] 2024-03-28T18:16:23.949 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5645b658511b] 2024-03-28T18:16:23.949 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5645b65856bc] 2024-03-28T18:16:23.950 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5645b656f1ff] 2024-03-28T18:16:23.950 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f41eeb64488] 2024-03-28T18:16:23.950 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 16: (DispatchQueue::entry()+0x51f) [0x7f41eeb6160f] 2024-03-28T18:16:23.950 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f41eec34e81] 2024-03-28T18:16:23.950 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f41ed64f1ca] 2024-03-28T18:16:23.950 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 19: clone() 2024-03-28T18:16:23.950 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2024-03-28T18:16:23.718+0000 7f41e3000700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:16:23.950 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: in thread 7f41e3000700 thread_name:ms_dispatch 2024-03-28T18:16:23.950 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2024-03-28T18:16:23.950 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:23.950 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f41ed659d20] 2024-03-28T18:16:23.950 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2: gsignal() 2024-03-28T18:16:23.950 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 3: abort() 2024-03-28T18:16:23.950 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f41ee8d24fb] 2024-03-28T18:16:23.951 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f41ee8d2667] 2024-03-28T18:16:23.951 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5645b67b7223] 2024-03-28T18:16:23.951 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5645b66315d6] 2024-03-28T18:16:23.951 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5645b6631da1] 2024-03-28T18:16:23.951 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5645b66342d5] 2024-03-28T18:16:23.951 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5645b6684233] 2024-03-28T18:16:23.951 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5645b6686197] 2024-03-28T18:16:23.952 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5645b658511b] 2024-03-28T18:16:23.952 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5645b65856bc] 2024-03-28T18:16:23.952 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5645b656f1ff] 2024-03-28T18:16:23.952 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f41eeb64488] 2024-03-28T18:16:23.952 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 16: (DispatchQueue::entry()+0x51f) [0x7f41eeb6160f] 2024-03-28T18:16:23.952 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f41eec34e81] 2024-03-28T18:16:23.952 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f41ed64f1ca] 2024-03-28T18:16:23.952 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 19: clone() 2024-03-28T18:16:23.952 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:16:23.952 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2024-03-28T18:16:23.952 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: -1> 2024-03-28T18:16:23.715+0000 7f41e3000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f41e3000700 time 2024-03-28T18:16:23.713804+0000 2024-03-28T18:16:23.952 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:16:23.952 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2024-03-28T18:16:23.953 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:23.953 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f41ee8d24a1] 2024-03-28T18:16:23.953 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f41ee8d2667] 2024-03-28T18:16:23.953 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5645b67b7223] 2024-03-28T18:16:23.953 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5645b66315d6] 2024-03-28T18:16:23.953 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5645b6631da1] 2024-03-28T18:16:23.953 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5645b66342d5] 2024-03-28T18:16:23.953 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5645b6684233] 2024-03-28T18:16:23.953 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5645b6686197] 2024-03-28T18:16:23.953 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5645b658511b] 2024-03-28T18:16:23.953 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5645b65856bc] 2024-03-28T18:16:23.953 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5645b656f1ff] 2024-03-28T18:16:23.953 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f41eeb64488] 2024-03-28T18:16:23.953 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 13: (DispatchQueue::entry()+0x51f) [0x7f41eeb6160f] 2024-03-28T18:16:23.954 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f41eec34e81] 2024-03-28T18:16:23.954 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f41ed64f1ca] 2024-03-28T18:16:23.954 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 16: clone() 2024-03-28T18:16:23.954 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2024-03-28T18:16:23.954 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 0> 2024-03-28T18:16:23.718+0000 7f41e3000700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:16:23.954 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: in thread 7f41e3000700 thread_name:ms_dispatch 2024-03-28T18:16:23.954 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2024-03-28T18:16:23.954 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:23.954 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f41ed659d20] 2024-03-28T18:16:23.954 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2: gsignal() 2024-03-28T18:16:23.954 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 3: abort() 2024-03-28T18:16:23.954 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f41ee8d24fb] 2024-03-28T18:16:23.954 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f41ee8d2667] 2024-03-28T18:16:23.954 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5645b67b7223] 2024-03-28T18:16:23.955 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5645b66315d6] 2024-03-28T18:16:23.955 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5645b6631da1] 2024-03-28T18:16:23.955 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5645b66342d5] 2024-03-28T18:16:23.955 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5645b6684233] 2024-03-28T18:16:23.956 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5645b6686197] 2024-03-28T18:16:23.956 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5645b658511b] 2024-03-28T18:16:23.956 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5645b65856bc] 2024-03-28T18:16:23.956 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5645b656f1ff] 2024-03-28T18:16:23.956 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f41eeb64488] 2024-03-28T18:16:23.956 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 16: (DispatchQueue::entry()+0x51f) [0x7f41eeb6160f] 2024-03-28T18:16:23.956 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f41eec34e81] 2024-03-28T18:16:23.956 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f41ed64f1ca] 2024-03-28T18:16:23.956 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 19: clone() 2024-03-28T18:16:23.956 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:16:23.956 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2024-03-28T18:16:23.956 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: -4> 2024-03-28T18:16:23.715+0000 7f41e3000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f41e3000700 time 2024-03-28T18:16:23.713804+0000 2024-03-28T18:16:23.956 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:16:23.956 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2024-03-28T18:16:23.956 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:23.957 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f41ee8d24a1] 2024-03-28T18:16:23.957 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f41ee8d2667] 2024-03-28T18:16:23.957 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5645b67b7223] 2024-03-28T18:16:23.957 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5645b66315d6] 2024-03-28T18:16:23.957 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5645b6631da1] 2024-03-28T18:16:23.957 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5645b66342d5] 2024-03-28T18:16:23.957 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5645b6684233] 2024-03-28T18:16:23.957 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5645b6686197] 2024-03-28T18:16:23.957 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5645b658511b] 2024-03-28T18:16:23.957 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5645b65856bc] 2024-03-28T18:16:23.957 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5645b656f1ff] 2024-03-28T18:16:23.957 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f41eeb64488] 2024-03-28T18:16:23.957 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 13: (DispatchQueue::entry()+0x51f) [0x7f41eeb6160f] 2024-03-28T18:16:23.957 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f41eec34e81] 2024-03-28T18:16:23.957 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f41ed64f1ca] 2024-03-28T18:16:23.957 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 16: clone() 2024-03-28T18:16:23.958 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2024-03-28T18:16:23.958 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: -3> 2024-03-28T18:16:23.718+0000 7f41e3000700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:16:23.958 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: in thread 7f41e3000700 thread_name:ms_dispatch 2024-03-28T18:16:23.958 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2024-03-28T18:16:23.958 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:23.958 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f41ed659d20] 2024-03-28T18:16:23.958 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2: gsignal() 2024-03-28T18:16:23.958 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 3: abort() 2024-03-28T18:16:23.958 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f41ee8d24fb] 2024-03-28T18:16:23.958 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f41ee8d2667] 2024-03-28T18:16:23.958 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5645b67b7223] 2024-03-28T18:16:23.958 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5645b66315d6] 2024-03-28T18:16:23.959 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5645b6631da1] 2024-03-28T18:16:23.959 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5645b66342d5] 2024-03-28T18:16:23.959 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5645b6684233] 2024-03-28T18:16:23.959 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5645b6686197] 2024-03-28T18:16:23.959 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5645b658511b] 2024-03-28T18:16:23.959 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5645b65856bc] 2024-03-28T18:16:23.959 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5645b656f1ff] 2024-03-28T18:16:23.959 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f41eeb64488] 2024-03-28T18:16:23.959 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 16: (DispatchQueue::entry()+0x51f) [0x7f41eeb6160f] 2024-03-28T18:16:23.959 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f41eec34e81] 2024-03-28T18:16:23.959 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f41ed64f1ca] 2024-03-28T18:16:23.959 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 19: clone() 2024-03-28T18:16:23.959 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:16:23.959 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[70078]: 2024-03-28T18:16:23.959 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:23 smithi037 podman[87305]: 2024-03-28 18:16:23.866643903 +0000 UTC m=+0.035374814 container died 4b0671f3d1a0534f9ddce8a0b049c2139432de516dc34edaa49e43157365f884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5) 2024-03-28T18:16:24.263 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3972375740,v1:172.21.15.37:6837/3972375740] up:rejoin 2024-03-28T18:16:24.263 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:24.263 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-mon[31460]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-28T18:16:24.263 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:16:24.263 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:16:24.263 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3972375740,v1:172.21.15.37:6837/3972375740] up:active 2024-03-28T18:16:24.263 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:23 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T18:16:24.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:23 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3972375740,v1:172.21.15.37:6837/3972375740] up:rejoin 2024-03-28T18:16:24.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:23 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:24.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:23 smithi059 ceph-mon[36190]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-28T18:16:24.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:23 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:16:24.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:23 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:16:24.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:23 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3972375740,v1:172.21.15.37:6837/3972375740] up:active 2024-03-28T18:16:24.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:23 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T18:16:24.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:23 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3972375740,v1:172.21.15.37:6837/3972375740] up:rejoin 2024-03-28T18:16:24.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:23 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:24.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:23 smithi170 ceph-mon[36280]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-28T18:16:24.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:23 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:16:24.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:23 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:16:24.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:23 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3972375740,v1:172.21.15.37:6837/3972375740] up:active 2024-03-28T18:16:24.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:23 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T18:16:25.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:24 smithi037 ceph-mon[31460]: pgmap v170: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:16:25.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:24 smithi037 podman[87305]: 2024-03-28 18:16:24.743780959 +0000 UTC m=+0.912511916 container remove 4b0671f3d1a0534f9ddce8a0b049c2139432de516dc34edaa49e43157365f884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-28T18:16:25.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:24 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:16:25.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:24 smithi059 ceph-mon[36190]: pgmap v170: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:16:25.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:24 smithi170 ceph-mon[36280]: pgmap v170: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:16:25.469 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:25 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-28T18:16:25.469 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:25 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Consumed 2.088s CPU time. 2024-03-28T18:16:26.624 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:16:26 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:16:26.411+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:16:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:26 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T18:16:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:26 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:16:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:26 smithi059 ceph-mon[36190]: pgmap v171: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:16:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:26 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T18:16:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:26 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:16:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:26 smithi170 ceph-mon[36280]: pgmap v171: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:16:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:26 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T18:16:27.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:26 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:16:27.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:26 smithi037 ceph-mon[31460]: pgmap v171: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:16:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:27 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:16:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:27 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:16:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:27 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:16:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:27 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:16:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:27 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:16:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:27 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:16:28.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:27 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:16:28.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:27 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:16:28.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:27 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:16:29.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:28 smithi059 ceph-mon[36190]: pgmap v172: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:16:29.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:28 smithi170 ceph-mon[36280]: pgmap v172: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:16:29.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:28 smithi037 ceph-mon[31460]: pgmap v172: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:16:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:31 smithi037 ceph-mon[31460]: pgmap v173: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 183 B/s wr, 14 op/s 2024-03-28T18:16:31.988 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:31 smithi059 ceph-mon[36190]: pgmap v173: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 183 B/s wr, 14 op/s 2024-03-28T18:16:31.988 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:31 smithi170 ceph-mon[36280]: pgmap v173: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 183 B/s wr, 14 op/s 2024-03-28T18:16:33.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:33 smithi037 ceph-mon[31460]: pgmap v174: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:16:34.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:33 smithi059 ceph-mon[36190]: pgmap v174: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:16:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:33 smithi170 ceph-mon[36280]: pgmap v174: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:16:35.755 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:35 smithi037 ceph-mon[31460]: pgmap v175: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:16:35.756 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:35 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 1. 2024-03-28T18:16:35.756 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:35 smithi037 systemd[1]: Stopped Ceph mds.d for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:16:35.756 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:35 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Consumed 2.088s CPU time. 2024-03-28T18:16:35.756 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:35 smithi037 systemd[1]: Starting Ceph mds.d for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:16:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:35 smithi059 ceph-mon[36190]: pgmap v175: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:16:36.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:35 smithi170 ceph-mon[36280]: pgmap v175: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:16:36.065 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:35 smithi037 podman[87529]: 2024-03-28 18:16:35.661227141 +0000 UTC m=+0.032035231 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:16:36.065 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:35 smithi037 podman[87529]: 2024-03-28 18:16:35.803636126 +0000 UTC m=+0.174444261 container create 58605365ba926f64597e2a0285ac0255d4d18f054d97db091450f381b8437fce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-28T18:16:36.375 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:36 smithi037 podman[87529]: 2024-03-28 18:16:36.065152986 +0000 UTC m=+0.435961093 container init 58605365ba926f64597e2a0285ac0255d4d18f054d97db091450f381b8437fce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-28T18:16:36.375 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:36 smithi037 podman[87529]: 2024-03-28 18:16:36.074934128 +0000 UTC m=+0.445742214 container start 58605365ba926f64597e2a0285ac0255d4d18f054d97db091450f381b8437fce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:16:36.375 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:36 smithi037 ceph-mds[87547]: -- 172.21.15.37:0/4158401031 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5640fa73fa40 con 0x5640fa712c00 2024-03-28T18:16:36.375 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:36 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: starting mds.d at 2024-03-28T18:16:36.375 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:36 smithi037 bash[87529]: 58605365ba926f64597e2a0285ac0255d4d18f054d97db091450f381b8437fce 2024-03-28T18:16:36.375 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:16:36 smithi037 systemd[1]: Started Ceph mds.d for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:16:36.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:36 smithi037 ceph-mon[31460]: daemon mds.d restarted 2024-03-28T18:16:36.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:36 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:16:36.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:36 smithi037 ceph-mon[31460]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T18:16:36.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:36 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:16:36.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:36 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3340788991,v1:172.21.15.37:6837/3340788991] up:boot 2024-03-28T18:16:36.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:36 smithi037 ceph-mon[31460]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-28T18:16:36.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:36 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-03-28T18:16:36.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:36 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:replay,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:37.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:36 smithi059 ceph-mon[36190]: daemon mds.d restarted 2024-03-28T18:16:37.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:36 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:16:37.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:36 smithi059 ceph-mon[36190]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T18:16:37.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:36 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:16:37.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:36 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3340788991,v1:172.21.15.37:6837/3340788991] up:boot 2024-03-28T18:16:37.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:36 smithi059 ceph-mon[36190]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-28T18:16:37.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:36 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-03-28T18:16:37.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:36 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:replay,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:37.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:36 smithi170 ceph-mon[36280]: daemon mds.d restarted 2024-03-28T18:16:37.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:36 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:16:37.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:36 smithi170 ceph-mon[36280]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T18:16:37.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:36 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:16:37.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:36 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3340788991,v1:172.21.15.37:6837/3340788991] up:boot 2024-03-28T18:16:37.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:36 smithi170 ceph-mon[36280]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-28T18:16:37.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:36 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-03-28T18:16:37.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:36 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:replay,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:37.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:37 smithi037 ceph-mon[31460]: pgmap v177: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:16:38.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:37 smithi059 ceph-mon[36190]: pgmap v177: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:16:38.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:37 smithi170 ceph-mon[36280]: pgmap v177: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:16:38.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:38 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/1475489176,v1:172.21.15.170:6835/1475489176] up:resolve 2024-03-28T18:16:38.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:38 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:resolve,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:39.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:38 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/1475489176,v1:172.21.15.170:6835/1475489176] up:resolve 2024-03-28T18:16:39.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:38 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:resolve,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:39.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:38 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/1475489176,v1:172.21.15.170:6835/1475489176] up:resolve 2024-03-28T18:16:39.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:38 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:resolve,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:39.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:39 smithi037 ceph-mon[31460]: pgmap v178: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:16:39.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:39 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/1475489176,v1:172.21.15.170:6835/1475489176] up:reconnect 2024-03-28T18:16:39.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:39 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:reconnect,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:39.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:39 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089998 2024-03-28T18:16:39.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:39 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00999978 2024-03-28T18:16:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:39 smithi059 ceph-mon[36190]: pgmap v178: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:16:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:39 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/1475489176,v1:172.21.15.170:6835/1475489176] up:reconnect 2024-03-28T18:16:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:39 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:reconnect,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:39 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089998 2024-03-28T18:16:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:39 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00999978 2024-03-28T18:16:40.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:39 smithi170 ceph-mon[36280]: pgmap v178: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:16:40.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:39 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/1475489176,v1:172.21.15.170:6835/1475489176] up:reconnect 2024-03-28T18:16:40.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:39 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:reconnect,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:40.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:39 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089998 2024-03-28T18:16:40.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:39 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00999978 2024-03-28T18:16:40.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:40 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/1475489176,v1:172.21.15.170:6835/1475489176] up:rejoin 2024-03-28T18:16:40.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:40 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:rejoin,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:40.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:40 smithi037 ceph-mon[31460]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-28T18:16:41.010 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/1475489176,v1:172.21.15.170:6835/1475489176] up:rejoin 2024-03-28T18:16:41.010 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:rejoin,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:41.010 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-mon[36280]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-28T18:16:41.011 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2e8f3cd700 time 2024-03-28T18:16:40.645259+0000 2024-03-28T18:16:41.011 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:16:41.011 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:41.011 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2e999f74a1] 2024-03-28T18:16:41.011 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2e999f7667] 2024-03-28T18:16:41.011 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55840bf52223] 2024-03-28T18:16:41.011 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55840bdcc5d6] 2024-03-28T18:16:41.011 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55840bdccda1] 2024-03-28T18:16:41.011 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55840bdcf2d5] 2024-03-28T18:16:41.011 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55840be1f233] 2024-03-28T18:16:41.011 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55840be21197] 2024-03-28T18:16:41.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55840bd2011b] 2024-03-28T18:16:41.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55840bd206bc] 2024-03-28T18:16:41.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55840bd0a1ff] 2024-03-28T18:16:41.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2e99c89488] 2024-03-28T18:16:41.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 13: (DispatchQueue::entry()+0x51f) [0x7f2e99c8660f] 2024-03-28T18:16:41.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2e99d59e81] 2024-03-28T18:16:41.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2e987741ca] 2024-03-28T18:16:41.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 16: clone() 2024-03-28T18:16:41.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2024-03-28T18:16:40.647+0000 7f2e8f3cd700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2e8f3cd700 time 2024-03-28T18:16:40.645259+0000 2024-03-28T18:16:41.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:16:41.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2024-03-28T18:16:41.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:41.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2e999f74a1] 2024-03-28T18:16:41.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2e999f7667] 2024-03-28T18:16:41.012 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55840bf52223] 2024-03-28T18:16:41.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55840bdcc5d6] 2024-03-28T18:16:41.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55840bdccda1] 2024-03-28T18:16:41.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55840bdcf2d5] 2024-03-28T18:16:41.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55840be1f233] 2024-03-28T18:16:41.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55840be21197] 2024-03-28T18:16:41.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55840bd2011b] 2024-03-28T18:16:41.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55840bd206bc] 2024-03-28T18:16:41.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55840bd0a1ff] 2024-03-28T18:16:41.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2e99c89488] 2024-03-28T18:16:41.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 13: (DispatchQueue::entry()+0x51f) [0x7f2e99c8660f] 2024-03-28T18:16:41.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2e99d59e81] 2024-03-28T18:16:41.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2e987741ca] 2024-03-28T18:16:41.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 16: clone() 2024-03-28T18:16:41.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2024-03-28T18:16:41.013 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: *** Caught signal (Aborted) ** 2024-03-28T18:16:41.014 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: in thread 7f2e8f3cd700 thread_name:ms_dispatch 2024-03-28T18:16:41.014 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:41.014 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2e9877ed20] 2024-03-28T18:16:41.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2: gsignal() 2024-03-28T18:16:41.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 3: abort() 2024-03-28T18:16:41.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2e999f74fb] 2024-03-28T18:16:41.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2e999f7667] 2024-03-28T18:16:41.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55840bf52223] 2024-03-28T18:16:41.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55840bdcc5d6] 2024-03-28T18:16:41.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55840bdccda1] 2024-03-28T18:16:41.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55840bdcf2d5] 2024-03-28T18:16:41.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55840be1f233] 2024-03-28T18:16:41.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55840be21197] 2024-03-28T18:16:41.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55840bd2011b] 2024-03-28T18:16:41.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55840bd206bc] 2024-03-28T18:16:41.015 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55840bd0a1ff] 2024-03-28T18:16:41.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2e99c89488] 2024-03-28T18:16:41.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 16: (DispatchQueue::entry()+0x51f) [0x7f2e99c8660f] 2024-03-28T18:16:41.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2e99d59e81] 2024-03-28T18:16:41.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2e987741ca] 2024-03-28T18:16:41.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 19: clone() 2024-03-28T18:16:41.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2024-03-28T18:16:40.649+0000 7f2e8f3cd700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:16:41.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: in thread 7f2e8f3cd700 thread_name:ms_dispatch 2024-03-28T18:16:41.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2024-03-28T18:16:41.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:41.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2e9877ed20] 2024-03-28T18:16:41.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2: gsignal() 2024-03-28T18:16:41.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 3: abort() 2024-03-28T18:16:41.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2e999f74fb] 2024-03-28T18:16:41.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2e999f7667] 2024-03-28T18:16:41.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55840bf52223] 2024-03-28T18:16:41.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55840bdcc5d6] 2024-03-28T18:16:41.016 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55840bdccda1] 2024-03-28T18:16:41.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55840bdcf2d5] 2024-03-28T18:16:41.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55840be1f233] 2024-03-28T18:16:41.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55840be21197] 2024-03-28T18:16:41.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55840bd2011b] 2024-03-28T18:16:41.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55840bd206bc] 2024-03-28T18:16:41.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55840bd0a1ff] 2024-03-28T18:16:41.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2e99c89488] 2024-03-28T18:16:41.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 16: (DispatchQueue::entry()+0x51f) [0x7f2e99c8660f] 2024-03-28T18:16:41.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2e99d59e81] 2024-03-28T18:16:41.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2e987741ca] 2024-03-28T18:16:41.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 19: clone() 2024-03-28T18:16:41.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:16:41.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2024-03-28T18:16:41.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: -1> 2024-03-28T18:16:40.647+0000 7f2e8f3cd700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2e8f3cd700 time 2024-03-28T18:16:40.645259+0000 2024-03-28T18:16:41.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:16:41.017 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2024-03-28T18:16:41.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:41.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2e999f74a1] 2024-03-28T18:16:41.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2e999f7667] 2024-03-28T18:16:41.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55840bf52223] 2024-03-28T18:16:41.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55840bdcc5d6] 2024-03-28T18:16:41.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55840bdccda1] 2024-03-28T18:16:41.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55840bdcf2d5] 2024-03-28T18:16:41.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55840be1f233] 2024-03-28T18:16:41.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55840be21197] 2024-03-28T18:16:41.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55840bd2011b] 2024-03-28T18:16:41.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55840bd206bc] 2024-03-28T18:16:41.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55840bd0a1ff] 2024-03-28T18:16:41.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2e99c89488] 2024-03-28T18:16:41.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 13: (DispatchQueue::entry()+0x51f) [0x7f2e99c8660f] 2024-03-28T18:16:41.018 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2e99d59e81] 2024-03-28T18:16:41.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2e987741ca] 2024-03-28T18:16:41.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 16: clone() 2024-03-28T18:16:41.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2024-03-28T18:16:41.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 0> 2024-03-28T18:16:40.649+0000 7f2e8f3cd700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:16:41.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: in thread 7f2e8f3cd700 thread_name:ms_dispatch 2024-03-28T18:16:41.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2024-03-28T18:16:41.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:41.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2e9877ed20] 2024-03-28T18:16:41.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2: gsignal() 2024-03-28T18:16:41.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 3: abort() 2024-03-28T18:16:41.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2e999f74fb] 2024-03-28T18:16:41.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2e999f7667] 2024-03-28T18:16:41.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55840bf52223] 2024-03-28T18:16:41.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55840bdcc5d6] 2024-03-28T18:16:41.019 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55840bdccda1] 2024-03-28T18:16:41.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55840bdcf2d5] 2024-03-28T18:16:41.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55840be1f233] 2024-03-28T18:16:41.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55840be21197] 2024-03-28T18:16:41.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55840bd2011b] 2024-03-28T18:16:41.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55840bd206bc] 2024-03-28T18:16:41.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55840bd0a1ff] 2024-03-28T18:16:41.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2e99c89488] 2024-03-28T18:16:41.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 16: (DispatchQueue::entry()+0x51f) [0x7f2e99c8660f] 2024-03-28T18:16:41.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2e99d59e81] 2024-03-28T18:16:41.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2e987741ca] 2024-03-28T18:16:41.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 19: clone() 2024-03-28T18:16:41.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:16:41.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2024-03-28T18:16:41.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: -1> 2024-03-28T18:16:40.647+0000 7f2e8f3cd700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2e8f3cd700 time 2024-03-28T18:16:40.645259+0000 2024-03-28T18:16:41.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:16:41.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2024-03-28T18:16:41.020 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:41.021 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2e999f74a1] 2024-03-28T18:16:41.021 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2e999f7667] 2024-03-28T18:16:41.021 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55840bf52223] 2024-03-28T18:16:41.021 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55840bdcc5d6] 2024-03-28T18:16:41.021 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55840bdccda1] 2024-03-28T18:16:41.021 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55840bdcf2d5] 2024-03-28T18:16:41.021 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55840be1f233] 2024-03-28T18:16:41.021 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55840be21197] 2024-03-28T18:16:41.021 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55840bd2011b] 2024-03-28T18:16:41.021 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55840bd206bc] 2024-03-28T18:16:41.021 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55840bd0a1ff] 2024-03-28T18:16:41.021 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2e99c89488] 2024-03-28T18:16:41.021 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 13: (DispatchQueue::entry()+0x51f) [0x7f2e99c8660f] 2024-03-28T18:16:41.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2e99d59e81] 2024-03-28T18:16:41.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2e987741ca] 2024-03-28T18:16:41.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 16: clone() 2024-03-28T18:16:41.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2024-03-28T18:16:41.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 0> 2024-03-28T18:16:40.649+0000 7f2e8f3cd700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:16:41.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: in thread 7f2e8f3cd700 thread_name:ms_dispatch 2024-03-28T18:16:41.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2024-03-28T18:16:41.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:16:41.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2e9877ed20] 2024-03-28T18:16:41.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2: gsignal() 2024-03-28T18:16:41.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 3: abort() 2024-03-28T18:16:41.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2e999f74fb] 2024-03-28T18:16:41.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2e999f7667] 2024-03-28T18:16:41.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55840bf52223] 2024-03-28T18:16:41.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55840bdcc5d6] 2024-03-28T18:16:41.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55840bdccda1] 2024-03-28T18:16:41.022 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55840bdcf2d5] 2024-03-28T18:16:41.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55840be1f233] 2024-03-28T18:16:41.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55840be21197] 2024-03-28T18:16:41.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55840bd2011b] 2024-03-28T18:16:41.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55840bd206bc] 2024-03-28T18:16:41.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55840bd0a1ff] 2024-03-28T18:16:41.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2e99c89488] 2024-03-28T18:16:41.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 16: (DispatchQueue::entry()+0x51f) [0x7f2e99c8660f] 2024-03-28T18:16:41.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2e99d59e81] 2024-03-28T18:16:41.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2e987741ca] 2024-03-28T18:16:41.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 19: clone() 2024-03-28T18:16:41.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:16:41.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f[71397]: 2024-03-28T18:16:41.023 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:40 smithi170 podman[71564]: 2024-03-28 18:16:40.71082569 +0000 UTC m=+0.022753881 container died 7bfaedb0cfb493dd53151e8a7c5c6503fbf40c4c16b3fd315903713e89b2880b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-28T18:16:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:40 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/1475489176,v1:172.21.15.170:6835/1475489176] up:rejoin 2024-03-28T18:16:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:40 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=f=up:rejoin,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:40 smithi059 ceph-mon[36190]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-28T18:16:41.602 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:41 smithi170 podman[71564]: 2024-03-28 18:16:41.345732476 +0000 UTC m=+0.657660653 container remove 7bfaedb0cfb493dd53151e8a7c5c6503fbf40c4c16b3fd315903713e89b2880b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-28T18:16:41.602 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:41 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:16:41.602 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:41 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-28T18:16:41.855 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:41 smithi037 ceph-mon[31460]: pgmap v179: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:16:41.855 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:41 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:16:41.855 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:41 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:16:41.855 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:41 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6834/1475489176,v1:172.21.15.170:6835/1475489176] up:active 2024-03-28T18:16:41.855 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:41 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:41.855 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:41 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T18:16:41.856 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:16:41 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:16:41.412+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:16:42.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:41 smithi059 ceph-mon[36190]: pgmap v179: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:16:42.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:41 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:16:42.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:41 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:16:42.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:41 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6834/1475489176,v1:172.21.15.170:6835/1475489176] up:active 2024-03-28T18:16:42.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:41 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:42.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:41 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T18:16:42.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:41 smithi170 ceph-mon[36280]: pgmap v179: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:16:42.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:41 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:16:42.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:41 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:16:42.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:41 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6834/1475489176,v1:172.21.15.170:6835/1475489176] up:active 2024-03-28T18:16:42.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:41 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T18:16:42.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:41 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=f=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T18:16:43.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:43 smithi037 ceph-mon[31460]: pgmap v180: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:16:44.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:43 smithi059 ceph-mon[36190]: pgmap v180: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:16:44.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:43 smithi170 ceph-mon[36280]: pgmap v180: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:16:45.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:45 smithi037 ceph-mon[31460]: pgmap v181: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:16:46.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:45 smithi059 ceph-mon[36190]: pgmap v181: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:16:46.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:45 smithi170 ceph-mon[36280]: pgmap v181: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:16:47.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:47 smithi037 ceph-mon[31460]: pgmap v182: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 197 B/s wr, 18 op/s 2024-03-28T18:16:48.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:47 smithi059 ceph-mon[36190]: pgmap v182: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 197 B/s wr, 18 op/s 2024-03-28T18:16:48.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:47 smithi170 ceph-mon[36280]: pgmap v182: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 197 B/s wr, 18 op/s 2024-03-28T18:16:49.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:49 smithi037 ceph-mon[31460]: pgmap v183: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:16:50.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:49 smithi059 ceph-mon[36190]: pgmap v183: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:16:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:49 smithi170 ceph-mon[36280]: pgmap v183: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:16:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:51 smithi059 ceph-mon[36190]: pgmap v184: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:16:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:51 smithi170 ceph-mon[36280]: pgmap v184: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:16:52.035 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:51 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 6. 2024-03-28T18:16:52.035 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:51 smithi170 systemd[1]: Stopped Ceph mds.f for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:16:52.035 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:51 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Start request repeated too quickly. 2024-03-28T18:16:52.035 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:51 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-28T18:16:52.036 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 28 18:16:51 smithi170 systemd[1]: Failed to start Ceph mds.f for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:16:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:51 smithi037 ceph-mon[31460]: pgmap v184: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:16:54.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:53 smithi059 ceph-mon[36190]: pgmap v185: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:16:54.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:53 smithi170 ceph-mon[36280]: pgmap v185: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:16:54.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:53 smithi037 ceph-mon[31460]: pgmap v185: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:16:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:55 smithi059 ceph-mon[36190]: pgmap v186: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:56.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:55 smithi170 ceph-mon[36280]: pgmap v186: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:56.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:55 smithi037 ceph-mon[31460]: pgmap v186: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:57 smithi059 ceph-mon[36190]: pgmap v187: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:16:58.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:57 smithi170 ceph-mon[36280]: pgmap v187: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:16:58.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:57 smithi037 ceph-mon[31460]: pgmap v187: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:17:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:59 smithi059 ceph-mon[36190]: pgmap v188: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:59 smithi059 ceph-mon[36190]: Replacing daemon mds.f as rank 1 with standby daemon mds.d 2024-03-28T18:17:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:59 smithi059 ceph-mon[36190]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-03-28T18:17:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:59 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:17:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:59 smithi059 ceph-mon[36190]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T18:17:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:16:59 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:00.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:59 smithi170 ceph-mon[36280]: pgmap v188: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:00.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:59 smithi170 ceph-mon[36280]: Replacing daemon mds.f as rank 1 with standby daemon mds.d 2024-03-28T18:17:00.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:59 smithi170 ceph-mon[36280]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-03-28T18:17:00.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:59 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:17:00.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:59 smithi170 ceph-mon[36280]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T18:17:00.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:16:59 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:00.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:59 smithi037 ceph-mon[31460]: pgmap v188: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:00.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:59 smithi037 ceph-mon[31460]: Replacing daemon mds.f as rank 1 with standby daemon mds.d 2024-03-28T18:17:00.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:59 smithi037 ceph-mon[31460]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-03-28T18:17:00.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:59 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:17:00.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:59 smithi037 ceph-mon[31460]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T18:17:00.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:16:59 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:00 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3340788991,v1:172.21.15.37:6837/3340788991] up:resolve 2024-03-28T18:17:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:00 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:01.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:00 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3340788991,v1:172.21.15.37:6837/3340788991] up:resolve 2024-03-28T18:17:01.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:00 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:01.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:00 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3340788991,v1:172.21.15.37:6837/3340788991] up:resolve 2024-03-28T18:17:01.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:00 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:01 smithi059 ceph-mon[36190]: pgmap v190: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:17:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:01 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:17:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:01 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3340788991,v1:172.21.15.37:6837/3340788991] up:reconnect 2024-03-28T18:17:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:01 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:01 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0129998 2024-03-28T18:17:02.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:01 smithi170 ceph-mon[36280]: pgmap v190: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:17:02.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:01 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:17:02.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:01 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3340788991,v1:172.21.15.37:6837/3340788991] up:reconnect 2024-03-28T18:17:02.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:01 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:02.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:01 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0129998 2024-03-28T18:17:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:01 smithi037 ceph-mon[31460]: pgmap v190: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:17:02.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:01 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:17:02.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:01 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3340788991,v1:172.21.15.37:6837/3340788991] up:reconnect 2024-03-28T18:17:02.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:01 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:02.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:01 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0129998 2024-03-28T18:17:03.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:02 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3340788991,v1:172.21.15.37:6837/3340788991] up:rejoin 2024-03-28T18:17:03.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:02 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:03.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:02 smithi059 ceph-mon[36190]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-28T18:17:03.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:02 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3340788991,v1:172.21.15.37:6837/3340788991] up:rejoin 2024-03-28T18:17:03.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:02 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:03.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:02 smithi170 ceph-mon[36280]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-28T18:17:03.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3340788991,v1:172.21.15.37:6837/3340788991] up:rejoin 2024-03-28T18:17:03.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:03.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-mon[31460]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-28T18:17:03.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff600c00700 time 2024-03-28T18:17:02.778044+0000 2024-03-28T18:17:03.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:17:03.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:03.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff60c42b4a1] 2024-03-28T18:17:03.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff60c42b667] 2024-03-28T18:17:03.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5640f7eab223] 2024-03-28T18:17:03.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5640f7d255d6] 2024-03-28T18:17:03.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5640f7d25da1] 2024-03-28T18:17:03.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5640f7d282d5] 2024-03-28T18:17:03.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5640f7d78233] 2024-03-28T18:17:03.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5640f7d7a197] 2024-03-28T18:17:03.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5640f7c7911b] 2024-03-28T18:17:03.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5640f7c796bc] 2024-03-28T18:17:03.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5640f7c631ff] 2024-03-28T18:17:03.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff60c6bd488] 2024-03-28T18:17:03.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 13: (DispatchQueue::entry()+0x51f) [0x7ff60c6ba60f] 2024-03-28T18:17:03.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff60c78de81] 2024-03-28T18:17:03.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff60b1a81ca] 2024-03-28T18:17:03.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 16: clone() 2024-03-28T18:17:03.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2024-03-28T18:17:02.780+0000 7ff600c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff600c00700 time 2024-03-28T18:17:02.778044+0000 2024-03-28T18:17:03.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:17:03.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2024-03-28T18:17:03.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:03.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff60c42b4a1] 2024-03-28T18:17:03.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff60c42b667] 2024-03-28T18:17:03.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5640f7eab223] 2024-03-28T18:17:03.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5640f7d255d6] 2024-03-28T18:17:03.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5640f7d25da1] 2024-03-28T18:17:03.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5640f7d282d5] 2024-03-28T18:17:03.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5640f7d78233] 2024-03-28T18:17:03.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5640f7d7a197] 2024-03-28T18:17:03.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5640f7c7911b] 2024-03-28T18:17:03.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5640f7c796bc] 2024-03-28T18:17:03.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5640f7c631ff] 2024-03-28T18:17:03.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff60c6bd488] 2024-03-28T18:17:03.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 13: (DispatchQueue::entry()+0x51f) [0x7ff60c6ba60f] 2024-03-28T18:17:03.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff60c78de81] 2024-03-28T18:17:03.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff60b1a81ca] 2024-03-28T18:17:03.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 16: clone() 2024-03-28T18:17:03.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2024-03-28T18:17:03.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: *** Caught signal (Aborted) ** 2024-03-28T18:17:03.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: in thread 7ff600c00700 thread_name:ms_dispatch 2024-03-28T18:17:03.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:03.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff60b1b2d20] 2024-03-28T18:17:03.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2: gsignal() 2024-03-28T18:17:03.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 3: abort() 2024-03-28T18:17:03.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff60c42b4fb] 2024-03-28T18:17:03.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff60c42b667] 2024-03-28T18:17:03.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5640f7eab223] 2024-03-28T18:17:03.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5640f7d255d6] 2024-03-28T18:17:03.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5640f7d25da1] 2024-03-28T18:17:03.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5640f7d282d5] 2024-03-28T18:17:03.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5640f7d78233] 2024-03-28T18:17:03.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5640f7d7a197] 2024-03-28T18:17:03.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5640f7c7911b] 2024-03-28T18:17:03.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5640f7c796bc] 2024-03-28T18:17:03.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5640f7c631ff] 2024-03-28T18:17:03.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff60c6bd488] 2024-03-28T18:17:03.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 16: (DispatchQueue::entry()+0x51f) [0x7ff60c6ba60f] 2024-03-28T18:17:03.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff60c78de81] 2024-03-28T18:17:03.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff60b1a81ca] 2024-03-28T18:17:03.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 19: clone() 2024-03-28T18:17:03.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2024-03-28T18:17:02.784+0000 7ff600c00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:17:03.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: in thread 7ff600c00700 thread_name:ms_dispatch 2024-03-28T18:17:03.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2024-03-28T18:17:03.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:03.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff60b1b2d20] 2024-03-28T18:17:03.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2: gsignal() 2024-03-28T18:17:03.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 3: abort() 2024-03-28T18:17:03.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff60c42b4fb] 2024-03-28T18:17:03.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff60c42b667] 2024-03-28T18:17:03.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5640f7eab223] 2024-03-28T18:17:03.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5640f7d255d6] 2024-03-28T18:17:03.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5640f7d25da1] 2024-03-28T18:17:03.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5640f7d282d5] 2024-03-28T18:17:03.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5640f7d78233] 2024-03-28T18:17:03.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5640f7d7a197] 2024-03-28T18:17:03.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5640f7c7911b] 2024-03-28T18:17:03.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5640f7c796bc] 2024-03-28T18:17:03.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5640f7c631ff] 2024-03-28T18:17:03.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff60c6bd488] 2024-03-28T18:17:03.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 16: (DispatchQueue::entry()+0x51f) [0x7ff60c6ba60f] 2024-03-28T18:17:03.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff60c78de81] 2024-03-28T18:17:03.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff60b1a81ca] 2024-03-28T18:17:03.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 19: clone() 2024-03-28T18:17:03.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:17:03.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2024-03-28T18:17:03.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: -1> 2024-03-28T18:17:02.780+0000 7ff600c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff600c00700 time 2024-03-28T18:17:02.778044+0000 2024-03-28T18:17:03.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:17:03.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2024-03-28T18:17:03.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:03.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff60c42b4a1] 2024-03-28T18:17:03.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff60c42b667] 2024-03-28T18:17:03.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5640f7eab223] 2024-03-28T18:17:03.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5640f7d255d6] 2024-03-28T18:17:03.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5640f7d25da1] 2024-03-28T18:17:03.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5640f7d282d5] 2024-03-28T18:17:03.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5640f7d78233] 2024-03-28T18:17:03.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5640f7d7a197] 2024-03-28T18:17:03.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5640f7c7911b] 2024-03-28T18:17:03.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5640f7c796bc] 2024-03-28T18:17:03.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5640f7c631ff] 2024-03-28T18:17:03.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff60c6bd488] 2024-03-28T18:17:03.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 13: (DispatchQueue::entry()+0x51f) [0x7ff60c6ba60f] 2024-03-28T18:17:03.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff60c78de81] 2024-03-28T18:17:03.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff60b1a81ca] 2024-03-28T18:17:03.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 16: clone() 2024-03-28T18:17:03.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2024-03-28T18:17:03.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 0> 2024-03-28T18:17:02.784+0000 7ff600c00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:17:03.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: in thread 7ff600c00700 thread_name:ms_dispatch 2024-03-28T18:17:03.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2024-03-28T18:17:03.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:03.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff60b1b2d20] 2024-03-28T18:17:03.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2: gsignal() 2024-03-28T18:17:03.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 3: abort() 2024-03-28T18:17:03.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff60c42b4fb] 2024-03-28T18:17:03.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff60c42b667] 2024-03-28T18:17:03.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5640f7eab223] 2024-03-28T18:17:03.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5640f7d255d6] 2024-03-28T18:17:03.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5640f7d25da1] 2024-03-28T18:17:03.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5640f7d282d5] 2024-03-28T18:17:03.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5640f7d78233] 2024-03-28T18:17:03.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5640f7d7a197] 2024-03-28T18:17:03.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5640f7c7911b] 2024-03-28T18:17:03.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5640f7c796bc] 2024-03-28T18:17:03.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5640f7c631ff] 2024-03-28T18:17:03.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff60c6bd488] 2024-03-28T18:17:03.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 16: (DispatchQueue::entry()+0x51f) [0x7ff60c6ba60f] 2024-03-28T18:17:03.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff60c78de81] 2024-03-28T18:17:03.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff60b1a81ca] 2024-03-28T18:17:03.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 19: clone() 2024-03-28T18:17:03.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:17:03.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2024-03-28T18:17:03.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: -1> 2024-03-28T18:17:02.780+0000 7ff600c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff600c00700 time 2024-03-28T18:17:02.778044+0000 2024-03-28T18:17:03.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:17:03.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2024-03-28T18:17:03.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:03.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff60c42b4a1] 2024-03-28T18:17:03.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff60c42b667] 2024-03-28T18:17:03.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5640f7eab223] 2024-03-28T18:17:03.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5640f7d255d6] 2024-03-28T18:17:03.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5640f7d25da1] 2024-03-28T18:17:03.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5640f7d282d5] 2024-03-28T18:17:03.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5640f7d78233] 2024-03-28T18:17:03.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5640f7d7a197] 2024-03-28T18:17:03.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5640f7c7911b] 2024-03-28T18:17:03.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5640f7c796bc] 2024-03-28T18:17:03.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5640f7c631ff] 2024-03-28T18:17:03.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff60c6bd488] 2024-03-28T18:17:03.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 13: (DispatchQueue::entry()+0x51f) [0x7ff60c6ba60f] 2024-03-28T18:17:03.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff60c78de81] 2024-03-28T18:17:03.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff60b1a81ca] 2024-03-28T18:17:03.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 16: clone() 2024-03-28T18:17:03.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2024-03-28T18:17:03.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 0> 2024-03-28T18:17:02.784+0000 7ff600c00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:17:03.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: in thread 7ff600c00700 thread_name:ms_dispatch 2024-03-28T18:17:03.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2024-03-28T18:17:03.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:03.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff60b1b2d20] 2024-03-28T18:17:03.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2: gsignal() 2024-03-28T18:17:03.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 3: abort() 2024-03-28T18:17:03.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff60c42b4fb] 2024-03-28T18:17:03.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff60c42b667] 2024-03-28T18:17:03.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5640f7eab223] 2024-03-28T18:17:03.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5640f7d255d6] 2024-03-28T18:17:03.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5640f7d25da1] 2024-03-28T18:17:03.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5640f7d282d5] 2024-03-28T18:17:03.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5640f7d78233] 2024-03-28T18:17:03.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5640f7d7a197] 2024-03-28T18:17:03.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5640f7c7911b] 2024-03-28T18:17:03.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5640f7c796bc] 2024-03-28T18:17:03.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5640f7c631ff] 2024-03-28T18:17:03.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff60c6bd488] 2024-03-28T18:17:03.141 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 16: (DispatchQueue::entry()+0x51f) [0x7ff60c6ba60f] 2024-03-28T18:17:03.141 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff60c78de81] 2024-03-28T18:17:03.141 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff60b1a81ca] 2024-03-28T18:17:03.141 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 19: clone() 2024-03-28T18:17:03.141 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:17:03.141 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87543]: 2024-03-28T18:17:03.141 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:02 smithi037 podman[87594]: 2024-03-28 18:17:02.911669464 +0000 UTC m=+0.033423388 container died 58605365ba926f64597e2a0285ac0255d4d18f054d97db091450f381b8437fce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS) 2024-03-28T18:17:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:03 smithi059 ceph-mon[36190]: pgmap v191: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:17:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:03 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:17:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:03 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:17:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:03 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3340788991,v1:172.21.15.37:6837/3340788991] up:active 2024-03-28T18:17:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:03 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:03 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:17:04.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:03 smithi170 ceph-mon[36280]: pgmap v191: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:17:04.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:03 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:17:04.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:03 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:17:04.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:03 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3340788991,v1:172.21.15.37:6837/3340788991] up:active 2024-03-28T18:17:04.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:03 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:04.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:03 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:17:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:03 smithi037 ceph-mon[31460]: pgmap v191: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:17:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:03 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:17:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:03 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:17:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:03 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3340788991,v1:172.21.15.37:6837/3340788991] up:active 2024-03-28T18:17:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:03 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:03 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:17:04.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:03 smithi037 podman[87594]: 2024-03-28 18:17:03.813095981 +0000 UTC m=+0.934849952 container remove 58605365ba926f64597e2a0285ac0255d4d18f054d97db091450f381b8437fce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-28T18:17:04.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:03 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:17:04.624 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:04 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-28T18:17:04.625 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:04 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.799s CPU time. 2024-03-28T18:17:06.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:05 smithi059 ceph-mon[36190]: pgmap v192: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:17:06.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:05 smithi170 ceph-mon[36280]: pgmap v192: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:17:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:05 smithi037 ceph-mon[31460]: pgmap v192: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:17:06.874 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:17:06 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:17:06.415+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:17:08.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:07 smithi059 ceph-mon[36190]: pgmap v193: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:17:08.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:07 smithi170 ceph-mon[36280]: pgmap v193: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:17:08.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:07 smithi037 ceph-mon[31460]: pgmap v193: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:17:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:09 smithi059 ceph-mon[36190]: pgmap v194: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:17:10.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:09 smithi170 ceph-mon[36280]: pgmap v194: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:17:10.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:09 smithi037 ceph-mon[31460]: pgmap v194: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:17:12.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:11 smithi059 ceph-mon[36190]: pgmap v195: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-03-28T18:17:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:11 smithi170 ceph-mon[36280]: pgmap v195: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-03-28T18:17:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:11 smithi037 ceph-mon[31460]: pgmap v195: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-03-28T18:17:14.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:13 smithi059 ceph-mon[36190]: pgmap v196: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 23 KiB/s wr, 31 op/s 2024-03-28T18:17:14.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:13 smithi170 ceph-mon[36280]: pgmap v196: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 23 KiB/s wr, 31 op/s 2024-03-28T18:17:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:13 smithi037 ceph-mon[31460]: pgmap v196: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 23 KiB/s wr, 31 op/s 2024-03-28T18:17:14.776 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:14 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 2. 2024-03-28T18:17:14.776 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:14 smithi037 systemd[1]: Stopped Ceph mds.d for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:17:14.776 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:14 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.799s CPU time. 2024-03-28T18:17:14.776 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:14 smithi037 systemd[1]: Starting Ceph mds.d for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:17:15.101 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:14 smithi037 podman[87684]: 2024-03-28 18:17:14.681179144 +0000 UTC m=+0.039989305 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:17:15.102 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:14 smithi037 podman[87684]: 2024-03-28 18:17:14.817482109 +0000 UTC m=+0.176292303 container create 1d449f4b18a2539f14978f58a93852b88abfd3cf52115b7e7c677554960db2b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD) 2024-03-28T18:17:15.375 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:15 smithi037 podman[87684]: 2024-03-28 18:17:15.102271832 +0000 UTC m=+0.461082013 container init 1d449f4b18a2539f14978f58a93852b88abfd3cf52115b7e7c677554960db2b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-28T18:17:15.375 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:15 smithi037 podman[87684]: 2024-03-28 18:17:15.115593259 +0000 UTC m=+0.474403440 container start 1d449f4b18a2539f14978f58a93852b88abfd3cf52115b7e7c677554960db2b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-03-28T18:17:15.375 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:15 smithi037 ceph-mds[87703]: -- 172.21.15.37:0/2803403578 <== mon.0 v2:172.21.15.37:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558c9f32ba40 con 0x558c9f2fec00 2024-03-28T18:17:15.375 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:15 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: starting mds.d at 2024-03-28T18:17:15.375 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:15 smithi037 bash[87684]: 1d449f4b18a2539f14978f58a93852b88abfd3cf52115b7e7c677554960db2b6 2024-03-28T18:17:15.375 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:15 smithi037 systemd[1]: Started Ceph mds.d for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:17:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:15 smithi059 ceph-mon[36190]: pgmap v197: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-28T18:17:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:15 smithi059 ceph-mon[36190]: daemon mds.d restarted 2024-03-28T18:17:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:15 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:17:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:15 smithi059 ceph-mon[36190]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T18:17:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:15 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/2156393437,v1:172.21.15.37:6837/2156393437] up:boot 2024-03-28T18:17:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:15 smithi059 ceph-mon[36190]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-03-28T18:17:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:15 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:17:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:15 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:17:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:15 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:15 smithi170 ceph-mon[36280]: pgmap v197: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-28T18:17:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:15 smithi170 ceph-mon[36280]: daemon mds.d restarted 2024-03-28T18:17:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:15 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:17:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:15 smithi170 ceph-mon[36280]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T18:17:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:15 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/2156393437,v1:172.21.15.37:6837/2156393437] up:boot 2024-03-28T18:17:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:15 smithi170 ceph-mon[36280]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-03-28T18:17:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:15 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:17:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:15 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:17:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:15 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:16.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:15 smithi037 ceph-mon[31460]: pgmap v197: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-28T18:17:16.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:15 smithi037 ceph-mon[31460]: daemon mds.d restarted 2024-03-28T18:17:16.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:15 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:17:16.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:15 smithi037 ceph-mon[31460]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T18:17:16.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:15 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/2156393437,v1:172.21.15.37:6837/2156393437] up:boot 2024-03-28T18:17:16.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:15 smithi037 ceph-mon[31460]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-03-28T18:17:16.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:15 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:17:16.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:15 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:17:16.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:15 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:18.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:17 smithi059 ceph-mon[36190]: pgmap v199: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:17:18.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:17 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/2253706559,v1:172.21.15.59:6835/2253706559] up:resolve 2024-03-28T18:17:18.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:17 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:18.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:17 smithi170 ceph-mon[36280]: pgmap v199: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:17:18.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:17 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/2253706559,v1:172.21.15.59:6835/2253706559] up:resolve 2024-03-28T18:17:18.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:17 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:18.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:17 smithi037 ceph-mon[31460]: pgmap v199: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:17:18.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:17 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/2253706559,v1:172.21.15.59:6835/2253706559] up:resolve 2024-03-28T18:17:18.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:17 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:18 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/2253706559,v1:172.21.15.59:6835/2253706559] up:reconnect 2024-03-28T18:17:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:18 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:18 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999989 2024-03-28T18:17:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:18 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00999989 2024-03-28T18:17:19.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:18 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/2253706559,v1:172.21.15.59:6835/2253706559] up:reconnect 2024-03-28T18:17:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:18 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:18 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999989 2024-03-28T18:17:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:18 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00999989 2024-03-28T18:17:19.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:18 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/2253706559,v1:172.21.15.59:6835/2253706559] up:reconnect 2024-03-28T18:17:19.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:18 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:19.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:18 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999989 2024-03-28T18:17:19.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:18 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00999989 2024-03-28T18:17:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-mon[36190]: pgmap v200: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:17:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/2253706559,v1:172.21.15.59:6835/2253706559] up:rejoin 2024-03-28T18:17:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-mon[36190]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-03-28T18:17:20.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f50a4c00700 time 2024-03-28T18:17:19.809349+0000 2024-03-28T18:17:20.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:17:20.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:20.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f50af22a4a1] 2024-03-28T18:17:20.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f50af22a667] 2024-03-28T18:17:20.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557f795a1223] 2024-03-28T18:17:20.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557f7941b5d6] 2024-03-28T18:17:20.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557f7941bda1] 2024-03-28T18:17:20.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557f7941e2d5] 2024-03-28T18:17:20.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557f7946e233] 2024-03-28T18:17:20.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557f79470197] 2024-03-28T18:17:20.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557f7936f11b] 2024-03-28T18:17:20.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557f7936f6bc] 2024-03-28T18:17:20.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557f793591ff] 2024-03-28T18:17:20.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f50af4bc488] 2024-03-28T18:17:20.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 13: (DispatchQueue::entry()+0x51f) [0x7f50af4b960f] 2024-03-28T18:17:20.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f50af58ce81] 2024-03-28T18:17:20.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f50adfa71ca] 2024-03-28T18:17:20.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 16: clone() 2024-03-28T18:17:20.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2024-03-28T18:17:19.810+0000 7f50a4c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f50a4c00700 time 2024-03-28T18:17:19.809349+0000 2024-03-28T18:17:20.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:17:20.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2024-03-28T18:17:20.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:20.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f50af22a4a1] 2024-03-28T18:17:20.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f50af22a667] 2024-03-28T18:17:20.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557f795a1223] 2024-03-28T18:17:20.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557f7941b5d6] 2024-03-28T18:17:20.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557f7941bda1] 2024-03-28T18:17:20.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557f7941e2d5] 2024-03-28T18:17:20.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557f7946e233] 2024-03-28T18:17:20.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557f79470197] 2024-03-28T18:17:20.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557f7936f11b] 2024-03-28T18:17:20.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557f7936f6bc] 2024-03-28T18:17:20.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557f793591ff] 2024-03-28T18:17:20.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f50af4bc488] 2024-03-28T18:17:20.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 13: (DispatchQueue::entry()+0x51f) [0x7f50af4b960f] 2024-03-28T18:17:20.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f50af58ce81] 2024-03-28T18:17:20.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f50adfa71ca] 2024-03-28T18:17:20.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 16: clone() 2024-03-28T18:17:20.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2024-03-28T18:17:20.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: *** Caught signal (Aborted) ** 2024-03-28T18:17:20.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: in thread 7f50a4c00700 thread_name:ms_dispatch 2024-03-28T18:17:20.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:20.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f50adfb1d20] 2024-03-28T18:17:20.039 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:19 smithi170 ceph-mon[36280]: pgmap v200: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:17:20.039 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:19 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/2253706559,v1:172.21.15.59:6835/2253706559] up:rejoin 2024-03-28T18:17:20.040 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:19 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:20.040 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:19 smithi170 ceph-mon[36280]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-03-28T18:17:20.040 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2: gsignal() 2024-03-28T18:17:20.040 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 3: abort() 2024-03-28T18:17:20.040 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f50af22a4fb] 2024-03-28T18:17:20.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f50af22a667] 2024-03-28T18:17:20.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557f795a1223] 2024-03-28T18:17:20.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557f7941b5d6] 2024-03-28T18:17:20.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557f7941bda1] 2024-03-28T18:17:20.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557f7941e2d5] 2024-03-28T18:17:20.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557f7946e233] 2024-03-28T18:17:20.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557f79470197] 2024-03-28T18:17:20.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557f7936f11b] 2024-03-28T18:17:20.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557f7936f6bc] 2024-03-28T18:17:20.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557f793591ff] 2024-03-28T18:17:20.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f50af4bc488] 2024-03-28T18:17:20.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 16: (DispatchQueue::entry()+0x51f) [0x7f50af4b960f] 2024-03-28T18:17:20.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f50af58ce81] 2024-03-28T18:17:20.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f50adfa71ca] 2024-03-28T18:17:20.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 19: clone() 2024-03-28T18:17:20.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2024-03-28T18:17:19.812+0000 7f50a4c00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:17:20.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: in thread 7f50a4c00700 thread_name:ms_dispatch 2024-03-28T18:17:20.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2024-03-28T18:17:20.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:20.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f50adfb1d20] 2024-03-28T18:17:20.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2: gsignal() 2024-03-28T18:17:20.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 3: abort() 2024-03-28T18:17:20.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f50af22a4fb] 2024-03-28T18:17:20.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f50af22a667] 2024-03-28T18:17:20.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557f795a1223] 2024-03-28T18:17:20.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557f7941b5d6] 2024-03-28T18:17:20.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557f7941bda1] 2024-03-28T18:17:20.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557f7941e2d5] 2024-03-28T18:17:20.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557f7946e233] 2024-03-28T18:17:20.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557f79470197] 2024-03-28T18:17:20.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557f7936f11b] 2024-03-28T18:17:20.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557f7936f6bc] 2024-03-28T18:17:20.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557f793591ff] 2024-03-28T18:17:20.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f50af4bc488] 2024-03-28T18:17:20.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 16: (DispatchQueue::entry()+0x51f) [0x7f50af4b960f] 2024-03-28T18:17:20.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f50af58ce81] 2024-03-28T18:17:20.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f50adfa71ca] 2024-03-28T18:17:20.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 19: clone() 2024-03-28T18:17:20.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:17:20.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2024-03-28T18:17:20.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: -1> 2024-03-28T18:17:19.810+0000 7f50a4c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f50a4c00700 time 2024-03-28T18:17:19.809349+0000 2024-03-28T18:17:20.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:17:20.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2024-03-28T18:17:20.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:20.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f50af22a4a1] 2024-03-28T18:17:20.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f50af22a667] 2024-03-28T18:17:20.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557f795a1223] 2024-03-28T18:17:20.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557f7941b5d6] 2024-03-28T18:17:20.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557f7941bda1] 2024-03-28T18:17:20.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557f7941e2d5] 2024-03-28T18:17:20.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557f7946e233] 2024-03-28T18:17:20.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557f79470197] 2024-03-28T18:17:20.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557f7936f11b] 2024-03-28T18:17:20.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557f7936f6bc] 2024-03-28T18:17:20.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557f793591ff] 2024-03-28T18:17:20.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f50af4bc488] 2024-03-28T18:17:20.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 13: (DispatchQueue::entry()+0x51f) [0x7f50af4b960f] 2024-03-28T18:17:20.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f50af58ce81] 2024-03-28T18:17:20.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f50adfa71ca] 2024-03-28T18:17:20.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 16: clone() 2024-03-28T18:17:20.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2024-03-28T18:17:20.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 0> 2024-03-28T18:17:19.812+0000 7f50a4c00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:17:20.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: in thread 7f50a4c00700 thread_name:ms_dispatch 2024-03-28T18:17:20.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2024-03-28T18:17:20.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:20.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f50adfb1d20] 2024-03-28T18:17:20.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2: gsignal() 2024-03-28T18:17:20.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 3: abort() 2024-03-28T18:17:20.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f50af22a4fb] 2024-03-28T18:17:20.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f50af22a667] 2024-03-28T18:17:20.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557f795a1223] 2024-03-28T18:17:20.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557f7941b5d6] 2024-03-28T18:17:20.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557f7941bda1] 2024-03-28T18:17:20.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557f7941e2d5] 2024-03-28T18:17:20.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557f7946e233] 2024-03-28T18:17:20.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557f79470197] 2024-03-28T18:17:20.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557f7936f11b] 2024-03-28T18:17:20.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557f7936f6bc] 2024-03-28T18:17:20.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557f793591ff] 2024-03-28T18:17:20.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f50af4bc488] 2024-03-28T18:17:20.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 16: (DispatchQueue::entry()+0x51f) [0x7f50af4b960f] 2024-03-28T18:17:20.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f50af58ce81] 2024-03-28T18:17:20.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f50adfa71ca] 2024-03-28T18:17:20.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 19: clone() 2024-03-28T18:17:20.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:17:20.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2024-03-28T18:17:20.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: -1> 2024-03-28T18:17:19.810+0000 7f50a4c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f50a4c00700 time 2024-03-28T18:17:19.809349+0000 2024-03-28T18:17:20.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:17:20.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2024-03-28T18:17:20.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:20.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f50af22a4a1] 2024-03-28T18:17:20.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f50af22a667] 2024-03-28T18:17:20.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557f795a1223] 2024-03-28T18:17:20.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557f7941b5d6] 2024-03-28T18:17:20.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557f7941bda1] 2024-03-28T18:17:20.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557f7941e2d5] 2024-03-28T18:17:20.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557f7946e233] 2024-03-28T18:17:20.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557f79470197] 2024-03-28T18:17:20.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557f7936f11b] 2024-03-28T18:17:20.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557f7936f6bc] 2024-03-28T18:17:20.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557f793591ff] 2024-03-28T18:17:20.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f50af4bc488] 2024-03-28T18:17:20.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 13: (DispatchQueue::entry()+0x51f) [0x7f50af4b960f] 2024-03-28T18:17:20.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f50af58ce81] 2024-03-28T18:17:20.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f50adfa71ca] 2024-03-28T18:17:20.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 16: clone() 2024-03-28T18:17:20.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2024-03-28T18:17:20.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 0> 2024-03-28T18:17:19.812+0000 7f50a4c00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:17:20.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: in thread 7f50a4c00700 thread_name:ms_dispatch 2024-03-28T18:17:20.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2024-03-28T18:17:20.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:20.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f50adfb1d20] 2024-03-28T18:17:20.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2: gsignal() 2024-03-28T18:17:20.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 3: abort() 2024-03-28T18:17:20.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f50af22a4fb] 2024-03-28T18:17:20.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f50af22a667] 2024-03-28T18:17:20.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557f795a1223] 2024-03-28T18:17:20.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557f7941b5d6] 2024-03-28T18:17:20.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557f7941bda1] 2024-03-28T18:17:20.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557f7941e2d5] 2024-03-28T18:17:20.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557f7946e233] 2024-03-28T18:17:20.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557f79470197] 2024-03-28T18:17:20.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557f7936f11b] 2024-03-28T18:17:20.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557f7936f6bc] 2024-03-28T18:17:20.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557f793591ff] 2024-03-28T18:17:20.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f50af4bc488] 2024-03-28T18:17:20.049 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 16: (DispatchQueue::entry()+0x51f) [0x7f50af4b960f] 2024-03-28T18:17:20.049 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f50af58ce81] 2024-03-28T18:17:20.049 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f50adfa71ca] 2024-03-28T18:17:20.049 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 19: clone() 2024-03-28T18:17:20.049 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:17:20.049 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[66167]: 2024-03-28T18:17:20.049 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:19 smithi059 podman[71574]: 2024-03-28 18:17:19.874434436 +0000 UTC m=+0.020355067 container died e836f5f0fe333e76e353802ed768f297f18f5d171dc1c10771ec0729586a0077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:17:20.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:19 smithi037 ceph-mon[31460]: pgmap v200: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:17:20.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:19 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/2253706559,v1:172.21.15.59:6835/2253706559] up:rejoin 2024-03-28T18:17:20.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:19 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:20.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:19 smithi037 ceph-mon[31460]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-03-28T18:17:20.913 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:20 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:17:20.913 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:20 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:17:20.913 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:20 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/2253706559,v1:172.21.15.59:6835/2253706559] up:active 2024-03-28T18:17:20.913 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:20 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:20.914 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:20 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:17:20.914 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:20 smithi059 podman[71574]: 2024-03-28 18:17:20.653028982 +0000 UTC m=+0.798949598 container remove e836f5f0fe333e76e353802ed768f297f18f5d171dc1c10771ec0729586a0077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:17:20.914 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:20 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:17:21.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:20 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:17:21.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:20 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:17:21.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:20 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/2253706559,v1:172.21.15.59:6835/2253706559] up:active 2024-03-28T18:17:21.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:20 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:21.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:20 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:17:21.285 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:20 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service: Failed with result 'exit-code'. 2024-03-28T18:17:21.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:20 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:17:21.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:20 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:17:21.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:20 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/2253706559,v1:172.21.15.59:6835/2253706559] up:active 2024-03-28T18:17:21.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:20 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:21.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:20 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:17:21.790 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:17:21 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:17:21.415+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:17:22.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:21 smithi037 ceph-mon[31460]: pgmap v201: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:17:22.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:21 smithi059 ceph-mon[36190]: pgmap v201: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:17:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:21 smithi170 ceph-mon[36280]: pgmap v201: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:17:24.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:23 smithi037 ceph-mon[31460]: pgmap v202: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:17:24.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:23 smithi059 ceph-mon[36190]: pgmap v202: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:17:24.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:23 smithi170 ceph-mon[36280]: pgmap v202: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:17:26.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:25 smithi037 ceph-mon[31460]: pgmap v203: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:17:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:25 smithi059 ceph-mon[36190]: pgmap v203: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:17:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:25 smithi170 ceph-mon[36280]: pgmap v203: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:17:28.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:27 smithi037 ceph-mon[31460]: pgmap v204: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 180 B/s wr, 17 op/s 2024-03-28T18:17:28.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:27 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:17:28.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:27 smithi059 ceph-mon[36190]: pgmap v204: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 180 B/s wr, 17 op/s 2024-03-28T18:17:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:27 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:17:28.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:27 smithi170 ceph-mon[36280]: pgmap v204: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 180 B/s wr, 17 op/s 2024-03-28T18:17:28.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:27 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:17:30.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:29 smithi037 ceph-mon[31460]: pgmap v205: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:17:30.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:29 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:17:30.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:29 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:17:30.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:29 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:17:30.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:29 smithi059 ceph-mon[36190]: pgmap v205: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:17:30.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:29 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:17:30.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:29 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:17:30.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:29 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:17:30.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:29 smithi170 ceph-mon[36280]: pgmap v205: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:17:30.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:29 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:17:30.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:29 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:17:30.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:29 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:17:31.337 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:31 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service: Scheduled restart job, restart counter is at 1. 2024-03-28T18:17:31.337 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:31 smithi059 systemd[1]: Stopped Ceph mds.e for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:17:31.337 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:31 smithi059 systemd[1]: Starting Ceph mds.e for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:17:31.337 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:31 smithi059 podman[71799]: 2024-03-28 18:17:31.177309125 +0000 UTC m=+0.018559773 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:17:31.643 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:31 smithi059 podman[71799]: 2024-03-28 18:17:31.337687535 +0000 UTC m=+0.178938168 container create 21e52087456820a2908804e8d8ec55cc34258e4ca0ab1ff3001513de7c36673c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-28T18:17:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:31 smithi059 ceph-mon[36190]: pgmap v206: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:17:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:31 smithi059 ceph-mon[36190]: daemon mds.e restarted 2024-03-28T18:17:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:31 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:17:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:31 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:17:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:31 smithi059 ceph-mon[36190]: osdmap e102: 12 total, 12 up, 12 in 2024-03-28T18:17:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:31 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/1971908336,v1:172.21.15.59:6835/1971908336] up:boot 2024-03-28T18:17:32.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:31 smithi059 ceph-mon[36190]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-28T18:17:32.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:31 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:17:32.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:31 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:32.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:31 smithi059 podman[71799]: 2024-03-28 18:17:31.707759033 +0000 UTC m=+0.549009660 container init 21e52087456820a2908804e8d8ec55cc34258e4ca0ab1ff3001513de7c36673c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:17:32.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:31 smithi059 podman[71799]: 2024-03-28 18:17:31.710836551 +0000 UTC m=+0.552087187 container start 21e52087456820a2908804e8d8ec55cc34258e4ca0ab1ff3001513de7c36673c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0) 2024-03-28T18:17:32.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:31 smithi059 ceph-mds[71827]: -- 172.21.15.59:0/3450305441 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557ab8a7da40 con 0x557ab8a50c00 2024-03-28T18:17:32.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:31 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: starting mds.e at 2024-03-28T18:17:32.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:31 smithi059 bash[71799]: 21e52087456820a2908804e8d8ec55cc34258e4ca0ab1ff3001513de7c36673c 2024-03-28T18:17:32.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:31 smithi059 systemd[1]: Started Ceph mds.e for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:17:32.087 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:31 smithi170 ceph-mon[36280]: pgmap v206: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:17:32.087 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:31 smithi170 ceph-mon[36280]: daemon mds.e restarted 2024-03-28T18:17:32.087 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:31 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:17:32.088 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:31 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:17:32.088 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:31 smithi170 ceph-mon[36280]: osdmap e102: 12 total, 12 up, 12 in 2024-03-28T18:17:32.088 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:31 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/1971908336,v1:172.21.15.59:6835/1971908336] up:boot 2024-03-28T18:17:32.088 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:31 smithi170 ceph-mon[36280]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-28T18:17:32.088 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:31 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:17:32.088 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:31 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:32.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:31 smithi037 ceph-mon[31460]: pgmap v206: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:17:32.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:31 smithi037 ceph-mon[31460]: daemon mds.e restarted 2024-03-28T18:17:32.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:31 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:17:32.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:31 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:17:32.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:31 smithi037 ceph-mon[31460]: osdmap e102: 12 total, 12 up, 12 in 2024-03-28T18:17:32.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:31 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/1971908336,v1:172.21.15.59:6835/1971908336] up:boot 2024-03-28T18:17:32.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:31 smithi037 ceph-mon[31460]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-28T18:17:32.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:31 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:17:32.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:31 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:34.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:33 smithi037 ceph-mon[31460]: pgmap v208: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:17:34.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:33 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/2156393437,v1:172.21.15.37:6837/2156393437] up:resolve 2024-03-28T18:17:34.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:33 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:33 smithi059 ceph-mon[36190]: pgmap v208: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:17:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:33 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/2156393437,v1:172.21.15.37:6837/2156393437] up:resolve 2024-03-28T18:17:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:33 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:34.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:33 smithi170 ceph-mon[36280]: pgmap v208: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:17:34.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:33 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/2156393437,v1:172.21.15.37:6837/2156393437] up:resolve 2024-03-28T18:17:34.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:33 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:35.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:34 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199997 2024-03-28T18:17:35.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:34 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/2156393437,v1:172.21.15.37:6837/2156393437] up:reconnect 2024-03-28T18:17:35.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:34 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:35.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:34 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0079999 2024-03-28T18:17:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:34 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199997 2024-03-28T18:17:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:34 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/2156393437,v1:172.21.15.37:6837/2156393437] up:reconnect 2024-03-28T18:17:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:34 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:34 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0079999 2024-03-28T18:17:35.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:34 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199997 2024-03-28T18:17:35.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:34 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/2156393437,v1:172.21.15.37:6837/2156393437] up:reconnect 2024-03-28T18:17:35.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:34 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:35.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:34 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0079999 2024-03-28T18:17:36.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-mon[31460]: pgmap v209: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:17:36.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/2156393437,v1:172.21.15.37:6837/2156393437] up:rejoin 2024-03-28T18:17:36.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:36.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-mon[31460]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-28T18:17:36.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1de3800700 time 2024-03-28T18:17:35.912158+0000 2024-03-28T18:17:36.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:17:36.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:36.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1def0a94a1] 2024-03-28T18:17:36.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1def0a9667] 2024-03-28T18:17:36.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558c9dfee223] 2024-03-28T18:17:36.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558c9de685d6] 2024-03-28T18:17:36.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558c9de68da1] 2024-03-28T18:17:36.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558c9de6b2d5] 2024-03-28T18:17:36.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558c9debb233] 2024-03-28T18:17:36.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558c9debd197] 2024-03-28T18:17:36.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558c9ddbc11b] 2024-03-28T18:17:36.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558c9ddbc6bc] 2024-03-28T18:17:36.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558c9dda61ff] 2024-03-28T18:17:36.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1def33b488] 2024-03-28T18:17:36.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 13: (DispatchQueue::entry()+0x51f) [0x7f1def33860f] 2024-03-28T18:17:36.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1def40be81] 2024-03-28T18:17:36.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1dede261ca] 2024-03-28T18:17:36.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 16: clone() 2024-03-28T18:17:36.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2024-03-28T18:17:35.914+0000 7f1de3800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1de3800700 time 2024-03-28T18:17:35.912158+0000 2024-03-28T18:17:36.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:17:36.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2024-03-28T18:17:36.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:36.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1def0a94a1] 2024-03-28T18:17:36.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1def0a9667] 2024-03-28T18:17:36.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558c9dfee223] 2024-03-28T18:17:36.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558c9de685d6] 2024-03-28T18:17:36.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558c9de68da1] 2024-03-28T18:17:36.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558c9de6b2d5] 2024-03-28T18:17:36.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558c9debb233] 2024-03-28T18:17:36.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558c9debd197] 2024-03-28T18:17:36.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558c9ddbc11b] 2024-03-28T18:17:36.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558c9ddbc6bc] 2024-03-28T18:17:36.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558c9dda61ff] 2024-03-28T18:17:36.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1def33b488] 2024-03-28T18:17:36.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 13: (DispatchQueue::entry()+0x51f) [0x7f1def33860f] 2024-03-28T18:17:36.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1def40be81] 2024-03-28T18:17:36.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1dede261ca] 2024-03-28T18:17:36.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 16: clone() 2024-03-28T18:17:36.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2024-03-28T18:17:36.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: *** Caught signal (Aborted) ** 2024-03-28T18:17:36.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: in thread 7f1de3800700 thread_name:ms_dispatch 2024-03-28T18:17:36.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:36.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1dede30d20] 2024-03-28T18:17:36.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2: gsignal() 2024-03-28T18:17:36.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 3: abort() 2024-03-28T18:17:36.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1def0a94fb] 2024-03-28T18:17:36.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1def0a9667] 2024-03-28T18:17:36.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558c9dfee223] 2024-03-28T18:17:36.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558c9de685d6] 2024-03-28T18:17:36.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558c9de68da1] 2024-03-28T18:17:36.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558c9de6b2d5] 2024-03-28T18:17:36.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558c9debb233] 2024-03-28T18:17:36.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558c9debd197] 2024-03-28T18:17:36.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558c9ddbc11b] 2024-03-28T18:17:36.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558c9ddbc6bc] 2024-03-28T18:17:36.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558c9dda61ff] 2024-03-28T18:17:36.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1def33b488] 2024-03-28T18:17:36.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 16: (DispatchQueue::entry()+0x51f) [0x7f1def33860f] 2024-03-28T18:17:36.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1def40be81] 2024-03-28T18:17:36.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1dede261ca] 2024-03-28T18:17:36.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 19: clone() 2024-03-28T18:17:36.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2024-03-28T18:17:35.918+0000 7f1de3800700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:17:36.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: in thread 7f1de3800700 thread_name:ms_dispatch 2024-03-28T18:17:36.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2024-03-28T18:17:36.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:36.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1dede30d20] 2024-03-28T18:17:36.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2: gsignal() 2024-03-28T18:17:36.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 3: abort() 2024-03-28T18:17:36.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1def0a94fb] 2024-03-28T18:17:36.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1def0a9667] 2024-03-28T18:17:36.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558c9dfee223] 2024-03-28T18:17:36.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558c9de685d6] 2024-03-28T18:17:36.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558c9de68da1] 2024-03-28T18:17:36.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558c9de6b2d5] 2024-03-28T18:17:36.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558c9debb233] 2024-03-28T18:17:36.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558c9debd197] 2024-03-28T18:17:36.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558c9ddbc11b] 2024-03-28T18:17:36.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558c9ddbc6bc] 2024-03-28T18:17:36.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558c9dda61ff] 2024-03-28T18:17:36.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1def33b488] 2024-03-28T18:17:36.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 16: (DispatchQueue::entry()+0x51f) [0x7f1def33860f] 2024-03-28T18:17:36.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1def40be81] 2024-03-28T18:17:36.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1dede261ca] 2024-03-28T18:17:36.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 19: clone() 2024-03-28T18:17:36.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:17:36.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2024-03-28T18:17:36.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: -1> 2024-03-28T18:17:35.914+0000 7f1de3800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1de3800700 time 2024-03-28T18:17:35.912158+0000 2024-03-28T18:17:36.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:17:36.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2024-03-28T18:17:36.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:36.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1def0a94a1] 2024-03-28T18:17:36.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1def0a9667] 2024-03-28T18:17:36.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558c9dfee223] 2024-03-28T18:17:36.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558c9de685d6] 2024-03-28T18:17:36.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558c9de68da1] 2024-03-28T18:17:36.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558c9de6b2d5] 2024-03-28T18:17:36.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558c9debb233] 2024-03-28T18:17:36.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558c9debd197] 2024-03-28T18:17:36.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558c9ddbc11b] 2024-03-28T18:17:36.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558c9ddbc6bc] 2024-03-28T18:17:36.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558c9dda61ff] 2024-03-28T18:17:36.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1def33b488] 2024-03-28T18:17:36.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 13: (DispatchQueue::entry()+0x51f) [0x7f1def33860f] 2024-03-28T18:17:36.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1def40be81] 2024-03-28T18:17:36.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1dede261ca] 2024-03-28T18:17:36.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 16: clone() 2024-03-28T18:17:36.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2024-03-28T18:17:36.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 0> 2024-03-28T18:17:35.918+0000 7f1de3800700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:17:36.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: in thread 7f1de3800700 thread_name:ms_dispatch 2024-03-28T18:17:36.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2024-03-28T18:17:36.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:36.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1dede30d20] 2024-03-28T18:17:36.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2: gsignal() 2024-03-28T18:17:36.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 3: abort() 2024-03-28T18:17:36.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1def0a94fb] 2024-03-28T18:17:36.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1def0a9667] 2024-03-28T18:17:36.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558c9dfee223] 2024-03-28T18:17:36.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558c9de685d6] 2024-03-28T18:17:36.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558c9de68da1] 2024-03-28T18:17:36.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558c9de6b2d5] 2024-03-28T18:17:36.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558c9debb233] 2024-03-28T18:17:36.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558c9debd197] 2024-03-28T18:17:36.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558c9ddbc11b] 2024-03-28T18:17:36.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558c9ddbc6bc] 2024-03-28T18:17:36.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558c9dda61ff] 2024-03-28T18:17:36.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1def33b488] 2024-03-28T18:17:36.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 16: (DispatchQueue::entry()+0x51f) [0x7f1def33860f] 2024-03-28T18:17:36.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1def40be81] 2024-03-28T18:17:36.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1dede261ca] 2024-03-28T18:17:36.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 19: clone() 2024-03-28T18:17:36.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:17:36.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2024-03-28T18:17:36.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: -1> 2024-03-28T18:17:35.914+0000 7f1de3800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1de3800700 time 2024-03-28T18:17:35.912158+0000 2024-03-28T18:17:36.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:17:36.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2024-03-28T18:17:36.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:36.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1def0a94a1] 2024-03-28T18:17:36.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1def0a9667] 2024-03-28T18:17:36.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558c9dfee223] 2024-03-28T18:17:36.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558c9de685d6] 2024-03-28T18:17:36.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558c9de68da1] 2024-03-28T18:17:36.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558c9de6b2d5] 2024-03-28T18:17:36.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558c9debb233] 2024-03-28T18:17:36.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558c9debd197] 2024-03-28T18:17:36.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558c9ddbc11b] 2024-03-28T18:17:36.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558c9ddbc6bc] 2024-03-28T18:17:36.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558c9dda61ff] 2024-03-28T18:17:36.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1def33b488] 2024-03-28T18:17:36.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 13: (DispatchQueue::entry()+0x51f) [0x7f1def33860f] 2024-03-28T18:17:36.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1def40be81] 2024-03-28T18:17:36.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1dede261ca] 2024-03-28T18:17:36.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 16: clone() 2024-03-28T18:17:36.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2024-03-28T18:17:36.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 0> 2024-03-28T18:17:35.918+0000 7f1de3800700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:17:36.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: in thread 7f1de3800700 thread_name:ms_dispatch 2024-03-28T18:17:36.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2024-03-28T18:17:36.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:36.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1dede30d20] 2024-03-28T18:17:36.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2: gsignal() 2024-03-28T18:17:36.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 3: abort() 2024-03-28T18:17:36.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1def0a94fb] 2024-03-28T18:17:36.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1def0a9667] 2024-03-28T18:17:36.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558c9dfee223] 2024-03-28T18:17:36.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558c9de685d6] 2024-03-28T18:17:36.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558c9de68da1] 2024-03-28T18:17:36.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558c9de6b2d5] 2024-03-28T18:17:36.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558c9debb233] 2024-03-28T18:17:36.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558c9debd197] 2024-03-28T18:17:36.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558c9ddbc11b] 2024-03-28T18:17:36.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558c9ddbc6bc] 2024-03-28T18:17:36.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558c9dda61ff] 2024-03-28T18:17:36.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1def33b488] 2024-03-28T18:17:36.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 16: (DispatchQueue::entry()+0x51f) [0x7f1def33860f] 2024-03-28T18:17:36.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1def40be81] 2024-03-28T18:17:36.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1dede261ca] 2024-03-28T18:17:36.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 19: clone() 2024-03-28T18:17:36.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:17:36.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:35 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87699]: 2024-03-28T18:17:36.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:36 smithi037 podman[87865]: 2024-03-28 18:17:36.088239654 +0000 UTC m=+0.051458814 container died 1d449f4b18a2539f14978f58a93852b88abfd3cf52115b7e7c677554960db2b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-28T18:17:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:35 smithi059 ceph-mon[36190]: pgmap v209: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:17:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:35 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/2156393437,v1:172.21.15.37:6837/2156393437] up:rejoin 2024-03-28T18:17:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:35 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:35 smithi059 ceph-mon[36190]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-28T18:17:36.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:35 smithi170 ceph-mon[36280]: pgmap v209: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:17:36.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:35 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/2156393437,v1:172.21.15.37:6837/2156393437] up:rejoin 2024-03-28T18:17:36.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:35 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:35 smithi170 ceph-mon[36280]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-28T18:17:36.839 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:17:36 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:17:36.418+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:17:37.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:36 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:17:37.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:36 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:17:37.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:36 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/2156393437,v1:172.21.15.37:6837/2156393437] up:active 2024-03-28T18:17:37.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:36 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:37.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:36 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:17:37.124 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:36 smithi037 podman[87865]: 2024-03-28 18:17:36.839517618 +0000 UTC m=+0.802736828 container remove 1d449f4b18a2539f14978f58a93852b88abfd3cf52115b7e7c677554960db2b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:17:37.124 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:36 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:17:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:36 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:17:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:36 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:17:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:36 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/2156393437,v1:172.21.15.37:6837/2156393437] up:active 2024-03-28T18:17:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:36 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:36 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:17:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:36 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:17:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:36 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:17:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:36 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/2156393437,v1:172.21.15.37:6837/2156393437] up:active 2024-03-28T18:17:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:36 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:37.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:36 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:17:37.412 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:37 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-28T18:17:37.412 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:37 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.718s CPU time. 2024-03-28T18:17:38.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:37 smithi037 ceph-mon[31460]: pgmap v210: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:17:38.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:37 smithi059 ceph-mon[36190]: pgmap v210: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:17:38.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:37 smithi170 ceph-mon[36280]: pgmap v210: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:17:39.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:38 smithi059 ceph-mon[36190]: pgmap v211: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:17:39.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:38 smithi170 ceph-mon[36280]: pgmap v211: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:17:39.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:38 smithi037 ceph-mon[31460]: pgmap v211: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:17:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:41 smithi037 ceph-mon[31460]: pgmap v212: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:17:41.930 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:41 smithi059 ceph-mon[36190]: pgmap v212: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:17:42.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:41 smithi170 ceph-mon[36280]: pgmap v212: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:17:43.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:43 smithi170 ceph-mon[36280]: pgmap v213: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 190 B/s wr, 15 op/s 2024-03-28T18:17:43.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:43 smithi037 ceph-mon[31460]: pgmap v213: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 190 B/s wr, 15 op/s 2024-03-28T18:17:44.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:43 smithi059 ceph-mon[36190]: pgmap v213: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 190 B/s wr, 15 op/s 2024-03-28T18:17:45.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:45 smithi037 ceph-mon[31460]: pgmap v214: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:17:46.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:45 smithi059 ceph-mon[36190]: pgmap v214: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:17:46.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:45 smithi170 ceph-mon[36280]: pgmap v214: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:17:47.757 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:47 smithi037 ceph-mon[31460]: pgmap v215: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:17:47.757 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:47 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 3. 2024-03-28T18:17:47.757 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:47 smithi037 systemd[1]: Stopped Ceph mds.d for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:17:47.757 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:47 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.718s CPU time. 2024-03-28T18:17:47.757 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:47 smithi037 systemd[1]: Starting Ceph mds.d for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:17:48.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:47 smithi059 ceph-mon[36190]: pgmap v215: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:17:48.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:47 smithi170 ceph-mon[36280]: pgmap v215: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:17:48.124 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:47 smithi037 podman[87969]: 2024-03-28 18:17:47.661040885 +0000 UTC m=+0.034814822 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:17:48.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:47 smithi037 podman[87969]: 2024-03-28 18:17:47.791054389 +0000 UTC m=+0.164828351 container create e2a512f368b3247e9d2cd2bddc159236f3004bc1d9fc0a34b20ecfd71e1a1124 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-03-28T18:17:48.596 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:48 smithi037 podman[87969]: 2024-03-28 18:17:48.177628986 +0000 UTC m=+0.551402934 container init e2a512f368b3247e9d2cd2bddc159236f3004bc1d9fc0a34b20ecfd71e1a1124 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-28T18:17:48.596 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:48 smithi037 podman[87969]: 2024-03-28 18:17:48.187361119 +0000 UTC m=+0.561135051 container start e2a512f368b3247e9d2cd2bddc159236f3004bc1d9fc0a34b20ecfd71e1a1124 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-28T18:17:48.596 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:48 smithi037 ceph-mds[87988]: -- 172.21.15.37:0/2232133007 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55baca2d5a40 con 0x55baca2a9000 2024-03-28T18:17:48.596 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:48 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: starting mds.d at 2024-03-28T18:17:48.596 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:48 smithi037 bash[87969]: e2a512f368b3247e9d2cd2bddc159236f3004bc1d9fc0a34b20ecfd71e1a1124 2024-03-28T18:17:48.596 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:17:48 smithi037 systemd[1]: Started Ceph mds.d for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:17:48.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:48 smithi037 ceph-mon[31460]: daemon mds.d restarted 2024-03-28T18:17:48.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:48 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:17:48.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:48 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:17:48.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:48 smithi037 ceph-mon[31460]: osdmap e103: 12 total, 12 up, 12 in 2024-03-28T18:17:48.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:48 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3060578049,v1:172.21.15.37:6837/3060578049] up:boot 2024-03-28T18:17:48.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:48 smithi037 ceph-mon[31460]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-03-28T18:17:48.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:48 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:17:48.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:48 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:49.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:48 smithi059 ceph-mon[36190]: daemon mds.d restarted 2024-03-28T18:17:49.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:48 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:17:49.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:48 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:17:49.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:48 smithi059 ceph-mon[36190]: osdmap e103: 12 total, 12 up, 12 in 2024-03-28T18:17:49.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:48 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3060578049,v1:172.21.15.37:6837/3060578049] up:boot 2024-03-28T18:17:49.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:48 smithi059 ceph-mon[36190]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-03-28T18:17:49.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:48 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:17:49.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:48 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:49.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:48 smithi170 ceph-mon[36280]: daemon mds.d restarted 2024-03-28T18:17:49.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:48 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:17:49.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:48 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:17:49.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:48 smithi170 ceph-mon[36280]: osdmap e103: 12 total, 12 up, 12 in 2024-03-28T18:17:49.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:48 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3060578049,v1:172.21.15.37:6837/3060578049] up:boot 2024-03-28T18:17:49.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:48 smithi170 ceph-mon[36280]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-03-28T18:17:49.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:48 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:17:49.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:48 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:49.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:49 smithi037 ceph-mon[31460]: pgmap v217: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:17:50.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:49 smithi059 ceph-mon[36190]: pgmap v217: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:17:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:49 smithi170 ceph-mon[36280]: pgmap v217: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:17:50.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:50 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/1971908336,v1:172.21.15.59:6835/1971908336] up:resolve 2024-03-28T18:17:50.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:50 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:51.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:50 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/1971908336,v1:172.21.15.59:6835/1971908336] up:resolve 2024-03-28T18:17:51.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:50 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:51.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:50 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/1971908336,v1:172.21.15.59:6835/1971908336] up:resolve 2024-03-28T18:17:51.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:50 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:51 smithi059 ceph-mon[36190]: pgmap v218: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:17:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:51 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199998 2024-03-28T18:17:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:51 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/1971908336,v1:172.21.15.59:6835/1971908336] up:reconnect 2024-03-28T18:17:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:51 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:51 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089999 2024-03-28T18:17:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:51 smithi170 ceph-mon[36280]: pgmap v218: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:17:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:51 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199998 2024-03-28T18:17:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:51 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/1971908336,v1:172.21.15.59:6835/1971908336] up:reconnect 2024-03-28T18:17:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:51 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:51 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089999 2024-03-28T18:17:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:51 smithi037 ceph-mon[31460]: pgmap v218: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:17:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:51 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199998 2024-03-28T18:17:52.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:51 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/1971908336,v1:172.21.15.59:6835/1971908336] up:reconnect 2024-03-28T18:17:52.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:51 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:52.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:51 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089999 2024-03-28T18:17:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/1971908336,v1:172.21.15.59:6835/1971908336] up:rejoin 2024-03-28T18:17:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-mon[36190]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-03-28T18:17:53.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:52 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/1971908336,v1:172.21.15.59:6835/1971908336] up:rejoin 2024-03-28T18:17:53.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:52 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:53.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:52 smithi170 ceph-mon[36280]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-03-28T18:17:53.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f6b49f8d700 time 2024-03-28T18:17:52.679011+0000 2024-03-28T18:17:53.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:17:53.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:53.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6b545b74a1] 2024-03-28T18:17:53.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6b545b7667] 2024-03-28T18:17:53.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557ab66ee223] 2024-03-28T18:17:53.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557ab65685d6] 2024-03-28T18:17:53.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557ab6568da1] 2024-03-28T18:17:53.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557ab656b2d5] 2024-03-28T18:17:53.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557ab65bb233] 2024-03-28T18:17:53.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557ab65bd197] 2024-03-28T18:17:53.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557ab64bc11b] 2024-03-28T18:17:53.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557ab64bc6bc] 2024-03-28T18:17:53.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557ab64a61ff] 2024-03-28T18:17:53.037 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6b54849488] 2024-03-28T18:17:53.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 13: (DispatchQueue::entry()+0x51f) [0x7f6b5484660f] 2024-03-28T18:17:53.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6b54919e81] 2024-03-28T18:17:53.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6b533341ca] 2024-03-28T18:17:53.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 16: clone() 2024-03-28T18:17:53.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2024-03-28T18:17:52.679+0000 7f6b49f8d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f6b49f8d700 time 2024-03-28T18:17:52.679011+0000 2024-03-28T18:17:53.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:17:53.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2024-03-28T18:17:53.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:53.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6b545b74a1] 2024-03-28T18:17:53.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6b545b7667] 2024-03-28T18:17:53.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557ab66ee223] 2024-03-28T18:17:53.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557ab65685d6] 2024-03-28T18:17:53.038 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557ab6568da1] 2024-03-28T18:17:53.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557ab656b2d5] 2024-03-28T18:17:53.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557ab65bb233] 2024-03-28T18:17:53.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557ab65bd197] 2024-03-28T18:17:53.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557ab64bc11b] 2024-03-28T18:17:53.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557ab64bc6bc] 2024-03-28T18:17:53.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557ab64a61ff] 2024-03-28T18:17:53.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6b54849488] 2024-03-28T18:17:53.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 13: (DispatchQueue::entry()+0x51f) [0x7f6b5484660f] 2024-03-28T18:17:53.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6b54919e81] 2024-03-28T18:17:53.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6b533341ca] 2024-03-28T18:17:53.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 16: clone() 2024-03-28T18:17:53.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2024-03-28T18:17:53.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: *** Caught signal (Aborted) ** 2024-03-28T18:17:53.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: in thread 7f6b49f8d700 thread_name:ms_dispatch 2024-03-28T18:17:53.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:53.039 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6b5333ed20] 2024-03-28T18:17:53.040 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2: gsignal() 2024-03-28T18:17:53.040 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 3: abort() 2024-03-28T18:17:53.040 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6b545b74fb] 2024-03-28T18:17:53.040 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6b545b7667] 2024-03-28T18:17:53.040 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557ab66ee223] 2024-03-28T18:17:53.040 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557ab65685d6] 2024-03-28T18:17:53.040 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557ab6568da1] 2024-03-28T18:17:53.040 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557ab656b2d5] 2024-03-28T18:17:53.040 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557ab65bb233] 2024-03-28T18:17:53.040 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557ab65bd197] 2024-03-28T18:17:53.040 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557ab64bc11b] 2024-03-28T18:17:53.040 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557ab64bc6bc] 2024-03-28T18:17:53.040 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557ab64a61ff] 2024-03-28T18:17:53.040 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6b54849488] 2024-03-28T18:17:53.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 16: (DispatchQueue::entry()+0x51f) [0x7f6b5484660f] 2024-03-28T18:17:53.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6b54919e81] 2024-03-28T18:17:53.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6b533341ca] 2024-03-28T18:17:53.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 19: clone() 2024-03-28T18:17:53.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2024-03-28T18:17:52.681+0000 7f6b49f8d700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:17:53.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: in thread 7f6b49f8d700 thread_name:ms_dispatch 2024-03-28T18:17:53.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2024-03-28T18:17:53.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:53.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6b5333ed20] 2024-03-28T18:17:53.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2: gsignal() 2024-03-28T18:17:53.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 3: abort() 2024-03-28T18:17:53.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6b545b74fb] 2024-03-28T18:17:53.041 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6b545b7667] 2024-03-28T18:17:53.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557ab66ee223] 2024-03-28T18:17:53.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557ab65685d6] 2024-03-28T18:17:53.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557ab6568da1] 2024-03-28T18:17:53.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557ab656b2d5] 2024-03-28T18:17:53.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557ab65bb233] 2024-03-28T18:17:53.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557ab65bd197] 2024-03-28T18:17:53.042 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557ab64bc11b] 2024-03-28T18:17:53.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557ab64bc6bc] 2024-03-28T18:17:53.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557ab64a61ff] 2024-03-28T18:17:53.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6b54849488] 2024-03-28T18:17:53.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 16: (DispatchQueue::entry()+0x51f) [0x7f6b5484660f] 2024-03-28T18:17:53.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6b54919e81] 2024-03-28T18:17:53.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6b533341ca] 2024-03-28T18:17:53.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 19: clone() 2024-03-28T18:17:53.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:17:53.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2024-03-28T18:17:53.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: -1> 2024-03-28T18:17:52.679+0000 7f6b49f8d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f6b49f8d700 time 2024-03-28T18:17:52.679011+0000 2024-03-28T18:17:53.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:17:53.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2024-03-28T18:17:53.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:53.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6b545b74a1] 2024-03-28T18:17:53.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6b545b7667] 2024-03-28T18:17:53.043 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557ab66ee223] 2024-03-28T18:17:53.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557ab65685d6] 2024-03-28T18:17:53.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557ab6568da1] 2024-03-28T18:17:53.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557ab656b2d5] 2024-03-28T18:17:53.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557ab65bb233] 2024-03-28T18:17:53.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557ab65bd197] 2024-03-28T18:17:53.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557ab64bc11b] 2024-03-28T18:17:53.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557ab64bc6bc] 2024-03-28T18:17:53.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557ab64a61ff] 2024-03-28T18:17:53.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6b54849488] 2024-03-28T18:17:53.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 13: (DispatchQueue::entry()+0x51f) [0x7f6b5484660f] 2024-03-28T18:17:53.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6b54919e81] 2024-03-28T18:17:53.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6b533341ca] 2024-03-28T18:17:53.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 16: clone() 2024-03-28T18:17:53.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2024-03-28T18:17:53.044 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 0> 2024-03-28T18:17:52.681+0000 7f6b49f8d700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:17:53.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: in thread 7f6b49f8d700 thread_name:ms_dispatch 2024-03-28T18:17:53.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2024-03-28T18:17:53.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:53.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6b5333ed20] 2024-03-28T18:17:53.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2: gsignal() 2024-03-28T18:17:53.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 3: abort() 2024-03-28T18:17:53.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6b545b74fb] 2024-03-28T18:17:53.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6b545b7667] 2024-03-28T18:17:53.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557ab66ee223] 2024-03-28T18:17:53.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557ab65685d6] 2024-03-28T18:17:53.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557ab6568da1] 2024-03-28T18:17:53.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557ab656b2d5] 2024-03-28T18:17:53.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557ab65bb233] 2024-03-28T18:17:53.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557ab65bd197] 2024-03-28T18:17:53.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557ab64bc11b] 2024-03-28T18:17:53.045 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557ab64bc6bc] 2024-03-28T18:17:53.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557ab64a61ff] 2024-03-28T18:17:53.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6b54849488] 2024-03-28T18:17:53.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 16: (DispatchQueue::entry()+0x51f) [0x7f6b5484660f] 2024-03-28T18:17:53.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6b54919e81] 2024-03-28T18:17:53.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6b533341ca] 2024-03-28T18:17:53.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 19: clone() 2024-03-28T18:17:53.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:17:53.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2024-03-28T18:17:53.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: -1> 2024-03-28T18:17:52.679+0000 7f6b49f8d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f6b49f8d700 time 2024-03-28T18:17:52.679011+0000 2024-03-28T18:17:53.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:17:53.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2024-03-28T18:17:53.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:53.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6b545b74a1] 2024-03-28T18:17:53.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6b545b7667] 2024-03-28T18:17:53.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557ab66ee223] 2024-03-28T18:17:53.046 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557ab65685d6] 2024-03-28T18:17:53.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557ab6568da1] 2024-03-28T18:17:53.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557ab656b2d5] 2024-03-28T18:17:53.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557ab65bb233] 2024-03-28T18:17:53.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557ab65bd197] 2024-03-28T18:17:53.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557ab64bc11b] 2024-03-28T18:17:53.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557ab64bc6bc] 2024-03-28T18:17:53.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557ab64a61ff] 2024-03-28T18:17:53.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6b54849488] 2024-03-28T18:17:53.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 13: (DispatchQueue::entry()+0x51f) [0x7f6b5484660f] 2024-03-28T18:17:53.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6b54919e81] 2024-03-28T18:17:53.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6b533341ca] 2024-03-28T18:17:53.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 16: clone() 2024-03-28T18:17:53.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2024-03-28T18:17:53.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 0> 2024-03-28T18:17:52.681+0000 7f6b49f8d700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:17:53.047 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: in thread 7f6b49f8d700 thread_name:ms_dispatch 2024-03-28T18:17:53.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2024-03-28T18:17:53.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:17:53.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6b5333ed20] 2024-03-28T18:17:53.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2: gsignal() 2024-03-28T18:17:53.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 3: abort() 2024-03-28T18:17:53.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6b545b74fb] 2024-03-28T18:17:53.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6b545b7667] 2024-03-28T18:17:53.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557ab66ee223] 2024-03-28T18:17:53.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557ab65685d6] 2024-03-28T18:17:53.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557ab6568da1] 2024-03-28T18:17:53.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557ab656b2d5] 2024-03-28T18:17:53.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557ab65bb233] 2024-03-28T18:17:53.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557ab65bd197] 2024-03-28T18:17:53.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557ab64bc11b] 2024-03-28T18:17:53.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557ab64bc6bc] 2024-03-28T18:17:53.048 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557ab64a61ff] 2024-03-28T18:17:53.049 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6b54849488] 2024-03-28T18:17:53.049 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 16: (DispatchQueue::entry()+0x51f) [0x7f6b5484660f] 2024-03-28T18:17:53.049 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6b54919e81] 2024-03-28T18:17:53.049 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6b533341ca] 2024-03-28T18:17:53.049 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 19: clone() 2024-03-28T18:17:53.049 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:17:53.049 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71815]: 2024-03-28T18:17:53.049 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:52 smithi059 podman[71865]: 2024-03-28 18:17:52.744400323 +0000 UTC m=+0.020479488 container died 21e52087456820a2908804e8d8ec55cc34258e4ca0ab1ff3001513de7c36673c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-28T18:17:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:52 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/1971908336,v1:172.21.15.59:6835/1971908336] up:rejoin 2024-03-28T18:17:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:52 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:53.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:52 smithi037 ceph-mon[31460]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-03-28T18:17:53.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:53 smithi059 ceph-mon[36190]: pgmap v219: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-28T18:17:53.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:53 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:17:53.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:53 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:17:53.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:53 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/1971908336,v1:172.21.15.59:6835/1971908336] up:active 2024-03-28T18:17:53.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:53 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:53.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:53 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:17:53.785 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:53 smithi059 podman[71865]: 2024-03-28 18:17:53.521360505 +0000 UTC m=+0.797439668 container remove 21e52087456820a2908804e8d8ec55cc34258e4ca0ab1ff3001513de7c36673c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True) 2024-03-28T18:17:53.785 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:53 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:17:53.785 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:17:53 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service: Failed with result 'exit-code'. 2024-03-28T18:17:53.798 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:53 smithi037 ceph-mon[31460]: pgmap v219: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-28T18:17:53.798 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:53 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:17:53.798 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:53 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:17:53.798 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:53 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/1971908336,v1:172.21.15.59:6835/1971908336] up:active 2024-03-28T18:17:53.798 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:53 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:53.798 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:53 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:17:54.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:53 smithi170 ceph-mon[36280]: pgmap v219: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-28T18:17:54.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:53 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:17:54.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:53 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:17:54.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:53 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/1971908336,v1:172.21.15.59:6835/1971908336] up:active 2024-03-28T18:17:54.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:53 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:17:54.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:53 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:17:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:55 smithi059 ceph-mon[36190]: pgmap v220: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-28T18:17:56.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:55 smithi170 ceph-mon[36280]: pgmap v220: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-28T18:17:56.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:55 smithi037 ceph-mon[31460]: pgmap v220: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-28T18:17:56.874 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:17:56 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:17:56.419+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:17:57.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:57 smithi037 ceph-mon[31460]: pgmap v221: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:17:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:57 smithi059 ceph-mon[36190]: pgmap v221: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:17:58.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:57 smithi170 ceph-mon[36280]: pgmap v221: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:18:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:17:59 smithi059 ceph-mon[36190]: pgmap v222: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 199 B/s wr, 15 op/s 2024-03-28T18:18:00.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:17:59 smithi170 ceph-mon[36280]: pgmap v222: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 199 B/s wr, 15 op/s 2024-03-28T18:18:00.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:17:59 smithi037 ceph-mon[31460]: pgmap v222: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 199 B/s wr, 15 op/s 2024-03-28T18:18:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:01 smithi059 ceph-mon[36190]: pgmap v223: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:18:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:01 smithi170 ceph-mon[36280]: pgmap v223: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:18:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:01 smithi037 ceph-mon[31460]: pgmap v223: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:18:03.839 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:03 smithi170 ceph-mon[36280]: pgmap v224: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 17 op/s 2024-03-28T18:18:04.027 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:03 smithi059 ceph-mon[36190]: pgmap v224: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 17 op/s 2024-03-28T18:18:04.027 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:03 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service: Scheduled restart job, restart counter is at 2. 2024-03-28T18:18:04.027 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:03 smithi059 systemd[1]: Stopped Ceph mds.e for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:18:04.027 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:03 smithi059 systemd[1]: Starting Ceph mds.e for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:18:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:03 smithi037 ceph-mon[31460]: pgmap v224: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 17 op/s 2024-03-28T18:18:04.282 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:04 smithi059 podman[71962]: 2024-03-28 18:18:03.930916097 +0000 UTC m=+0.018481906 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:18:04.282 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:04 smithi059 podman[71962]: 2024-03-28 18:18:04.030411352 +0000 UTC m=+0.117977149 container create 8436f838003179d234782abe38ef28f3ac6edd69b19c62ef46a8b6ba5951318c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:18:04.282 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:04 smithi059 podman[71962]: 2024-03-28 18:18:04.258019352 +0000 UTC m=+0.345585158 container init 8436f838003179d234782abe38ef28f3ac6edd69b19c62ef46a8b6ba5951318c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:18:04.282 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:04 smithi059 podman[71962]: 2024-03-28 18:18:04.260716253 +0000 UTC m=+0.348282052 container start 8436f838003179d234782abe38ef28f3ac6edd69b19c62ef46a8b6ba5951318c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-28T18:18:04.282 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:04 smithi059 ceph-mds[71981]: -- 172.21.15.59:0/1738847973 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c19cc25a40 con 0x55c19cbf9000 2024-03-28T18:18:04.535 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:04 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: starting mds.e at 2024-03-28T18:18:04.535 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:04 smithi059 bash[71962]: 8436f838003179d234782abe38ef28f3ac6edd69b19c62ef46a8b6ba5951318c 2024-03-28T18:18:04.535 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:04 smithi059 systemd[1]: Started Ceph mds.e for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:18:05.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:04 smithi059 ceph-mon[36190]: daemon mds.e restarted 2024-03-28T18:18:05.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:04 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:18:05.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:04 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:18:05.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:04 smithi059 ceph-mon[36190]: osdmap e104: 12 total, 12 up, 12 in 2024-03-28T18:18:05.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:04 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/2809705173,v1:172.21.15.59:6835/2809705173] up:boot 2024-03-28T18:18:05.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:04 smithi059 ceph-mon[36190]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-28T18:18:05.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:04 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:18:05.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:04 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:05.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:04 smithi170 ceph-mon[36280]: daemon mds.e restarted 2024-03-28T18:18:05.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:04 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:18:05.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:04 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:18:05.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:04 smithi170 ceph-mon[36280]: osdmap e104: 12 total, 12 up, 12 in 2024-03-28T18:18:05.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:04 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/2809705173,v1:172.21.15.59:6835/2809705173] up:boot 2024-03-28T18:18:05.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:04 smithi170 ceph-mon[36280]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-28T18:18:05.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:04 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:18:05.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:04 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:05.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:04 smithi037 ceph-mon[31460]: daemon mds.e restarted 2024-03-28T18:18:05.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:04 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:18:05.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:04 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:18:05.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:04 smithi037 ceph-mon[31460]: osdmap e104: 12 total, 12 up, 12 in 2024-03-28T18:18:05.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:04 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/2809705173,v1:172.21.15.59:6835/2809705173] up:boot 2024-03-28T18:18:05.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:04 smithi037 ceph-mon[31460]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-28T18:18:05.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:04 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:18:05.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:04 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:06.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:05 smithi059 ceph-mon[36190]: pgmap v226: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 17 op/s 2024-03-28T18:18:06.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:05 smithi170 ceph-mon[36280]: pgmap v226: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 17 op/s 2024-03-28T18:18:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:05 smithi037 ceph-mon[31460]: pgmap v226: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 17 op/s 2024-03-28T18:18:07.026 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:06 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3060578049,v1:172.21.15.37:6837/3060578049] up:resolve 2024-03-28T18:18:07.026 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:06 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:07.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:06 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3060578049,v1:172.21.15.37:6837/3060578049] up:resolve 2024-03-28T18:18:07.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:06 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:07.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:06 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3060578049,v1:172.21.15.37:6837/3060578049] up:resolve 2024-03-28T18:18:07.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:06 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:07 smithi059 ceph-mon[36190]: pgmap v227: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:18:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:07 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00399995 2024-03-28T18:18:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:07 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3060578049,v1:172.21.15.37:6837/3060578049] up:reconnect 2024-03-28T18:18:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:07 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:07 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999987 2024-03-28T18:18:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:07 smithi170 ceph-mon[36280]: pgmap v227: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:18:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:07 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00399995 2024-03-28T18:18:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:07 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3060578049,v1:172.21.15.37:6837/3060578049] up:reconnect 2024-03-28T18:18:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:07 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:07 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999987 2024-03-28T18:18:08.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:07 smithi037 ceph-mon[31460]: pgmap v227: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:18:08.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:07 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00399995 2024-03-28T18:18:08.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:07 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3060578049,v1:172.21.15.37:6837/3060578049] up:reconnect 2024-03-28T18:18:08.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:07 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:08.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:07 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999987 2024-03-28T18:18:09.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:08 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3060578049,v1:172.21.15.37:6837/3060578049] up:rejoin 2024-03-28T18:18:09.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:08 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:09.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:08 smithi059 ceph-mon[36190]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-28T18:18:09.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:08 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3060578049,v1:172.21.15.37:6837/3060578049] up:rejoin 2024-03-28T18:18:09.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:08 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:09.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:08 smithi170 ceph-mon[36280]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-28T18:18:09.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3060578049,v1:172.21.15.37:6837/3060578049] up:rejoin 2024-03-28T18:18:09.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:09.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-mon[31460]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-28T18:18:09.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f6eebe00700 time 2024-03-28T18:18:08.786104+0000 2024-03-28T18:18:09.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:18:09.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:09.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6ef77464a1] 2024-03-28T18:18:09.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6ef7746667] 2024-03-28T18:18:09.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bac8d9a223] 2024-03-28T18:18:09.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bac8c145d6] 2024-03-28T18:18:09.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bac8c14da1] 2024-03-28T18:18:09.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bac8c172d5] 2024-03-28T18:18:09.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bac8c67233] 2024-03-28T18:18:09.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bac8c69197] 2024-03-28T18:18:09.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bac8b6811b] 2024-03-28T18:18:09.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bac8b686bc] 2024-03-28T18:18:09.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bac8b521ff] 2024-03-28T18:18:09.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6ef79d8488] 2024-03-28T18:18:09.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 13: (DispatchQueue::entry()+0x51f) [0x7f6ef79d560f] 2024-03-28T18:18:09.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6ef7aa8e81] 2024-03-28T18:18:09.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6ef64c31ca] 2024-03-28T18:18:09.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 16: clone() 2024-03-28T18:18:09.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2024-03-28T18:18:08.787+0000 7f6eebe00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f6eebe00700 time 2024-03-28T18:18:08.786104+0000 2024-03-28T18:18:09.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:18:09.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2024-03-28T18:18:09.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:09.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6ef77464a1] 2024-03-28T18:18:09.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6ef7746667] 2024-03-28T18:18:09.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bac8d9a223] 2024-03-28T18:18:09.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bac8c145d6] 2024-03-28T18:18:09.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bac8c14da1] 2024-03-28T18:18:09.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bac8c172d5] 2024-03-28T18:18:09.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bac8c67233] 2024-03-28T18:18:09.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bac8c69197] 2024-03-28T18:18:09.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bac8b6811b] 2024-03-28T18:18:09.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bac8b686bc] 2024-03-28T18:18:09.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bac8b521ff] 2024-03-28T18:18:09.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6ef79d8488] 2024-03-28T18:18:09.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 13: (DispatchQueue::entry()+0x51f) [0x7f6ef79d560f] 2024-03-28T18:18:09.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6ef7aa8e81] 2024-03-28T18:18:09.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6ef64c31ca] 2024-03-28T18:18:09.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 16: clone() 2024-03-28T18:18:09.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2024-03-28T18:18:09.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: *** Caught signal (Aborted) ** 2024-03-28T18:18:09.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: in thread 7f6eebe00700 thread_name:ms_dispatch 2024-03-28T18:18:09.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:09.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6ef64cdd20] 2024-03-28T18:18:09.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2: gsignal() 2024-03-28T18:18:09.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 3: abort() 2024-03-28T18:18:09.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6ef77464fb] 2024-03-28T18:18:09.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6ef7746667] 2024-03-28T18:18:09.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bac8d9a223] 2024-03-28T18:18:09.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bac8c145d6] 2024-03-28T18:18:09.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bac8c14da1] 2024-03-28T18:18:09.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bac8c172d5] 2024-03-28T18:18:09.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bac8c67233] 2024-03-28T18:18:09.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bac8c69197] 2024-03-28T18:18:09.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bac8b6811b] 2024-03-28T18:18:09.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bac8b686bc] 2024-03-28T18:18:09.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bac8b521ff] 2024-03-28T18:18:09.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6ef79d8488] 2024-03-28T18:18:09.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 16: (DispatchQueue::entry()+0x51f) [0x7f6ef79d560f] 2024-03-28T18:18:09.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6ef7aa8e81] 2024-03-28T18:18:09.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6ef64c31ca] 2024-03-28T18:18:09.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 19: clone() 2024-03-28T18:18:09.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2024-03-28T18:18:08.789+0000 7f6eebe00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:18:09.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: in thread 7f6eebe00700 thread_name:ms_dispatch 2024-03-28T18:18:09.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2024-03-28T18:18:09.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:09.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6ef64cdd20] 2024-03-28T18:18:09.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2: gsignal() 2024-03-28T18:18:09.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 3: abort() 2024-03-28T18:18:09.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6ef77464fb] 2024-03-28T18:18:09.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6ef7746667] 2024-03-28T18:18:09.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bac8d9a223] 2024-03-28T18:18:09.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bac8c145d6] 2024-03-28T18:18:09.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bac8c14da1] 2024-03-28T18:18:09.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bac8c172d5] 2024-03-28T18:18:09.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bac8c67233] 2024-03-28T18:18:09.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bac8c69197] 2024-03-28T18:18:09.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bac8b6811b] 2024-03-28T18:18:09.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bac8b686bc] 2024-03-28T18:18:09.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bac8b521ff] 2024-03-28T18:18:09.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6ef79d8488] 2024-03-28T18:18:09.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 16: (DispatchQueue::entry()+0x51f) [0x7f6ef79d560f] 2024-03-28T18:18:09.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6ef7aa8e81] 2024-03-28T18:18:09.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6ef64c31ca] 2024-03-28T18:18:09.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 19: clone() 2024-03-28T18:18:09.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:18:09.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2024-03-28T18:18:09.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: -1> 2024-03-28T18:18:08.787+0000 7f6eebe00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f6eebe00700 time 2024-03-28T18:18:08.786104+0000 2024-03-28T18:18:09.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:18:09.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2024-03-28T18:18:09.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:09.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6ef77464a1] 2024-03-28T18:18:09.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6ef7746667] 2024-03-28T18:18:09.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bac8d9a223] 2024-03-28T18:18:09.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bac8c145d6] 2024-03-28T18:18:09.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bac8c14da1] 2024-03-28T18:18:09.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bac8c172d5] 2024-03-28T18:18:09.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bac8c67233] 2024-03-28T18:18:09.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bac8c69197] 2024-03-28T18:18:09.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bac8b6811b] 2024-03-28T18:18:09.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bac8b686bc] 2024-03-28T18:18:09.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bac8b521ff] 2024-03-28T18:18:09.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6ef79d8488] 2024-03-28T18:18:09.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 13: (DispatchQueue::entry()+0x51f) [0x7f6ef79d560f] 2024-03-28T18:18:09.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6ef7aa8e81] 2024-03-28T18:18:09.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6ef64c31ca] 2024-03-28T18:18:09.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 16: clone() 2024-03-28T18:18:09.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2024-03-28T18:18:09.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 0> 2024-03-28T18:18:08.789+0000 7f6eebe00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:18:09.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: in thread 7f6eebe00700 thread_name:ms_dispatch 2024-03-28T18:18:09.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2024-03-28T18:18:09.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:09.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6ef64cdd20] 2024-03-28T18:18:09.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2: gsignal() 2024-03-28T18:18:09.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 3: abort() 2024-03-28T18:18:09.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6ef77464fb] 2024-03-28T18:18:09.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6ef7746667] 2024-03-28T18:18:09.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bac8d9a223] 2024-03-28T18:18:09.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bac8c145d6] 2024-03-28T18:18:09.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bac8c14da1] 2024-03-28T18:18:09.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bac8c172d5] 2024-03-28T18:18:09.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bac8c67233] 2024-03-28T18:18:09.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bac8c69197] 2024-03-28T18:18:09.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bac8b6811b] 2024-03-28T18:18:09.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bac8b686bc] 2024-03-28T18:18:09.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bac8b521ff] 2024-03-28T18:18:09.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6ef79d8488] 2024-03-28T18:18:09.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 16: (DispatchQueue::entry()+0x51f) [0x7f6ef79d560f] 2024-03-28T18:18:09.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6ef7aa8e81] 2024-03-28T18:18:09.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6ef64c31ca] 2024-03-28T18:18:09.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 19: clone() 2024-03-28T18:18:09.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:18:09.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2024-03-28T18:18:09.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: -1> 2024-03-28T18:18:08.787+0000 7f6eebe00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f6eebe00700 time 2024-03-28T18:18:08.786104+0000 2024-03-28T18:18:09.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:18:09.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2024-03-28T18:18:09.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:09.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6ef77464a1] 2024-03-28T18:18:09.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6ef7746667] 2024-03-28T18:18:09.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bac8d9a223] 2024-03-28T18:18:09.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bac8c145d6] 2024-03-28T18:18:09.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bac8c14da1] 2024-03-28T18:18:09.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bac8c172d5] 2024-03-28T18:18:09.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bac8c67233] 2024-03-28T18:18:09.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bac8c69197] 2024-03-28T18:18:09.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bac8b6811b] 2024-03-28T18:18:09.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bac8b686bc] 2024-03-28T18:18:09.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bac8b521ff] 2024-03-28T18:18:09.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6ef79d8488] 2024-03-28T18:18:09.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 13: (DispatchQueue::entry()+0x51f) [0x7f6ef79d560f] 2024-03-28T18:18:09.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6ef7aa8e81] 2024-03-28T18:18:09.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6ef64c31ca] 2024-03-28T18:18:09.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 16: clone() 2024-03-28T18:18:09.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2024-03-28T18:18:09.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 0> 2024-03-28T18:18:08.789+0000 7f6eebe00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:18:09.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: in thread 7f6eebe00700 thread_name:ms_dispatch 2024-03-28T18:18:09.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2024-03-28T18:18:09.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:09.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6ef64cdd20] 2024-03-28T18:18:09.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2: gsignal() 2024-03-28T18:18:09.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 3: abort() 2024-03-28T18:18:09.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6ef77464fb] 2024-03-28T18:18:09.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6ef7746667] 2024-03-28T18:18:09.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bac8d9a223] 2024-03-28T18:18:09.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bac8c145d6] 2024-03-28T18:18:09.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bac8c14da1] 2024-03-28T18:18:09.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bac8c172d5] 2024-03-28T18:18:09.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bac8c67233] 2024-03-28T18:18:09.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bac8c69197] 2024-03-28T18:18:09.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bac8b6811b] 2024-03-28T18:18:09.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bac8b686bc] 2024-03-28T18:18:09.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bac8b521ff] 2024-03-28T18:18:09.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6ef79d8488] 2024-03-28T18:18:09.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 16: (DispatchQueue::entry()+0x51f) [0x7f6ef79d560f] 2024-03-28T18:18:09.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6ef7aa8e81] 2024-03-28T18:18:09.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6ef64c31ca] 2024-03-28T18:18:09.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 19: clone() 2024-03-28T18:18:09.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:18:09.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[87984]: 2024-03-28T18:18:09.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:08 smithi037 podman[88024]: 2024-03-28 18:18:08.916966593 +0000 UTC m=+0.033597681 container died e2a512f368b3247e9d2cd2bddc159236f3004bc1d9fc0a34b20ecfd71e1a1124 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:18:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:09 smithi059 ceph-mon[36190]: pgmap v228: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:18:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:09 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:18:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:09 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:18:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:09 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3060578049,v1:172.21.15.37:6837/3060578049] up:active 2024-03-28T18:18:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:09 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:18:10.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:09 smithi170 ceph-mon[36280]: pgmap v228: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:18:10.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:09 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:18:10.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:09 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:18:10.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:09 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3060578049,v1:172.21.15.37:6837/3060578049] up:active 2024-03-28T18:18:10.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:09 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:18:10.101 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:09 smithi037 podman[88024]: 2024-03-28 18:18:09.666086367 +0000 UTC m=+0.782717495 container remove e2a512f368b3247e9d2cd2bddc159236f3004bc1d9fc0a34b20ecfd71e1a1124 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-28T18:18:10.101 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:09 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:18:10.101 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:09 smithi037 ceph-mon[31460]: pgmap v228: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:18:10.101 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:09 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:18:10.101 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:09 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:18:10.101 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:09 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3060578049,v1:172.21.15.37:6837/3060578049] up:active 2024-03-28T18:18:10.101 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:09 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:18:10.374 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:10 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-28T18:18:10.374 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:10 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.722s CPU time. 2024-03-28T18:18:11.625 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:18:11 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:18:11.421+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:18:12.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:11 smithi059 ceph-mon[36190]: pgmap v229: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:18:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:11 smithi170 ceph-mon[36280]: pgmap v229: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:18:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:11 smithi037 ceph-mon[31460]: pgmap v229: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:18:14.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:13 smithi059 ceph-mon[36190]: pgmap v230: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 32 op/s 2024-03-28T18:18:14.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:13 smithi170 ceph-mon[36280]: pgmap v230: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 32 op/s 2024-03-28T18:18:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:13 smithi037 ceph-mon[31460]: pgmap v230: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 32 op/s 2024-03-28T18:18:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:15 smithi059 ceph-mon[36190]: pgmap v231: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 26 KiB/s wr, 32 op/s 2024-03-28T18:18:16.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:15 smithi170 ceph-mon[36280]: pgmap v231: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 26 KiB/s wr, 32 op/s 2024-03-28T18:18:16.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:15 smithi037 ceph-mon[31460]: pgmap v231: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 26 KiB/s wr, 32 op/s 2024-03-28T18:18:18.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:17 smithi059 ceph-mon[36190]: pgmap v232: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 23 KiB/s wr, 30 op/s 2024-03-28T18:18:18.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:17 smithi170 ceph-mon[36280]: pgmap v232: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 23 KiB/s wr, 30 op/s 2024-03-28T18:18:18.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:17 smithi037 ceph-mon[31460]: pgmap v232: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 23 KiB/s wr, 30 op/s 2024-03-28T18:18:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:19 smithi059 ceph-mon[36190]: pgmap v233: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-28T18:18:20.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:19 smithi170 ceph-mon[36280]: pgmap v233: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-28T18:18:20.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:19 smithi037 ceph-mon[31460]: pgmap v233: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-28T18:18:20.536 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:20 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 4. 2024-03-28T18:18:20.536 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:20 smithi037 systemd[1]: Stopped Ceph mds.d for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:18:20.536 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:20 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.722s CPU time. 2024-03-28T18:18:20.536 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:20 smithi037 systemd[1]: Starting Ceph mds.d for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:18:20.863 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:20 smithi037 podman[88116]: 2024-03-28 18:18:20.441317047 +0000 UTC m=+0.039876438 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:18:20.863 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:20 smithi037 podman[88116]: 2024-03-28 18:18:20.575618413 +0000 UTC m=+0.174177867 container create 248bbd8bbca0ae6c4fe75c54403268debd1f6eec8fd695c2e7a8ea61e75a966e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:18:20.863 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:20 smithi037 podman[88116]: 2024-03-28 18:18:20.863173805 +0000 UTC m=+0.461733276 container init 248bbd8bbca0ae6c4fe75c54403268debd1f6eec8fd695c2e7a8ea61e75a966e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-28T18:18:21.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:20 smithi037 podman[88116]: 2024-03-28 18:18:20.87394611 +0000 UTC m=+0.472505523 container start 248bbd8bbca0ae6c4fe75c54403268debd1f6eec8fd695c2e7a8ea61e75a966e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-28T18:18:21.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:20 smithi037 ceph-mds[88135]: -- 172.21.15.37:0/3162996615 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561204a0ba40 con 0x5612049dec00 2024-03-28T18:18:21.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:20 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: starting mds.d at 2024-03-28T18:18:21.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:21 smithi037 bash[88116]: 248bbd8bbca0ae6c4fe75c54403268debd1f6eec8fd695c2e7a8ea61e75a966e 2024-03-28T18:18:21.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:18:21 smithi037 systemd[1]: Started Ceph mds.d for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:18:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:21 smithi059 ceph-mon[36190]: pgmap v234: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-28T18:18:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:21 smithi059 ceph-mon[36190]: daemon mds.d restarted 2024-03-28T18:18:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:21 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:18:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:21 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:18:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:21 smithi059 ceph-mon[36190]: osdmap e105: 12 total, 12 up, 12 in 2024-03-28T18:18:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:21 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/192494457,v1:172.21.15.37:6837/192494457] up:boot 2024-03-28T18:18:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:21 smithi059 ceph-mon[36190]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-03-28T18:18:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:21 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:18:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:21 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:22.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:21 smithi170 ceph-mon[36280]: pgmap v234: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-28T18:18:22.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:21 smithi170 ceph-mon[36280]: daemon mds.d restarted 2024-03-28T18:18:22.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:21 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:18:22.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:21 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:18:22.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:21 smithi170 ceph-mon[36280]: osdmap e105: 12 total, 12 up, 12 in 2024-03-28T18:18:22.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:21 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/192494457,v1:172.21.15.37:6837/192494457] up:boot 2024-03-28T18:18:22.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:21 smithi170 ceph-mon[36280]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-03-28T18:18:22.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:21 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:18:22.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:21 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:21 smithi037 ceph-mon[31460]: pgmap v234: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-28T18:18:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:21 smithi037 ceph-mon[31460]: daemon mds.d restarted 2024-03-28T18:18:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:21 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:18:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:21 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:18:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:21 smithi037 ceph-mon[31460]: osdmap e105: 12 total, 12 up, 12 in 2024-03-28T18:18:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:21 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/192494457,v1:172.21.15.37:6837/192494457] up:boot 2024-03-28T18:18:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:21 smithi037 ceph-mon[31460]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-03-28T18:18:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:21 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:18:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:21 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:23 smithi059 ceph-mon[36190]: pgmap v236: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:18:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:23 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/2809705173,v1:172.21.15.59:6835/2809705173] up:resolve 2024-03-28T18:18:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:23 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:23 smithi170 ceph-mon[36280]: pgmap v236: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:18:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:23 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/2809705173,v1:172.21.15.59:6835/2809705173] up:resolve 2024-03-28T18:18:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:23 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:24.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:23 smithi037 ceph-mon[31460]: pgmap v236: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:18:24.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:23 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/2809705173,v1:172.21.15.59:6835/2809705173] up:resolve 2024-03-28T18:18:24.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:23 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:25.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:24 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299997 2024-03-28T18:18:25.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:24 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/2809705173,v1:172.21.15.59:6835/2809705173] up:reconnect 2024-03-28T18:18:25.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:24 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:25.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:24 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999989 2024-03-28T18:18:25.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:24 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299997 2024-03-28T18:18:25.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:24 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/2809705173,v1:172.21.15.59:6835/2809705173] up:reconnect 2024-03-28T18:18:25.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:24 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:25.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:24 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999989 2024-03-28T18:18:25.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:24 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299997 2024-03-28T18:18:25.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:24 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/2809705173,v1:172.21.15.59:6835/2809705173] up:reconnect 2024-03-28T18:18:25.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:24 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:25.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:24 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999989 2024-03-28T18:18:26.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:25 smithi037 ceph-mon[31460]: pgmap v237: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:18:26.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:25 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/2809705173,v1:172.21.15.59:6835/2809705173] up:rejoin 2024-03-28T18:18:26.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:25 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:25 smithi037 ceph-mon[31460]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-03-28T18:18:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-mon[36190]: pgmap v237: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:18:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/2809705173,v1:172.21.15.59:6835/2809705173] up:rejoin 2024-03-28T18:18:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-mon[36190]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-03-28T18:18:26.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:25 smithi170 ceph-mon[36280]: pgmap v237: 97 pgs: 97 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:18:26.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:25 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/2809705173,v1:172.21.15.59:6835/2809705173] up:rejoin 2024-03-28T18:18:26.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:25 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:26.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:25 smithi170 ceph-mon[36280]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-03-28T18:18:26.287 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff536c61700 time 2024-03-28T18:18:25.829717+0000 2024-03-28T18:18:26.287 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:18:26.287 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:26.287 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff54128b4a1] 2024-03-28T18:18:26.287 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff54128b667] 2024-03-28T18:18:26.287 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c19acd9223] 2024-03-28T18:18:26.287 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c19ab535d6] 2024-03-28T18:18:26.287 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c19ab53da1] 2024-03-28T18:18:26.287 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c19ab562d5] 2024-03-28T18:18:26.287 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c19aba6233] 2024-03-28T18:18:26.287 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c19aba8197] 2024-03-28T18:18:26.287 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c19aaa711b] 2024-03-28T18:18:26.288 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c19aaa76bc] 2024-03-28T18:18:26.288 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c19aa911ff] 2024-03-28T18:18:26.288 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff54151d488] 2024-03-28T18:18:26.288 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 13: (DispatchQueue::entry()+0x51f) [0x7ff54151a60f] 2024-03-28T18:18:26.288 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff5415ede81] 2024-03-28T18:18:26.288 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff5400081ca] 2024-03-28T18:18:26.288 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 16: clone() 2024-03-28T18:18:26.288 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2024-03-28T18:18:25.830+0000 7ff536c61700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff536c61700 time 2024-03-28T18:18:25.829717+0000 2024-03-28T18:18:26.288 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:18:26.288 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2024-03-28T18:18:26.288 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:26.288 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff54128b4a1] 2024-03-28T18:18:26.288 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff54128b667] 2024-03-28T18:18:26.289 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c19acd9223] 2024-03-28T18:18:26.289 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c19ab535d6] 2024-03-28T18:18:26.289 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c19ab53da1] 2024-03-28T18:18:26.289 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c19ab562d5] 2024-03-28T18:18:26.289 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c19aba6233] 2024-03-28T18:18:26.289 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c19aba8197] 2024-03-28T18:18:26.289 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c19aaa711b] 2024-03-28T18:18:26.289 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c19aaa76bc] 2024-03-28T18:18:26.289 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c19aa911ff] 2024-03-28T18:18:26.289 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff54151d488] 2024-03-28T18:18:26.289 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 13: (DispatchQueue::entry()+0x51f) [0x7ff54151a60f] 2024-03-28T18:18:26.289 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff5415ede81] 2024-03-28T18:18:26.289 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff5400081ca] 2024-03-28T18:18:26.290 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 16: clone() 2024-03-28T18:18:26.290 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2024-03-28T18:18:26.290 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: *** Caught signal (Aborted) ** 2024-03-28T18:18:26.290 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: in thread 7ff536c61700 thread_name:ms_dispatch 2024-03-28T18:18:26.290 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:26.290 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff540012d20] 2024-03-28T18:18:26.290 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2: gsignal() 2024-03-28T18:18:26.290 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 3: abort() 2024-03-28T18:18:26.290 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff54128b4fb] 2024-03-28T18:18:26.290 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff54128b667] 2024-03-28T18:18:26.290 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c19acd9223] 2024-03-28T18:18:26.290 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c19ab535d6] 2024-03-28T18:18:26.291 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c19ab53da1] 2024-03-28T18:18:26.291 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c19ab562d5] 2024-03-28T18:18:26.291 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c19aba6233] 2024-03-28T18:18:26.291 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c19aba8197] 2024-03-28T18:18:26.291 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c19aaa711b] 2024-03-28T18:18:26.291 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c19aaa76bc] 2024-03-28T18:18:26.291 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c19aa911ff] 2024-03-28T18:18:26.291 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff54151d488] 2024-03-28T18:18:26.291 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 16: (DispatchQueue::entry()+0x51f) [0x7ff54151a60f] 2024-03-28T18:18:26.291 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff5415ede81] 2024-03-28T18:18:26.291 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff5400081ca] 2024-03-28T18:18:26.291 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 19: clone() 2024-03-28T18:18:26.291 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2024-03-28T18:18:25.832+0000 7ff536c61700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:18:26.291 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: in thread 7ff536c61700 thread_name:ms_dispatch 2024-03-28T18:18:26.292 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2024-03-28T18:18:26.292 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:26.292 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff540012d20] 2024-03-28T18:18:26.292 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2: gsignal() 2024-03-28T18:18:26.292 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 3: abort() 2024-03-28T18:18:26.292 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff54128b4fb] 2024-03-28T18:18:26.292 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff54128b667] 2024-03-28T18:18:26.293 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c19acd9223] 2024-03-28T18:18:26.293 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c19ab535d6] 2024-03-28T18:18:26.293 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c19ab53da1] 2024-03-28T18:18:26.293 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c19ab562d5] 2024-03-28T18:18:26.293 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c19aba6233] 2024-03-28T18:18:26.293 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c19aba8197] 2024-03-28T18:18:26.294 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c19aaa711b] 2024-03-28T18:18:26.294 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c19aaa76bc] 2024-03-28T18:18:26.294 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c19aa911ff] 2024-03-28T18:18:26.294 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff54151d488] 2024-03-28T18:18:26.294 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 16: (DispatchQueue::entry()+0x51f) [0x7ff54151a60f] 2024-03-28T18:18:26.294 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff5415ede81] 2024-03-28T18:18:26.294 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff5400081ca] 2024-03-28T18:18:26.294 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 19: clone() 2024-03-28T18:18:26.294 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:18:26.294 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2024-03-28T18:18:26.294 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: -1> 2024-03-28T18:18:25.830+0000 7ff536c61700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff536c61700 time 2024-03-28T18:18:25.829717+0000 2024-03-28T18:18:26.294 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:18:26.294 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2024-03-28T18:18:26.295 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:26.295 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff54128b4a1] 2024-03-28T18:18:26.295 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff54128b667] 2024-03-28T18:18:26.295 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c19acd9223] 2024-03-28T18:18:26.295 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c19ab535d6] 2024-03-28T18:18:26.295 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c19ab53da1] 2024-03-28T18:18:26.295 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c19ab562d5] 2024-03-28T18:18:26.295 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c19aba6233] 2024-03-28T18:18:26.295 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c19aba8197] 2024-03-28T18:18:26.295 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c19aaa711b] 2024-03-28T18:18:26.295 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c19aaa76bc] 2024-03-28T18:18:26.295 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c19aa911ff] 2024-03-28T18:18:26.295 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff54151d488] 2024-03-28T18:18:26.295 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 13: (DispatchQueue::entry()+0x51f) [0x7ff54151a60f] 2024-03-28T18:18:26.296 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff5415ede81] 2024-03-28T18:18:26.296 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff5400081ca] 2024-03-28T18:18:26.296 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 16: clone() 2024-03-28T18:18:26.296 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2024-03-28T18:18:26.296 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 0> 2024-03-28T18:18:25.832+0000 7ff536c61700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:18:26.296 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: in thread 7ff536c61700 thread_name:ms_dispatch 2024-03-28T18:18:26.296 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2024-03-28T18:18:26.296 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:26.296 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff540012d20] 2024-03-28T18:18:26.296 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2: gsignal() 2024-03-28T18:18:26.296 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 3: abort() 2024-03-28T18:18:26.296 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff54128b4fb] 2024-03-28T18:18:26.296 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff54128b667] 2024-03-28T18:18:26.296 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c19acd9223] 2024-03-28T18:18:26.296 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c19ab535d6] 2024-03-28T18:18:26.297 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c19ab53da1] 2024-03-28T18:18:26.297 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c19ab562d5] 2024-03-28T18:18:26.297 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c19aba6233] 2024-03-28T18:18:26.297 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c19aba8197] 2024-03-28T18:18:26.297 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c19aaa711b] 2024-03-28T18:18:26.297 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c19aaa76bc] 2024-03-28T18:18:26.297 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c19aa911ff] 2024-03-28T18:18:26.297 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff54151d488] 2024-03-28T18:18:26.297 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 16: (DispatchQueue::entry()+0x51f) [0x7ff54151a60f] 2024-03-28T18:18:26.297 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff5415ede81] 2024-03-28T18:18:26.297 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff5400081ca] 2024-03-28T18:18:26.297 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 19: clone() 2024-03-28T18:18:26.297 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:18:26.297 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2024-03-28T18:18:26.298 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: -1> 2024-03-28T18:18:25.830+0000 7ff536c61700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff536c61700 time 2024-03-28T18:18:25.829717+0000 2024-03-28T18:18:26.298 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:18:26.298 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2024-03-28T18:18:26.298 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:26.298 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff54128b4a1] 2024-03-28T18:18:26.298 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff54128b667] 2024-03-28T18:18:26.298 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c19acd9223] 2024-03-28T18:18:26.298 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c19ab535d6] 2024-03-28T18:18:26.298 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c19ab53da1] 2024-03-28T18:18:26.298 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c19ab562d5] 2024-03-28T18:18:26.298 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c19aba6233] 2024-03-28T18:18:26.298 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c19aba8197] 2024-03-28T18:18:26.298 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c19aaa711b] 2024-03-28T18:18:26.299 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c19aaa76bc] 2024-03-28T18:18:26.299 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c19aa911ff] 2024-03-28T18:18:26.299 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff54151d488] 2024-03-28T18:18:26.299 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 13: (DispatchQueue::entry()+0x51f) [0x7ff54151a60f] 2024-03-28T18:18:26.299 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff5415ede81] 2024-03-28T18:18:26.299 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff5400081ca] 2024-03-28T18:18:26.299 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 16: clone() 2024-03-28T18:18:26.299 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2024-03-28T18:18:26.299 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 0> 2024-03-28T18:18:25.832+0000 7ff536c61700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:18:26.299 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: in thread 7ff536c61700 thread_name:ms_dispatch 2024-03-28T18:18:26.299 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2024-03-28T18:18:26.299 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:26.299 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff540012d20] 2024-03-28T18:18:26.299 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2: gsignal() 2024-03-28T18:18:26.299 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 3: abort() 2024-03-28T18:18:26.300 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff54128b4fb] 2024-03-28T18:18:26.300 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff54128b667] 2024-03-28T18:18:26.300 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c19acd9223] 2024-03-28T18:18:26.300 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c19ab535d6] 2024-03-28T18:18:26.300 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c19ab53da1] 2024-03-28T18:18:26.300 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c19ab562d5] 2024-03-28T18:18:26.300 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c19aba6233] 2024-03-28T18:18:26.300 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c19aba8197] 2024-03-28T18:18:26.300 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c19aaa711b] 2024-03-28T18:18:26.300 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c19aaa76bc] 2024-03-28T18:18:26.300 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c19aa911ff] 2024-03-28T18:18:26.300 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff54151d488] 2024-03-28T18:18:26.300 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 16: (DispatchQueue::entry()+0x51f) [0x7ff54151a60f] 2024-03-28T18:18:26.300 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff5415ede81] 2024-03-28T18:18:26.301 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff5400081ca] 2024-03-28T18:18:26.301 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 19: clone() 2024-03-28T18:18:26.301 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:18:26.301 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[71976]: 2024-03-28T18:18:26.301 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:25 smithi059 podman[72028]: 2024-03-28 18:18:25.892733612 +0000 UTC m=+0.019575785 container died 8436f838003179d234782abe38ef28f3ac6edd69b19c62ef46a8b6ba5951318c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True) 2024-03-28T18:18:26.810 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:18:26 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:18:26.422+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:18:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:26 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:18:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:26 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:18:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:26 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/2809705173,v1:172.21.15.59:6835/2809705173] up:active 2024-03-28T18:18:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:26 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:26 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:18:27.035 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:26 smithi059 podman[72028]: 2024-03-28 18:18:26.672621809 +0000 UTC m=+0.799463983 container remove 8436f838003179d234782abe38ef28f3ac6edd69b19c62ef46a8b6ba5951318c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-28T18:18:27.035 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:26 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:18:27.035 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:26 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service: Failed with result 'exit-code'. 2024-03-28T18:18:27.078 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:26 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:18:27.078 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:26 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:18:27.078 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:26 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/2809705173,v1:172.21.15.59:6835/2809705173] up:active 2024-03-28T18:18:27.078 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:26 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:27.078 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:26 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:18:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:26 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:18:27.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:26 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:18:27.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:26 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/2809705173,v1:172.21.15.59:6835/2809705173] up:active 2024-03-28T18:18:27.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:26 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:27.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:26 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:18:27.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:27 smithi037 ceph-mon[31460]: pgmap v238: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:18:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:27 smithi059 ceph-mon[36190]: pgmap v238: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:18:28.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:27 smithi170 ceph-mon[36280]: pgmap v238: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:18:30.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:29 smithi037 ceph-mon[31460]: pgmap v239: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:18:30.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:29 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:18:30.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:29 smithi059 ceph-mon[36190]: pgmap v239: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:18:30.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:29 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:18:30.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:29 smithi170 ceph-mon[36280]: pgmap v239: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:18:30.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:29 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:18:31.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:30 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:18:31.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:30 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:18:31.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:30 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:18:31.186 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:30 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:18:31.186 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:30 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:18:31.186 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:30 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:18:31.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:30 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:18:31.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:30 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:18:31.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:30 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:18:32.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:31 smithi037 ceph-mon[31460]: pgmap v240: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:18:32.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:31 smithi059 ceph-mon[36190]: pgmap v240: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:18:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:31 smithi170 ceph-mon[36280]: pgmap v240: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:18:34.087 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:33 smithi170 ceph-mon[36280]: pgmap v241: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-28T18:18:34.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:33 smithi037 ceph-mon[31460]: pgmap v241: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-28T18:18:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:33 smithi059 ceph-mon[36190]: pgmap v241: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-28T18:18:36.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:35 smithi037 ceph-mon[31460]: pgmap v242: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-03-28T18:18:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:35 smithi059 ceph-mon[36190]: pgmap v242: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-03-28T18:18:36.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:35 smithi170 ceph-mon[36280]: pgmap v242: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-03-28T18:18:37.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:36 smithi037 ceph-mon[31460]: osdmap e106: 12 total, 12 up, 12 in 2024-03-28T18:18:37.277 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:36 smithi059 ceph-mon[36190]: osdmap e106: 12 total, 12 up, 12 in 2024-03-28T18:18:37.278 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:37 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service: Scheduled restart job, restart counter is at 3. 2024-03-28T18:18:37.278 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:37 smithi059 systemd[1]: Stopped Ceph mds.e for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:18:37.278 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:37 smithi059 systemd[1]: Starting Ceph mds.e for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:18:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:36 smithi170 ceph-mon[36280]: osdmap e106: 12 total, 12 up, 12 in 2024-03-28T18:18:37.535 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:37 smithi059 podman[72255]: 2024-03-28 18:18:37.181146075 +0000 UTC m=+0.018886832 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:18:37.535 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:37 smithi059 podman[72255]: 2024-03-28 18:18:37.323695096 +0000 UTC m=+0.161435881 container create 40d7de18ba698ea2db9589bd03f9a81f2d8473797bebc6af58b3fa342d534525 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:18:37.857 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:37 smithi059 podman[72255]: 2024-03-28 18:18:37.576450404 +0000 UTC m=+0.414191147 container init 40d7de18ba698ea2db9589bd03f9a81f2d8473797bebc6af58b3fa342d534525 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-28T18:18:37.857 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:37 smithi059 podman[72255]: 2024-03-28 18:18:37.579791766 +0000 UTC m=+0.417532508 container start 40d7de18ba698ea2db9589bd03f9a81f2d8473797bebc6af58b3fa342d534525 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-28T18:18:37.857 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:37 smithi059 ceph-mds[72274]: -- 172.21.15.59:0/4194628575 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5579ace6fa40 con 0x5579ace42800 2024-03-28T18:18:37.857 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:37 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: starting mds.e at 2024-03-28T18:18:37.858 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:37 smithi059 bash[72255]: 40d7de18ba698ea2db9589bd03f9a81f2d8473797bebc6af58b3fa342d534525 2024-03-28T18:18:37.858 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:37 smithi059 systemd[1]: Started Ceph mds.e for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:18:38.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:37 smithi037 ceph-mon[31460]: pgmap v244: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:18:38.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:37 smithi037 ceph-mon[31460]: daemon mds.e restarted 2024-03-28T18:18:38.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:37 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:18:38.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:37 smithi037 ceph-mon[31460]: osdmap e107: 12 total, 12 up, 12 in 2024-03-28T18:18:38.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:37 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/3462024636,v1:172.21.15.59:6835/3462024636] up:boot 2024-03-28T18:18:38.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:37 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:18:38.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:37 smithi037 ceph-mon[31460]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-28T18:18:38.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:37 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:18:38.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:37 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:38.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:37 smithi059 ceph-mon[36190]: pgmap v244: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:18:38.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:37 smithi059 ceph-mon[36190]: daemon mds.e restarted 2024-03-28T18:18:38.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:37 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:18:38.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:37 smithi059 ceph-mon[36190]: osdmap e107: 12 total, 12 up, 12 in 2024-03-28T18:18:38.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:37 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/3462024636,v1:172.21.15.59:6835/3462024636] up:boot 2024-03-28T18:18:38.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:37 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:18:38.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:37 smithi059 ceph-mon[36190]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-28T18:18:38.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:37 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:18:38.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:37 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:38.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:37 smithi170 ceph-mon[36280]: pgmap v244: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:18:38.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:37 smithi170 ceph-mon[36280]: daemon mds.e restarted 2024-03-28T18:18:38.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:37 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:18:38.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:37 smithi170 ceph-mon[36280]: osdmap e107: 12 total, 12 up, 12 in 2024-03-28T18:18:38.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:37 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/3462024636,v1:172.21.15.59:6835/3462024636] up:boot 2024-03-28T18:18:38.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:37 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:18:38.288 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:37 smithi170 ceph-mon[36280]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-28T18:18:38.288 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:37 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:18:38.288 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:37 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:39 smithi059 ceph-mon[36190]: pgmap v246: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-28T18:18:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:39 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/576965351,v1:172.21.15.170:6833/576965351] up:resolve 2024-03-28T18:18:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:39 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:40.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:39 smithi170 ceph-mon[36280]: pgmap v246: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-28T18:18:40.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:39 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/576965351,v1:172.21.15.170:6833/576965351] up:resolve 2024-03-28T18:18:40.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:39 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:40.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:39 smithi037 ceph-mon[31460]: pgmap v246: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-28T18:18:40.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:39 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/576965351,v1:172.21.15.170:6833/576965351] up:resolve 2024-03-28T18:18:40.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:39 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:40 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199995 2024-03-28T18:18:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:40 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/576965351,v1:172.21.15.170:6833/576965351] up:reconnect 2024-03-28T18:18:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:40 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:40 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999978 2024-03-28T18:18:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:40 smithi059 ceph-mon[36190]: pgmap v247: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T18:18:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:40 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199995 2024-03-28T18:18:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:40 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/576965351,v1:172.21.15.170:6833/576965351] up:reconnect 2024-03-28T18:18:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:40 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:40 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999978 2024-03-28T18:18:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:40 smithi170 ceph-mon[36280]: pgmap v247: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T18:18:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:40 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199995 2024-03-28T18:18:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:40 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/576965351,v1:172.21.15.170:6833/576965351] up:reconnect 2024-03-28T18:18:41.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:40 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:41.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:40 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999978 2024-03-28T18:18:41.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:40 smithi037 ceph-mon[31460]: pgmap v247: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T18:18:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:41 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/576965351,v1:172.21.15.170:6833/576965351] up:rejoin 2024-03-28T18:18:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:41 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:42.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:41 smithi059 ceph-mon[36190]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-28T18:18:42.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/576965351,v1:172.21.15.170:6833/576965351] up:rejoin 2024-03-28T18:18:42.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:42.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-mon[36280]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-28T18:18:42.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa05d44b700 time 2024-03-28T18:18:41.930653+0000 2024-03-28T18:18:42.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:18:42.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:42.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa067a754a1] 2024-03-28T18:18:42.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa067a75667] 2024-03-28T18:18:42.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557c96dfa223] 2024-03-28T18:18:42.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557c96c745d6] 2024-03-28T18:18:42.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557c96c74da1] 2024-03-28T18:18:42.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557c96c772d5] 2024-03-28T18:18:42.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557c96cc7233] 2024-03-28T18:18:42.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557c96cc9197] 2024-03-28T18:18:42.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557c96bc811b] 2024-03-28T18:18:42.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557c96bc86bc] 2024-03-28T18:18:42.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557c96bb21ff] 2024-03-28T18:18:42.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa067d07488] 2024-03-28T18:18:42.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 13: (DispatchQueue::entry()+0x51f) [0x7fa067d0460f] 2024-03-28T18:18:42.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa067dd7e81] 2024-03-28T18:18:42.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa0667f21ca] 2024-03-28T18:18:42.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 16: clone() 2024-03-28T18:18:42.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2024-03-28T18:18:41.932+0000 7fa05d44b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa05d44b700 time 2024-03-28T18:18:41.930653+0000 2024-03-28T18:18:42.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:18:42.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2024-03-28T18:18:42.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:42.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa067a754a1] 2024-03-28T18:18:42.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa067a75667] 2024-03-28T18:18:42.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557c96dfa223] 2024-03-28T18:18:42.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557c96c745d6] 2024-03-28T18:18:42.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557c96c74da1] 2024-03-28T18:18:42.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557c96c772d5] 2024-03-28T18:18:42.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557c96cc7233] 2024-03-28T18:18:42.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557c96cc9197] 2024-03-28T18:18:42.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557c96bc811b] 2024-03-28T18:18:42.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557c96bc86bc] 2024-03-28T18:18:42.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557c96bb21ff] 2024-03-28T18:18:42.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa067d07488] 2024-03-28T18:18:42.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 13: (DispatchQueue::entry()+0x51f) [0x7fa067d0460f] 2024-03-28T18:18:42.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa067dd7e81] 2024-03-28T18:18:42.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa0667f21ca] 2024-03-28T18:18:42.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 16: clone() 2024-03-28T18:18:42.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2024-03-28T18:18:42.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: *** Caught signal (Aborted) ** 2024-03-28T18:18:42.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: in thread 7fa05d44b700 thread_name:ms_dispatch 2024-03-28T18:18:42.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:42.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa0667fcd20] 2024-03-28T18:18:42.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2: gsignal() 2024-03-28T18:18:42.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 3: abort() 2024-03-28T18:18:42.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa067a754fb] 2024-03-28T18:18:42.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa067a75667] 2024-03-28T18:18:42.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557c96dfa223] 2024-03-28T18:18:42.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557c96c745d6] 2024-03-28T18:18:42.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557c96c74da1] 2024-03-28T18:18:42.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557c96c772d5] 2024-03-28T18:18:42.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557c96cc7233] 2024-03-28T18:18:42.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557c96cc9197] 2024-03-28T18:18:42.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557c96bc811b] 2024-03-28T18:18:42.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557c96bc86bc] 2024-03-28T18:18:42.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557c96bb21ff] 2024-03-28T18:18:42.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa067d07488] 2024-03-28T18:18:42.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 16: (DispatchQueue::entry()+0x51f) [0x7fa067d0460f] 2024-03-28T18:18:42.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa067dd7e81] 2024-03-28T18:18:42.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa0667f21ca] 2024-03-28T18:18:42.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 19: clone() 2024-03-28T18:18:42.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2024-03-28T18:18:41.934+0000 7fa05d44b700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:18:42.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: in thread 7fa05d44b700 thread_name:ms_dispatch 2024-03-28T18:18:42.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2024-03-28T18:18:42.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:42.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa0667fcd20] 2024-03-28T18:18:42.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2: gsignal() 2024-03-28T18:18:42.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 3: abort() 2024-03-28T18:18:42.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa067a754fb] 2024-03-28T18:18:42.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa067a75667] 2024-03-28T18:18:42.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557c96dfa223] 2024-03-28T18:18:42.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557c96c745d6] 2024-03-28T18:18:42.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557c96c74da1] 2024-03-28T18:18:42.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557c96c772d5] 2024-03-28T18:18:42.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557c96cc7233] 2024-03-28T18:18:42.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557c96cc9197] 2024-03-28T18:18:42.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557c96bc811b] 2024-03-28T18:18:42.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557c96bc86bc] 2024-03-28T18:18:42.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557c96bb21ff] 2024-03-28T18:18:42.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa067d07488] 2024-03-28T18:18:42.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 16: (DispatchQueue::entry()+0x51f) [0x7fa067d0460f] 2024-03-28T18:18:42.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa067dd7e81] 2024-03-28T18:18:42.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa0667f21ca] 2024-03-28T18:18:42.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 19: clone() 2024-03-28T18:18:42.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:18:42.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2024-03-28T18:18:42.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: -1> 2024-03-28T18:18:41.932+0000 7fa05d44b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa05d44b700 time 2024-03-28T18:18:41.930653+0000 2024-03-28T18:18:42.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:18:42.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2024-03-28T18:18:42.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:42.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa067a754a1] 2024-03-28T18:18:42.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa067a75667] 2024-03-28T18:18:42.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557c96dfa223] 2024-03-28T18:18:42.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557c96c745d6] 2024-03-28T18:18:42.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557c96c74da1] 2024-03-28T18:18:42.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557c96c772d5] 2024-03-28T18:18:42.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557c96cc7233] 2024-03-28T18:18:42.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557c96cc9197] 2024-03-28T18:18:42.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557c96bc811b] 2024-03-28T18:18:42.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557c96bc86bc] 2024-03-28T18:18:42.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557c96bb21ff] 2024-03-28T18:18:42.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa067d07488] 2024-03-28T18:18:42.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 13: (DispatchQueue::entry()+0x51f) [0x7fa067d0460f] 2024-03-28T18:18:42.300 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa067dd7e81] 2024-03-28T18:18:42.300 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa0667f21ca] 2024-03-28T18:18:42.300 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 16: clone() 2024-03-28T18:18:42.300 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2024-03-28T18:18:42.300 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 0> 2024-03-28T18:18:41.934+0000 7fa05d44b700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:18:42.300 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: in thread 7fa05d44b700 thread_name:ms_dispatch 2024-03-28T18:18:42.300 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2024-03-28T18:18:42.300 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:42.300 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa0667fcd20] 2024-03-28T18:18:42.300 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2: gsignal() 2024-03-28T18:18:42.300 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 3: abort() 2024-03-28T18:18:42.301 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa067a754fb] 2024-03-28T18:18:42.301 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa067a75667] 2024-03-28T18:18:42.301 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557c96dfa223] 2024-03-28T18:18:42.301 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557c96c745d6] 2024-03-28T18:18:42.301 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557c96c74da1] 2024-03-28T18:18:42.301 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557c96c772d5] 2024-03-28T18:18:42.301 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557c96cc7233] 2024-03-28T18:18:42.301 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557c96cc9197] 2024-03-28T18:18:42.301 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557c96bc811b] 2024-03-28T18:18:42.301 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557c96bc86bc] 2024-03-28T18:18:42.302 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557c96bb21ff] 2024-03-28T18:18:42.302 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa067d07488] 2024-03-28T18:18:42.302 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 16: (DispatchQueue::entry()+0x51f) [0x7fa067d0460f] 2024-03-28T18:18:42.302 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa067dd7e81] 2024-03-28T18:18:42.302 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa0667f21ca] 2024-03-28T18:18:42.302 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 19: clone() 2024-03-28T18:18:42.302 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:18:42.302 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2024-03-28T18:18:42.302 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: -1> 2024-03-28T18:18:41.932+0000 7fa05d44b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa05d44b700 time 2024-03-28T18:18:41.930653+0000 2024-03-28T18:18:42.302 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:18:42.303 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2024-03-28T18:18:42.303 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:42.303 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa067a754a1] 2024-03-28T18:18:42.303 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa067a75667] 2024-03-28T18:18:42.303 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557c96dfa223] 2024-03-28T18:18:42.303 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557c96c745d6] 2024-03-28T18:18:42.303 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557c96c74da1] 2024-03-28T18:18:42.303 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557c96c772d5] 2024-03-28T18:18:42.303 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557c96cc7233] 2024-03-28T18:18:42.304 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557c96cc9197] 2024-03-28T18:18:42.304 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557c96bc811b] 2024-03-28T18:18:42.304 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557c96bc86bc] 2024-03-28T18:18:42.304 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557c96bb21ff] 2024-03-28T18:18:42.304 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa067d07488] 2024-03-28T18:18:42.304 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 13: (DispatchQueue::entry()+0x51f) [0x7fa067d0460f] 2024-03-28T18:18:42.304 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa067dd7e81] 2024-03-28T18:18:42.304 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa0667f21ca] 2024-03-28T18:18:42.304 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 16: clone() 2024-03-28T18:18:42.304 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2024-03-28T18:18:42.304 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 0> 2024-03-28T18:18:41.934+0000 7fa05d44b700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:18:42.304 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: in thread 7fa05d44b700 thread_name:ms_dispatch 2024-03-28T18:18:42.305 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2024-03-28T18:18:42.305 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:42.305 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa0667fcd20] 2024-03-28T18:18:42.305 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2: gsignal() 2024-03-28T18:18:42.305 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 3: abort() 2024-03-28T18:18:42.305 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa067a754fb] 2024-03-28T18:18:42.305 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa067a75667] 2024-03-28T18:18:42.305 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557c96dfa223] 2024-03-28T18:18:42.305 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557c96c745d6] 2024-03-28T18:18:42.305 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557c96c74da1] 2024-03-28T18:18:42.305 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557c96c772d5] 2024-03-28T18:18:42.306 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557c96cc7233] 2024-03-28T18:18:42.306 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557c96cc9197] 2024-03-28T18:18:42.306 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557c96bc811b] 2024-03-28T18:18:42.306 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557c96bc86bc] 2024-03-28T18:18:42.306 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557c96bb21ff] 2024-03-28T18:18:42.306 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa067d07488] 2024-03-28T18:18:42.306 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 16: (DispatchQueue::entry()+0x51f) [0x7fa067d0460f] 2024-03-28T18:18:42.306 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa067dd7e81] 2024-03-28T18:18:42.306 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa0667f21ca] 2024-03-28T18:18:42.306 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 19: clone() 2024-03-28T18:18:42.306 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:18:42.307 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:41 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[65725]: 2024-03-28T18:18:42.307 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:42 smithi170 podman[71905]: 2024-03-28 18:18:42.00215566 +0000 UTC m=+0.022144552 container died 8a70934209e5848cca87708a2602393becd1f8addba647cc9bcd632a46ee8c92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-28T18:18:42.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:41 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/576965351,v1:172.21.15.170:6833/576965351] up:rejoin 2024-03-28T18:18:42.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:41 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:42.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:41 smithi037 ceph-mon[31460]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-28T18:18:42.913 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:42 smithi170 podman[71905]: 2024-03-28 18:18:42.627775423 +0000 UTC m=+0.647764297 container remove 8a70934209e5848cca87708a2602393becd1f8addba647cc9bcd632a46ee8c92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-28T18:18:42.913 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:42 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:18:42.913 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:42 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-28T18:18:42.950 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:42 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:18:42.950 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:42 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:18:42.950 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:42 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/576965351,v1:172.21.15.170:6833/576965351] up:active 2024-03-28T18:18:42.951 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:42 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:42.951 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:42 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:18:42.951 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:42 smithi037 ceph-mon[31460]: pgmap v248: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T18:18:43.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:42 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:18:43.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:42 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:18:43.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:42 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/576965351,v1:172.21.15.170:6833/576965351] up:active 2024-03-28T18:18:43.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:42 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:43.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:42 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:18:43.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:42 smithi059 ceph-mon[36190]: pgmap v248: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T18:18:43.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:42 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:18:43.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:42 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:18:43.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:42 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/576965351,v1:172.21.15.170:6833/576965351] up:active 2024-03-28T18:18:43.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:42 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:43.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:42 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:18:43.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:42 smithi170 ceph-mon[36280]: pgmap v248: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T18:18:45.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:44 smithi059 ceph-mon[36190]: osdmap e108: 12 total, 12 up, 12 in 2024-03-28T18:18:45.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:44 smithi170 ceph-mon[36280]: osdmap e108: 12 total, 12 up, 12 in 2024-03-28T18:18:45.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:44 smithi037 ceph-mon[31460]: osdmap e108: 12 total, 12 up, 12 in 2024-03-28T18:18:46.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:45 smithi059 ceph-mon[36190]: pgmap v250: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 383 B/s wr, 4 op/s 2024-03-28T18:18:46.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:45 smithi170 ceph-mon[36280]: pgmap v250: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 383 B/s wr, 4 op/s 2024-03-28T18:18:46.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:45 smithi037 ceph-mon[31460]: pgmap v250: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 383 B/s wr, 4 op/s 2024-03-28T18:18:46.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:18:46 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:18:46.423+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:18:48.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:47 smithi059 ceph-mon[36190]: pgmap v251: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 344 B/s wr, 18 op/s 2024-03-28T18:18:48.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:47 smithi170 ceph-mon[36280]: pgmap v251: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 344 B/s wr, 18 op/s 2024-03-28T18:18:48.089 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:47 smithi037 ceph-mon[31460]: pgmap v251: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 344 B/s wr, 18 op/s 2024-03-28T18:18:50.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:49 smithi059 ceph-mon[36190]: pgmap v252: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 307 B/s wr, 16 op/s 2024-03-28T18:18:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:49 smithi170 ceph-mon[36280]: pgmap v252: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 307 B/s wr, 16 op/s 2024-03-28T18:18:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:49 smithi037 ceph-mon[31460]: pgmap v252: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 307 B/s wr, 16 op/s 2024-03-28T18:18:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:51 smithi059 ceph-mon[36190]: pgmap v253: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 307 B/s wr, 16 op/s 2024-03-28T18:18:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:51 smithi170 ceph-mon[36280]: pgmap v253: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 307 B/s wr, 16 op/s 2024-03-28T18:18:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:51 smithi037 ceph-mon[31460]: pgmap v253: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 307 B/s wr, 16 op/s 2024-03-28T18:18:53.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:53 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 1. 2024-03-28T18:18:53.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:53 smithi170 systemd[1]: Stopped Ceph mds.l for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:18:53.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:53 smithi170 systemd[1]: Starting Ceph mds.l for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:18:53.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:53 smithi170 podman[71999]: 2024-03-28 18:18:53.185901917 +0000 UTC m=+0.023270746 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:18:53.581 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:53 smithi170 podman[71999]: 2024-03-28 18:18:53.295028601 +0000 UTC m=+0.132397424 container create e4a7ca1cc8b09533c060cb335f487728ac944b8413d9644dc6546aca172ee27e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True) 2024-03-28T18:18:53.581 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:53 smithi170 podman[71999]: 2024-03-28 18:18:53.506660602 +0000 UTC m=+0.344029440 container init e4a7ca1cc8b09533c060cb335f487728ac944b8413d9644dc6546aca172ee27e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-28T18:18:53.581 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:53 smithi170 podman[71999]: 2024-03-28 18:18:53.510132443 +0000 UTC m=+0.347501282 container start e4a7ca1cc8b09533c060cb335f487728ac944b8413d9644dc6546aca172ee27e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-28T18:18:53.582 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:53 smithi170 ceph-mds[72017]: -- 172.21.15.170:0/3168413898 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555869c17a40 con 0x555869bea800 2024-03-28T18:18:53.582 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:53 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: starting mds.l at 2024-03-28T18:18:54.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:53 smithi059 ceph-mon[36190]: pgmap v254: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 16 op/s 2024-03-28T18:18:54.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:53 smithi059 ceph-mon[36190]: daemon mds.l restarted 2024-03-28T18:18:54.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:53 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:18:54.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:53 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:18:54.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:53 smithi059 ceph-mon[36190]: osdmap e109: 12 total, 12 up, 12 in 2024-03-28T18:18:54.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:53 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/3058605036,v1:172.21.15.170:6833/3058605036] up:boot 2024-03-28T18:18:54.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:53 smithi059 ceph-mon[36190]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-03-28T18:18:54.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:53 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:18:54.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:53 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:54.037 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:53 smithi170 bash[71999]: e4a7ca1cc8b09533c060cb335f487728ac944b8413d9644dc6546aca172ee27e 2024-03-28T18:18:54.037 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:18:53 smithi170 systemd[1]: Started Ceph mds.l for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:18:54.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:53 smithi170 ceph-mon[36280]: pgmap v254: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 16 op/s 2024-03-28T18:18:54.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:53 smithi170 ceph-mon[36280]: daemon mds.l restarted 2024-03-28T18:18:54.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:53 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:18:54.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:53 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:18:54.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:53 smithi170 ceph-mon[36280]: osdmap e109: 12 total, 12 up, 12 in 2024-03-28T18:18:54.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:53 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/3058605036,v1:172.21.15.170:6833/3058605036] up:boot 2024-03-28T18:18:54.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:53 smithi170 ceph-mon[36280]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-03-28T18:18:54.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:53 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:18:54.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:53 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:53 smithi037 ceph-mon[31460]: pgmap v254: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 16 op/s 2024-03-28T18:18:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:53 smithi037 ceph-mon[31460]: daemon mds.l restarted 2024-03-28T18:18:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:53 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:18:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:53 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:18:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:53 smithi037 ceph-mon[31460]: osdmap e109: 12 total, 12 up, 12 in 2024-03-28T18:18:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:53 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/3058605036,v1:172.21.15.170:6833/3058605036] up:boot 2024-03-28T18:18:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:53 smithi037 ceph-mon[31460]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-03-28T18:18:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:53 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:18:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:53 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:55 smithi170 ceph-mon[36280]: pgmap v256: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 16 op/s 2024-03-28T18:18:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:55 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/3462024636,v1:172.21.15.59:6835/3462024636] up:resolve 2024-03-28T18:18:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:55 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:56.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:55 smithi037 ceph-mon[31460]: pgmap v256: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 16 op/s 2024-03-28T18:18:56.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:55 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/3462024636,v1:172.21.15.59:6835/3462024636] up:resolve 2024-03-28T18:18:56.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:55 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:55 smithi059 ceph-mon[36190]: pgmap v256: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 16 op/s 2024-03-28T18:18:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:55 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/3462024636,v1:172.21.15.59:6835/3462024636] up:resolve 2024-03-28T18:18:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:55 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:56 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/3462024636,v1:172.21.15.59:6835/3462024636] up:reconnect 2024-03-28T18:18:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:56 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:56 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089999 2024-03-28T18:18:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:56 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.0089999 2024-03-28T18:18:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:56 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/3462024636,v1:172.21.15.59:6835/3462024636] up:reconnect 2024-03-28T18:18:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:56 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:56 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089999 2024-03-28T18:18:57.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:56 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.0089999 2024-03-28T18:18:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:56 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/3462024636,v1:172.21.15.59:6835/3462024636] up:reconnect 2024-03-28T18:18:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:56 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:56 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089999 2024-03-28T18:18:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:56 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.0089999 2024-03-28T18:18:58.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:57 smithi037 ceph-mon[31460]: pgmap v257: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-03-28T18:18:58.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:57 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/3462024636,v1:172.21.15.59:6835/3462024636] up:rejoin 2024-03-28T18:18:58.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:57 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:57 smithi037 ceph-mon[31460]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-03-28T18:18:58.195 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-mon[36190]: pgmap v257: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-03-28T18:18:58.196 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/3462024636,v1:172.21.15.59:6835/3462024636] up:rejoin 2024-03-28T18:18:58.196 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:58.196 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-mon[36190]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-03-28T18:18:58.196 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fdba8206700 time 2024-03-28T18:18:57.836556+0000 2024-03-28T18:18:58.196 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:18:58.196 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:58.196 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fdbb28304a1] 2024-03-28T18:18:58.196 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdbb2830667] 2024-03-28T18:18:58.196 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5579ab97c223] 2024-03-28T18:18:58.197 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5579ab7f65d6] 2024-03-28T18:18:58.197 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5579ab7f6da1] 2024-03-28T18:18:58.197 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5579ab7f92d5] 2024-03-28T18:18:58.197 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5579ab849233] 2024-03-28T18:18:58.197 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5579ab84b197] 2024-03-28T18:18:58.197 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5579ab74a11b] 2024-03-28T18:18:58.197 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5579ab74a6bc] 2024-03-28T18:18:58.197 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5579ab7341ff] 2024-03-28T18:18:58.197 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdbb2ac2488] 2024-03-28T18:18:58.197 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 13: (DispatchQueue::entry()+0x51f) [0x7fdbb2abf60f] 2024-03-28T18:18:58.197 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdbb2b92e81] 2024-03-28T18:18:58.197 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fdbb15ad1ca] 2024-03-28T18:18:58.197 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 16: clone() 2024-03-28T18:18:58.198 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2024-03-28T18:18:57.837+0000 7fdba8206700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fdba8206700 time 2024-03-28T18:18:57.836556+0000 2024-03-28T18:18:58.198 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:18:58.198 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2024-03-28T18:18:58.198 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:58.199 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fdbb28304a1] 2024-03-28T18:18:58.199 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdbb2830667] 2024-03-28T18:18:58.199 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5579ab97c223] 2024-03-28T18:18:58.199 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5579ab7f65d6] 2024-03-28T18:18:58.199 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5579ab7f6da1] 2024-03-28T18:18:58.199 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5579ab7f92d5] 2024-03-28T18:18:58.199 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5579ab849233] 2024-03-28T18:18:58.199 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5579ab84b197] 2024-03-28T18:18:58.199 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5579ab74a11b] 2024-03-28T18:18:58.199 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5579ab74a6bc] 2024-03-28T18:18:58.199 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5579ab7341ff] 2024-03-28T18:18:58.199 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdbb2ac2488] 2024-03-28T18:18:58.199 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 13: (DispatchQueue::entry()+0x51f) [0x7fdbb2abf60f] 2024-03-28T18:18:58.200 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdbb2b92e81] 2024-03-28T18:18:58.200 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fdbb15ad1ca] 2024-03-28T18:18:58.200 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 16: clone() 2024-03-28T18:18:58.200 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2024-03-28T18:18:58.200 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: *** Caught signal (Aborted) ** 2024-03-28T18:18:58.200 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: in thread 7fdba8206700 thread_name:ms_dispatch 2024-03-28T18:18:58.200 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:58.200 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fdbb15b7d20] 2024-03-28T18:18:58.200 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2: gsignal() 2024-03-28T18:18:58.200 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 3: abort() 2024-03-28T18:18:58.200 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fdbb28304fb] 2024-03-28T18:18:58.200 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdbb2830667] 2024-03-28T18:18:58.200 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5579ab97c223] 2024-03-28T18:18:58.200 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5579ab7f65d6] 2024-03-28T18:18:58.200 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5579ab7f6da1] 2024-03-28T18:18:58.201 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5579ab7f92d5] 2024-03-28T18:18:58.201 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5579ab849233] 2024-03-28T18:18:58.201 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5579ab84b197] 2024-03-28T18:18:58.201 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5579ab74a11b] 2024-03-28T18:18:58.201 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5579ab74a6bc] 2024-03-28T18:18:58.201 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5579ab7341ff] 2024-03-28T18:18:58.201 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdbb2ac2488] 2024-03-28T18:18:58.201 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 16: (DispatchQueue::entry()+0x51f) [0x7fdbb2abf60f] 2024-03-28T18:18:58.201 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdbb2b92e81] 2024-03-28T18:18:58.201 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fdbb15ad1ca] 2024-03-28T18:18:58.201 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 19: clone() 2024-03-28T18:18:58.201 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2024-03-28T18:18:57.840+0000 7fdba8206700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:18:58.201 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: in thread 7fdba8206700 thread_name:ms_dispatch 2024-03-28T18:18:58.201 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2024-03-28T18:18:58.202 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:58.202 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fdbb15b7d20] 2024-03-28T18:18:58.202 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2: gsignal() 2024-03-28T18:18:58.202 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 3: abort() 2024-03-28T18:18:58.202 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fdbb28304fb] 2024-03-28T18:18:58.202 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdbb2830667] 2024-03-28T18:18:58.202 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5579ab97c223] 2024-03-28T18:18:58.202 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5579ab7f65d6] 2024-03-28T18:18:58.202 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5579ab7f6da1] 2024-03-28T18:18:58.202 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5579ab7f92d5] 2024-03-28T18:18:58.202 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5579ab849233] 2024-03-28T18:18:58.202 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5579ab84b197] 2024-03-28T18:18:58.202 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5579ab74a11b] 2024-03-28T18:18:58.202 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5579ab74a6bc] 2024-03-28T18:18:58.203 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5579ab7341ff] 2024-03-28T18:18:58.203 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdbb2ac2488] 2024-03-28T18:18:58.203 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 16: (DispatchQueue::entry()+0x51f) [0x7fdbb2abf60f] 2024-03-28T18:18:58.203 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdbb2b92e81] 2024-03-28T18:18:58.203 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fdbb15ad1ca] 2024-03-28T18:18:58.203 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 19: clone() 2024-03-28T18:18:58.203 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:18:58.203 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2024-03-28T18:18:58.203 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: -1> 2024-03-28T18:18:57.837+0000 7fdba8206700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fdba8206700 time 2024-03-28T18:18:57.836556+0000 2024-03-28T18:18:58.203 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:18:58.203 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2024-03-28T18:18:58.203 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:58.203 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fdbb28304a1] 2024-03-28T18:18:58.203 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdbb2830667] 2024-03-28T18:18:58.203 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5579ab97c223] 2024-03-28T18:18:58.203 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5579ab7f65d6] 2024-03-28T18:18:58.204 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5579ab7f6da1] 2024-03-28T18:18:58.204 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5579ab7f92d5] 2024-03-28T18:18:58.204 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5579ab849233] 2024-03-28T18:18:58.204 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5579ab84b197] 2024-03-28T18:18:58.204 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5579ab74a11b] 2024-03-28T18:18:58.204 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5579ab74a6bc] 2024-03-28T18:18:58.204 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5579ab7341ff] 2024-03-28T18:18:58.204 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdbb2ac2488] 2024-03-28T18:18:58.204 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 13: (DispatchQueue::entry()+0x51f) [0x7fdbb2abf60f] 2024-03-28T18:18:58.204 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdbb2b92e81] 2024-03-28T18:18:58.204 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fdbb15ad1ca] 2024-03-28T18:18:58.204 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 16: clone() 2024-03-28T18:18:58.204 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2024-03-28T18:18:58.204 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 0> 2024-03-28T18:18:57.840+0000 7fdba8206700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:18:58.204 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: in thread 7fdba8206700 thread_name:ms_dispatch 2024-03-28T18:18:58.204 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2024-03-28T18:18:58.205 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:58.205 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fdbb15b7d20] 2024-03-28T18:18:58.205 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2: gsignal() 2024-03-28T18:18:58.205 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 3: abort() 2024-03-28T18:18:58.205 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fdbb28304fb] 2024-03-28T18:18:58.205 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdbb2830667] 2024-03-28T18:18:58.205 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5579ab97c223] 2024-03-28T18:18:58.205 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5579ab7f65d6] 2024-03-28T18:18:58.206 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5579ab7f6da1] 2024-03-28T18:18:58.206 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5579ab7f92d5] 2024-03-28T18:18:58.206 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5579ab849233] 2024-03-28T18:18:58.206 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5579ab84b197] 2024-03-28T18:18:58.206 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5579ab74a11b] 2024-03-28T18:18:58.206 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5579ab74a6bc] 2024-03-28T18:18:58.206 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5579ab7341ff] 2024-03-28T18:18:58.206 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdbb2ac2488] 2024-03-28T18:18:58.206 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 16: (DispatchQueue::entry()+0x51f) [0x7fdbb2abf60f] 2024-03-28T18:18:58.206 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdbb2b92e81] 2024-03-28T18:18:58.223 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fdbb15ad1ca] 2024-03-28T18:18:58.223 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 19: clone() 2024-03-28T18:18:58.223 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:18:58.223 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2024-03-28T18:18:58.223 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: -1> 2024-03-28T18:18:57.837+0000 7fdba8206700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fdba8206700 time 2024-03-28T18:18:57.836556+0000 2024-03-28T18:18:58.223 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:18:58.223 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2024-03-28T18:18:58.223 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:58.223 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fdbb28304a1] 2024-03-28T18:18:58.223 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdbb2830667] 2024-03-28T18:18:58.223 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5579ab97c223] 2024-03-28T18:18:58.223 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5579ab7f65d6] 2024-03-28T18:18:58.223 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5579ab7f6da1] 2024-03-28T18:18:58.224 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5579ab7f92d5] 2024-03-28T18:18:58.224 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5579ab849233] 2024-03-28T18:18:58.224 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5579ab84b197] 2024-03-28T18:18:58.224 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5579ab74a11b] 2024-03-28T18:18:58.224 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5579ab74a6bc] 2024-03-28T18:18:58.224 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5579ab7341ff] 2024-03-28T18:18:58.224 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdbb2ac2488] 2024-03-28T18:18:58.224 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 13: (DispatchQueue::entry()+0x51f) [0x7fdbb2abf60f] 2024-03-28T18:18:58.224 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdbb2b92e81] 2024-03-28T18:18:58.224 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fdbb15ad1ca] 2024-03-28T18:18:58.224 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 16: clone() 2024-03-28T18:18:58.224 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2024-03-28T18:18:58.224 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 0> 2024-03-28T18:18:57.840+0000 7fdba8206700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:18:58.224 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: in thread 7fdba8206700 thread_name:ms_dispatch 2024-03-28T18:18:58.224 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2024-03-28T18:18:58.224 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:18:58.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fdbb15b7d20] 2024-03-28T18:18:58.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2: gsignal() 2024-03-28T18:18:58.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 3: abort() 2024-03-28T18:18:58.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fdbb28304fb] 2024-03-28T18:18:58.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdbb2830667] 2024-03-28T18:18:58.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5579ab97c223] 2024-03-28T18:18:58.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5579ab7f65d6] 2024-03-28T18:18:58.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5579ab7f6da1] 2024-03-28T18:18:58.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5579ab7f92d5] 2024-03-28T18:18:58.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5579ab849233] 2024-03-28T18:18:58.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5579ab84b197] 2024-03-28T18:18:58.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5579ab74a11b] 2024-03-28T18:18:58.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5579ab74a6bc] 2024-03-28T18:18:58.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5579ab7341ff] 2024-03-28T18:18:58.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdbb2ac2488] 2024-03-28T18:18:58.226 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 16: (DispatchQueue::entry()+0x51f) [0x7fdbb2abf60f] 2024-03-28T18:18:58.226 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdbb2b92e81] 2024-03-28T18:18:58.226 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fdbb15ad1ca] 2024-03-28T18:18:58.226 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 19: clone() 2024-03-28T18:18:58.226 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:18:58.226 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72270]: 2024-03-28T18:18:58.226 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:57 smithi059 podman[72308]: 2024-03-28 18:18:57.925695391 +0000 UTC m=+0.019446261 container died 40d7de18ba698ea2db9589bd03f9a81f2d8473797bebc6af58b3fa342d534525 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-28T18:18:58.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:57 smithi170 ceph-mon[36280]: pgmap v257: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-03-28T18:18:58.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:57 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/3462024636,v1:172.21.15.59:6835/3462024636] up:rejoin 2024-03-28T18:18:58.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:57 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:58.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:57 smithi170 ceph-mon[36280]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-03-28T18:18:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:58 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:18:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:58 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:18:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:58 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/3462024636,v1:172.21.15.59:6835/3462024636] up:active 2024-03-28T18:18:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:58 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:58 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:18:59.035 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:58 smithi059 podman[72308]: 2024-03-28 18:18:58.781915944 +0000 UTC m=+0.875666811 container remove 40d7de18ba698ea2db9589bd03f9a81f2d8473797bebc6af58b3fa342d534525 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-28T18:18:59.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:58 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:18:59.036 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:18:59 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service: Failed with result 'exit-code'. 2024-03-28T18:18:59.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:58 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:18:59.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:58 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:18:59.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:58 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/3462024636,v1:172.21.15.59:6835/3462024636] up:active 2024-03-28T18:18:59.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:58 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:59.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:58 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:18:59.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:58 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:18:59.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:58 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:18:59.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:58 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/3462024636,v1:172.21.15.59:6835/3462024636] up:active 2024-03-28T18:18:59.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:58 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:18:59.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:58 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:18:59.846 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:18:59 smithi037 ceph-mon[31460]: pgmap v258: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-03-28T18:19:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:18:59 smithi059 ceph-mon[36190]: pgmap v258: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-03-28T18:19:00.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:18:59 smithi170 ceph-mon[36280]: pgmap v258: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-03-28T18:19:01.826 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:19:01 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:19:01.424+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:19:02.087 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:01 smithi059 ceph-mon[36190]: pgmap v259: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-03-28T18:19:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:01 smithi037 ceph-mon[31460]: pgmap v259: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-03-28T18:19:02.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:01 smithi170 ceph-mon[36280]: pgmap v259: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-03-28T18:19:04.088 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:03 smithi170 ceph-mon[36280]: pgmap v260: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T18:19:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:03 smithi037 ceph-mon[31460]: pgmap v260: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T18:19:04.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:03 smithi059 ceph-mon[36190]: pgmap v260: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T18:19:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:05 smithi037 ceph-mon[31460]: pgmap v261: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 186 B/s wr, 17 op/s 2024-03-28T18:19:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:05 smithi059 ceph-mon[36190]: pgmap v261: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 186 B/s wr, 17 op/s 2024-03-28T18:19:06.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:05 smithi170 ceph-mon[36280]: pgmap v261: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 186 B/s wr, 17 op/s 2024-03-28T18:19:08.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:07 smithi037 ceph-mon[31460]: pgmap v262: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 170 B/s wr, 18 op/s 2024-03-28T18:19:08.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:07 smithi059 ceph-mon[36190]: pgmap v262: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 170 B/s wr, 18 op/s 2024-03-28T18:19:08.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:07 smithi170 ceph-mon[36280]: pgmap v262: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 170 B/s wr, 18 op/s 2024-03-28T18:19:09.299 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:09 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service: Scheduled restart job, restart counter is at 4. 2024-03-28T18:19:09.299 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:09 smithi059 systemd[1]: Stopped Ceph mds.e for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:19:09.299 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:09 smithi059 systemd[1]: Starting Ceph mds.e for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:19:09.299 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:09 smithi059 podman[72406]: 2024-03-28 18:19:09.178331479 +0000 UTC m=+0.018891449 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:19:09.299 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:09 smithi059 podman[72406]: 2024-03-28 18:19:09.299394017 +0000 UTC m=+0.139953960 container create 451ce61079ea1e8061cf418fb22d078e2861a6644823de0a7931cec0848910cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-28T18:19:09.553 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:09 smithi059 podman[72406]: 2024-03-28 18:19:09.527073718 +0000 UTC m=+0.367633669 container init 451ce61079ea1e8061cf418fb22d078e2861a6644823de0a7931cec0848910cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:19:09.553 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:09 smithi059 podman[72406]: 2024-03-28 18:19:09.530264913 +0000 UTC m=+0.370824871 container start 451ce61079ea1e8061cf418fb22d078e2861a6644823de0a7931cec0848910cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:19:09.844 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:09 smithi059 ceph-mds[72425]: -- 172.21.15.59:0/2856096264 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55988163da40 con 0x559881610800 2024-03-28T18:19:09.845 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:09 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: starting mds.e at 2024-03-28T18:19:09.845 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:09 smithi059 bash[72406]: 451ce61079ea1e8061cf418fb22d078e2861a6644823de0a7931cec0848910cd 2024-03-28T18:19:09.845 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:09 smithi059 systemd[1]: Started Ceph mds.e for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:19:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:09 smithi037 ceph-mon[31460]: pgmap v263: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 15 op/s 2024-03-28T18:19:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:09 smithi037 ceph-mon[31460]: daemon mds.e restarted 2024-03-28T18:19:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:09 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:19:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:09 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:19:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:09 smithi037 ceph-mon[31460]: osdmap e110: 12 total, 12 up, 12 in 2024-03-28T18:19:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:09 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/3249002570,v1:172.21.15.59:6835/3249002570] up:boot 2024-03-28T18:19:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:09 smithi037 ceph-mon[31460]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-28T18:19:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:09 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:19:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:09 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:10.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:09 smithi059 ceph-mon[36190]: pgmap v263: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 15 op/s 2024-03-28T18:19:10.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:09 smithi059 ceph-mon[36190]: daemon mds.e restarted 2024-03-28T18:19:10.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:09 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:19:10.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:09 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:19:10.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:09 smithi059 ceph-mon[36190]: osdmap e110: 12 total, 12 up, 12 in 2024-03-28T18:19:10.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:09 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/3249002570,v1:172.21.15.59:6835/3249002570] up:boot 2024-03-28T18:19:10.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:09 smithi059 ceph-mon[36190]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-28T18:19:10.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:09 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:19:10.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:09 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:10.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:09 smithi170 ceph-mon[36280]: pgmap v263: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 15 op/s 2024-03-28T18:19:10.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:09 smithi170 ceph-mon[36280]: daemon mds.e restarted 2024-03-28T18:19:10.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:09 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:19:10.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:09 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:19:10.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:09 smithi170 ceph-mon[36280]: osdmap e110: 12 total, 12 up, 12 in 2024-03-28T18:19:10.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:09 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/3249002570,v1:172.21.15.59:6835/3249002570] up:boot 2024-03-28T18:19:10.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:09 smithi170 ceph-mon[36280]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-28T18:19:10.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:09 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:19:10.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:09 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:11 smithi037 ceph-mon[31460]: pgmap v265: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T18:19:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:11 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/3058605036,v1:172.21.15.170:6833/3058605036] up:resolve 2024-03-28T18:19:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:11 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:11 smithi059 ceph-mon[36190]: pgmap v265: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T18:19:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:11 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/3058605036,v1:172.21.15.170:6833/3058605036] up:resolve 2024-03-28T18:19:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:11 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:12.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:11 smithi170 ceph-mon[36280]: pgmap v265: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T18:19:12.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:11 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/3058605036,v1:172.21.15.170:6833/3058605036] up:resolve 2024-03-28T18:19:12.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:11 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:13.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:12 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199995 2024-03-28T18:19:13.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:12 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/3058605036,v1:172.21.15.170:6833/3058605036] up:reconnect 2024-03-28T18:19:13.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:12 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:13.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:12 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999978 2024-03-28T18:19:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:12 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199995 2024-03-28T18:19:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:12 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/3058605036,v1:172.21.15.170:6833/3058605036] up:reconnect 2024-03-28T18:19:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:12 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:12 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999978 2024-03-28T18:19:13.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:12 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199995 2024-03-28T18:19:13.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:12 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/3058605036,v1:172.21.15.170:6833/3058605036] up:reconnect 2024-03-28T18:19:13.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:12 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:13.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:12 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999978 2024-03-28T18:19:14.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:13 smithi037 ceph-mon[31460]: pgmap v266: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-03-28T18:19:14.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:13 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/3058605036,v1:172.21.15.170:6833/3058605036] up:rejoin 2024-03-28T18:19:14.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:13 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:14.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:13 smithi037 ceph-mon[31460]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-28T18:19:14.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:13 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:19:14.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:13 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:19:14.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:13 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/3058605036,v1:172.21.15.170:6833/3058605036] up:active 2024-03-28T18:19:14.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:13 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:19:14.219 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-mon[36280]: pgmap v266: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-03-28T18:19:14.219 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/3058605036,v1:172.21.15.170:6833/3058605036] up:rejoin 2024-03-28T18:19:14.219 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:14.219 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-mon[36280]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-28T18:19:14.219 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:19:14.219 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:19:14.219 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/3058605036,v1:172.21.15.170:6833/3058605036] up:active 2024-03-28T18:19:14.219 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:19:14.220 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff0da2fb700 time 2024-03-28T18:19:13.786800+0000 2024-03-28T18:19:14.220 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:19:14.220 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:14.220 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff0e49254a1] 2024-03-28T18:19:14.220 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff0e4925667] 2024-03-28T18:19:14.220 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555867a70223] 2024-03-28T18:19:14.220 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5558678ea5d6] 2024-03-28T18:19:14.220 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5558678eada1] 2024-03-28T18:19:14.220 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5558678ed2d5] 2024-03-28T18:19:14.221 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55586793d233] 2024-03-28T18:19:14.221 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55586793f197] 2024-03-28T18:19:14.221 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55586783e11b] 2024-03-28T18:19:14.221 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55586783e6bc] 2024-03-28T18:19:14.221 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5558678281ff] 2024-03-28T18:19:14.221 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff0e4bb7488] 2024-03-28T18:19:14.221 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 13: (DispatchQueue::entry()+0x51f) [0x7ff0e4bb460f] 2024-03-28T18:19:14.221 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff0e4c87e81] 2024-03-28T18:19:14.221 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff0e36a21ca] 2024-03-28T18:19:14.221 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 16: clone() 2024-03-28T18:19:14.221 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2024-03-28T18:19:13.786+0000 7ff0da2fb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff0da2fb700 time 2024-03-28T18:19:13.786800+0000 2024-03-28T18:19:14.221 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:19:14.221 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2024-03-28T18:19:14.221 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:14.221 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff0e49254a1] 2024-03-28T18:19:14.222 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff0e4925667] 2024-03-28T18:19:14.222 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555867a70223] 2024-03-28T18:19:14.222 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5558678ea5d6] 2024-03-28T18:19:14.222 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5558678eada1] 2024-03-28T18:19:14.222 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5558678ed2d5] 2024-03-28T18:19:14.222 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55586793d233] 2024-03-28T18:19:14.222 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55586793f197] 2024-03-28T18:19:14.222 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55586783e11b] 2024-03-28T18:19:14.222 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55586783e6bc] 2024-03-28T18:19:14.222 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5558678281ff] 2024-03-28T18:19:14.222 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff0e4bb7488] 2024-03-28T18:19:14.222 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 13: (DispatchQueue::entry()+0x51f) [0x7ff0e4bb460f] 2024-03-28T18:19:14.222 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff0e4c87e81] 2024-03-28T18:19:14.223 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff0e36a21ca] 2024-03-28T18:19:14.223 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 16: clone() 2024-03-28T18:19:14.223 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2024-03-28T18:19:14.223 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: *** Caught signal (Aborted) ** 2024-03-28T18:19:14.223 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: in thread 7ff0da2fb700 thread_name:ms_dispatch 2024-03-28T18:19:14.223 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:14.223 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff0e36acd20] 2024-03-28T18:19:14.224 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2: gsignal() 2024-03-28T18:19:14.224 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 3: abort() 2024-03-28T18:19:14.224 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff0e49254fb] 2024-03-28T18:19:14.224 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff0e4925667] 2024-03-28T18:19:14.224 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555867a70223] 2024-03-28T18:19:14.224 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5558678ea5d6] 2024-03-28T18:19:14.224 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5558678eada1] 2024-03-28T18:19:14.224 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5558678ed2d5] 2024-03-28T18:19:14.224 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55586793d233] 2024-03-28T18:19:14.225 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55586793f197] 2024-03-28T18:19:14.225 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55586783e11b] 2024-03-28T18:19:14.225 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55586783e6bc] 2024-03-28T18:19:14.225 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5558678281ff] 2024-03-28T18:19:14.225 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff0e4bb7488] 2024-03-28T18:19:14.225 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 16: (DispatchQueue::entry()+0x51f) [0x7ff0e4bb460f] 2024-03-28T18:19:14.225 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff0e4c87e81] 2024-03-28T18:19:14.225 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff0e36a21ca] 2024-03-28T18:19:14.225 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 19: clone() 2024-03-28T18:19:14.225 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2024-03-28T18:19:13.788+0000 7ff0da2fb700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:19:14.225 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: in thread 7ff0da2fb700 thread_name:ms_dispatch 2024-03-28T18:19:14.225 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2024-03-28T18:19:14.225 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:14.225 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff0e36acd20] 2024-03-28T18:19:14.225 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2: gsignal() 2024-03-28T18:19:14.226 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 3: abort() 2024-03-28T18:19:14.226 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff0e49254fb] 2024-03-28T18:19:14.226 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff0e4925667] 2024-03-28T18:19:14.226 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555867a70223] 2024-03-28T18:19:14.226 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5558678ea5d6] 2024-03-28T18:19:14.226 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5558678eada1] 2024-03-28T18:19:14.226 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5558678ed2d5] 2024-03-28T18:19:14.226 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55586793d233] 2024-03-28T18:19:14.226 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55586793f197] 2024-03-28T18:19:14.226 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55586783e11b] 2024-03-28T18:19:14.226 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55586783e6bc] 2024-03-28T18:19:14.226 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5558678281ff] 2024-03-28T18:19:14.226 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff0e4bb7488] 2024-03-28T18:19:14.226 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 16: (DispatchQueue::entry()+0x51f) [0x7ff0e4bb460f] 2024-03-28T18:19:14.226 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff0e4c87e81] 2024-03-28T18:19:14.226 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff0e36a21ca] 2024-03-28T18:19:14.226 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 19: clone() 2024-03-28T18:19:14.227 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:19:14.227 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2024-03-28T18:19:14.227 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: -1> 2024-03-28T18:19:13.786+0000 7ff0da2fb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff0da2fb700 time 2024-03-28T18:19:13.786800+0000 2024-03-28T18:19:14.227 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:19:14.227 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2024-03-28T18:19:14.227 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:14.227 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff0e49254a1] 2024-03-28T18:19:14.227 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff0e4925667] 2024-03-28T18:19:14.227 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555867a70223] 2024-03-28T18:19:14.227 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5558678ea5d6] 2024-03-28T18:19:14.227 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5558678eada1] 2024-03-28T18:19:14.227 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5558678ed2d5] 2024-03-28T18:19:14.227 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55586793d233] 2024-03-28T18:19:14.227 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55586793f197] 2024-03-28T18:19:14.227 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55586783e11b] 2024-03-28T18:19:14.227 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55586783e6bc] 2024-03-28T18:19:14.228 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5558678281ff] 2024-03-28T18:19:14.228 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff0e4bb7488] 2024-03-28T18:19:14.228 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 13: (DispatchQueue::entry()+0x51f) [0x7ff0e4bb460f] 2024-03-28T18:19:14.228 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff0e4c87e81] 2024-03-28T18:19:14.228 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff0e36a21ca] 2024-03-28T18:19:14.228 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 16: clone() 2024-03-28T18:19:14.228 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2024-03-28T18:19:14.228 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 0> 2024-03-28T18:19:13.788+0000 7ff0da2fb700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:19:14.228 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: in thread 7ff0da2fb700 thread_name:ms_dispatch 2024-03-28T18:19:14.228 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2024-03-28T18:19:14.228 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:14.228 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff0e36acd20] 2024-03-28T18:19:14.228 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2: gsignal() 2024-03-28T18:19:14.228 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 3: abort() 2024-03-28T18:19:14.229 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff0e49254fb] 2024-03-28T18:19:14.229 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff0e4925667] 2024-03-28T18:19:14.229 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555867a70223] 2024-03-28T18:19:14.229 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5558678ea5d6] 2024-03-28T18:19:14.229 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5558678eada1] 2024-03-28T18:19:14.229 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5558678ed2d5] 2024-03-28T18:19:14.229 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55586793d233] 2024-03-28T18:19:14.229 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55586793f197] 2024-03-28T18:19:14.229 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55586783e11b] 2024-03-28T18:19:14.229 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55586783e6bc] 2024-03-28T18:19:14.229 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5558678281ff] 2024-03-28T18:19:14.229 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff0e4bb7488] 2024-03-28T18:19:14.229 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 16: (DispatchQueue::entry()+0x51f) [0x7ff0e4bb460f] 2024-03-28T18:19:14.229 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff0e4c87e81] 2024-03-28T18:19:14.229 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff0e36a21ca] 2024-03-28T18:19:14.229 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 19: clone() 2024-03-28T18:19:14.230 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:19:14.230 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2024-03-28T18:19:14.230 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: -1> 2024-03-28T18:19:13.786+0000 7ff0da2fb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff0da2fb700 time 2024-03-28T18:19:13.786800+0000 2024-03-28T18:19:14.230 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:19:14.230 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2024-03-28T18:19:14.230 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:14.230 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff0e49254a1] 2024-03-28T18:19:14.230 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff0e4925667] 2024-03-28T18:19:14.230 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555867a70223] 2024-03-28T18:19:14.230 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5558678ea5d6] 2024-03-28T18:19:14.230 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5558678eada1] 2024-03-28T18:19:14.230 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5558678ed2d5] 2024-03-28T18:19:14.230 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55586793d233] 2024-03-28T18:19:14.230 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55586793f197] 2024-03-28T18:19:14.230 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55586783e11b] 2024-03-28T18:19:14.230 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55586783e6bc] 2024-03-28T18:19:14.231 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5558678281ff] 2024-03-28T18:19:14.231 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff0e4bb7488] 2024-03-28T18:19:14.231 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 13: (DispatchQueue::entry()+0x51f) [0x7ff0e4bb460f] 2024-03-28T18:19:14.231 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff0e4c87e81] 2024-03-28T18:19:14.231 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff0e36a21ca] 2024-03-28T18:19:14.231 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 16: clone() 2024-03-28T18:19:14.231 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2024-03-28T18:19:14.231 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 0> 2024-03-28T18:19:13.788+0000 7ff0da2fb700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:19:14.231 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: in thread 7ff0da2fb700 thread_name:ms_dispatch 2024-03-28T18:19:14.231 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2024-03-28T18:19:14.231 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:14.231 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff0e36acd20] 2024-03-28T18:19:14.231 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2: gsignal() 2024-03-28T18:19:14.231 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 3: abort() 2024-03-28T18:19:14.231 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff0e49254fb] 2024-03-28T18:19:14.231 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff0e4925667] 2024-03-28T18:19:14.232 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555867a70223] 2024-03-28T18:19:14.232 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5558678ea5d6] 2024-03-28T18:19:14.232 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5558678eada1] 2024-03-28T18:19:14.232 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5558678ed2d5] 2024-03-28T18:19:14.232 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55586793d233] 2024-03-28T18:19:14.232 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55586793f197] 2024-03-28T18:19:14.232 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55586783e11b] 2024-03-28T18:19:14.232 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55586783e6bc] 2024-03-28T18:19:14.232 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5558678281ff] 2024-03-28T18:19:14.232 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff0e4bb7488] 2024-03-28T18:19:14.232 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 16: (DispatchQueue::entry()+0x51f) [0x7ff0e4bb460f] 2024-03-28T18:19:14.232 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff0e4c87e81] 2024-03-28T18:19:14.232 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff0e36a21ca] 2024-03-28T18:19:14.232 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 19: clone() 2024-03-28T18:19:14.232 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:19:14.233 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72013]: 2024-03-28T18:19:14.233 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:13 smithi170 podman[72051]: 2024-03-28 18:19:13.860166503 +0000 UTC m=+0.021596742 container died e4a7ca1cc8b09533c060cb335f487728ac944b8413d9644dc6546aca172ee27e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-28T18:19:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:13 smithi059 ceph-mon[36190]: pgmap v266: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-03-28T18:19:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:13 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/3058605036,v1:172.21.15.170:6833/3058605036] up:rejoin 2024-03-28T18:19:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:13 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:13 smithi059 ceph-mon[36190]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-28T18:19:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:13 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:19:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:13 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:19:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:13 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/3058605036,v1:172.21.15.170:6833/3058605036] up:active 2024-03-28T18:19:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:13 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:19:14.915 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:14 smithi170 podman[72051]: 2024-03-28 18:19:14.644808362 +0000 UTC m=+0.806238583 container remove e4a7ca1cc8b09533c060cb335f487728ac944b8413d9644dc6546aca172ee27e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:19:14.915 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:14 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:19:15.285 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:14 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-28T18:19:16.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:15 smithi059 ceph-mon[36190]: pgmap v267: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-03-28T18:19:16.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:15 smithi170 ceph-mon[36280]: pgmap v267: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-03-28T18:19:16.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:15 smithi037 ceph-mon[31460]: pgmap v267: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-03-28T18:19:16.742 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:19:16 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:19:16.425+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:19:17.254 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:16 smithi037 ceph-mon[31460]: pgmap v268: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:19:17.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:16 smithi059 ceph-mon[36190]: pgmap v268: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:19:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:16 smithi170 ceph-mon[36280]: pgmap v268: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:19:19.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:19 smithi037 ceph-mon[31460]: pgmap v269: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:19:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:19 smithi059 ceph-mon[36190]: pgmap v269: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:19:20.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:19 smithi170 ceph-mon[36280]: pgmap v269: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:19:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:21 smithi059 ceph-mon[36190]: pgmap v270: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 25 KiB/s wr, 31 op/s 2024-03-28T18:19:22.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:21 smithi170 ceph-mon[36280]: pgmap v270: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 25 KiB/s wr, 31 op/s 2024-03-28T18:19:22.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:21 smithi037 ceph-mon[31460]: pgmap v270: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 25 KiB/s wr, 31 op/s 2024-03-28T18:19:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:23 smithi059 ceph-mon[36190]: pgmap v271: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 23 KiB/s wr, 31 op/s 2024-03-28T18:19:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:23 smithi170 ceph-mon[36280]: pgmap v271: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 23 KiB/s wr, 31 op/s 2024-03-28T18:19:24.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:23 smithi037 ceph-mon[31460]: pgmap v271: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 23 KiB/s wr, 31 op/s 2024-03-28T18:19:25.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:25 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 2. 2024-03-28T18:19:25.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:25 smithi170 systemd[1]: Stopped Ceph mds.l for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:19:25.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:25 smithi170 systemd[1]: Starting Ceph mds.l for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:19:25.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:25 smithi170 podman[72150]: 2024-03-28 18:19:25.181092728 +0000 UTC m=+0.018377878 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:19:25.563 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:25 smithi170 podman[72150]: 2024-03-28 18:19:25.295041677 +0000 UTC m=+0.132326824 container create 0fb55f9537e9afecf7f847476a3c26905691eeb5e5e6a136bb3375d2a4a150f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-28T18:19:25.563 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:25 smithi170 podman[72150]: 2024-03-28 18:19:25.539569671 +0000 UTC m=+0.376854818 container init 0fb55f9537e9afecf7f847476a3c26905691eeb5e5e6a136bb3375d2a4a150f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-28T18:19:25.563 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:25 smithi170 podman[72150]: 2024-03-28 18:19:25.542182926 +0000 UTC m=+0.379468068 container start 0fb55f9537e9afecf7f847476a3c26905691eeb5e5e6a136bb3375d2a4a150f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-28T18:19:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:25 smithi059 ceph-mon[36190]: pgmap v272: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-28T18:19:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:25 smithi059 ceph-mon[36190]: daemon mds.l restarted 2024-03-28T18:19:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:25 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:19:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:25 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:19:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:25 smithi059 ceph-mon[36190]: osdmap e111: 12 total, 12 up, 12 in 2024-03-28T18:19:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:25 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/749986964,v1:172.21.15.170:6833/749986964] up:boot 2024-03-28T18:19:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:25 smithi059 ceph-mon[36190]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-03-28T18:19:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:25 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:19:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:25 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:26.036 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:25 smithi170 ceph-mds[72169]: -- 172.21.15.170:0/3905166427 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5614dc83ba40 con 0x5614dc80ec00 2024-03-28T18:19:26.036 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:25 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: starting mds.l at 2024-03-28T18:19:26.036 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:25 smithi170 bash[72150]: 0fb55f9537e9afecf7f847476a3c26905691eeb5e5e6a136bb3375d2a4a150f7 2024-03-28T18:19:26.036 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:25 smithi170 systemd[1]: Started Ceph mds.l for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:19:26.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:25 smithi170 ceph-mon[36280]: pgmap v272: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-28T18:19:26.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:25 smithi170 ceph-mon[36280]: daemon mds.l restarted 2024-03-28T18:19:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:25 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:19:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:25 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:19:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:25 smithi170 ceph-mon[36280]: osdmap e111: 12 total, 12 up, 12 in 2024-03-28T18:19:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:25 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/749986964,v1:172.21.15.170:6833/749986964] up:boot 2024-03-28T18:19:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:25 smithi170 ceph-mon[36280]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-03-28T18:19:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:25 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:19:26.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:25 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:25 smithi037 ceph-mon[31460]: pgmap v272: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-28T18:19:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:25 smithi037 ceph-mon[31460]: daemon mds.l restarted 2024-03-28T18:19:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:25 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:19:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:25 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:19:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:25 smithi037 ceph-mon[31460]: osdmap e111: 12 total, 12 up, 12 in 2024-03-28T18:19:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:25 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/749986964,v1:172.21.15.170:6833/749986964] up:boot 2024-03-28T18:19:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:25 smithi037 ceph-mon[31460]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-03-28T18:19:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:25 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:19:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:25 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:27.927 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:27 smithi037 ceph-mon[31460]: pgmap v274: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:19:27.927 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:27 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/3249002570,v1:172.21.15.59:6835/3249002570] up:resolve 2024-03-28T18:19:27.927 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:27 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:27 smithi059 ceph-mon[36190]: pgmap v274: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:19:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:27 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/3249002570,v1:172.21.15.59:6835/3249002570] up:resolve 2024-03-28T18:19:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:27 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:28.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:27 smithi170 ceph-mon[36280]: pgmap v274: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:19:28.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:27 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/3249002570,v1:172.21.15.59:6835/3249002570] up:resolve 2024-03-28T18:19:28.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:27 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:28.811 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199998 2024-03-28T18:19:28.811 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/3249002570,v1:172.21.15.59:6835/3249002570] up:reconnect 2024-03-28T18:19:28.811 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:28.811 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999989 2024-03-28T18:19:28.812 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1815400700 time 2024-03-28T18:19:28.806187+0000 2024-03-28T18:19:28.812 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:19:28.812 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:28.812 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f181fa2a4a1] 2024-03-28T18:19:28.812 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f181fa2a667] 2024-03-28T18:19:28.812 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55987f28a223] 2024-03-28T18:19:28.812 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55987f1045d6] 2024-03-28T18:19:28.812 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55987f104da1] 2024-03-28T18:19:28.812 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55987f1072d5] 2024-03-28T18:19:28.812 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55987f157233] 2024-03-28T18:19:28.812 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55987f159197] 2024-03-28T18:19:28.812 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55987f05811b] 2024-03-28T18:19:28.812 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55987f0586bc] 2024-03-28T18:19:28.813 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55987f0421ff] 2024-03-28T18:19:28.813 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f181fcbc488] 2024-03-28T18:19:28.813 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 13: (DispatchQueue::entry()+0x51f) [0x7f181fcb960f] 2024-03-28T18:19:28.813 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f181fd8ce81] 2024-03-28T18:19:28.813 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f181e7a71ca] 2024-03-28T18:19:28.813 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 16: clone() 2024-03-28T18:19:28.813 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2024-03-28T18:19:28.808+0000 7f1815400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1815400700 time 2024-03-28T18:19:28.806187+0000 2024-03-28T18:19:28.814 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:19:28.814 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2024-03-28T18:19:28.814 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:28.814 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f181fa2a4a1] 2024-03-28T18:19:28.814 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f181fa2a667] 2024-03-28T18:19:28.814 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55987f28a223] 2024-03-28T18:19:28.814 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55987f1045d6] 2024-03-28T18:19:28.814 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55987f104da1] 2024-03-28T18:19:28.814 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55987f1072d5] 2024-03-28T18:19:28.814 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55987f157233] 2024-03-28T18:19:28.814 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55987f159197] 2024-03-28T18:19:28.814 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55987f05811b] 2024-03-28T18:19:28.814 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55987f0586bc] 2024-03-28T18:19:28.814 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55987f0421ff] 2024-03-28T18:19:28.815 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f181fcbc488] 2024-03-28T18:19:28.815 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 13: (DispatchQueue::entry()+0x51f) [0x7f181fcb960f] 2024-03-28T18:19:28.815 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f181fd8ce81] 2024-03-28T18:19:28.815 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f181e7a71ca] 2024-03-28T18:19:28.815 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 16: clone() 2024-03-28T18:19:28.815 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2024-03-28T18:19:28.815 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: *** Caught signal (Aborted) ** 2024-03-28T18:19:28.815 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: in thread 7f1815400700 thread_name:ms_dispatch 2024-03-28T18:19:28.815 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:28.815 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f181e7b1d20] 2024-03-28T18:19:28.815 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2: gsignal() 2024-03-28T18:19:28.815 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 3: abort() 2024-03-28T18:19:28.815 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f181fa2a4fb] 2024-03-28T18:19:28.815 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f181fa2a667] 2024-03-28T18:19:28.815 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55987f28a223] 2024-03-28T18:19:28.816 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55987f1045d6] 2024-03-28T18:19:28.816 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55987f104da1] 2024-03-28T18:19:28.816 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55987f1072d5] 2024-03-28T18:19:28.816 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55987f157233] 2024-03-28T18:19:28.816 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55987f159197] 2024-03-28T18:19:28.816 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55987f05811b] 2024-03-28T18:19:28.816 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55987f0586bc] 2024-03-28T18:19:28.816 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55987f0421ff] 2024-03-28T18:19:28.816 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f181fcbc488] 2024-03-28T18:19:28.816 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 16: (DispatchQueue::entry()+0x51f) [0x7f181fcb960f] 2024-03-28T18:19:28.816 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f181fd8ce81] 2024-03-28T18:19:28.816 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f181e7a71ca] 2024-03-28T18:19:28.816 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 19: clone() 2024-03-28T18:19:28.816 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2024-03-28T18:19:28.810+0000 7f1815400700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:19:28.816 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: in thread 7f1815400700 thread_name:ms_dispatch 2024-03-28T18:19:28.816 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2024-03-28T18:19:28.817 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:28.817 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f181e7b1d20] 2024-03-28T18:19:28.817 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2: gsignal() 2024-03-28T18:19:28.817 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 3: abort() 2024-03-28T18:19:28.817 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f181fa2a4fb] 2024-03-28T18:19:28.817 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f181fa2a667] 2024-03-28T18:19:28.817 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55987f28a223] 2024-03-28T18:19:28.817 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55987f1045d6] 2024-03-28T18:19:28.817 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55987f104da1] 2024-03-28T18:19:28.817 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55987f1072d5] 2024-03-28T18:19:28.817 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55987f157233] 2024-03-28T18:19:28.817 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55987f159197] 2024-03-28T18:19:28.817 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55987f05811b] 2024-03-28T18:19:28.817 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55987f0586bc] 2024-03-28T18:19:28.817 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55987f0421ff] 2024-03-28T18:19:28.817 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f181fcbc488] 2024-03-28T18:19:28.818 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 16: (DispatchQueue::entry()+0x51f) [0x7f181fcb960f] 2024-03-28T18:19:28.818 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f181fd8ce81] 2024-03-28T18:19:28.818 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f181e7a71ca] 2024-03-28T18:19:28.818 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 19: clone() 2024-03-28T18:19:28.818 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:19:28.818 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2024-03-28T18:19:29.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:28 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199998 2024-03-28T18:19:29.340 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:28 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/3249002570,v1:172.21.15.59:6835/3249002570] up:reconnect 2024-03-28T18:19:29.340 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:28 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:29.340 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:28 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999989 2024-03-28T18:19:29.340 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: -1> 2024-03-28T18:19:28.808+0000 7f1815400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1815400700 time 2024-03-28T18:19:28.806187+0000 2024-03-28T18:19:29.341 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:19:29.341 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2024-03-28T18:19:29.341 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:29.341 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f181fa2a4a1] 2024-03-28T18:19:29.341 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f181fa2a667] 2024-03-28T18:19:29.341 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55987f28a223] 2024-03-28T18:19:29.341 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55987f1045d6] 2024-03-28T18:19:29.341 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55987f104da1] 2024-03-28T18:19:29.341 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55987f1072d5] 2024-03-28T18:19:29.341 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55987f157233] 2024-03-28T18:19:29.341 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55987f159197] 2024-03-28T18:19:29.342 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55987f05811b] 2024-03-28T18:19:29.342 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55987f0586bc] 2024-03-28T18:19:29.342 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55987f0421ff] 2024-03-28T18:19:29.342 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f181fcbc488] 2024-03-28T18:19:29.342 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 13: (DispatchQueue::entry()+0x51f) [0x7f181fcb960f] 2024-03-28T18:19:29.342 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f181fd8ce81] 2024-03-28T18:19:29.342 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f181e7a71ca] 2024-03-28T18:19:29.342 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 16: clone() 2024-03-28T18:19:29.342 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2024-03-28T18:19:29.342 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 0> 2024-03-28T18:19:28.810+0000 7f1815400700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:19:29.342 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: in thread 7f1815400700 thread_name:ms_dispatch 2024-03-28T18:19:29.342 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2024-03-28T18:19:29.343 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:29.343 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f181e7b1d20] 2024-03-28T18:19:29.343 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2: gsignal() 2024-03-28T18:19:29.343 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 3: abort() 2024-03-28T18:19:29.343 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f181fa2a4fb] 2024-03-28T18:19:29.343 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f181fa2a667] 2024-03-28T18:19:29.343 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55987f28a223] 2024-03-28T18:19:29.343 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55987f1045d6] 2024-03-28T18:19:29.343 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55987f104da1] 2024-03-28T18:19:29.343 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55987f1072d5] 2024-03-28T18:19:29.343 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55987f157233] 2024-03-28T18:19:29.343 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55987f159197] 2024-03-28T18:19:29.343 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55987f05811b] 2024-03-28T18:19:29.344 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55987f0586bc] 2024-03-28T18:19:29.344 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55987f0421ff] 2024-03-28T18:19:29.344 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f181fcbc488] 2024-03-28T18:19:29.344 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 16: (DispatchQueue::entry()+0x51f) [0x7f181fcb960f] 2024-03-28T18:19:29.344 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f181fd8ce81] 2024-03-28T18:19:29.344 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f181e7a71ca] 2024-03-28T18:19:29.344 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 19: clone() 2024-03-28T18:19:29.344 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:19:29.344 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2024-03-28T18:19:29.344 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: -1> 2024-03-28T18:19:28.808+0000 7f1815400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1815400700 time 2024-03-28T18:19:28.806187+0000 2024-03-28T18:19:29.344 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:19:29.344 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2024-03-28T18:19:29.344 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:29.344 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f181fa2a4a1] 2024-03-28T18:19:29.345 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f181fa2a667] 2024-03-28T18:19:29.345 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55987f28a223] 2024-03-28T18:19:29.345 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55987f1045d6] 2024-03-28T18:19:29.345 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55987f104da1] 2024-03-28T18:19:29.345 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55987f1072d5] 2024-03-28T18:19:29.345 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55987f157233] 2024-03-28T18:19:29.345 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55987f159197] 2024-03-28T18:19:29.345 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55987f05811b] 2024-03-28T18:19:29.345 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55987f0586bc] 2024-03-28T18:19:29.345 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55987f0421ff] 2024-03-28T18:19:29.345 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f181fcbc488] 2024-03-28T18:19:29.345 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 13: (DispatchQueue::entry()+0x51f) [0x7f181fcb960f] 2024-03-28T18:19:29.346 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f181fd8ce81] 2024-03-28T18:19:29.346 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f181e7a71ca] 2024-03-28T18:19:29.346 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 16: clone() 2024-03-28T18:19:29.346 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2024-03-28T18:19:29.346 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 0> 2024-03-28T18:19:28.810+0000 7f1815400700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:19:29.346 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: in thread 7f1815400700 thread_name:ms_dispatch 2024-03-28T18:19:29.346 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2024-03-28T18:19:29.346 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:29.346 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f181e7b1d20] 2024-03-28T18:19:29.346 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2: gsignal() 2024-03-28T18:19:29.346 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 3: abort() 2024-03-28T18:19:29.346 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f181fa2a4fb] 2024-03-28T18:19:29.346 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f181fa2a667] 2024-03-28T18:19:29.346 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55987f28a223] 2024-03-28T18:19:29.347 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55987f1045d6] 2024-03-28T18:19:29.347 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55987f104da1] 2024-03-28T18:19:29.347 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55987f1072d5] 2024-03-28T18:19:29.347 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55987f157233] 2024-03-28T18:19:29.347 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55987f159197] 2024-03-28T18:19:29.347 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55987f05811b] 2024-03-28T18:19:29.347 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55987f0586bc] 2024-03-28T18:19:29.347 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55987f0421ff] 2024-03-28T18:19:29.347 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f181fcbc488] 2024-03-28T18:19:29.347 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 16: (DispatchQueue::entry()+0x51f) [0x7f181fcb960f] 2024-03-28T18:19:29.347 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f181fd8ce81] 2024-03-28T18:19:29.347 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f181e7a71ca] 2024-03-28T18:19:29.348 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 19: clone() 2024-03-28T18:19:29.348 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:19:29.348 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72421]: 2024-03-28T18:19:29.348 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:28 smithi059 podman[72459]: 2024-03-28 18:19:28.896687873 +0000 UTC m=+0.019301190 container died 451ce61079ea1e8061cf418fb22d078e2861a6644823de0a7931cec0848910cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:19:29.348 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:28 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199998 2024-03-28T18:19:29.348 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:28 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/3249002570,v1:172.21.15.59:6835/3249002570] up:reconnect 2024-03-28T18:19:29.348 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:28 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:29.349 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:28 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999989 2024-03-28T18:19:29.933 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:29 smithi059 ceph-mon[36190]: pgmap v275: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:19:29.933 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:29 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/3249002570,v1:172.21.15.59:6835/3249002570] up:rejoin 2024-03-28T18:19:29.933 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:29 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:29.934 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:29 smithi059 ceph-mon[36190]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-03-28T18:19:29.934 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:29 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:19:29.934 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:29 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:19:29.934 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:29 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/3249002570,v1:172.21.15.59:6835/3249002570] up:active 2024-03-28T18:19:29.934 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:29 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:19:29.934 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:29 smithi059 podman[72459]: 2024-03-28 18:19:29.690891988 +0000 UTC m=+0.813505323 container remove 451ce61079ea1e8061cf418fb22d078e2861a6644823de0a7931cec0848910cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:19:29.934 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:29 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:19:30.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:29 smithi170 ceph-mon[36280]: pgmap v275: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:19:30.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:29 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/3249002570,v1:172.21.15.59:6835/3249002570] up:rejoin 2024-03-28T18:19:30.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:29 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:30.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:29 smithi170 ceph-mon[36280]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-03-28T18:19:30.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:29 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:19:30.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:29 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:19:30.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:29 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/3249002570,v1:172.21.15.59:6835/3249002570] up:active 2024-03-28T18:19:30.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:29 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:19:30.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:29 smithi037 ceph-mon[31460]: pgmap v275: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:19:30.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:29 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/3249002570,v1:172.21.15.59:6835/3249002570] up:rejoin 2024-03-28T18:19:30.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:29 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:30.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:29 smithi037 ceph-mon[31460]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-03-28T18:19:30.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:29 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:19:30.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:29 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:19:30.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:29 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/3249002570,v1:172.21.15.59:6835/3249002570] up:active 2024-03-28T18:19:30.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:29 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:19:30.285 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:29 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service: Failed with result 'exit-code'. 2024-03-28T18:19:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:31 smithi037 ceph-mon[31460]: pgmap v276: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:19:31.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:31 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:19:31.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:19:31 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:19:31.427+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:19:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:31 smithi059 ceph-mon[36190]: pgmap v276: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:19:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:31 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:19:32.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:31 smithi170 ceph-mon[36280]: pgmap v276: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:19:32.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:31 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:19:33.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:32 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:19:33.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:32 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:19:33.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:32 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:19:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:32 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:19:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:32 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:19:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:32 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:19:33.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:32 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:19:33.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:32 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:19:33.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:32 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:19:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:33 smithi170 ceph-mon[36280]: pgmap v277: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 27 KiB/s wr, 21 op/s 2024-03-28T18:19:34.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:33 smithi059 ceph-mon[36190]: pgmap v277: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 27 KiB/s wr, 21 op/s 2024-03-28T18:19:34.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:33 smithi037 ceph-mon[31460]: pgmap v277: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 27 KiB/s wr, 21 op/s 2024-03-28T18:19:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:35 smithi059 ceph-mon[36190]: pgmap v278: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 27 KiB/s wr, 21 op/s 2024-03-28T18:19:36.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:35 smithi170 ceph-mon[36280]: pgmap v278: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 27 KiB/s wr, 21 op/s 2024-03-28T18:19:36.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:35 smithi037 ceph-mon[31460]: pgmap v278: 97 pgs: 97 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 27 KiB/s wr, 21 op/s 2024-03-28T18:19:37.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:36 smithi059 ceph-mon[36190]: osdmap e112: 12 total, 12 up, 12 in 2024-03-28T18:19:37.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:36 smithi170 ceph-mon[36280]: osdmap e112: 12 total, 12 up, 12 in 2024-03-28T18:19:37.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:36 smithi037 ceph-mon[31460]: osdmap e112: 12 total, 12 up, 12 in 2024-03-28T18:19:38.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:37 smithi059 ceph-mon[36190]: pgmap v280: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:19:38.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:37 smithi170 ceph-mon[36280]: pgmap v280: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:19:38.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:37 smithi037 ceph-mon[31460]: pgmap v280: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:19:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:39 smithi059 ceph-mon[36190]: pgmap v281: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:19:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:39 smithi059 ceph-mon[36190]: osdmap e113: 12 total, 12 up, 12 in 2024-03-28T18:19:40.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:39 smithi170 ceph-mon[36280]: pgmap v281: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:19:40.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:39 smithi170 ceph-mon[36280]: osdmap e113: 12 total, 12 up, 12 in 2024-03-28T18:19:40.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:39 smithi037 ceph-mon[31460]: pgmap v281: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:19:40.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:39 smithi037 ceph-mon[31460]: osdmap e113: 12 total, 12 up, 12 in 2024-03-28T18:19:40.307 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:40 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service: Scheduled restart job, restart counter is at 5. 2024-03-28T18:19:40.308 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:40 smithi059 systemd[1]: Stopped Ceph mds.e for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:19:40.308 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:40 smithi059 systemd[1]: Starting Ceph mds.e for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:19:40.308 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:40 smithi059 podman[72697]: 2024-03-28 18:19:40.17890663 +0000 UTC m=+0.018575081 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:19:40.308 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:40 smithi059 podman[72697]: 2024-03-28 18:19:40.308763192 +0000 UTC m=+0.148431625 container create d4c823fb00dd6c5a548307965258f2bf05d7564849384d47eca27c37b30e7f4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS) 2024-03-28T18:19:40.569 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:40 smithi059 podman[72697]: 2024-03-28 18:19:40.544697998 +0000 UTC m=+0.384366431 container init d4c823fb00dd6c5a548307965258f2bf05d7564849384d47eca27c37b30e7f4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-28T18:19:40.569 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:40 smithi059 podman[72697]: 2024-03-28 18:19:40.54736759 +0000 UTC m=+0.387036023 container start d4c823fb00dd6c5a548307965258f2bf05d7564849384d47eca27c37b30e7f4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True) 2024-03-28T18:19:41.035 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:40 smithi059 ceph-mds[72716]: -- 172.21.15.59:0/2710226562 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557479abfa40 con 0x557479a93000 2024-03-28T18:19:41.035 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:40 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: starting mds.e at 2024-03-28T18:19:41.035 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:40 smithi059 bash[72697]: d4c823fb00dd6c5a548307965258f2bf05d7564849384d47eca27c37b30e7f4f 2024-03-28T18:19:41.035 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:19:40 smithi059 systemd[1]: Started Ceph mds.e for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:19:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:40 smithi059 ceph-mon[36190]: daemon mds.e restarted 2024-03-28T18:19:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:40 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:19:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:40 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:19:41.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:40 smithi059 ceph-mon[36190]: osdmap e114: 12 total, 12 up, 12 in 2024-03-28T18:19:41.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:40 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/3246699021,v1:172.21.15.59:6835/3246699021] up:boot 2024-03-28T18:19:41.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:40 smithi059 ceph-mon[36190]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-28T18:19:41.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:40 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:19:41.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:40 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:40 smithi170 ceph-mon[36280]: daemon mds.e restarted 2024-03-28T18:19:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:40 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:19:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:40 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:19:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:40 smithi170 ceph-mon[36280]: osdmap e114: 12 total, 12 up, 12 in 2024-03-28T18:19:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:40 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/3246699021,v1:172.21.15.59:6835/3246699021] up:boot 2024-03-28T18:19:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:40 smithi170 ceph-mon[36280]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-28T18:19:41.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:40 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:19:41.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:40 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:41.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:40 smithi037 ceph-mon[31460]: daemon mds.e restarted 2024-03-28T18:19:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:40 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:19:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:40 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:19:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:40 smithi037 ceph-mon[31460]: osdmap e114: 12 total, 12 up, 12 in 2024-03-28T18:19:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:40 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/3246699021,v1:172.21.15.59:6835/3246699021] up:boot 2024-03-28T18:19:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:40 smithi037 ceph-mon[31460]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-28T18:19:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:40 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:19:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:40 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:42.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:41 smithi059 ceph-mon[36190]: pgmap v283: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T18:19:42.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:41 smithi170 ceph-mon[36280]: pgmap v283: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T18:19:42.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:41 smithi037 ceph-mon[31460]: pgmap v283: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T18:19:43.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:42 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/749986964,v1:172.21.15.170:6833/749986964] up:resolve 2024-03-28T18:19:43.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:42 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:43.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:42 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/749986964,v1:172.21.15.170:6833/749986964] up:resolve 2024-03-28T18:19:43.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:42 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:43.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:42 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/749986964,v1:172.21.15.170:6833/749986964] up:resolve 2024-03-28T18:19:43.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:42 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:44.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:43 smithi037 ceph-mon[31460]: pgmap v285: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 297 B/s wr, 10 op/s 2024-03-28T18:19:44.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:43 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/749986964,v1:172.21.15.170:6833/749986964] up:reconnect 2024-03-28T18:19:44.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:43 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:44.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:43 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999978 2024-03-28T18:19:44.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:43 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00999978 2024-03-28T18:19:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:43 smithi059 ceph-mon[36190]: pgmap v285: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 297 B/s wr, 10 op/s 2024-03-28T18:19:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:43 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/749986964,v1:172.21.15.170:6833/749986964] up:reconnect 2024-03-28T18:19:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:43 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:43 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999978 2024-03-28T18:19:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:43 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00999978 2024-03-28T18:19:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:43 smithi170 ceph-mon[36280]: pgmap v285: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 297 B/s wr, 10 op/s 2024-03-28T18:19:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:43 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/749986964,v1:172.21.15.170:6833/749986964] up:reconnect 2024-03-28T18:19:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:43 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:43 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999978 2024-03-28T18:19:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:43 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00999978 2024-03-28T18:19:45.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:44 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/749986964,v1:172.21.15.170:6833/749986964] up:rejoin 2024-03-28T18:19:45.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:44 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:45.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:44 smithi037 ceph-mon[31460]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-28T18:19:45.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/749986964,v1:172.21.15.170:6833/749986964] up:rejoin 2024-03-28T18:19:45.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:45.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-mon[36280]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-28T18:19:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:44 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/749986964,v1:172.21.15.170:6833/749986964] up:rejoin 2024-03-28T18:19:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:44 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:44 smithi059 ceph-mon[36190]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-28T18:19:45.285 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc182fa0700 time 2024-03-28T18:19:44.863456+0000 2024-03-28T18:19:45.285 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:19:45.285 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:45.286 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc18d5ca4a1] 2024-03-28T18:19:45.286 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc18d5ca667] 2024-03-28T18:19:45.286 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5614d9ef1223] 2024-03-28T18:19:45.286 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5614d9d6b5d6] 2024-03-28T18:19:45.286 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5614d9d6bda1] 2024-03-28T18:19:45.286 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5614d9d6e2d5] 2024-03-28T18:19:45.286 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5614d9dbe233] 2024-03-28T18:19:45.286 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5614d9dc0197] 2024-03-28T18:19:45.286 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5614d9cbf11b] 2024-03-28T18:19:45.286 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5614d9cbf6bc] 2024-03-28T18:19:45.286 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5614d9ca91ff] 2024-03-28T18:19:45.286 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc18d85c488] 2024-03-28T18:19:45.286 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 13: (DispatchQueue::entry()+0x51f) [0x7fc18d85960f] 2024-03-28T18:19:45.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc18d92ce81] 2024-03-28T18:19:45.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc18c3471ca] 2024-03-28T18:19:45.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 16: clone() 2024-03-28T18:19:45.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2024-03-28T18:19:44.864+0000 7fc182fa0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc182fa0700 time 2024-03-28T18:19:44.863456+0000 2024-03-28T18:19:45.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:19:45.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2024-03-28T18:19:45.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:45.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc18d5ca4a1] 2024-03-28T18:19:45.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc18d5ca667] 2024-03-28T18:19:45.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5614d9ef1223] 2024-03-28T18:19:45.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5614d9d6b5d6] 2024-03-28T18:19:45.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5614d9d6bda1] 2024-03-28T18:19:45.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5614d9d6e2d5] 2024-03-28T18:19:45.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5614d9dbe233] 2024-03-28T18:19:45.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5614d9dc0197] 2024-03-28T18:19:45.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5614d9cbf11b] 2024-03-28T18:19:45.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5614d9cbf6bc] 2024-03-28T18:19:45.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5614d9ca91ff] 2024-03-28T18:19:45.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc18d85c488] 2024-03-28T18:19:45.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 13: (DispatchQueue::entry()+0x51f) [0x7fc18d85960f] 2024-03-28T18:19:45.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc18d92ce81] 2024-03-28T18:19:45.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc18c3471ca] 2024-03-28T18:19:45.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 16: clone() 2024-03-28T18:19:45.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2024-03-28T18:19:45.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: *** Caught signal (Aborted) ** 2024-03-28T18:19:45.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: in thread 7fc182fa0700 thread_name:ms_dispatch 2024-03-28T18:19:45.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:45.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc18c351d20] 2024-03-28T18:19:45.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2: gsignal() 2024-03-28T18:19:45.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 3: abort() 2024-03-28T18:19:45.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc18d5ca4fb] 2024-03-28T18:19:45.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc18d5ca667] 2024-03-28T18:19:45.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5614d9ef1223] 2024-03-28T18:19:45.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5614d9d6b5d6] 2024-03-28T18:19:45.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5614d9d6bda1] 2024-03-28T18:19:45.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5614d9d6e2d5] 2024-03-28T18:19:45.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5614d9dbe233] 2024-03-28T18:19:45.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5614d9dc0197] 2024-03-28T18:19:45.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5614d9cbf11b] 2024-03-28T18:19:45.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5614d9cbf6bc] 2024-03-28T18:19:45.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5614d9ca91ff] 2024-03-28T18:19:45.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc18d85c488] 2024-03-28T18:19:45.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 16: (DispatchQueue::entry()+0x51f) [0x7fc18d85960f] 2024-03-28T18:19:45.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc18d92ce81] 2024-03-28T18:19:45.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc18c3471ca] 2024-03-28T18:19:45.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 19: clone() 2024-03-28T18:19:45.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2024-03-28T18:19:44.866+0000 7fc182fa0700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:19:45.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: in thread 7fc182fa0700 thread_name:ms_dispatch 2024-03-28T18:19:45.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2024-03-28T18:19:45.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:45.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc18c351d20] 2024-03-28T18:19:45.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2: gsignal() 2024-03-28T18:19:45.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 3: abort() 2024-03-28T18:19:45.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc18d5ca4fb] 2024-03-28T18:19:45.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc18d5ca667] 2024-03-28T18:19:45.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5614d9ef1223] 2024-03-28T18:19:45.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5614d9d6b5d6] 2024-03-28T18:19:45.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5614d9d6bda1] 2024-03-28T18:19:45.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5614d9d6e2d5] 2024-03-28T18:19:45.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5614d9dbe233] 2024-03-28T18:19:45.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5614d9dc0197] 2024-03-28T18:19:45.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5614d9cbf11b] 2024-03-28T18:19:45.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5614d9cbf6bc] 2024-03-28T18:19:45.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5614d9ca91ff] 2024-03-28T18:19:45.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc18d85c488] 2024-03-28T18:19:45.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 16: (DispatchQueue::entry()+0x51f) [0x7fc18d85960f] 2024-03-28T18:19:45.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc18d92ce81] 2024-03-28T18:19:45.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc18c3471ca] 2024-03-28T18:19:45.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 19: clone() 2024-03-28T18:19:45.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:19:45.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2024-03-28T18:19:45.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: -1> 2024-03-28T18:19:44.864+0000 7fc182fa0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc182fa0700 time 2024-03-28T18:19:44.863456+0000 2024-03-28T18:19:45.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:19:45.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2024-03-28T18:19:45.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:45.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc18d5ca4a1] 2024-03-28T18:19:45.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc18d5ca667] 2024-03-28T18:19:45.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5614d9ef1223] 2024-03-28T18:19:45.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5614d9d6b5d6] 2024-03-28T18:19:45.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5614d9d6bda1] 2024-03-28T18:19:45.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5614d9d6e2d5] 2024-03-28T18:19:45.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5614d9dbe233] 2024-03-28T18:19:45.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5614d9dc0197] 2024-03-28T18:19:45.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5614d9cbf11b] 2024-03-28T18:19:45.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5614d9cbf6bc] 2024-03-28T18:19:45.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5614d9ca91ff] 2024-03-28T18:19:45.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc18d85c488] 2024-03-28T18:19:45.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 13: (DispatchQueue::entry()+0x51f) [0x7fc18d85960f] 2024-03-28T18:19:45.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc18d92ce81] 2024-03-28T18:19:45.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc18c3471ca] 2024-03-28T18:19:45.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 16: clone() 2024-03-28T18:19:45.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2024-03-28T18:19:45.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 0> 2024-03-28T18:19:44.866+0000 7fc182fa0700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:19:45.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: in thread 7fc182fa0700 thread_name:ms_dispatch 2024-03-28T18:19:45.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2024-03-28T18:19:45.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:45.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc18c351d20] 2024-03-28T18:19:45.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2: gsignal() 2024-03-28T18:19:45.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 3: abort() 2024-03-28T18:19:45.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc18d5ca4fb] 2024-03-28T18:19:45.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc18d5ca667] 2024-03-28T18:19:45.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5614d9ef1223] 2024-03-28T18:19:45.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5614d9d6b5d6] 2024-03-28T18:19:45.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5614d9d6bda1] 2024-03-28T18:19:45.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5614d9d6e2d5] 2024-03-28T18:19:45.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5614d9dbe233] 2024-03-28T18:19:45.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5614d9dc0197] 2024-03-28T18:19:45.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5614d9cbf11b] 2024-03-28T18:19:45.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5614d9cbf6bc] 2024-03-28T18:19:45.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5614d9ca91ff] 2024-03-28T18:19:45.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc18d85c488] 2024-03-28T18:19:45.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 16: (DispatchQueue::entry()+0x51f) [0x7fc18d85960f] 2024-03-28T18:19:45.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc18d92ce81] 2024-03-28T18:19:45.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc18c3471ca] 2024-03-28T18:19:45.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 19: clone() 2024-03-28T18:19:45.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:19:45.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2024-03-28T18:19:45.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: -1> 2024-03-28T18:19:44.864+0000 7fc182fa0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc182fa0700 time 2024-03-28T18:19:44.863456+0000 2024-03-28T18:19:45.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:19:45.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2024-03-28T18:19:45.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:45.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc18d5ca4a1] 2024-03-28T18:19:45.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc18d5ca667] 2024-03-28T18:19:45.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5614d9ef1223] 2024-03-28T18:19:45.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5614d9d6b5d6] 2024-03-28T18:19:45.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5614d9d6bda1] 2024-03-28T18:19:45.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5614d9d6e2d5] 2024-03-28T18:19:45.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5614d9dbe233] 2024-03-28T18:19:45.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5614d9dc0197] 2024-03-28T18:19:45.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5614d9cbf11b] 2024-03-28T18:19:45.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5614d9cbf6bc] 2024-03-28T18:19:45.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5614d9ca91ff] 2024-03-28T18:19:45.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc18d85c488] 2024-03-28T18:19:45.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 13: (DispatchQueue::entry()+0x51f) [0x7fc18d85960f] 2024-03-28T18:19:45.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc18d92ce81] 2024-03-28T18:19:45.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc18c3471ca] 2024-03-28T18:19:45.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 16: clone() 2024-03-28T18:19:45.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2024-03-28T18:19:45.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 0> 2024-03-28T18:19:44.866+0000 7fc182fa0700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:19:45.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: in thread 7fc182fa0700 thread_name:ms_dispatch 2024-03-28T18:19:45.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2024-03-28T18:19:45.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:19:45.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc18c351d20] 2024-03-28T18:19:45.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2: gsignal() 2024-03-28T18:19:45.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 3: abort() 2024-03-28T18:19:45.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc18d5ca4fb] 2024-03-28T18:19:45.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc18d5ca667] 2024-03-28T18:19:45.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5614d9ef1223] 2024-03-28T18:19:45.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5614d9d6b5d6] 2024-03-28T18:19:45.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5614d9d6bda1] 2024-03-28T18:19:45.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5614d9d6e2d5] 2024-03-28T18:19:45.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5614d9dbe233] 2024-03-28T18:19:45.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5614d9dc0197] 2024-03-28T18:19:45.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5614d9cbf11b] 2024-03-28T18:19:45.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5614d9cbf6bc] 2024-03-28T18:19:45.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5614d9ca91ff] 2024-03-28T18:19:45.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc18d85c488] 2024-03-28T18:19:45.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 16: (DispatchQueue::entry()+0x51f) [0x7fc18d85960f] 2024-03-28T18:19:45.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc18d92ce81] 2024-03-28T18:19:45.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc18c3471ca] 2024-03-28T18:19:45.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 19: clone() 2024-03-28T18:19:45.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:19:45.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72165]: 2024-03-28T18:19:45.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:44 smithi170 podman[72350]: 2024-03-28 18:19:44.924394688 +0000 UTC m=+0.019859111 container died 0fb55f9537e9afecf7f847476a3c26905691eeb5e5e6a136bb3375d2a4a150f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-28T18:19:46.035 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:45 smithi170 podman[72350]: 2024-03-28 18:19:45.678571733 +0000 UTC m=+0.774036147 container remove 0fb55f9537e9afecf7f847476a3c26905691eeb5e5e6a136bb3375d2a4a150f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-03-28T18:19:46.036 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:45 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:19:46.036 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:45 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-28T18:19:46.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:45 smithi170 ceph-mon[36280]: pgmap v286: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:19:46.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:45 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:19:46.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:45 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:19:46.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:45 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/749986964,v1:172.21.15.170:6833/749986964] up:active 2024-03-28T18:19:46.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:45 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:46.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:45 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:19:46.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:45 smithi037 ceph-mon[31460]: pgmap v286: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:19:46.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:45 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:19:46.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:45 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:19:46.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:45 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/749986964,v1:172.21.15.170:6833/749986964] up:active 2024-03-28T18:19:46.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:45 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:46.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:45 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:19:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:45 smithi059 ceph-mon[36190]: pgmap v286: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:19:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:45 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:19:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:45 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:19:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:45 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/749986964,v1:172.21.15.170:6833/749986964] up:active 2024-03-28T18:19:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:45 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:45 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:19:46.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:19:46 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:19:46.428+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:19:48.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:47 smithi037 ceph-mon[31460]: pgmap v287: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 639 B/s wr, 21 op/s 2024-03-28T18:19:48.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:47 smithi059 ceph-mon[36190]: pgmap v287: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 639 B/s wr, 21 op/s 2024-03-28T18:19:48.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:47 smithi170 ceph-mon[36280]: pgmap v287: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 639 B/s wr, 21 op/s 2024-03-28T18:19:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:49 smithi037 ceph-mon[31460]: pgmap v288: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 522 B/s wr, 17 op/s 2024-03-28T18:19:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:49 smithi059 ceph-mon[36190]: pgmap v288: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 522 B/s wr, 17 op/s 2024-03-28T18:19:50.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:49 smithi170 ceph-mon[36280]: pgmap v288: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 522 B/s wr, 17 op/s 2024-03-28T18:19:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:51 smithi037 ceph-mon[31460]: pgmap v289: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 511 B/s wr, 17 op/s 2024-03-28T18:19:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:51 smithi059 ceph-mon[36190]: pgmap v289: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 511 B/s wr, 17 op/s 2024-03-28T18:19:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:51 smithi170 ceph-mon[36280]: pgmap v289: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 511 B/s wr, 17 op/s 2024-03-28T18:19:54.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:53 smithi037 ceph-mon[31460]: pgmap v290: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 427 B/s wr, 17 op/s 2024-03-28T18:19:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:53 smithi059 ceph-mon[36190]: pgmap v290: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 427 B/s wr, 17 op/s 2024-03-28T18:19:54.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:53 smithi170 ceph-mon[36280]: pgmap v290: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 427 B/s wr, 17 op/s 2024-03-28T18:19:56.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:55 smithi037 ceph-mon[31460]: pgmap v291: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 341 B/s wr, 14 op/s 2024-03-28T18:19:56.277 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:55 smithi170 ceph-mon[36280]: pgmap v291: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 341 B/s wr, 14 op/s 2024-03-28T18:19:56.277 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:56 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 3. 2024-03-28T18:19:56.277 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:56 smithi170 systemd[1]: Stopped Ceph mds.l for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:19:56.277 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:56 smithi170 systemd[1]: Starting Ceph mds.l for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:19:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:55 smithi059 ceph-mon[36190]: pgmap v291: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 341 B/s wr, 14 op/s 2024-03-28T18:19:56.535 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:56 smithi170 podman[72447]: 2024-03-28 18:19:56.180526108 +0000 UTC m=+0.024317131 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:19:56.536 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:56 smithi170 podman[72447]: 2024-03-28 18:19:56.316555472 +0000 UTC m=+0.160346481 container create 0d678da002aea92635cce92a59b64d232612487370fe68fa962cece35c230b3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-28T18:19:56.848 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:56 smithi170 podman[72447]: 2024-03-28 18:19:56.540151246 +0000 UTC m=+0.383942258 container init 0d678da002aea92635cce92a59b64d232612487370fe68fa962cece35c230b3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:19:56.848 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:56 smithi170 podman[72447]: 2024-03-28 18:19:56.543518769 +0000 UTC m=+0.387309768 container start 0d678da002aea92635cce92a59b64d232612487370fe68fa962cece35c230b3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-28T18:19:56.848 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:56 smithi170 ceph-mds[72466]: -- 172.21.15.170:0/374780051 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559825ca5a40 con 0x559825c79000 2024-03-28T18:19:56.848 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:56 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: starting mds.l at 2024-03-28T18:19:56.848 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:56 smithi170 bash[72447]: 0d678da002aea92635cce92a59b64d232612487370fe68fa962cece35c230b3b 2024-03-28T18:19:56.848 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:19:56 smithi170 systemd[1]: Started Ceph mds.l for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:19:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:56 smithi037 ceph-mon[31460]: daemon mds.l restarted 2024-03-28T18:19:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:56 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:19:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:56 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:19:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:56 smithi037 ceph-mon[31460]: osdmap e115: 12 total, 12 up, 12 in 2024-03-28T18:19:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:56 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/3028165361,v1:172.21.15.170:6833/3028165361] up:boot 2024-03-28T18:19:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:56 smithi037 ceph-mon[31460]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-03-28T18:19:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:56 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:19:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:56 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:56 smithi059 ceph-mon[36190]: daemon mds.l restarted 2024-03-28T18:19:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:56 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:19:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:56 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:19:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:56 smithi059 ceph-mon[36190]: osdmap e115: 12 total, 12 up, 12 in 2024-03-28T18:19:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:56 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/3028165361,v1:172.21.15.170:6833/3028165361] up:boot 2024-03-28T18:19:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:56 smithi059 ceph-mon[36190]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-03-28T18:19:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:56 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:19:57.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:56 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:56 smithi170 ceph-mon[36280]: daemon mds.l restarted 2024-03-28T18:19:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:56 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:19:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:56 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:19:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:56 smithi170 ceph-mon[36280]: osdmap e115: 12 total, 12 up, 12 in 2024-03-28T18:19:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:56 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/3028165361,v1:172.21.15.170:6833/3028165361] up:boot 2024-03-28T18:19:57.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:56 smithi170 ceph-mon[36280]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-03-28T18:19:57.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:56 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T18:19:57.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:56 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:58.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:57 smithi037 ceph-mon[31460]: pgmap v292: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 341 B/s wr, 16 op/s 2024-03-28T18:19:58.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:57 smithi059 ceph-mon[36190]: pgmap v292: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 341 B/s wr, 16 op/s 2024-03-28T18:19:58.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:57 smithi170 ceph-mon[36280]: pgmap v292: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 341 B/s wr, 16 op/s 2024-03-28T18:19:59.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:58 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/3246699021,v1:172.21.15.59:6835/3246699021] up:resolve 2024-03-28T18:19:59.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:58 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:59.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:58 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/3246699021,v1:172.21.15.59:6835/3246699021] up:resolve 2024-03-28T18:19:59.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:58 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:19:59.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:58 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/3246699021,v1:172.21.15.59:6835/3246699021] up:resolve 2024-03-28T18:19:59.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:58 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:20:00.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:59 smithi037 ceph-mon[31460]: pgmap v294: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-28T18:20:00.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:59 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299997 2024-03-28T18:20:00.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:59 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/3246699021,v1:172.21.15.59:6835/3246699021] up:reconnect 2024-03-28T18:20:00.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:59 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:20:00.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:19:59 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089999 2024-03-28T18:20:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:59 smithi059 ceph-mon[36190]: pgmap v294: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-28T18:20:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:59 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299997 2024-03-28T18:20:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:59 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/3246699021,v1:172.21.15.59:6835/3246699021] up:reconnect 2024-03-28T18:20:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:59 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:20:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:19:59 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089999 2024-03-28T18:20:00.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:59 smithi170 ceph-mon[36280]: pgmap v294: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-28T18:20:00.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:59 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299997 2024-03-28T18:20:00.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:59 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/3246699021,v1:172.21.15.59:6835/3246699021] up:reconnect 2024-03-28T18:20:00.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:59 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:20:00.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:19:59 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089999 2024-03-28T18:20:01.223 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/3246699021,v1:172.21.15.59:6835/3246699021] up:rejoin 2024-03-28T18:20:01.223 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:20:01.223 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-mon[36190]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-03-28T18:20:01.223 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-mon[36190]: Health detail: HEALTH_WARN 1 filesystem is degraded 2024-03-28T18:20:01.223 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-mon[36190]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-28T18:20:01.223 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-mon[36190]: fs cephfs is degraded 2024-03-28T18:20:01.224 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f58b00cc700 time 2024-03-28T18:20:00.930660+0000 2024-03-28T18:20:01.224 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:20:01.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:01.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f58ba6f64a1] 2024-03-28T18:20:01.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f58ba6f6667] 2024-03-28T18:20:01.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557477940223] 2024-03-28T18:20:01.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5574777ba5d6] 2024-03-28T18:20:01.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5574777bada1] 2024-03-28T18:20:01.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5574777bd2d5] 2024-03-28T18:20:01.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55747780d233] 2024-03-28T18:20:01.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55747780f197] 2024-03-28T18:20:01.225 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55747770e11b] 2024-03-28T18:20:01.226 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55747770e6bc] 2024-03-28T18:20:01.226 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5574776f81ff] 2024-03-28T18:20:01.226 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f58ba988488] 2024-03-28T18:20:01.226 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 13: (DispatchQueue::entry()+0x51f) [0x7f58ba98560f] 2024-03-28T18:20:01.226 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f58baa58e81] 2024-03-28T18:20:01.226 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f58b94731ca] 2024-03-28T18:20:01.226 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 16: clone() 2024-03-28T18:20:01.226 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2024-03-28T18:20:00.931+0000 7f58b00cc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f58b00cc700 time 2024-03-28T18:20:00.930660+0000 2024-03-28T18:20:01.226 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:20:01.226 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2024-03-28T18:20:01.226 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:01.226 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f58ba6f64a1] 2024-03-28T18:20:01.227 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f58ba6f6667] 2024-03-28T18:20:01.227 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557477940223] 2024-03-28T18:20:01.227 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5574777ba5d6] 2024-03-28T18:20:01.227 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5574777bada1] 2024-03-28T18:20:01.227 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5574777bd2d5] 2024-03-28T18:20:01.227 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55747780d233] 2024-03-28T18:20:01.227 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55747780f197] 2024-03-28T18:20:01.227 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55747770e11b] 2024-03-28T18:20:01.227 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55747770e6bc] 2024-03-28T18:20:01.227 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5574776f81ff] 2024-03-28T18:20:01.227 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f58ba988488] 2024-03-28T18:20:01.227 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 13: (DispatchQueue::entry()+0x51f) [0x7f58ba98560f] 2024-03-28T18:20:01.227 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f58baa58e81] 2024-03-28T18:20:01.227 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f58b94731ca] 2024-03-28T18:20:01.228 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 16: clone() 2024-03-28T18:20:01.228 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2024-03-28T18:20:01.228 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: *** Caught signal (Aborted) ** 2024-03-28T18:20:01.228 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: in thread 7f58b00cc700 thread_name:ms_dispatch 2024-03-28T18:20:01.228 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:01.228 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f58b947dd20] 2024-03-28T18:20:01.228 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2: gsignal() 2024-03-28T18:20:01.228 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 3: abort() 2024-03-28T18:20:01.228 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f58ba6f64fb] 2024-03-28T18:20:01.228 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f58ba6f6667] 2024-03-28T18:20:01.228 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557477940223] 2024-03-28T18:20:01.228 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5574777ba5d6] 2024-03-28T18:20:01.228 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5574777bada1] 2024-03-28T18:20:01.229 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5574777bd2d5] 2024-03-28T18:20:01.229 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55747780d233] 2024-03-28T18:20:01.229 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55747780f197] 2024-03-28T18:20:01.229 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55747770e11b] 2024-03-28T18:20:01.229 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55747770e6bc] 2024-03-28T18:20:01.229 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5574776f81ff] 2024-03-28T18:20:01.229 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f58ba988488] 2024-03-28T18:20:01.229 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 16: (DispatchQueue::entry()+0x51f) [0x7f58ba98560f] 2024-03-28T18:20:01.229 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f58baa58e81] 2024-03-28T18:20:01.229 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f58b94731ca] 2024-03-28T18:20:01.229 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 19: clone() 2024-03-28T18:20:01.229 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2024-03-28T18:20:00.932+0000 7f58b00cc700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:20:01.229 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: in thread 7f58b00cc700 thread_name:ms_dispatch 2024-03-28T18:20:01.229 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2024-03-28T18:20:01.230 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:01.230 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f58b947dd20] 2024-03-28T18:20:01.230 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2: gsignal() 2024-03-28T18:20:01.230 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 3: abort() 2024-03-28T18:20:01.230 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f58ba6f64fb] 2024-03-28T18:20:01.230 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f58ba6f6667] 2024-03-28T18:20:01.230 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557477940223] 2024-03-28T18:20:01.230 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5574777ba5d6] 2024-03-28T18:20:01.230 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5574777bada1] 2024-03-28T18:20:01.230 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5574777bd2d5] 2024-03-28T18:20:01.230 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55747780d233] 2024-03-28T18:20:01.230 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55747780f197] 2024-03-28T18:20:01.230 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55747770e11b] 2024-03-28T18:20:01.230 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55747770e6bc] 2024-03-28T18:20:01.231 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5574776f81ff] 2024-03-28T18:20:01.231 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f58ba988488] 2024-03-28T18:20:01.231 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 16: (DispatchQueue::entry()+0x51f) [0x7f58ba98560f] 2024-03-28T18:20:01.231 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f58baa58e81] 2024-03-28T18:20:01.231 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f58b94731ca] 2024-03-28T18:20:01.231 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 19: clone() 2024-03-28T18:20:01.231 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:20:01.231 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2024-03-28T18:20:01.231 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: -1> 2024-03-28T18:20:00.931+0000 7f58b00cc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f58b00cc700 time 2024-03-28T18:20:00.930660+0000 2024-03-28T18:20:01.231 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:20:01.231 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2024-03-28T18:20:01.231 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:01.231 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f58ba6f64a1] 2024-03-28T18:20:01.231 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f58ba6f6667] 2024-03-28T18:20:01.232 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557477940223] 2024-03-28T18:20:01.232 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5574777ba5d6] 2024-03-28T18:20:01.232 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5574777bada1] 2024-03-28T18:20:01.233 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5574777bd2d5] 2024-03-28T18:20:01.233 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55747780d233] 2024-03-28T18:20:01.233 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55747780f197] 2024-03-28T18:20:01.233 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55747770e11b] 2024-03-28T18:20:01.233 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55747770e6bc] 2024-03-28T18:20:01.233 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5574776f81ff] 2024-03-28T18:20:01.233 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f58ba988488] 2024-03-28T18:20:01.233 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 13: (DispatchQueue::entry()+0x51f) [0x7f58ba98560f] 2024-03-28T18:20:01.233 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f58baa58e81] 2024-03-28T18:20:01.233 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f58b94731ca] 2024-03-28T18:20:01.233 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 16: clone() 2024-03-28T18:20:01.233 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2024-03-28T18:20:01.233 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 0> 2024-03-28T18:20:00.932+0000 7f58b00cc700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:20:01.233 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: in thread 7f58b00cc700 thread_name:ms_dispatch 2024-03-28T18:20:01.234 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2024-03-28T18:20:01.234 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:01.234 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f58b947dd20] 2024-03-28T18:20:01.234 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2: gsignal() 2024-03-28T18:20:01.234 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 3: abort() 2024-03-28T18:20:01.234 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f58ba6f64fb] 2024-03-28T18:20:01.234 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f58ba6f6667] 2024-03-28T18:20:01.234 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557477940223] 2024-03-28T18:20:01.234 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5574777ba5d6] 2024-03-28T18:20:01.234 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5574777bada1] 2024-03-28T18:20:01.234 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5574777bd2d5] 2024-03-28T18:20:01.234 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55747780d233] 2024-03-28T18:20:01.234 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55747780f197] 2024-03-28T18:20:01.234 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55747770e11b] 2024-03-28T18:20:01.235 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55747770e6bc] 2024-03-28T18:20:01.235 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5574776f81ff] 2024-03-28T18:20:01.235 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f58ba988488] 2024-03-28T18:20:01.235 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 16: (DispatchQueue::entry()+0x51f) [0x7f58ba98560f] 2024-03-28T18:20:01.235 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f58baa58e81] 2024-03-28T18:20:01.235 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f58b94731ca] 2024-03-28T18:20:01.235 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 19: clone() 2024-03-28T18:20:01.235 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:20:01.235 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2024-03-28T18:20:01.235 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: -1> 2024-03-28T18:20:00.931+0000 7f58b00cc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f58b00cc700 time 2024-03-28T18:20:00.930660+0000 2024-03-28T18:20:01.235 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:20:01.235 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2024-03-28T18:20:01.235 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:01.235 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f58ba6f64a1] 2024-03-28T18:20:01.236 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f58ba6f6667] 2024-03-28T18:20:01.236 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557477940223] 2024-03-28T18:20:01.236 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5574777ba5d6] 2024-03-28T18:20:01.236 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5574777bada1] 2024-03-28T18:20:01.236 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5574777bd2d5] 2024-03-28T18:20:01.236 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55747780d233] 2024-03-28T18:20:01.236 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55747780f197] 2024-03-28T18:20:01.236 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55747770e11b] 2024-03-28T18:20:01.236 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55747770e6bc] 2024-03-28T18:20:01.236 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5574776f81ff] 2024-03-28T18:20:01.237 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f58ba988488] 2024-03-28T18:20:01.237 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 13: (DispatchQueue::entry()+0x51f) [0x7f58ba98560f] 2024-03-28T18:20:01.237 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f58baa58e81] 2024-03-28T18:20:01.237 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f58b94731ca] 2024-03-28T18:20:01.237 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 16: clone() 2024-03-28T18:20:01.237 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2024-03-28T18:20:01.237 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 0> 2024-03-28T18:20:00.932+0000 7f58b00cc700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:20:01.237 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: in thread 7f58b00cc700 thread_name:ms_dispatch 2024-03-28T18:20:01.237 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2024-03-28T18:20:01.237 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:01.237 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f58b947dd20] 2024-03-28T18:20:01.237 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2: gsignal() 2024-03-28T18:20:01.237 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 3: abort() 2024-03-28T18:20:01.237 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f58ba6f64fb] 2024-03-28T18:20:01.237 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f58ba6f6667] 2024-03-28T18:20:01.238 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557477940223] 2024-03-28T18:20:01.238 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5574777ba5d6] 2024-03-28T18:20:01.238 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5574777bada1] 2024-03-28T18:20:01.238 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5574777bd2d5] 2024-03-28T18:20:01.238 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55747780d233] 2024-03-28T18:20:01.238 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55747780f197] 2024-03-28T18:20:01.238 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55747770e11b] 2024-03-28T18:20:01.238 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55747770e6bc] 2024-03-28T18:20:01.238 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5574776f81ff] 2024-03-28T18:20:01.238 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f58ba988488] 2024-03-28T18:20:01.238 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 16: (DispatchQueue::entry()+0x51f) [0x7f58ba98560f] 2024-03-28T18:20:01.238 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f58baa58e81] 2024-03-28T18:20:01.239 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f58b94731ca] 2024-03-28T18:20:01.239 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 19: clone() 2024-03-28T18:20:01.239 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:20:01.239 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e[72712]: 2024-03-28T18:20:01.239 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:00 smithi059 podman[72750]: 2024-03-28 18:20:00.995056614 +0000 UTC m=+0.020187072 container died d4c823fb00dd6c5a548307965258f2bf05d7564849384d47eca27c37b30e7f4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-28T18:20:01.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:00 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/3246699021,v1:172.21.15.59:6835/3246699021] up:rejoin 2024-03-28T18:20:01.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:00 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:20:01.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:00 smithi170 ceph-mon[36280]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-03-28T18:20:01.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:00 smithi170 ceph-mon[36280]: Health detail: HEALTH_WARN 1 filesystem is degraded 2024-03-28T18:20:01.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:00 smithi170 ceph-mon[36280]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-28T18:20:01.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:00 smithi170 ceph-mon[36280]: fs cephfs is degraded 2024-03-28T18:20:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:00 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/3246699021,v1:172.21.15.59:6835/3246699021] up:rejoin 2024-03-28T18:20:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:00 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:20:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:00 smithi037 ceph-mon[31460]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-03-28T18:20:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:00 smithi037 ceph-mon[31460]: Health detail: HEALTH_WARN 1 filesystem is degraded 2024-03-28T18:20:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:00 smithi037 ceph-mon[31460]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-28T18:20:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:00 smithi037 ceph-mon[31460]: fs cephfs is degraded 2024-03-28T18:20:01.874 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:20:01 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:20:01.430+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:20:02.240 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:01 smithi059 ceph-mon[36190]: pgmap v295: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-28T18:20:02.240 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:01 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:20:02.240 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:01 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:20:02.240 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:01 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6834/3246699021,v1:172.21.15.59:6835/3246699021] up:active 2024-03-28T18:20:02.240 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:01 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:20:02.240 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:01 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:20:02.241 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:02 smithi059 podman[72750]: 2024-03-28 18:20:02.00041875 +0000 UTC m=+1.025549199 container remove d4c823fb00dd6c5a548307965258f2bf05d7564849384d47eca27c37b30e7f4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-e, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-28T18:20:02.241 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:02 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:20:02.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:01 smithi170 ceph-mon[36280]: pgmap v295: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-28T18:20:02.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:01 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:20:02.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:01 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:20:02.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:01 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6834/3246699021,v1:172.21.15.59:6835/3246699021] up:active 2024-03-28T18:20:02.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:01 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:20:02.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:01 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:20:02.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:01 smithi037 ceph-mon[31460]: pgmap v295: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-28T18:20:02.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:01 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:20:02.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:01 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:20:02.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:01 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6834/3246699021,v1:172.21.15.59:6835/3246699021] up:active 2024-03-28T18:20:02.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:01 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T18:20:02.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:01 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T18:20:02.535 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:02 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service: Failed with result 'exit-code'. 2024-03-28T18:20:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:02 smithi059 ceph-mon[36190]: pgmap v296: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 27 KiB/s wr, 35 op/s 2024-03-28T18:20:03.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:02 smithi170 ceph-mon[36280]: pgmap v296: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 27 KiB/s wr, 35 op/s 2024-03-28T18:20:03.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:02 smithi037 ceph-mon[31460]: pgmap v296: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 27 KiB/s wr, 35 op/s 2024-03-28T18:20:05.893 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:05 smithi037 ceph-mon[31460]: pgmap v297: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 27 KiB/s wr, 35 op/s 2024-03-28T18:20:05.988 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:05 smithi170 ceph-mon[36280]: pgmap v297: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 27 KiB/s wr, 35 op/s 2024-03-28T18:20:06.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:05 smithi059 ceph-mon[36190]: pgmap v297: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 27 KiB/s wr, 35 op/s 2024-03-28T18:20:07.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:07 smithi037 ceph-mon[31460]: pgmap v298: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 27 KiB/s wr, 36 op/s 2024-03-28T18:20:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:07 smithi059 ceph-mon[36190]: pgmap v298: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 27 KiB/s wr, 36 op/s 2024-03-28T18:20:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:07 smithi170 ceph-mon[36280]: pgmap v298: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 27 KiB/s wr, 36 op/s 2024-03-28T18:20:09.677 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:09 smithi037 ceph-mon[31460]: pgmap v299: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 23 KiB/s wr, 30 op/s 2024-03-28T18:20:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:09 smithi059 ceph-mon[36190]: pgmap v299: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 23 KiB/s wr, 30 op/s 2024-03-28T18:20:10.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:09 smithi170 ceph-mon[36280]: pgmap v299: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 23 KiB/s wr, 30 op/s 2024-03-28T18:20:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:11 smithi059 ceph-mon[36190]: pgmap v300: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:20:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:11 smithi170 ceph-mon[36280]: pgmap v300: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:20:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:11 smithi037 ceph-mon[31460]: pgmap v300: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:20:12.785 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:12 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service: Scheduled restart job, restart counter is at 6. 2024-03-28T18:20:12.785 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:12 smithi059 systemd[1]: Stopped Ceph mds.e for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:20:12.785 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:12 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service: Start request repeated too quickly. 2024-03-28T18:20:12.785 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:12 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.e.service: Failed with result 'exit-code'. 2024-03-28T18:20:12.785 INFO:journalctl@ceph.mds.e.smithi059.stdout:Mar 28 18:20:12 smithi059 systemd[1]: Failed to start Ceph mds.e for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:20:13.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:13 smithi037 ceph-mon[31460]: pgmap v301: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 23 KiB/s wr, 32 op/s 2024-03-28T18:20:14.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:13 smithi059 ceph-mon[36190]: pgmap v301: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 23 KiB/s wr, 32 op/s 2024-03-28T18:20:14.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:13 smithi170 ceph-mon[36280]: pgmap v301: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 23 KiB/s wr, 32 op/s 2024-03-28T18:20:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:15 smithi059 ceph-mon[36190]: pgmap v302: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:20:16.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:15 smithi170 ceph-mon[36280]: pgmap v302: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:20:16.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:15 smithi037 ceph-mon[31460]: pgmap v302: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:20:18.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:17 smithi059 ceph-mon[36190]: pgmap v303: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-28T18:20:18.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:17 smithi170 ceph-mon[36280]: pgmap v303: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-28T18:20:18.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:17 smithi037 ceph-mon[31460]: pgmap v303: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-28T18:20:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:19 smithi059 ceph-mon[36190]: pgmap v304: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T18:20:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:19 smithi059 ceph-mon[36190]: Replacing daemon mds.e as rank 1 with standby daemon mds.l 2024-03-28T18:20:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:19 smithi059 ceph-mon[36190]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-03-28T18:20:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:19 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:20:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:19 smithi059 ceph-mon[36190]: osdmap e116: 12 total, 12 up, 12 in 2024-03-28T18:20:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:19 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:20.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:19 smithi170 ceph-mon[36280]: pgmap v304: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T18:20:20.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:19 smithi170 ceph-mon[36280]: Replacing daemon mds.e as rank 1 with standby daemon mds.l 2024-03-28T18:20:20.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:19 smithi170 ceph-mon[36280]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-03-28T18:20:20.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:19 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:20:20.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:19 smithi170 ceph-mon[36280]: osdmap e116: 12 total, 12 up, 12 in 2024-03-28T18:20:20.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:19 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:20.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:19 smithi037 ceph-mon[31460]: pgmap v304: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T18:20:20.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:19 smithi037 ceph-mon[31460]: Replacing daemon mds.e as rank 1 with standby daemon mds.l 2024-03-28T18:20:20.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:19 smithi037 ceph-mon[31460]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-03-28T18:20:20.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:19 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:20:20.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:19 smithi037 ceph-mon[31460]: osdmap e116: 12 total, 12 up, 12 in 2024-03-28T18:20:20.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:19 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:21.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:20 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/3028165361,v1:172.21.15.170:6833/3028165361] up:resolve 2024-03-28T18:20:21.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:20 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:20 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/3028165361,v1:172.21.15.170:6833/3028165361] up:resolve 2024-03-28T18:20:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:20 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:21.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:20 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/3028165361,v1:172.21.15.170:6833/3028165361] up:resolve 2024-03-28T18:20:21.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:20 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:21 smithi037 ceph-mon[31460]: pgmap v306: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:20:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:21 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/3028165361,v1:172.21.15.170:6833/3028165361] up:reconnect 2024-03-28T18:20:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:21 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:21 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999978 2024-03-28T18:20:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:21 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00999978 2024-03-28T18:20:22.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:21 smithi059 ceph-mon[36190]: pgmap v306: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:20:22.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:21 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/3028165361,v1:172.21.15.170:6833/3028165361] up:reconnect 2024-03-28T18:20:22.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:21 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:22.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:21 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999978 2024-03-28T18:20:22.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:21 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00999978 2024-03-28T18:20:22.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:21 smithi170 ceph-mon[36280]: pgmap v306: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:20:22.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:21 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/3028165361,v1:172.21.15.170:6833/3028165361] up:reconnect 2024-03-28T18:20:22.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:21 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:22.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:21 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999978 2024-03-28T18:20:22.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:21 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00999978 2024-03-28T18:20:23.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:22 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/3028165361,v1:172.21.15.170:6833/3028165361] up:rejoin 2024-03-28T18:20:23.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:22 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:23.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:22 smithi037 ceph-mon[31460]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-28T18:20:23.236 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/3028165361,v1:172.21.15.170:6833/3028165361] up:rejoin 2024-03-28T18:20:23.236 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:23.236 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-mon[36280]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-28T18:20:23.236 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcecfc41700 time 2024-03-28T18:20:22.861386+0000 2024-03-28T18:20:23.236 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:20:23.236 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:23.236 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fceda26b4a1] 2024-03-28T18:20:23.236 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fceda26b667] 2024-03-28T18:20:23.237 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559823cb0223] 2024-03-28T18:20:23.237 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559823b2a5d6] 2024-03-28T18:20:23.237 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559823b2ada1] 2024-03-28T18:20:23.237 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559823b2d2d5] 2024-03-28T18:20:23.237 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559823b7d233] 2024-03-28T18:20:23.237 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559823b7f197] 2024-03-28T18:20:23.237 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559823a7e11b] 2024-03-28T18:20:23.237 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559823a7e6bc] 2024-03-28T18:20:23.237 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559823a681ff] 2024-03-28T18:20:23.237 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fceda4fd488] 2024-03-28T18:20:23.237 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 13: (DispatchQueue::entry()+0x51f) [0x7fceda4fa60f] 2024-03-28T18:20:23.237 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fceda5cde81] 2024-03-28T18:20:23.237 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fced8fe81ca] 2024-03-28T18:20:23.237 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 16: clone() 2024-03-28T18:20:23.238 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2024-03-28T18:20:22.862+0000 7fcecfc41700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcecfc41700 time 2024-03-28T18:20:22.861386+0000 2024-03-28T18:20:23.238 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:20:23.238 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2024-03-28T18:20:23.239 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:23.239 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fceda26b4a1] 2024-03-28T18:20:23.239 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fceda26b667] 2024-03-28T18:20:23.239 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559823cb0223] 2024-03-28T18:20:23.239 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559823b2a5d6] 2024-03-28T18:20:23.239 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559823b2ada1] 2024-03-28T18:20:23.239 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559823b2d2d5] 2024-03-28T18:20:23.239 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559823b7d233] 2024-03-28T18:20:23.239 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559823b7f197] 2024-03-28T18:20:23.239 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559823a7e11b] 2024-03-28T18:20:23.239 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559823a7e6bc] 2024-03-28T18:20:23.239 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559823a681ff] 2024-03-28T18:20:23.239 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fceda4fd488] 2024-03-28T18:20:23.239 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 13: (DispatchQueue::entry()+0x51f) [0x7fceda4fa60f] 2024-03-28T18:20:23.240 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fceda5cde81] 2024-03-28T18:20:23.240 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fced8fe81ca] 2024-03-28T18:20:23.240 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 16: clone() 2024-03-28T18:20:23.240 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2024-03-28T18:20:23.240 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: *** Caught signal (Aborted) ** 2024-03-28T18:20:23.240 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: in thread 7fcecfc41700 thread_name:ms_dispatch 2024-03-28T18:20:23.240 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:23.240 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fced8ff2d20] 2024-03-28T18:20:23.240 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2: gsignal() 2024-03-28T18:20:23.240 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 3: abort() 2024-03-28T18:20:23.240 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fceda26b4fb] 2024-03-28T18:20:23.240 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fceda26b667] 2024-03-28T18:20:23.240 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559823cb0223] 2024-03-28T18:20:23.240 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559823b2a5d6] 2024-03-28T18:20:23.241 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559823b2ada1] 2024-03-28T18:20:23.241 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559823b2d2d5] 2024-03-28T18:20:23.241 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559823b7d233] 2024-03-28T18:20:23.241 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559823b7f197] 2024-03-28T18:20:23.241 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559823a7e11b] 2024-03-28T18:20:23.241 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559823a7e6bc] 2024-03-28T18:20:23.241 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559823a681ff] 2024-03-28T18:20:23.241 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fceda4fd488] 2024-03-28T18:20:23.241 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 16: (DispatchQueue::entry()+0x51f) [0x7fceda4fa60f] 2024-03-28T18:20:23.241 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fceda5cde81] 2024-03-28T18:20:23.241 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fced8fe81ca] 2024-03-28T18:20:23.241 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 19: clone() 2024-03-28T18:20:23.241 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2024-03-28T18:20:22.865+0000 7fcecfc41700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:20:23.241 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: in thread 7fcecfc41700 thread_name:ms_dispatch 2024-03-28T18:20:23.241 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2024-03-28T18:20:23.242 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:23.242 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fced8ff2d20] 2024-03-28T18:20:23.242 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2: gsignal() 2024-03-28T18:20:23.242 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 3: abort() 2024-03-28T18:20:23.242 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fceda26b4fb] 2024-03-28T18:20:23.242 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fceda26b667] 2024-03-28T18:20:23.242 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559823cb0223] 2024-03-28T18:20:23.242 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559823b2a5d6] 2024-03-28T18:20:23.242 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559823b2ada1] 2024-03-28T18:20:23.242 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559823b2d2d5] 2024-03-28T18:20:23.242 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559823b7d233] 2024-03-28T18:20:23.242 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559823b7f197] 2024-03-28T18:20:23.242 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559823a7e11b] 2024-03-28T18:20:23.242 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559823a7e6bc] 2024-03-28T18:20:23.242 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559823a681ff] 2024-03-28T18:20:23.243 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fceda4fd488] 2024-03-28T18:20:23.243 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 16: (DispatchQueue::entry()+0x51f) [0x7fceda4fa60f] 2024-03-28T18:20:23.243 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fceda5cde81] 2024-03-28T18:20:23.243 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fced8fe81ca] 2024-03-28T18:20:23.243 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 19: clone() 2024-03-28T18:20:23.243 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:20:23.243 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2024-03-28T18:20:23.243 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: -1> 2024-03-28T18:20:22.862+0000 7fcecfc41700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcecfc41700 time 2024-03-28T18:20:22.861386+0000 2024-03-28T18:20:23.243 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:20:23.243 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2024-03-28T18:20:23.243 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:23.243 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fceda26b4a1] 2024-03-28T18:20:23.243 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fceda26b667] 2024-03-28T18:20:23.243 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559823cb0223] 2024-03-28T18:20:23.243 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559823b2a5d6] 2024-03-28T18:20:23.244 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559823b2ada1] 2024-03-28T18:20:23.244 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559823b2d2d5] 2024-03-28T18:20:23.244 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559823b7d233] 2024-03-28T18:20:23.244 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559823b7f197] 2024-03-28T18:20:23.244 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559823a7e11b] 2024-03-28T18:20:23.244 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559823a7e6bc] 2024-03-28T18:20:23.244 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559823a681ff] 2024-03-28T18:20:23.244 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fceda4fd488] 2024-03-28T18:20:23.244 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 13: (DispatchQueue::entry()+0x51f) [0x7fceda4fa60f] 2024-03-28T18:20:23.244 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fceda5cde81] 2024-03-28T18:20:23.244 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fced8fe81ca] 2024-03-28T18:20:23.244 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 16: clone() 2024-03-28T18:20:23.244 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2024-03-28T18:20:23.244 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 0> 2024-03-28T18:20:22.865+0000 7fcecfc41700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:20:23.244 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: in thread 7fcecfc41700 thread_name:ms_dispatch 2024-03-28T18:20:23.244 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2024-03-28T18:20:23.244 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:23.245 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fced8ff2d20] 2024-03-28T18:20:23.245 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2: gsignal() 2024-03-28T18:20:23.245 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 3: abort() 2024-03-28T18:20:23.245 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fceda26b4fb] 2024-03-28T18:20:23.245 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fceda26b667] 2024-03-28T18:20:23.245 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559823cb0223] 2024-03-28T18:20:23.245 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559823b2a5d6] 2024-03-28T18:20:23.245 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559823b2ada1] 2024-03-28T18:20:23.246 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559823b2d2d5] 2024-03-28T18:20:23.246 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559823b7d233] 2024-03-28T18:20:23.246 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559823b7f197] 2024-03-28T18:20:23.246 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559823a7e11b] 2024-03-28T18:20:23.246 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559823a7e6bc] 2024-03-28T18:20:23.246 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559823a681ff] 2024-03-28T18:20:23.246 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fceda4fd488] 2024-03-28T18:20:23.246 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 16: (DispatchQueue::entry()+0x51f) [0x7fceda4fa60f] 2024-03-28T18:20:23.246 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fceda5cde81] 2024-03-28T18:20:23.246 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fced8fe81ca] 2024-03-28T18:20:23.246 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 19: clone() 2024-03-28T18:20:23.246 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:20:23.246 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2024-03-28T18:20:23.246 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: -1> 2024-03-28T18:20:22.862+0000 7fcecfc41700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcecfc41700 time 2024-03-28T18:20:22.861386+0000 2024-03-28T18:20:23.247 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:20:23.247 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2024-03-28T18:20:23.247 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:23.247 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fceda26b4a1] 2024-03-28T18:20:23.247 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fceda26b667] 2024-03-28T18:20:23.247 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559823cb0223] 2024-03-28T18:20:23.247 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559823b2a5d6] 2024-03-28T18:20:23.247 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559823b2ada1] 2024-03-28T18:20:23.247 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559823b2d2d5] 2024-03-28T18:20:23.247 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559823b7d233] 2024-03-28T18:20:23.247 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559823b7f197] 2024-03-28T18:20:23.247 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559823a7e11b] 2024-03-28T18:20:23.247 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559823a7e6bc] 2024-03-28T18:20:23.247 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559823a681ff] 2024-03-28T18:20:23.247 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fceda4fd488] 2024-03-28T18:20:23.247 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 13: (DispatchQueue::entry()+0x51f) [0x7fceda4fa60f] 2024-03-28T18:20:23.248 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fceda5cde81] 2024-03-28T18:20:23.248 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fced8fe81ca] 2024-03-28T18:20:23.248 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 16: clone() 2024-03-28T18:20:23.248 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2024-03-28T18:20:23.248 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 0> 2024-03-28T18:20:22.865+0000 7fcecfc41700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:20:23.248 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: in thread 7fcecfc41700 thread_name:ms_dispatch 2024-03-28T18:20:23.248 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2024-03-28T18:20:23.248 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:23.248 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fced8ff2d20] 2024-03-28T18:20:23.248 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2: gsignal() 2024-03-28T18:20:23.248 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 3: abort() 2024-03-28T18:20:23.248 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fceda26b4fb] 2024-03-28T18:20:23.248 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fceda26b667] 2024-03-28T18:20:23.248 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559823cb0223] 2024-03-28T18:20:23.248 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559823b2a5d6] 2024-03-28T18:20:23.249 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559823b2ada1] 2024-03-28T18:20:23.249 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559823b2d2d5] 2024-03-28T18:20:23.249 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559823b7d233] 2024-03-28T18:20:23.249 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559823b7f197] 2024-03-28T18:20:23.249 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559823a7e11b] 2024-03-28T18:20:23.249 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559823a7e6bc] 2024-03-28T18:20:23.249 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559823a681ff] 2024-03-28T18:20:23.249 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fceda4fd488] 2024-03-28T18:20:23.249 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 16: (DispatchQueue::entry()+0x51f) [0x7fceda4fa60f] 2024-03-28T18:20:23.249 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fceda5cde81] 2024-03-28T18:20:23.249 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fced8fe81ca] 2024-03-28T18:20:23.249 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 19: clone() 2024-03-28T18:20:23.249 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:20:23.249 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72462]: 2024-03-28T18:20:23.249 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:22 smithi170 podman[72503]: 2024-03-28 18:20:22.921503403 +0000 UTC m=+0.018441547 container died 0d678da002aea92635cce92a59b64d232612487370fe68fa962cece35c230b3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-28T18:20:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:22 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/3028165361,v1:172.21.15.170:6833/3028165361] up:rejoin 2024-03-28T18:20:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:22 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:22 smithi059 ceph-mon[36190]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-28T18:20:24.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:23 smithi037 ceph-mon[31460]: pgmap v307: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-03-28T18:20:24.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:23 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:20:24.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:23 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:20:24.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:23 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/3028165361,v1:172.21.15.170:6833/3028165361] up:active 2024-03-28T18:20:24.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:23 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:24.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:23 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T18:20:24.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:23 smithi059 ceph-mon[36190]: pgmap v307: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-03-28T18:20:24.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:23 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:20:24.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:23 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:20:24.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:23 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/3028165361,v1:172.21.15.170:6833/3028165361] up:active 2024-03-28T18:20:24.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:23 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:24.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:23 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T18:20:24.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:23 smithi170 ceph-mon[36280]: pgmap v307: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-03-28T18:20:24.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:23 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:20:24.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:23 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:20:24.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:23 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/3028165361,v1:172.21.15.170:6833/3028165361] up:active 2024-03-28T18:20:24.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:23 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:24.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:23 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T18:20:24.286 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:23 smithi170 podman[72503]: 2024-03-28 18:20:23.819878207 +0000 UTC m=+0.916816351 container remove 0d678da002aea92635cce92a59b64d232612487370fe68fa962cece35c230b3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-28T18:20:24.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:23 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:20:24.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:24 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-28T18:20:26.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:25 smithi037 ceph-mon[31460]: pgmap v308: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-03-28T18:20:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:25 smithi059 ceph-mon[36190]: pgmap v308: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-03-28T18:20:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:25 smithi170 ceph-mon[36280]: pgmap v308: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-03-28T18:20:26.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:20:26 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:20:26.432+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:20:27.910 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:27 smithi037 ceph-mon[31460]: pgmap v309: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:20:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:27 smithi059 ceph-mon[36190]: pgmap v309: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:20:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:27 smithi170 ceph-mon[36280]: pgmap v309: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:20:30.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:29 smithi037 ceph-mon[31460]: pgmap v310: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:20:30.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:29 smithi059 ceph-mon[36190]: pgmap v310: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:20:30.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:29 smithi170 ceph-mon[36280]: pgmap v310: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:20:32.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:31 smithi037 ceph-mon[31460]: pgmap v311: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-03-28T18:20:32.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:31 smithi037 ceph-mon[31460]: osdmap e117: 12 total, 12 up, 12 in 2024-03-28T18:20:32.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:31 smithi059 ceph-mon[36190]: pgmap v311: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-03-28T18:20:32.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:31 smithi059 ceph-mon[36190]: osdmap e117: 12 total, 12 up, 12 in 2024-03-28T18:20:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:31 smithi170 ceph-mon[36280]: pgmap v311: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-03-28T18:20:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:31 smithi170 ceph-mon[36280]: osdmap e117: 12 total, 12 up, 12 in 2024-03-28T18:20:33.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:32 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:20:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:32 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:20:33.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:32 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:20:34.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:33 smithi037 ceph-mon[31460]: pgmap v313: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:20:34.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:33 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:20:34.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:33 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:20:34.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:33 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:20:34.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:33 smithi037 ceph-mon[31460]: osdmap e118: 12 total, 12 up, 12 in 2024-03-28T18:20:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:33 smithi059 ceph-mon[36190]: pgmap v313: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:20:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:33 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:20:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:33 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:20:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:33 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:20:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:33 smithi059 ceph-mon[36190]: osdmap e118: 12 total, 12 up, 12 in 2024-03-28T18:20:34.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:33 smithi170 ceph-mon[36280]: pgmap v313: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:20:34.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:33 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:20:34.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:33 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:20:34.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:33 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:20:34.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:33 smithi170 ceph-mon[36280]: osdmap e118: 12 total, 12 up, 12 in 2024-03-28T18:20:34.644 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:34 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 4. 2024-03-28T18:20:34.644 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:34 smithi170 systemd[1]: Stopped Ceph mds.l for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:20:34.644 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:34 smithi170 systemd[1]: Starting Ceph mds.l for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:20:34.644 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:34 smithi170 podman[72732]: 2024-03-28 18:20:34.418916511 +0000 UTC m=+0.018612590 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:20:34.645 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:34 smithi170 podman[72732]: 2024-03-28 18:20:34.529259689 +0000 UTC m=+0.128955764 container create c33bf6d26ad2e5f6b18ef4cb2cf554d10407b216ba17311bab73da67b5487022 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:20:35.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:34 smithi170 ceph-mon[36280]: daemon mds.l restarted 2024-03-28T18:20:35.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:34 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:20:35.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:34 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:20:35.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:34 smithi170 ceph-mon[36280]: osdmap e119: 12 total, 12 up, 12 in 2024-03-28T18:20:35.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:34 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/1516510856,v1:172.21.15.170:6833/1516510856] up:boot 2024-03-28T18:20:35.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:34 smithi170 ceph-mon[36280]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-28T18:20:35.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:34 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-28T18:20:35.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:34 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:35.036 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:34 smithi170 podman[72732]: 2024-03-28 18:20:34.765426134 +0000 UTC m=+0.365122211 container init c33bf6d26ad2e5f6b18ef4cb2cf554d10407b216ba17311bab73da67b5487022 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:20:35.036 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:34 smithi170 podman[72732]: 2024-03-28 18:20:34.768164992 +0000 UTC m=+0.367861067 container start c33bf6d26ad2e5f6b18ef4cb2cf554d10407b216ba17311bab73da67b5487022 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-28T18:20:35.036 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:34 smithi170 ceph-mds[72751]: -- 172.21.15.170:0/2639692377 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e9910b3a40 con 0x55e991086800 2024-03-28T18:20:35.037 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:34 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: starting mds.l at 2024-03-28T18:20:35.037 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:34 smithi170 bash[72732]: c33bf6d26ad2e5f6b18ef4cb2cf554d10407b216ba17311bab73da67b5487022 2024-03-28T18:20:35.037 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:34 smithi170 systemd[1]: Started Ceph mds.l for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:20:35.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:34 smithi037 ceph-mon[31460]: daemon mds.l restarted 2024-03-28T18:20:35.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:34 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:20:35.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:34 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:20:35.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:34 smithi037 ceph-mon[31460]: osdmap e119: 12 total, 12 up, 12 in 2024-03-28T18:20:35.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:34 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/1516510856,v1:172.21.15.170:6833/1516510856] up:boot 2024-03-28T18:20:35.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:34 smithi037 ceph-mon[31460]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-28T18:20:35.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:34 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-28T18:20:35.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:34 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:34 smithi059 ceph-mon[36190]: daemon mds.l restarted 2024-03-28T18:20:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:34 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:20:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:34 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:20:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:34 smithi059 ceph-mon[36190]: osdmap e119: 12 total, 12 up, 12 in 2024-03-28T18:20:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:34 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/1516510856,v1:172.21.15.170:6833/1516510856] up:boot 2024-03-28T18:20:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:34 smithi059 ceph-mon[36190]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-28T18:20:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:34 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-28T18:20:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:34 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:35 smithi059 ceph-mon[36190]: pgmap v315: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T18:20:36.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:35 smithi170 ceph-mon[36280]: pgmap v315: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T18:20:36.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:35 smithi037 ceph-mon[31460]: pgmap v315: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T18:20:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:36 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/16451159,v1:172.21.15.59:6833/16451159] up:resolve 2024-03-28T18:20:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:36 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:36 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/16451159,v1:172.21.15.59:6833/16451159] up:resolve 2024-03-28T18:20:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:36 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:36 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/16451159,v1:172.21.15.59:6833/16451159] up:resolve 2024-03-28T18:20:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:36 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:38.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:37 smithi059 ceph-mon[36190]: pgmap v317: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-28T18:20:38.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:37 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.000999989 2024-03-28T18:20:38.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:37 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/16451159,v1:172.21.15.59:6833/16451159] up:reconnect 2024-03-28T18:20:38.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:37 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:38.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:37 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089999 2024-03-28T18:20:38.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:37 smithi170 ceph-mon[36280]: pgmap v317: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-28T18:20:38.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:37 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.000999989 2024-03-28T18:20:38.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:37 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/16451159,v1:172.21.15.59:6833/16451159] up:reconnect 2024-03-28T18:20:38.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:37 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:38.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:37 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089999 2024-03-28T18:20:38.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:37 smithi037 ceph-mon[31460]: pgmap v317: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-28T18:20:38.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:37 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.000999989 2024-03-28T18:20:38.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:37 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/16451159,v1:172.21.15.59:6833/16451159] up:reconnect 2024-03-28T18:20:38.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:37 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:38.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:37 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089999 2024-03-28T18:20:39.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:38 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/16451159,v1:172.21.15.59:6833/16451159] up:rejoin 2024-03-28T18:20:39.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:38 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:39.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:38 smithi037 ceph-mon[31460]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-28T18:20:39.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:38 smithi037 ceph-mon[31460]: pgmap v318: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 264 B/s wr, 8 op/s 2024-03-28T18:20:39.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:38 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:20:39.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:38 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:20:39.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:38 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/16451159,v1:172.21.15.59:6833/16451159] up:active 2024-03-28T18:20:39.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:38 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T18:20:39.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/16451159,v1:172.21.15.59:6833/16451159] up:rejoin 2024-03-28T18:20:39.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:39.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-mon[36190]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-28T18:20:39.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-mon[36190]: pgmap v318: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 264 B/s wr, 8 op/s 2024-03-28T18:20:39.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:20:39.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:20:39.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/16451159,v1:172.21.15.59:6833/16451159] up:active 2024-03-28T18:20:39.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T18:20:39.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff51c757700 time 2024-03-28T18:20:38.848537+0000 2024-03-28T18:20:39.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:20:39.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:39.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff526d814a1] 2024-03-28T18:20:39.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff526d81667] 2024-03-28T18:20:39.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c6ad807223] 2024-03-28T18:20:39.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c6ad6815d6] 2024-03-28T18:20:39.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c6ad681da1] 2024-03-28T18:20:39.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c6ad6842d5] 2024-03-28T18:20:39.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c6ad6d4233] 2024-03-28T18:20:39.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c6ad6d6197] 2024-03-28T18:20:39.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c6ad5d511b] 2024-03-28T18:20:39.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c6ad5d56bc] 2024-03-28T18:20:39.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c6ad5bf1ff] 2024-03-28T18:20:39.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff527013488] 2024-03-28T18:20:39.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 13: (DispatchQueue::entry()+0x51f) [0x7ff52701060f] 2024-03-28T18:20:39.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff5270e3e81] 2024-03-28T18:20:39.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff525afe1ca] 2024-03-28T18:20:39.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 16: clone() 2024-03-28T18:20:39.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2024-03-28T18:20:38.849+0000 7ff51c757700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff51c757700 time 2024-03-28T18:20:38.848537+0000 2024-03-28T18:20:39.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:20:39.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2024-03-28T18:20:39.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:39.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff526d814a1] 2024-03-28T18:20:39.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff526d81667] 2024-03-28T18:20:39.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c6ad807223] 2024-03-28T18:20:39.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c6ad6815d6] 2024-03-28T18:20:39.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c6ad681da1] 2024-03-28T18:20:39.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c6ad6842d5] 2024-03-28T18:20:39.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c6ad6d4233] 2024-03-28T18:20:39.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c6ad6d6197] 2024-03-28T18:20:39.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c6ad5d511b] 2024-03-28T18:20:39.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c6ad5d56bc] 2024-03-28T18:20:39.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c6ad5bf1ff] 2024-03-28T18:20:39.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff527013488] 2024-03-28T18:20:39.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 13: (DispatchQueue::entry()+0x51f) [0x7ff52701060f] 2024-03-28T18:20:39.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff5270e3e81] 2024-03-28T18:20:39.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff525afe1ca] 2024-03-28T18:20:39.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 16: clone() 2024-03-28T18:20:39.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2024-03-28T18:20:39.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: *** Caught signal (Aborted) ** 2024-03-28T18:20:39.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: in thread 7ff51c757700 thread_name:ms_dispatch 2024-03-28T18:20:39.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:39.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff525b08d20] 2024-03-28T18:20:39.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2: gsignal() 2024-03-28T18:20:39.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 3: abort() 2024-03-28T18:20:39.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff526d814fb] 2024-03-28T18:20:39.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff526d81667] 2024-03-28T18:20:39.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c6ad807223] 2024-03-28T18:20:39.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c6ad6815d6] 2024-03-28T18:20:39.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c6ad681da1] 2024-03-28T18:20:39.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c6ad6842d5] 2024-03-28T18:20:39.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c6ad6d4233] 2024-03-28T18:20:39.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c6ad6d6197] 2024-03-28T18:20:39.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c6ad5d511b] 2024-03-28T18:20:39.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c6ad5d56bc] 2024-03-28T18:20:39.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c6ad5bf1ff] 2024-03-28T18:20:39.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff527013488] 2024-03-28T18:20:39.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 16: (DispatchQueue::entry()+0x51f) [0x7ff52701060f] 2024-03-28T18:20:39.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff5270e3e81] 2024-03-28T18:20:39.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff525afe1ca] 2024-03-28T18:20:39.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 19: clone() 2024-03-28T18:20:39.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2024-03-28T18:20:38.851+0000 7ff51c757700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:20:39.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: in thread 7ff51c757700 thread_name:ms_dispatch 2024-03-28T18:20:39.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2024-03-28T18:20:39.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:39.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff525b08d20] 2024-03-28T18:20:39.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2: gsignal() 2024-03-28T18:20:39.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 3: abort() 2024-03-28T18:20:39.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff526d814fb] 2024-03-28T18:20:39.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff526d81667] 2024-03-28T18:20:39.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c6ad807223] 2024-03-28T18:20:39.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c6ad6815d6] 2024-03-28T18:20:39.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c6ad681da1] 2024-03-28T18:20:39.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c6ad6842d5] 2024-03-28T18:20:39.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c6ad6d4233] 2024-03-28T18:20:39.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c6ad6d6197] 2024-03-28T18:20:39.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c6ad5d511b] 2024-03-28T18:20:39.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c6ad5d56bc] 2024-03-28T18:20:39.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c6ad5bf1ff] 2024-03-28T18:20:39.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff527013488] 2024-03-28T18:20:39.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 16: (DispatchQueue::entry()+0x51f) [0x7ff52701060f] 2024-03-28T18:20:39.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff5270e3e81] 2024-03-28T18:20:39.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff525afe1ca] 2024-03-28T18:20:39.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 19: clone() 2024-03-28T18:20:39.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:20:39.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2024-03-28T18:20:39.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: -1> 2024-03-28T18:20:38.849+0000 7ff51c757700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff51c757700 time 2024-03-28T18:20:38.848537+0000 2024-03-28T18:20:39.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:20:39.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2024-03-28T18:20:39.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:39.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff526d814a1] 2024-03-28T18:20:39.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff526d81667] 2024-03-28T18:20:39.294 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:38 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/16451159,v1:172.21.15.59:6833/16451159] up:rejoin 2024-03-28T18:20:39.294 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:38 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:39.294 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:38 smithi170 ceph-mon[36280]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-28T18:20:39.294 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:38 smithi170 ceph-mon[36280]: pgmap v318: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 264 B/s wr, 8 op/s 2024-03-28T18:20:39.294 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:38 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:20:39.294 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:38 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:20:39.294 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:38 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/16451159,v1:172.21.15.59:6833/16451159] up:active 2024-03-28T18:20:39.295 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:38 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T18:20:39.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c6ad807223] 2024-03-28T18:20:39.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c6ad6815d6] 2024-03-28T18:20:39.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c6ad681da1] 2024-03-28T18:20:39.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c6ad6842d5] 2024-03-28T18:20:39.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c6ad6d4233] 2024-03-28T18:20:39.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c6ad6d6197] 2024-03-28T18:20:39.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c6ad5d511b] 2024-03-28T18:20:39.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c6ad5d56bc] 2024-03-28T18:20:39.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c6ad5bf1ff] 2024-03-28T18:20:39.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff527013488] 2024-03-28T18:20:39.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 13: (DispatchQueue::entry()+0x51f) [0x7ff52701060f] 2024-03-28T18:20:39.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff5270e3e81] 2024-03-28T18:20:39.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff525afe1ca] 2024-03-28T18:20:39.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 16: clone() 2024-03-28T18:20:39.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2024-03-28T18:20:39.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 0> 2024-03-28T18:20:38.851+0000 7ff51c757700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:20:39.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: in thread 7ff51c757700 thread_name:ms_dispatch 2024-03-28T18:20:39.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2024-03-28T18:20:39.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:39.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff525b08d20] 2024-03-28T18:20:39.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2: gsignal() 2024-03-28T18:20:39.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 3: abort() 2024-03-28T18:20:39.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff526d814fb] 2024-03-28T18:20:39.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff526d81667] 2024-03-28T18:20:39.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c6ad807223] 2024-03-28T18:20:39.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c6ad6815d6] 2024-03-28T18:20:39.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c6ad681da1] 2024-03-28T18:20:39.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c6ad6842d5] 2024-03-28T18:20:39.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c6ad6d4233] 2024-03-28T18:20:39.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c6ad6d6197] 2024-03-28T18:20:39.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c6ad5d511b] 2024-03-28T18:20:39.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c6ad5d56bc] 2024-03-28T18:20:39.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c6ad5bf1ff] 2024-03-28T18:20:39.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff527013488] 2024-03-28T18:20:39.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 16: (DispatchQueue::entry()+0x51f) [0x7ff52701060f] 2024-03-28T18:20:39.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff5270e3e81] 2024-03-28T18:20:39.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff525afe1ca] 2024-03-28T18:20:39.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 19: clone() 2024-03-28T18:20:39.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:20:39.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2024-03-28T18:20:39.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: -1> 2024-03-28T18:20:38.849+0000 7ff51c757700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff51c757700 time 2024-03-28T18:20:38.848537+0000 2024-03-28T18:20:39.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:20:39.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2024-03-28T18:20:39.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:39.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff526d814a1] 2024-03-28T18:20:39.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff526d81667] 2024-03-28T18:20:39.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c6ad807223] 2024-03-28T18:20:39.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c6ad6815d6] 2024-03-28T18:20:39.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c6ad681da1] 2024-03-28T18:20:39.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c6ad6842d5] 2024-03-28T18:20:39.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c6ad6d4233] 2024-03-28T18:20:39.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c6ad6d6197] 2024-03-28T18:20:39.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c6ad5d511b] 2024-03-28T18:20:39.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c6ad5d56bc] 2024-03-28T18:20:39.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c6ad5bf1ff] 2024-03-28T18:20:39.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff527013488] 2024-03-28T18:20:39.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 13: (DispatchQueue::entry()+0x51f) [0x7ff52701060f] 2024-03-28T18:20:39.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff5270e3e81] 2024-03-28T18:20:39.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff525afe1ca] 2024-03-28T18:20:39.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 16: clone() 2024-03-28T18:20:39.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2024-03-28T18:20:39.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 0> 2024-03-28T18:20:38.851+0000 7ff51c757700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:20:39.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: in thread 7ff51c757700 thread_name:ms_dispatch 2024-03-28T18:20:39.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2024-03-28T18:20:39.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:39.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff525b08d20] 2024-03-28T18:20:39.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2: gsignal() 2024-03-28T18:20:39.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 3: abort() 2024-03-28T18:20:39.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff526d814fb] 2024-03-28T18:20:39.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff526d81667] 2024-03-28T18:20:39.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c6ad807223] 2024-03-28T18:20:39.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c6ad6815d6] 2024-03-28T18:20:39.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c6ad681da1] 2024-03-28T18:20:39.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c6ad6842d5] 2024-03-28T18:20:39.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c6ad6d4233] 2024-03-28T18:20:39.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c6ad6d6197] 2024-03-28T18:20:39.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c6ad5d511b] 2024-03-28T18:20:39.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c6ad5d56bc] 2024-03-28T18:20:39.301 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c6ad5bf1ff] 2024-03-28T18:20:39.301 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff527013488] 2024-03-28T18:20:39.301 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 16: (DispatchQueue::entry()+0x51f) [0x7ff52701060f] 2024-03-28T18:20:39.301 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff5270e3e81] 2024-03-28T18:20:39.301 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff525afe1ca] 2024-03-28T18:20:39.301 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 19: clone() 2024-03-28T18:20:39.301 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:20:39.301 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[65861]: 2024-03-28T18:20:39.301 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:38 smithi059 podman[72939]: 2024-03-28 18:20:38.912709589 +0000 UTC m=+0.020675733 container died 9f1727b54bbe9cf3105f7c4b654df6a4a7f05f36861f880fc4158954c628835b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-28T18:20:40.080 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:39 smithi059 podman[72939]: 2024-03-28 18:20:39.830543167 +0000 UTC m=+0.938509331 container remove 9f1727b54bbe9cf3105f7c4b654df6a4a7f05f36861f880fc4158954c628835b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:20:40.080 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:39 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:20:40.535 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:40 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-28T18:20:40.535 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:40 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Consumed 1.051s CPU time. 2024-03-28T18:20:41.658 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:20:41 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:20:41.434+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:20:42.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:41 smithi059 ceph-mon[36190]: pgmap v319: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:20:42.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:41 smithi170 ceph-mon[36280]: pgmap v319: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:20:42.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:41 smithi037 ceph-mon[31460]: pgmap v319: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:20:43.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:43 smithi037 ceph-mon[31460]: pgmap v320: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 465 B/s wr, 19 op/s 2024-03-28T18:20:44.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:43 smithi059 ceph-mon[36190]: pgmap v320: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 465 B/s wr, 19 op/s 2024-03-28T18:20:44.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:43 smithi170 ceph-mon[36280]: pgmap v320: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 465 B/s wr, 19 op/s 2024-03-28T18:20:46.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:45 smithi059 ceph-mon[36190]: pgmap v321: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 409 B/s wr, 17 op/s 2024-03-28T18:20:46.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:45 smithi170 ceph-mon[36280]: pgmap v321: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 409 B/s wr, 17 op/s 2024-03-28T18:20:46.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:45 smithi037 ceph-mon[31460]: pgmap v321: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 409 B/s wr, 17 op/s 2024-03-28T18:20:48.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:47 smithi059 ceph-mon[36190]: pgmap v322: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 347 B/s wr, 17 op/s 2024-03-28T18:20:48.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:47 smithi170 ceph-mon[36280]: pgmap v322: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 347 B/s wr, 17 op/s 2024-03-28T18:20:48.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:47 smithi037 ceph-mon[31460]: pgmap v322: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 347 B/s wr, 17 op/s 2024-03-28T18:20:50.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:49 smithi059 ceph-mon[36190]: pgmap v323: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 255 B/s wr, 13 op/s 2024-03-28T18:20:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:49 smithi170 ceph-mon[36280]: pgmap v323: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 255 B/s wr, 13 op/s 2024-03-28T18:20:50.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:49 smithi037 ceph-mon[31460]: pgmap v323: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 255 B/s wr, 13 op/s 2024-03-28T18:20:50.547 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:50 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Scheduled restart job, restart counter is at 1. 2024-03-28T18:20:50.548 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:50 smithi059 systemd[1]: Stopped Ceph mds.k for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:20:50.548 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:50 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Consumed 1.051s CPU time. 2024-03-28T18:20:50.548 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:50 smithi059 systemd[1]: Starting Ceph mds.k for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:20:50.548 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:50 smithi059 podman[73037]: 2024-03-28 18:20:50.427067705 +0000 UTC m=+0.018954194 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:20:50.818 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:50 smithi059 podman[73037]: 2024-03-28 18:20:50.54825344 +0000 UTC m=+0.140139963 container create df4c02ed98f3157fc07d5d655138bf1eab5b3acea3a4094814356092a5b16b10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:20:50.818 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:50 smithi059 podman[73037]: 2024-03-28 18:20:50.792532562 +0000 UTC m=+0.384419054 container init df4c02ed98f3157fc07d5d655138bf1eab5b3acea3a4094814356092a5b16b10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-28T18:20:50.818 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:50 smithi059 podman[73037]: 2024-03-28 18:20:50.795526127 +0000 UTC m=+0.387412619 container start df4c02ed98f3157fc07d5d655138bf1eab5b3acea3a4094814356092a5b16b10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:20:51.103 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:50 smithi059 ceph-mds[73056]: -- 172.21.15.59:0/1800090932 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56169ef7da40 con 0x56169ef50800 2024-03-28T18:20:51.103 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:50 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: starting mds.k at 2024-03-28T18:20:51.103 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:51 smithi059 bash[73037]: df4c02ed98f3157fc07d5d655138bf1eab5b3acea3a4094814356092a5b16b10 2024-03-28T18:20:51.103 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:20:51 smithi059 systemd[1]: Started Ceph mds.k for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:20:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:51 smithi059 ceph-mon[36190]: pgmap v324: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 255 B/s wr, 13 op/s 2024-03-28T18:20:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:51 smithi059 ceph-mon[36190]: daemon mds.k restarted 2024-03-28T18:20:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:51 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:20:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:51 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:20:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:51 smithi059 ceph-mon[36190]: osdmap e120: 12 total, 12 up, 12 in 2024-03-28T18:20:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:51 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/717335239,v1:172.21.15.59:6833/717335239] up:boot 2024-03-28T18:20:52.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:51 smithi059 ceph-mon[36190]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-28T18:20:52.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:51 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-28T18:20:52.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:51 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:51 smithi170 ceph-mon[36280]: pgmap v324: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 255 B/s wr, 13 op/s 2024-03-28T18:20:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:51 smithi170 ceph-mon[36280]: daemon mds.k restarted 2024-03-28T18:20:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:51 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:20:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:51 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:20:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:51 smithi170 ceph-mon[36280]: osdmap e120: 12 total, 12 up, 12 in 2024-03-28T18:20:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:51 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/717335239,v1:172.21.15.59:6833/717335239] up:boot 2024-03-28T18:20:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:51 smithi170 ceph-mon[36280]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-28T18:20:52.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:51 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-28T18:20:52.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:51 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:52.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:51 smithi037 ceph-mon[31460]: pgmap v324: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 255 B/s wr, 13 op/s 2024-03-28T18:20:52.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:51 smithi037 ceph-mon[31460]: daemon mds.k restarted 2024-03-28T18:20:52.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:51 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:20:52.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:51 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:20:52.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:51 smithi037 ceph-mon[31460]: osdmap e120: 12 total, 12 up, 12 in 2024-03-28T18:20:52.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:51 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/717335239,v1:172.21.15.59:6833/717335239] up:boot 2024-03-28T18:20:52.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:51 smithi037 ceph-mon[31460]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-28T18:20:52.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:51 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-28T18:20:52.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:51 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:52 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/1516510856,v1:172.21.15.170:6833/1516510856] up:resolve 2024-03-28T18:20:53.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:52 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:52 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/1516510856,v1:172.21.15.170:6833/1516510856] up:resolve 2024-03-28T18:20:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:52 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:52 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/1516510856,v1:172.21.15.170:6833/1516510856] up:resolve 2024-03-28T18:20:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:52 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:53 smithi059 ceph-mon[36190]: pgmap v326: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T18:20:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:53 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.000999977 2024-03-28T18:20:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:53 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/1516510856,v1:172.21.15.170:6833/1516510856] up:reconnect 2024-03-28T18:20:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:53 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:53 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089998 2024-03-28T18:20:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:53 smithi170 ceph-mon[36280]: pgmap v326: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T18:20:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:53 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.000999977 2024-03-28T18:20:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:53 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/1516510856,v1:172.21.15.170:6833/1516510856] up:reconnect 2024-03-28T18:20:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:53 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:53 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089998 2024-03-28T18:20:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:53 smithi037 ceph-mon[31460]: pgmap v326: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T18:20:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:53 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.000999977 2024-03-28T18:20:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:53 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/1516510856,v1:172.21.15.170:6833/1516510856] up:reconnect 2024-03-28T18:20:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:53 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:53 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089998 2024-03-28T18:20:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:54 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/1516510856,v1:172.21.15.170:6833/1516510856] up:rejoin 2024-03-28T18:20:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:54 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:54 smithi059 ceph-mon[36190]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-28T18:20:55.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/1516510856,v1:172.21.15.170:6833/1516510856] up:rejoin 2024-03-28T18:20:55.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:55.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-mon[36280]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-28T18:20:55.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3408d34700 time 2024-03-28T18:20:54.923581+0000 2024-03-28T18:20:55.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:20:55.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:55.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f341335e4a1] 2024-03-28T18:20:55.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f341335e667] 2024-03-28T18:20:55.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e98ff86223] 2024-03-28T18:20:55.287 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e98fe005d6] 2024-03-28T18:20:55.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e98fe00da1] 2024-03-28T18:20:55.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e98fe032d5] 2024-03-28T18:20:55.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e98fe53233] 2024-03-28T18:20:55.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e98fe55197] 2024-03-28T18:20:55.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e98fd5411b] 2024-03-28T18:20:55.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e98fd546bc] 2024-03-28T18:20:55.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e98fd3e1ff] 2024-03-28T18:20:55.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f34135f0488] 2024-03-28T18:20:55.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 13: (DispatchQueue::entry()+0x51f) [0x7f34135ed60f] 2024-03-28T18:20:55.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f34136c0e81] 2024-03-28T18:20:55.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f34120db1ca] 2024-03-28T18:20:55.288 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 16: clone() 2024-03-28T18:20:55.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: *** Caught signal (Aborted) ** 2024-03-28T18:20:55.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: in thread 7f3408d34700 thread_name:ms_dispatch 2024-03-28T18:20:55.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2024-03-28T18:20:54.924+0000 7f3408d34700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3408d34700 time 2024-03-28T18:20:54.923581+0000 2024-03-28T18:20:55.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:20:55.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2024-03-28T18:20:55.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:55.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f341335e4a1] 2024-03-28T18:20:55.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f341335e667] 2024-03-28T18:20:55.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e98ff86223] 2024-03-28T18:20:55.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e98fe005d6] 2024-03-28T18:20:55.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e98fe00da1] 2024-03-28T18:20:55.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e98fe032d5] 2024-03-28T18:20:55.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e98fe53233] 2024-03-28T18:20:55.289 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e98fe55197] 2024-03-28T18:20:55.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e98fd5411b] 2024-03-28T18:20:55.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e98fd546bc] 2024-03-28T18:20:55.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e98fd3e1ff] 2024-03-28T18:20:55.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f34135f0488] 2024-03-28T18:20:55.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 13: (DispatchQueue::entry()+0x51f) [0x7f34135ed60f] 2024-03-28T18:20:55.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f34136c0e81] 2024-03-28T18:20:55.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f34120db1ca] 2024-03-28T18:20:55.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 16: clone() 2024-03-28T18:20:55.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2024-03-28T18:20:55.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:55.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f34120e5d20] 2024-03-28T18:20:55.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2: gsignal() 2024-03-28T18:20:55.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 3: abort() 2024-03-28T18:20:55.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f341335e4fb] 2024-03-28T18:20:55.290 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f341335e667] 2024-03-28T18:20:55.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e98ff86223] 2024-03-28T18:20:55.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e98fe005d6] 2024-03-28T18:20:55.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e98fe00da1] 2024-03-28T18:20:55.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e98fe032d5] 2024-03-28T18:20:55.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e98fe53233] 2024-03-28T18:20:55.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e98fe55197] 2024-03-28T18:20:55.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e98fd5411b] 2024-03-28T18:20:55.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e98fd546bc] 2024-03-28T18:20:55.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e98fd3e1ff] 2024-03-28T18:20:55.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f34135f0488] 2024-03-28T18:20:55.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 16: (DispatchQueue::entry()+0x51f) [0x7f34135ed60f] 2024-03-28T18:20:55.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f34136c0e81] 2024-03-28T18:20:55.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f34120db1ca] 2024-03-28T18:20:55.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 19: clone() 2024-03-28T18:20:55.291 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2024-03-28T18:20:54.927+0000 7f3408d34700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:20:55.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: in thread 7f3408d34700 thread_name:ms_dispatch 2024-03-28T18:20:55.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2024-03-28T18:20:55.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:55.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f34120e5d20] 2024-03-28T18:20:55.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2: gsignal() 2024-03-28T18:20:55.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 3: abort() 2024-03-28T18:20:55.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f341335e4fb] 2024-03-28T18:20:55.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f341335e667] 2024-03-28T18:20:55.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e98ff86223] 2024-03-28T18:20:55.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e98fe005d6] 2024-03-28T18:20:55.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e98fe00da1] 2024-03-28T18:20:55.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e98fe032d5] 2024-03-28T18:20:55.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e98fe53233] 2024-03-28T18:20:55.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e98fe55197] 2024-03-28T18:20:55.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e98fd5411b] 2024-03-28T18:20:55.292 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e98fd546bc] 2024-03-28T18:20:55.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e98fd3e1ff] 2024-03-28T18:20:55.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f34135f0488] 2024-03-28T18:20:55.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 16: (DispatchQueue::entry()+0x51f) [0x7f34135ed60f] 2024-03-28T18:20:55.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f34136c0e81] 2024-03-28T18:20:55.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f34120db1ca] 2024-03-28T18:20:55.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 19: clone() 2024-03-28T18:20:55.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:20:55.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2024-03-28T18:20:55.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: -1> 2024-03-28T18:20:54.924+0000 7f3408d34700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3408d34700 time 2024-03-28T18:20:54.923581+0000 2024-03-28T18:20:55.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:20:55.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2024-03-28T18:20:55.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:55.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f341335e4a1] 2024-03-28T18:20:55.293 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f341335e667] 2024-03-28T18:20:55.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e98ff86223] 2024-03-28T18:20:55.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e98fe005d6] 2024-03-28T18:20:55.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e98fe00da1] 2024-03-28T18:20:55.294 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e98fe032d5] 2024-03-28T18:20:55.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e98fe53233] 2024-03-28T18:20:55.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e98fe55197] 2024-03-28T18:20:55.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e98fd5411b] 2024-03-28T18:20:55.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e98fd546bc] 2024-03-28T18:20:55.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e98fd3e1ff] 2024-03-28T18:20:55.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f34135f0488] 2024-03-28T18:20:55.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 13: (DispatchQueue::entry()+0x51f) [0x7f34135ed60f] 2024-03-28T18:20:55.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f34136c0e81] 2024-03-28T18:20:55.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f34120db1ca] 2024-03-28T18:20:55.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 16: clone() 2024-03-28T18:20:55.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2024-03-28T18:20:55.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 0> 2024-03-28T18:20:54.927+0000 7f3408d34700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:20:55.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: in thread 7f3408d34700 thread_name:ms_dispatch 2024-03-28T18:20:55.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2024-03-28T18:20:55.295 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:55.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f34120e5d20] 2024-03-28T18:20:55.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2: gsignal() 2024-03-28T18:20:55.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 3: abort() 2024-03-28T18:20:55.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f341335e4fb] 2024-03-28T18:20:55.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f341335e667] 2024-03-28T18:20:55.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e98ff86223] 2024-03-28T18:20:55.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e98fe005d6] 2024-03-28T18:20:55.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e98fe00da1] 2024-03-28T18:20:55.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e98fe032d5] 2024-03-28T18:20:55.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e98fe53233] 2024-03-28T18:20:55.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e98fe55197] 2024-03-28T18:20:55.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e98fd5411b] 2024-03-28T18:20:55.296 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e98fd546bc] 2024-03-28T18:20:55.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e98fd3e1ff] 2024-03-28T18:20:55.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f34135f0488] 2024-03-28T18:20:55.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 16: (DispatchQueue::entry()+0x51f) [0x7f34135ed60f] 2024-03-28T18:20:55.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f34136c0e81] 2024-03-28T18:20:55.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f34120db1ca] 2024-03-28T18:20:55.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 19: clone() 2024-03-28T18:20:55.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:20:55.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2024-03-28T18:20:55.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: -1> 2024-03-28T18:20:54.924+0000 7f3408d34700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3408d34700 time 2024-03-28T18:20:54.923581+0000 2024-03-28T18:20:55.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:20:55.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2024-03-28T18:20:55.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:55.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f341335e4a1] 2024-03-28T18:20:55.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f341335e667] 2024-03-28T18:20:55.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e98ff86223] 2024-03-28T18:20:55.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e98fe005d6] 2024-03-28T18:20:55.297 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e98fe00da1] 2024-03-28T18:20:55.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e98fe032d5] 2024-03-28T18:20:55.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e98fe53233] 2024-03-28T18:20:55.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e98fe55197] 2024-03-28T18:20:55.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e98fd5411b] 2024-03-28T18:20:55.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e98fd546bc] 2024-03-28T18:20:55.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e98fd3e1ff] 2024-03-28T18:20:55.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f34135f0488] 2024-03-28T18:20:55.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 13: (DispatchQueue::entry()+0x51f) [0x7f34135ed60f] 2024-03-28T18:20:55.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f34136c0e81] 2024-03-28T18:20:55.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f34120db1ca] 2024-03-28T18:20:55.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 16: clone() 2024-03-28T18:20:55.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2024-03-28T18:20:55.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 0> 2024-03-28T18:20:54.927+0000 7f3408d34700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:20:55.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: in thread 7f3408d34700 thread_name:ms_dispatch 2024-03-28T18:20:55.298 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2024-03-28T18:20:55.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:20:55.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f34120e5d20] 2024-03-28T18:20:55.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2: gsignal() 2024-03-28T18:20:55.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 3: abort() 2024-03-28T18:20:55.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f341335e4fb] 2024-03-28T18:20:55.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f341335e667] 2024-03-28T18:20:55.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e98ff86223] 2024-03-28T18:20:55.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e98fe005d6] 2024-03-28T18:20:55.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e98fe00da1] 2024-03-28T18:20:55.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e98fe032d5] 2024-03-28T18:20:55.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e98fe53233] 2024-03-28T18:20:55.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e98fe55197] 2024-03-28T18:20:55.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e98fd5411b] 2024-03-28T18:20:55.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e98fd546bc] 2024-03-28T18:20:55.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e98fd3e1ff] 2024-03-28T18:20:55.299 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f34135f0488] 2024-03-28T18:20:55.300 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 16: (DispatchQueue::entry()+0x51f) [0x7f34135ed60f] 2024-03-28T18:20:55.300 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f34136c0e81] 2024-03-28T18:20:55.300 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f34120db1ca] 2024-03-28T18:20:55.300 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 19: clone() 2024-03-28T18:20:55.300 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:20:55.300 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72747]: 2024-03-28T18:20:55.300 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:54 smithi170 podman[72799]: 2024-03-28 18:20:54.986879455 +0000 UTC m=+0.021876137 container died c33bf6d26ad2e5f6b18ef4cb2cf554d10407b216ba17311bab73da67b5487022 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:20:55.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:54 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/1516510856,v1:172.21.15.170:6833/1516510856] up:rejoin 2024-03-28T18:20:55.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:54 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:55.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:54 smithi037 ceph-mon[31460]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-28T18:20:56.035 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:55 smithi170 podman[72799]: 2024-03-28 18:20:55.773000927 +0000 UTC m=+0.807997611 container remove c33bf6d26ad2e5f6b18ef4cb2cf554d10407b216ba17311bab73da67b5487022 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-28T18:20:56.036 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:55 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:20:56.036 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:20:56 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-28T18:20:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:55 smithi170 ceph-mon[36280]: pgmap v327: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T18:20:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:55 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:20:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:55 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:20:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:55 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/1516510856,v1:172.21.15.170:6833/1516510856] up:active 2024-03-28T18:20:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:55 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:55 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T18:20:56.070 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:55 smithi037 ceph-mon[31460]: pgmap v327: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T18:20:56.070 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:55 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:20:56.070 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:55 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:20:56.070 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:55 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/1516510856,v1:172.21.15.170:6833/1516510856] up:active 2024-03-28T18:20:56.070 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:55 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:56.070 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:55 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T18:20:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:55 smithi059 ceph-mon[36190]: pgmap v327: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T18:20:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:55 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:20:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:55 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:20:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:55 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/1516510856,v1:172.21.15.170:6833/1516510856] up:active 2024-03-28T18:20:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:55 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:20:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:55 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T18:20:56.874 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:20:56 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:20:56.436+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:20:58.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:57 smithi059 ceph-mon[36190]: pgmap v328: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:20:58.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:57 smithi170 ceph-mon[36280]: pgmap v328: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:20:58.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:57 smithi037 ceph-mon[31460]: pgmap v328: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:21:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:20:59 smithi059 ceph-mon[36190]: pgmap v329: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:21:00.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:20:59 smithi170 ceph-mon[36280]: pgmap v329: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:21:00.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:20:59 smithi037 ceph-mon[31460]: pgmap v329: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:21:02.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:01 smithi059 ceph-mon[36190]: pgmap v330: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:21:02.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:01 smithi170 ceph-mon[36280]: pgmap v330: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:21:02.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:01 smithi037 ceph-mon[31460]: pgmap v330: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:21:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:02 smithi059 ceph-mon[36190]: pgmap v331: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 174 B/s wr, 18 op/s 2024-03-28T18:21:03.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:02 smithi170 ceph-mon[36280]: pgmap v331: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 174 B/s wr, 18 op/s 2024-03-28T18:21:03.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:02 smithi037 ceph-mon[31460]: pgmap v331: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 174 B/s wr, 18 op/s 2024-03-28T18:21:05.988 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:05 smithi170 ceph-mon[36280]: pgmap v332: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:21:06.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:05 smithi059 ceph-mon[36190]: pgmap v332: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:21:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:05 smithi037 ceph-mon[31460]: pgmap v332: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:21:06.272 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:06 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 5. 2024-03-28T18:21:06.272 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:06 smithi170 systemd[1]: Stopped Ceph mds.l for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:21:06.273 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:06 smithi170 systemd[1]: Starting Ceph mds.l for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:21:06.535 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:06 smithi170 podman[72897]: 2024-03-28 18:21:06.178788837 +0000 UTC m=+0.025068888 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:21:06.536 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:06 smithi170 podman[72897]: 2024-03-28 18:21:06.281887462 +0000 UTC m=+0.128167497 container create 3635fdcdb7e85227c799433e6ab7e714693a26e8ea4f51cf215f683ba4b3da14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-28T18:21:06.536 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:06 smithi170 podman[72897]: 2024-03-28 18:21:06.518151433 +0000 UTC m=+0.364431467 container init 3635fdcdb7e85227c799433e6ab7e714693a26e8ea4f51cf215f683ba4b3da14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:21:06.536 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:06 smithi170 podman[72897]: 2024-03-28 18:21:06.520804845 +0000 UTC m=+0.367084882 container start 3635fdcdb7e85227c799433e6ab7e714693a26e8ea4f51cf215f683ba4b3da14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:21:07.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:06 smithi059 ceph-mon[36190]: daemon mds.l restarted 2024-03-28T18:21:07.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:06 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:21:07.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:06 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:21:07.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:06 smithi059 ceph-mon[36190]: osdmap e121: 12 total, 12 up, 12 in 2024-03-28T18:21:07.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:06 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/2558453356,v1:172.21.15.170:6833/2558453356] up:boot 2024-03-28T18:21:07.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:06 smithi059 ceph-mon[36190]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-28T18:21:07.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:06 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-28T18:21:07.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:06 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:07.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:06 smithi170 ceph-mon[36280]: daemon mds.l restarted 2024-03-28T18:21:07.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:06 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:21:07.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:06 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:21:07.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:06 smithi170 ceph-mon[36280]: osdmap e121: 12 total, 12 up, 12 in 2024-03-28T18:21:07.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:06 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/2558453356,v1:172.21.15.170:6833/2558453356] up:boot 2024-03-28T18:21:07.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:06 smithi170 ceph-mon[36280]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-28T18:21:07.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:06 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-28T18:21:07.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:06 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:07.037 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:06 smithi170 ceph-mds[72916]: -- 172.21.15.170:0/2159637089 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55860b57ba40 con 0x55860b54ec00 2024-03-28T18:21:07.037 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:06 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: starting mds.l at 2024-03-28T18:21:07.037 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:06 smithi170 bash[72897]: 3635fdcdb7e85227c799433e6ab7e714693a26e8ea4f51cf215f683ba4b3da14 2024-03-28T18:21:07.037 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:06 smithi170 systemd[1]: Started Ceph mds.l for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:21:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:06 smithi037 ceph-mon[31460]: daemon mds.l restarted 2024-03-28T18:21:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:06 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:21:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:06 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:21:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:06 smithi037 ceph-mon[31460]: osdmap e121: 12 total, 12 up, 12 in 2024-03-28T18:21:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:06 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/2558453356,v1:172.21.15.170:6833/2558453356] up:boot 2024-03-28T18:21:07.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:06 smithi037 ceph-mon[31460]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-28T18:21:07.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:06 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-28T18:21:07.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:06 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:07 smithi059 ceph-mon[36190]: pgmap v334: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:21:08.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:07 smithi170 ceph-mon[36280]: pgmap v334: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:21:08.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:07 smithi037 ceph-mon[31460]: pgmap v334: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:21:09.026 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:08 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/717335239,v1:172.21.15.59:6833/717335239] up:resolve 2024-03-28T18:21:09.026 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:08 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:09.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:08 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/717335239,v1:172.21.15.59:6833/717335239] up:resolve 2024-03-28T18:21:09.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:08 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:09.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:08 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/717335239,v1:172.21.15.59:6833/717335239] up:resolve 2024-03-28T18:21:09.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:08 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:09 smithi059 ceph-mon[36190]: pgmap v335: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:21:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:09 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.000999988 2024-03-28T18:21:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:09 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/717335239,v1:172.21.15.59:6833/717335239] up:reconnect 2024-03-28T18:21:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:09 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:09 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999988 2024-03-28T18:21:10.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:09 smithi170 ceph-mon[36280]: pgmap v335: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:21:10.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:09 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.000999988 2024-03-28T18:21:10.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:09 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/717335239,v1:172.21.15.59:6833/717335239] up:reconnect 2024-03-28T18:21:10.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:09 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:10.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:09 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999988 2024-03-28T18:21:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:09 smithi037 ceph-mon[31460]: pgmap v335: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:21:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:09 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.000999988 2024-03-28T18:21:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:09 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/717335239,v1:172.21.15.59:6833/717335239] up:reconnect 2024-03-28T18:21:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:09 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:09 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999988 2024-03-28T18:21:11.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/717335239,v1:172.21.15.59:6833/717335239] up:rejoin 2024-03-28T18:21:11.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:11.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-mon[36190]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-28T18:21:11.036 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb70a48f700 time 2024-03-28T18:21:10.748636+0000 2024-03-28T18:21:11.036 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:21:11.036 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:11.036 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb714ab94a1] 2024-03-28T18:21:11.036 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb714ab9667] 2024-03-28T18:21:11.036 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56169cf5a223] 2024-03-28T18:21:11.036 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56169cdd45d6] 2024-03-28T18:21:11.036 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56169cdd4da1] 2024-03-28T18:21:11.036 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56169cdd72d5] 2024-03-28T18:21:11.036 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56169ce27233] 2024-03-28T18:21:11.036 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56169ce29197] 2024-03-28T18:21:11.036 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56169cd2811b] 2024-03-28T18:21:11.037 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56169cd286bc] 2024-03-28T18:21:11.037 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56169cd121ff] 2024-03-28T18:21:11.037 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb714d4b488] 2024-03-28T18:21:11.037 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 13: (DispatchQueue::entry()+0x51f) [0x7fb714d4860f] 2024-03-28T18:21:11.037 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb714e1be81] 2024-03-28T18:21:11.037 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb7138361ca] 2024-03-28T18:21:11.037 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 16: clone() 2024-03-28T18:21:11.037 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2024-03-28T18:21:10.748+0000 7fb70a48f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb70a48f700 time 2024-03-28T18:21:10.748636+0000 2024-03-28T18:21:11.037 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:21:11.037 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2024-03-28T18:21:11.037 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:11.037 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb714ab94a1] 2024-03-28T18:21:11.037 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb714ab9667] 2024-03-28T18:21:11.038 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56169cf5a223] 2024-03-28T18:21:11.038 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56169cdd45d6] 2024-03-28T18:21:11.038 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56169cdd4da1] 2024-03-28T18:21:11.038 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56169cdd72d5] 2024-03-28T18:21:11.038 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56169ce27233] 2024-03-28T18:21:11.038 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56169ce29197] 2024-03-28T18:21:11.038 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56169cd2811b] 2024-03-28T18:21:11.038 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56169cd286bc] 2024-03-28T18:21:11.038 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56169cd121ff] 2024-03-28T18:21:11.038 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb714d4b488] 2024-03-28T18:21:11.038 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 13: (DispatchQueue::entry()+0x51f) [0x7fb714d4860f] 2024-03-28T18:21:11.038 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb714e1be81] 2024-03-28T18:21:11.038 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb7138361ca] 2024-03-28T18:21:11.039 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 16: clone() 2024-03-28T18:21:11.039 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2024-03-28T18:21:11.039 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: *** Caught signal (Aborted) ** 2024-03-28T18:21:11.039 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: in thread 7fb70a48f700 thread_name:ms_dispatch 2024-03-28T18:21:11.039 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:11.039 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb713840d20] 2024-03-28T18:21:11.039 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:10 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/717335239,v1:172.21.15.59:6833/717335239] up:rejoin 2024-03-28T18:21:11.040 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:10 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:11.040 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:10 smithi170 ceph-mon[36280]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-28T18:21:11.040 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2: gsignal() 2024-03-28T18:21:11.040 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 3: abort() 2024-03-28T18:21:11.040 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb714ab94fb] 2024-03-28T18:21:11.041 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb714ab9667] 2024-03-28T18:21:11.041 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56169cf5a223] 2024-03-28T18:21:11.041 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56169cdd45d6] 2024-03-28T18:21:11.041 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56169cdd4da1] 2024-03-28T18:21:11.041 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56169cdd72d5] 2024-03-28T18:21:11.041 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56169ce27233] 2024-03-28T18:21:11.041 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56169ce29197] 2024-03-28T18:21:11.041 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56169cd2811b] 2024-03-28T18:21:11.041 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56169cd286bc] 2024-03-28T18:21:11.041 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56169cd121ff] 2024-03-28T18:21:11.041 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb714d4b488] 2024-03-28T18:21:11.041 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 16: (DispatchQueue::entry()+0x51f) [0x7fb714d4860f] 2024-03-28T18:21:11.041 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb714e1be81] 2024-03-28T18:21:11.041 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb7138361ca] 2024-03-28T18:21:11.041 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 19: clone() 2024-03-28T18:21:11.041 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2024-03-28T18:21:10.750+0000 7fb70a48f700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:21:11.042 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: in thread 7fb70a48f700 thread_name:ms_dispatch 2024-03-28T18:21:11.042 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2024-03-28T18:21:11.042 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:11.042 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb713840d20] 2024-03-28T18:21:11.042 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2: gsignal() 2024-03-28T18:21:11.042 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 3: abort() 2024-03-28T18:21:11.042 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb714ab94fb] 2024-03-28T18:21:11.042 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb714ab9667] 2024-03-28T18:21:11.042 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56169cf5a223] 2024-03-28T18:21:11.042 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56169cdd45d6] 2024-03-28T18:21:11.042 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56169cdd4da1] 2024-03-28T18:21:11.042 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56169cdd72d5] 2024-03-28T18:21:11.042 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56169ce27233] 2024-03-28T18:21:11.042 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56169ce29197] 2024-03-28T18:21:11.042 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56169cd2811b] 2024-03-28T18:21:11.043 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56169cd286bc] 2024-03-28T18:21:11.043 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56169cd121ff] 2024-03-28T18:21:11.043 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb714d4b488] 2024-03-28T18:21:11.043 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 16: (DispatchQueue::entry()+0x51f) [0x7fb714d4860f] 2024-03-28T18:21:11.043 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb714e1be81] 2024-03-28T18:21:11.043 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb7138361ca] 2024-03-28T18:21:11.043 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 19: clone() 2024-03-28T18:21:11.043 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:21:11.043 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2024-03-28T18:21:11.043 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: -1> 2024-03-28T18:21:10.748+0000 7fb70a48f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb70a48f700 time 2024-03-28T18:21:10.748636+0000 2024-03-28T18:21:11.043 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:21:11.043 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2024-03-28T18:21:11.043 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:11.043 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb714ab94a1] 2024-03-28T18:21:11.043 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb714ab9667] 2024-03-28T18:21:11.044 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56169cf5a223] 2024-03-28T18:21:11.044 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56169cdd45d6] 2024-03-28T18:21:11.044 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56169cdd4da1] 2024-03-28T18:21:11.044 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56169cdd72d5] 2024-03-28T18:21:11.044 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56169ce27233] 2024-03-28T18:21:11.044 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56169ce29197] 2024-03-28T18:21:11.044 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56169cd2811b] 2024-03-28T18:21:11.044 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56169cd286bc] 2024-03-28T18:21:11.044 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56169cd121ff] 2024-03-28T18:21:11.044 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb714d4b488] 2024-03-28T18:21:11.044 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 13: (DispatchQueue::entry()+0x51f) [0x7fb714d4860f] 2024-03-28T18:21:11.044 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb714e1be81] 2024-03-28T18:21:11.044 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb7138361ca] 2024-03-28T18:21:11.044 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 16: clone() 2024-03-28T18:21:11.044 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2024-03-28T18:21:11.044 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 0> 2024-03-28T18:21:10.750+0000 7fb70a48f700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:21:11.045 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: in thread 7fb70a48f700 thread_name:ms_dispatch 2024-03-28T18:21:11.045 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2024-03-28T18:21:11.045 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:11.045 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb713840d20] 2024-03-28T18:21:11.045 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2: gsignal() 2024-03-28T18:21:11.045 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 3: abort() 2024-03-28T18:21:11.045 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb714ab94fb] 2024-03-28T18:21:11.045 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb714ab9667] 2024-03-28T18:21:11.045 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56169cf5a223] 2024-03-28T18:21:11.045 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56169cdd45d6] 2024-03-28T18:21:11.045 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56169cdd4da1] 2024-03-28T18:21:11.045 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56169cdd72d5] 2024-03-28T18:21:11.045 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56169ce27233] 2024-03-28T18:21:11.045 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56169ce29197] 2024-03-28T18:21:11.045 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56169cd2811b] 2024-03-28T18:21:11.046 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56169cd286bc] 2024-03-28T18:21:11.046 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56169cd121ff] 2024-03-28T18:21:11.046 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb714d4b488] 2024-03-28T18:21:11.046 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 16: (DispatchQueue::entry()+0x51f) [0x7fb714d4860f] 2024-03-28T18:21:11.046 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb714e1be81] 2024-03-28T18:21:11.046 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb7138361ca] 2024-03-28T18:21:11.046 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 19: clone() 2024-03-28T18:21:11.046 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:21:11.046 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2024-03-28T18:21:11.046 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: -1> 2024-03-28T18:21:10.748+0000 7fb70a48f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb70a48f700 time 2024-03-28T18:21:10.748636+0000 2024-03-28T18:21:11.046 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:21:11.046 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2024-03-28T18:21:11.047 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:11.047 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb714ab94a1] 2024-03-28T18:21:11.047 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb714ab9667] 2024-03-28T18:21:11.047 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56169cf5a223] 2024-03-28T18:21:11.047 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56169cdd45d6] 2024-03-28T18:21:11.047 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56169cdd4da1] 2024-03-28T18:21:11.047 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56169cdd72d5] 2024-03-28T18:21:11.047 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56169ce27233] 2024-03-28T18:21:11.047 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56169ce29197] 2024-03-28T18:21:11.047 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56169cd2811b] 2024-03-28T18:21:11.047 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56169cd286bc] 2024-03-28T18:21:11.047 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56169cd121ff] 2024-03-28T18:21:11.047 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb714d4b488] 2024-03-28T18:21:11.047 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 13: (DispatchQueue::entry()+0x51f) [0x7fb714d4860f] 2024-03-28T18:21:11.048 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb714e1be81] 2024-03-28T18:21:11.048 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb7138361ca] 2024-03-28T18:21:11.048 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 16: clone() 2024-03-28T18:21:11.048 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2024-03-28T18:21:11.048 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 0> 2024-03-28T18:21:10.750+0000 7fb70a48f700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:21:11.048 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: in thread 7fb70a48f700 thread_name:ms_dispatch 2024-03-28T18:21:11.048 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2024-03-28T18:21:11.048 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:11.048 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb713840d20] 2024-03-28T18:21:11.048 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2: gsignal() 2024-03-28T18:21:11.048 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 3: abort() 2024-03-28T18:21:11.048 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb714ab94fb] 2024-03-28T18:21:11.048 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb714ab9667] 2024-03-28T18:21:11.048 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56169cf5a223] 2024-03-28T18:21:11.048 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56169cdd45d6] 2024-03-28T18:21:11.048 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56169cdd4da1] 2024-03-28T18:21:11.049 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56169cdd72d5] 2024-03-28T18:21:11.049 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56169ce27233] 2024-03-28T18:21:11.049 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56169ce29197] 2024-03-28T18:21:11.049 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56169cd2811b] 2024-03-28T18:21:11.049 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56169cd286bc] 2024-03-28T18:21:11.049 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56169cd121ff] 2024-03-28T18:21:11.049 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb714d4b488] 2024-03-28T18:21:11.049 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 16: (DispatchQueue::entry()+0x51f) [0x7fb714d4860f] 2024-03-28T18:21:11.049 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb714e1be81] 2024-03-28T18:21:11.049 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb7138361ca] 2024-03-28T18:21:11.049 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 19: clone() 2024-03-28T18:21:11.049 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:21:11.049 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73052]: 2024-03-28T18:21:11.049 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:10 smithi059 podman[73103]: 2024-03-28 18:21:10.811754563 +0000 UTC m=+0.020082375 container died df4c02ed98f3157fc07d5d655138bf1eab5b3acea3a4094814356092a5b16b10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:21:11.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:10 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/717335239,v1:172.21.15.59:6833/717335239] up:rejoin 2024-03-28T18:21:11.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:10 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:11.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:10 smithi037 ceph-mon[31460]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-28T18:21:11.705 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:21:11 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:21:11.436+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:21:11.901 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:11 smithi059 podman[73103]: 2024-03-28 18:21:11.648629158 +0000 UTC m=+0.856957015 container remove df4c02ed98f3157fc07d5d655138bf1eab5b3acea3a4094814356092a5b16b10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-28T18:21:11.901 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:11 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:21:11.902 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:11 smithi059 ceph-mon[36190]: pgmap v336: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:21:11.902 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:11 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:21:11.902 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:11 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:21:11.902 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:11 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/717335239,v1:172.21.15.59:6833/717335239] up:active 2024-03-28T18:21:11.902 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:11 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:11.902 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:11 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T18:21:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:11 smithi170 ceph-mon[36280]: pgmap v336: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:21:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:11 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:21:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:11 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:21:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:11 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/717335239,v1:172.21.15.59:6833/717335239] up:active 2024-03-28T18:21:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:11 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:12.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:11 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T18:21:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:11 smithi037 ceph-mon[31460]: pgmap v336: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:21:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:11 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:21:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:11 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:21:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:11 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/717335239,v1:172.21.15.59:6833/717335239] up:active 2024-03-28T18:21:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:11 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:11 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T18:21:12.285 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:11 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-28T18:21:14.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:13 smithi059 ceph-mon[36190]: pgmap v337: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:21:14.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:13 smithi170 ceph-mon[36280]: pgmap v337: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:21:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:13 smithi037 ceph-mon[31460]: pgmap v337: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:21:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:15 smithi059 ceph-mon[36190]: pgmap v338: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:21:16.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:15 smithi170 ceph-mon[36280]: pgmap v338: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:21:16.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:15 smithi037 ceph-mon[31460]: pgmap v338: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:21:18.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:17 smithi059 ceph-mon[36190]: pgmap v339: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:21:18.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:17 smithi170 ceph-mon[36280]: pgmap v339: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:21:18.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:17 smithi037 ceph-mon[31460]: pgmap v339: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:21:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:19 smithi059 ceph-mon[36190]: pgmap v340: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-03-28T18:21:20.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:19 smithi170 ceph-mon[36280]: pgmap v340: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-03-28T18:21:20.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:19 smithi037 ceph-mon[31460]: pgmap v340: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-03-28T18:21:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:21 smithi059 ceph-mon[36190]: pgmap v341: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-03-28T18:21:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:21 smithi170 ceph-mon[36280]: pgmap v341: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-03-28T18:21:22.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:21 smithi037 ceph-mon[31460]: pgmap v341: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-03-28T18:21:22.415 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:22 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Scheduled restart job, restart counter is at 2. 2024-03-28T18:21:22.415 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:22 smithi059 systemd[1]: Stopped Ceph mds.k for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:21:22.416 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:22 smithi059 systemd[1]: Starting Ceph mds.k for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:21:22.416 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:22 smithi059 podman[73195]: 2024-03-28 18:21:22.1780525 +0000 UTC m=+0.019637423 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:21:22.416 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:22 smithi059 podman[73195]: 2024-03-28 18:21:22.41636005 +0000 UTC m=+0.257944964 container create f2576b6e31b2ce8a98518c39c3209a9b7f51984b1c23a1803decabbaa1587c49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True) 2024-03-28T18:21:23.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:22 smithi059 ceph-mon[36190]: daemon mds.k restarted 2024-03-28T18:21:23.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:22 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:21:23.035 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:22 smithi059 podman[73195]: 2024-03-28 18:21:22.669293721 +0000 UTC m=+0.510878639 container init f2576b6e31b2ce8a98518c39c3209a9b7f51984b1c23a1803decabbaa1587c49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-28T18:21:23.035 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:22 smithi059 podman[73195]: 2024-03-28 18:21:22.672065695 +0000 UTC m=+0.513650625 container start f2576b6e31b2ce8a98518c39c3209a9b7f51984b1c23a1803decabbaa1587c49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True) 2024-03-28T18:21:23.035 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:22 smithi059 ceph-mds[73214]: -- 172.21.15.59:0/1158159265 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5581140e9a40 con 0x5581140bcc00 2024-03-28T18:21:23.036 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:22 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: starting mds.k at 2024-03-28T18:21:23.036 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:22 smithi059 bash[73195]: f2576b6e31b2ce8a98518c39c3209a9b7f51984b1c23a1803decabbaa1587c49 2024-03-28T18:21:23.036 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:22 smithi059 systemd[1]: Started Ceph mds.k for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:21:23.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:22 smithi170 ceph-mon[36280]: daemon mds.k restarted 2024-03-28T18:21:23.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:22 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:21:23.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:22 smithi037 ceph-mon[31460]: daemon mds.k restarted 2024-03-28T18:21:23.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:22 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:21:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:23 smithi059 ceph-mon[36190]: pgmap v342: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:21:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:23 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:21:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:23 smithi059 ceph-mon[36190]: osdmap e122: 12 total, 12 up, 12 in 2024-03-28T18:21:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:23 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/10766106,v1:172.21.15.59:6833/10766106] up:boot 2024-03-28T18:21:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:23 smithi059 ceph-mon[36190]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-28T18:21:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:23 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-28T18:21:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:23 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:24.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:23 smithi170 ceph-mon[36280]: pgmap v342: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:21:24.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:23 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:21:24.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:23 smithi170 ceph-mon[36280]: osdmap e122: 12 total, 12 up, 12 in 2024-03-28T18:21:24.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:23 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/10766106,v1:172.21.15.59:6833/10766106] up:boot 2024-03-28T18:21:24.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:23 smithi170 ceph-mon[36280]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-28T18:21:24.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:23 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-28T18:21:24.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:23 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:24.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:23 smithi037 ceph-mon[31460]: pgmap v342: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:21:24.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:23 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:21:24.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:23 smithi037 ceph-mon[31460]: osdmap e122: 12 total, 12 up, 12 in 2024-03-28T18:21:24.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:23 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/10766106,v1:172.21.15.59:6833/10766106] up:boot 2024-03-28T18:21:24.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:23 smithi037 ceph-mon[31460]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-28T18:21:24.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:23 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-28T18:21:24.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:23 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:25.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:24 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/2558453356,v1:172.21.15.170:6833/2558453356] up:resolve 2024-03-28T18:21:25.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:24 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:25.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:24 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/2558453356,v1:172.21.15.170:6833/2558453356] up:resolve 2024-03-28T18:21:25.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:24 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:25.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:24 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/2558453356,v1:172.21.15.170:6833/2558453356] up:resolve 2024-03-28T18:21:25.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:24 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:25 smithi059 ceph-mon[36190]: pgmap v344: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:21:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:25 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199996 2024-03-28T18:21:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:25 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999978 2024-03-28T18:21:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:25 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/2558453356,v1:172.21.15.170:6833/2558453356] up:reconnect 2024-03-28T18:21:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:25 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:26.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:25 smithi170 ceph-mon[36280]: pgmap v344: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:21:26.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:25 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199996 2024-03-28T18:21:26.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:25 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999978 2024-03-28T18:21:26.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:25 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/2558453356,v1:172.21.15.170:6833/2558453356] up:reconnect 2024-03-28T18:21:26.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:25 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:26.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:25 smithi037 ceph-mon[31460]: pgmap v344: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:21:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:25 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199996 2024-03-28T18:21:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:25 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999978 2024-03-28T18:21:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:25 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/2558453356,v1:172.21.15.170:6833/2558453356] up:reconnect 2024-03-28T18:21:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:25 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:26 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/2558453356,v1:172.21.15.170:6833/2558453356] up:rejoin 2024-03-28T18:21:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:26 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:26 smithi059 ceph-mon[36190]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-28T18:21:27.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/2558453356,v1:172.21.15.170:6833/2558453356] up:rejoin 2024-03-28T18:21:27.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:27.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-mon[36280]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-28T18:21:27.037 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0046a1f700 time 2024-03-28T18:21:26.805203+0000 2024-03-28T18:21:27.037 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:21:27.037 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:27.037 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f00510494a1] 2024-03-28T18:21:27.037 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0051049667] 2024-03-28T18:21:27.037 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5586096cd223] 2024-03-28T18:21:27.037 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5586095475d6] 2024-03-28T18:21:27.037 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558609547da1] 2024-03-28T18:21:27.037 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55860954a2d5] 2024-03-28T18:21:27.038 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55860959a233] 2024-03-28T18:21:27.038 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55860959c197] 2024-03-28T18:21:27.038 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55860949b11b] 2024-03-28T18:21:27.038 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55860949b6bc] 2024-03-28T18:21:27.038 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5586094851ff] 2024-03-28T18:21:27.038 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f00512db488] 2024-03-28T18:21:27.038 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 13: (DispatchQueue::entry()+0x51f) [0x7f00512d860f] 2024-03-28T18:21:27.038 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f00513abe81] 2024-03-28T18:21:27.038 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f004fdc61ca] 2024-03-28T18:21:27.038 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 16: clone() 2024-03-28T18:21:27.038 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2024-03-28T18:21:26.805+0000 7f0046a1f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0046a1f700 time 2024-03-28T18:21:26.805203+0000 2024-03-28T18:21:27.039 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:21:27.039 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2024-03-28T18:21:27.039 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:27.039 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f00510494a1] 2024-03-28T18:21:27.039 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0051049667] 2024-03-28T18:21:27.039 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5586096cd223] 2024-03-28T18:21:27.039 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5586095475d6] 2024-03-28T18:21:27.039 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558609547da1] 2024-03-28T18:21:27.039 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55860954a2d5] 2024-03-28T18:21:27.039 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55860959a233] 2024-03-28T18:21:27.039 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55860959c197] 2024-03-28T18:21:27.039 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55860949b11b] 2024-03-28T18:21:27.039 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55860949b6bc] 2024-03-28T18:21:27.039 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5586094851ff] 2024-03-28T18:21:27.040 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f00512db488] 2024-03-28T18:21:27.040 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 13: (DispatchQueue::entry()+0x51f) [0x7f00512d860f] 2024-03-28T18:21:27.040 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f00513abe81] 2024-03-28T18:21:27.040 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f004fdc61ca] 2024-03-28T18:21:27.040 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 16: clone() 2024-03-28T18:21:27.040 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2024-03-28T18:21:27.040 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: *** Caught signal (Aborted) ** 2024-03-28T18:21:27.040 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: in thread 7f0046a1f700 thread_name:ms_dispatch 2024-03-28T18:21:27.040 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:27.040 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f004fdd0d20] 2024-03-28T18:21:27.040 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2: gsignal() 2024-03-28T18:21:27.040 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 3: abort() 2024-03-28T18:21:27.040 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f00510494fb] 2024-03-28T18:21:27.040 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0051049667] 2024-03-28T18:21:27.040 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5586096cd223] 2024-03-28T18:21:27.040 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5586095475d6] 2024-03-28T18:21:27.041 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558609547da1] 2024-03-28T18:21:27.041 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55860954a2d5] 2024-03-28T18:21:27.041 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55860959a233] 2024-03-28T18:21:27.041 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55860959c197] 2024-03-28T18:21:27.041 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55860949b11b] 2024-03-28T18:21:27.041 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55860949b6bc] 2024-03-28T18:21:27.041 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5586094851ff] 2024-03-28T18:21:27.041 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f00512db488] 2024-03-28T18:21:27.041 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 16: (DispatchQueue::entry()+0x51f) [0x7f00512d860f] 2024-03-28T18:21:27.041 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f00513abe81] 2024-03-28T18:21:27.041 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f004fdc61ca] 2024-03-28T18:21:27.041 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 19: clone() 2024-03-28T18:21:27.041 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2024-03-28T18:21:26.807+0000 7f0046a1f700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:21:27.041 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: in thread 7f0046a1f700 thread_name:ms_dispatch 2024-03-28T18:21:27.041 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2024-03-28T18:21:27.041 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:27.042 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f004fdd0d20] 2024-03-28T18:21:27.042 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2: gsignal() 2024-03-28T18:21:27.042 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 3: abort() 2024-03-28T18:21:27.042 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f00510494fb] 2024-03-28T18:21:27.042 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0051049667] 2024-03-28T18:21:27.042 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5586096cd223] 2024-03-28T18:21:27.042 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5586095475d6] 2024-03-28T18:21:27.042 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558609547da1] 2024-03-28T18:21:27.042 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55860954a2d5] 2024-03-28T18:21:27.042 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55860959a233] 2024-03-28T18:21:27.042 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55860959c197] 2024-03-28T18:21:27.042 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55860949b11b] 2024-03-28T18:21:27.042 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55860949b6bc] 2024-03-28T18:21:27.042 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5586094851ff] 2024-03-28T18:21:27.042 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f00512db488] 2024-03-28T18:21:27.043 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 16: (DispatchQueue::entry()+0x51f) [0x7f00512d860f] 2024-03-28T18:21:27.043 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f00513abe81] 2024-03-28T18:21:27.043 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f004fdc61ca] 2024-03-28T18:21:27.043 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 19: clone() 2024-03-28T18:21:27.043 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:21:27.043 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2024-03-28T18:21:27.043 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: -1> 2024-03-28T18:21:26.805+0000 7f0046a1f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0046a1f700 time 2024-03-28T18:21:26.805203+0000 2024-03-28T18:21:27.043 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:21:27.043 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2024-03-28T18:21:27.043 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:27.043 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f00510494a1] 2024-03-28T18:21:27.043 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0051049667] 2024-03-28T18:21:27.044 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5586096cd223] 2024-03-28T18:21:27.044 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5586095475d6] 2024-03-28T18:21:27.044 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558609547da1] 2024-03-28T18:21:27.044 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55860954a2d5] 2024-03-28T18:21:27.044 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55860959a233] 2024-03-28T18:21:27.044 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55860959c197] 2024-03-28T18:21:27.044 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55860949b11b] 2024-03-28T18:21:27.044 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55860949b6bc] 2024-03-28T18:21:27.044 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5586094851ff] 2024-03-28T18:21:27.044 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f00512db488] 2024-03-28T18:21:27.045 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 13: (DispatchQueue::entry()+0x51f) [0x7f00512d860f] 2024-03-28T18:21:27.045 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f00513abe81] 2024-03-28T18:21:27.045 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f004fdc61ca] 2024-03-28T18:21:27.045 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 16: clone() 2024-03-28T18:21:27.045 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2024-03-28T18:21:27.045 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 0> 2024-03-28T18:21:26.807+0000 7f0046a1f700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:21:27.045 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: in thread 7f0046a1f700 thread_name:ms_dispatch 2024-03-28T18:21:27.045 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2024-03-28T18:21:27.045 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:27.045 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f004fdd0d20] 2024-03-28T18:21:27.045 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2: gsignal() 2024-03-28T18:21:27.045 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 3: abort() 2024-03-28T18:21:27.045 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f00510494fb] 2024-03-28T18:21:27.045 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0051049667] 2024-03-28T18:21:27.045 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5586096cd223] 2024-03-28T18:21:27.046 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5586095475d6] 2024-03-28T18:21:27.046 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558609547da1] 2024-03-28T18:21:27.046 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55860954a2d5] 2024-03-28T18:21:27.046 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55860959a233] 2024-03-28T18:21:27.046 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55860959c197] 2024-03-28T18:21:27.046 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55860949b11b] 2024-03-28T18:21:27.046 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55860949b6bc] 2024-03-28T18:21:27.046 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5586094851ff] 2024-03-28T18:21:27.046 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f00512db488] 2024-03-28T18:21:27.046 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 16: (DispatchQueue::entry()+0x51f) [0x7f00512d860f] 2024-03-28T18:21:27.046 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f00513abe81] 2024-03-28T18:21:27.046 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f004fdc61ca] 2024-03-28T18:21:27.046 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 19: clone() 2024-03-28T18:21:27.046 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:21:27.046 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2024-03-28T18:21:27.046 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: -1> 2024-03-28T18:21:26.805+0000 7f0046a1f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0046a1f700 time 2024-03-28T18:21:26.805203+0000 2024-03-28T18:21:27.046 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:21:27.047 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2024-03-28T18:21:27.047 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:27.047 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f00510494a1] 2024-03-28T18:21:27.047 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0051049667] 2024-03-28T18:21:27.047 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5586096cd223] 2024-03-28T18:21:27.047 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5586095475d6] 2024-03-28T18:21:27.047 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558609547da1] 2024-03-28T18:21:27.047 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55860954a2d5] 2024-03-28T18:21:27.047 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55860959a233] 2024-03-28T18:21:27.047 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55860959c197] 2024-03-28T18:21:27.047 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55860949b11b] 2024-03-28T18:21:27.047 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55860949b6bc] 2024-03-28T18:21:27.047 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5586094851ff] 2024-03-28T18:21:27.048 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f00512db488] 2024-03-28T18:21:27.048 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 13: (DispatchQueue::entry()+0x51f) [0x7f00512d860f] 2024-03-28T18:21:27.048 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f00513abe81] 2024-03-28T18:21:27.048 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f004fdc61ca] 2024-03-28T18:21:27.048 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 16: clone() 2024-03-28T18:21:27.048 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2024-03-28T18:21:27.048 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 0> 2024-03-28T18:21:26.807+0000 7f0046a1f700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:21:27.048 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: in thread 7f0046a1f700 thread_name:ms_dispatch 2024-03-28T18:21:27.048 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2024-03-28T18:21:27.048 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:27.048 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f004fdd0d20] 2024-03-28T18:21:27.048 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2: gsignal() 2024-03-28T18:21:27.048 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 3: abort() 2024-03-28T18:21:27.048 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f00510494fb] 2024-03-28T18:21:27.048 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0051049667] 2024-03-28T18:21:27.048 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5586096cd223] 2024-03-28T18:21:27.049 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5586095475d6] 2024-03-28T18:21:27.049 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558609547da1] 2024-03-28T18:21:27.049 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55860954a2d5] 2024-03-28T18:21:27.049 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55860959a233] 2024-03-28T18:21:27.049 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55860959c197] 2024-03-28T18:21:27.049 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55860949b11b] 2024-03-28T18:21:27.049 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55860949b6bc] 2024-03-28T18:21:27.049 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5586094851ff] 2024-03-28T18:21:27.049 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f00512db488] 2024-03-28T18:21:27.049 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 16: (DispatchQueue::entry()+0x51f) [0x7f00512d860f] 2024-03-28T18:21:27.049 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f00513abe81] 2024-03-28T18:21:27.049 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f004fdc61ca] 2024-03-28T18:21:27.049 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 19: clone() 2024-03-28T18:21:27.049 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:21:27.049 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l[72912]: 2024-03-28T18:21:27.049 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:26 smithi170 podman[72953]: 2024-03-28 18:21:26.873667018 +0000 UTC m=+0.020554208 container died 3635fdcdb7e85227c799433e6ab7e714693a26e8ea4f51cf215f683ba4b3da14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:21:27.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:26 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/2558453356,v1:172.21.15.170:6833/2558453356] up:rejoin 2024-03-28T18:21:27.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:26 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:27.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:26 smithi037 ceph-mon[31460]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-28T18:21:27.892 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:27 smithi170 podman[72953]: 2024-03-28 18:21:27.641106492 +0000 UTC m=+0.787993670 container remove 3635fdcdb7e85227c799433e6ab7e714693a26e8ea4f51cf215f683ba4b3da14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-l, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-28T18:21:27.893 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:27 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:21:27.893 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:27 smithi170 ceph-mon[36280]: pgmap v345: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2024-03-28T18:21:27.893 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:27 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:21:27.893 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:27 smithi170 ceph-mon[36280]: Cluster is now healthy 2024-03-28T18:21:27.893 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:27 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.170:6832/2558453356,v1:172.21.15.170:6833/2558453356] up:active 2024-03-28T18:21:27.893 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:27 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:27.893 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:27 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T18:21:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:27 smithi059 ceph-mon[36190]: pgmap v345: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2024-03-28T18:21:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:27 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:21:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:27 smithi059 ceph-mon[36190]: Cluster is now healthy 2024-03-28T18:21:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:27 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.170:6832/2558453356,v1:172.21.15.170:6833/2558453356] up:active 2024-03-28T18:21:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:27 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:27 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T18:21:28.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:27 smithi037 ceph-mon[31460]: pgmap v345: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2024-03-28T18:21:28.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:27 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:21:28.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:27 smithi037 ceph-mon[31460]: Cluster is now healthy 2024-03-28T18:21:28.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:27 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.170:6832/2558453356,v1:172.21.15.170:6833/2558453356] up:active 2024-03-28T18:21:28.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:27 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T18:21:28.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:27 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T18:21:28.285 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:27 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-28T18:21:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:29 smithi059 ceph-mon[36190]: pgmap v346: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2024-03-28T18:21:30.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:29 smithi170 ceph-mon[36280]: pgmap v346: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2024-03-28T18:21:30.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:29 smithi037 ceph-mon[31460]: pgmap v346: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2024-03-28T18:21:31.788 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:21:31 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:21:31.439+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:21:32.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:31 smithi037 ceph-mon[31460]: pgmap v347: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2024-03-28T18:21:32.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:31 smithi059 ceph-mon[36190]: pgmap v347: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2024-03-28T18:21:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:31 smithi170 ceph-mon[36280]: pgmap v347: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2024-03-28T18:21:34.087 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:33 smithi059 ceph-mon[36190]: pgmap v348: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:21:34.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:33 smithi037 ceph-mon[31460]: pgmap v348: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:21:34.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:33 smithi170 ceph-mon[36280]: pgmap v348: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:21:35.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:34 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:21:35.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:34 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:35.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:34 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:35.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:34 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:35.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:34 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:35.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:34 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:35.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:34 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:35.187 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:34 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:21:35.187 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:34 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:35.187 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:34 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:35.187 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:34 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:35.187 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:34 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:35.187 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:34 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:35.187 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:34 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:34 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:21:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:34 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:34 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:34 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:34 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:34 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:34 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:35 smithi059 ceph-mon[36190]: pgmap v349: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-03-28T18:21:36.109 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:35 smithi037 ceph-mon[31460]: pgmap v349: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-03-28T18:21:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:35 smithi170 ceph-mon[36280]: pgmap v349: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-03-28T18:21:37.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: osdmap e123: 12 total, 12 up, 12 in 2024-03-28T18:21:37.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: pgmap v351: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:21:37.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:37.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:37.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:37.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:37.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:37 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:37.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: osdmap e123: 12 total, 12 up, 12 in 2024-03-28T18:21:37.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: pgmap v351: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:21:37.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:37.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:37.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:37.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:37.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:37 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:37.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: osdmap e123: 12 total, 12 up, 12 in 2024-03-28T18:21:37.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: pgmap v351: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:21:37.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:37.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:37.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:37.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:37.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:37.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:37 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:38.523 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:38 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 6. 2024-03-28T18:21:38.523 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:38 smithi170 systemd[1]: Stopped Ceph mds.l for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:21:38.524 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:38 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service: Start request repeated too quickly. 2024-03-28T18:21:38.524 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:38 smithi170 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-28T18:21:38.524 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 28 18:21:38 smithi170 systemd[1]: Failed to start Ceph mds.l for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:21:38.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:38 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T18:21:38.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:38 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T18:21:38.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:38 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T18:21:38.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:38 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T18:21:38.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:38 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:38.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:38 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T18:21:38.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:38 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T18:21:38.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:38 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T18:21:38.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:38 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T18:21:38.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:38 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:38.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:38 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T18:21:38.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:38 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T18:21:38.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:38 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T18:21:38.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:38 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T18:21:38.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:38 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:39.778 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:39 smithi037 ceph-mon[31460]: pgmap v352: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:21:39.778 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:39 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.778 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:39 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.778 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:39 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.778 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:39 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.778 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:39 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.778 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:39 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.778 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:39 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.778 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:39 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.778 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:39 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T18:21:39.778 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:39 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:21:39.779 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:39 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:21:39.779 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:39 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:21:39.779 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:39 smithi037 ceph-mon[31460]: pgmap v353: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 33 KiB/s wr, 25 op/s 2024-03-28T18:21:39.779 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:39 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:39.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:39 smithi059 ceph-mon[36190]: pgmap v352: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:21:39.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:39 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:39 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:39 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:39 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:39 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:39 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:39 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:39 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:39 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T18:21:39.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:39 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:21:39.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:39 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:21:39.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:39 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:21:39.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:39 smithi059 ceph-mon[36190]: pgmap v353: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 33 KiB/s wr, 25 op/s 2024-03-28T18:21:39.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:39 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:39.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:39 smithi170 ceph-mon[36280]: pgmap v352: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-28T18:21:39.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:39 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:39 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:39 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:39 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:39 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:39 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:39 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:39 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:21:39.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:39 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T18:21:39.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:39 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:21:39.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:39 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:21:39.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:39 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:21:39.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:39 smithi170 ceph-mon[36280]: pgmap v353: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 33 KiB/s wr, 25 op/s 2024-03-28T18:21:39.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:39 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:21:40.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:40 smithi059 ceph-mon[36190]: Health check failed: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-28T18:21:40.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:40 smithi170 ceph-mon[36280]: Health check failed: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-28T18:21:40.801 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:40 smithi037 ceph-mon[31460]: Health check failed: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-28T18:21:41.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:41 smithi170 ceph-mon[36280]: pgmap v354: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 124 B/s wr, 3 op/s 2024-03-28T18:21:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:41 smithi037 ceph-mon[31460]: pgmap v354: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 124 B/s wr, 3 op/s 2024-03-28T18:21:42.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:41 smithi059 ceph-mon[36190]: pgmap v354: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 124 B/s wr, 3 op/s 2024-03-28T18:21:44.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:43 smithi037 ceph-mon[31460]: pgmap v355: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 249 B/s wr, 7 op/s 2024-03-28T18:21:44.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:43 smithi037 ceph-mon[31460]: Replacing daemon mds.l as rank 1 with standby daemon mds.k 2024-03-28T18:21:44.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:43 smithi037 ceph-mon[31460]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-03-28T18:21:44.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:43 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:21:44.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:43 smithi037 ceph-mon[31460]: osdmap e124: 12 total, 12 up, 12 in 2024-03-28T18:21:44.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:43 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:21:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:43 smithi059 ceph-mon[36190]: pgmap v355: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 249 B/s wr, 7 op/s 2024-03-28T18:21:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:43 smithi059 ceph-mon[36190]: Replacing daemon mds.l as rank 1 with standby daemon mds.k 2024-03-28T18:21:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:43 smithi059 ceph-mon[36190]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-03-28T18:21:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:43 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:21:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:43 smithi059 ceph-mon[36190]: osdmap e124: 12 total, 12 up, 12 in 2024-03-28T18:21:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:43 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:21:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:43 smithi170 ceph-mon[36280]: pgmap v355: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 249 B/s wr, 7 op/s 2024-03-28T18:21:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:43 smithi170 ceph-mon[36280]: Replacing daemon mds.l as rank 1 with standby daemon mds.k 2024-03-28T18:21:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:43 smithi170 ceph-mon[36280]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-03-28T18:21:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:43 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:21:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:43 smithi170 ceph-mon[36280]: osdmap e124: 12 total, 12 up, 12 in 2024-03-28T18:21:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:43 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:21:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:45 smithi059 ceph-mon[36190]: pgmap v357: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 124 B/s wr, 3 op/s 2024-03-28T18:21:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:45 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/10766106,v1:172.21.15.59:6833/10766106] up:resolve 2024-03-28T18:21:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:45 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:21:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:45 smithi170 ceph-mon[36280]: pgmap v357: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 124 B/s wr, 3 op/s 2024-03-28T18:21:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:45 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/10766106,v1:172.21.15.59:6833/10766106] up:resolve 2024-03-28T18:21:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:45 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:21:46.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:45 smithi037 ceph-mon[31460]: pgmap v357: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 124 B/s wr, 3 op/s 2024-03-28T18:21:46.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:45 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/10766106,v1:172.21.15.59:6833/10766106] up:resolve 2024-03-28T18:21:46.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:45 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:21:47.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:46 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:21:47.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:46 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/10766106,v1:172.21.15.59:6833/10766106] up:reconnect 2024-03-28T18:21:47.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:46 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:21:47.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:46 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089999 2024-03-28T18:21:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:46 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:21:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:46 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/10766106,v1:172.21.15.59:6833/10766106] up:reconnect 2024-03-28T18:21:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:46 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:21:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:46 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089999 2024-03-28T18:21:47.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:46 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:21:47.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:46 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/10766106,v1:172.21.15.59:6833/10766106] up:reconnect 2024-03-28T18:21:47.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:46 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:21:47.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:46 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0089999 2024-03-28T18:21:48.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-mon[36190]: pgmap v358: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 124 B/s wr, 7 op/s 2024-03-28T18:21:48.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/10766106,v1:172.21.15.59:6833/10766106] up:rejoin 2024-03-28T18:21:48.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:21:48.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-mon[36190]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-28T18:21:48.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:47 smithi170 ceph-mon[36280]: pgmap v358: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 124 B/s wr, 7 op/s 2024-03-28T18:21:48.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:47 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/10766106,v1:172.21.15.59:6833/10766106] up:rejoin 2024-03-28T18:21:48.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:47 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:21:48.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:47 smithi170 ceph-mon[36280]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-28T18:21:48.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f019a045700 time 2024-03-28T18:21:47.950473+0000 2024-03-28T18:21:48.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:21:48.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:48.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f01a466f4a1] 2024-03-28T18:21:48.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f01a466f667] 2024-03-28T18:21:48.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5581116bf223] 2024-03-28T18:21:48.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5581115395d6] 2024-03-28T18:21:48.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558111539da1] 2024-03-28T18:21:48.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55811153c2d5] 2024-03-28T18:21:48.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55811158c233] 2024-03-28T18:21:48.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55811158e197] 2024-03-28T18:21:48.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55811148d11b] 2024-03-28T18:21:48.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55811148d6bc] 2024-03-28T18:21:48.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5581114771ff] 2024-03-28T18:21:48.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f01a4901488] 2024-03-28T18:21:48.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 13: (DispatchQueue::entry()+0x51f) [0x7f01a48fe60f] 2024-03-28T18:21:48.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f01a49d1e81] 2024-03-28T18:21:48.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f01a33ec1ca] 2024-03-28T18:21:48.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 16: clone() 2024-03-28T18:21:48.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: *** Caught signal (Aborted) ** 2024-03-28T18:21:48.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: in thread 7f019a045700 thread_name:ms_dispatch 2024-03-28T18:21:48.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2024-03-28T18:21:47.951+0000 7f019a045700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f019a045700 time 2024-03-28T18:21:47.950473+0000 2024-03-28T18:21:48.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:21:48.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2024-03-28T18:21:48.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:48.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f01a466f4a1] 2024-03-28T18:21:48.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f01a466f667] 2024-03-28T18:21:48.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5581116bf223] 2024-03-28T18:21:48.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5581115395d6] 2024-03-28T18:21:48.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558111539da1] 2024-03-28T18:21:48.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55811153c2d5] 2024-03-28T18:21:48.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55811158c233] 2024-03-28T18:21:48.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55811158e197] 2024-03-28T18:21:48.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55811148d11b] 2024-03-28T18:21:48.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55811148d6bc] 2024-03-28T18:21:48.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5581114771ff] 2024-03-28T18:21:48.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f01a4901488] 2024-03-28T18:21:48.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 13: (DispatchQueue::entry()+0x51f) [0x7f01a48fe60f] 2024-03-28T18:21:48.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f01a49d1e81] 2024-03-28T18:21:48.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f01a33ec1ca] 2024-03-28T18:21:48.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 16: clone() 2024-03-28T18:21:48.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2024-03-28T18:21:48.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:48.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f01a33f6d20] 2024-03-28T18:21:48.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2: gsignal() 2024-03-28T18:21:48.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 3: abort() 2024-03-28T18:21:48.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f01a466f4fb] 2024-03-28T18:21:48.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f01a466f667] 2024-03-28T18:21:48.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5581116bf223] 2024-03-28T18:21:48.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5581115395d6] 2024-03-28T18:21:48.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558111539da1] 2024-03-28T18:21:48.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55811153c2d5] 2024-03-28T18:21:48.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55811158c233] 2024-03-28T18:21:48.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55811158e197] 2024-03-28T18:21:48.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55811148d11b] 2024-03-28T18:21:48.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55811148d6bc] 2024-03-28T18:21:48.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5581114771ff] 2024-03-28T18:21:48.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f01a4901488] 2024-03-28T18:21:48.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 16: (DispatchQueue::entry()+0x51f) [0x7f01a48fe60f] 2024-03-28T18:21:48.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f01a49d1e81] 2024-03-28T18:21:48.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f01a33ec1ca] 2024-03-28T18:21:48.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 19: clone() 2024-03-28T18:21:48.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2024-03-28T18:21:47.953+0000 7f019a045700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:21:48.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: in thread 7f019a045700 thread_name:ms_dispatch 2024-03-28T18:21:48.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2024-03-28T18:21:48.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:48.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f01a33f6d20] 2024-03-28T18:21:48.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2: gsignal() 2024-03-28T18:21:48.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 3: abort() 2024-03-28T18:21:48.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f01a466f4fb] 2024-03-28T18:21:48.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f01a466f667] 2024-03-28T18:21:48.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5581116bf223] 2024-03-28T18:21:48.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5581115395d6] 2024-03-28T18:21:48.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558111539da1] 2024-03-28T18:21:48.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55811153c2d5] 2024-03-28T18:21:48.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55811158c233] 2024-03-28T18:21:48.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55811158e197] 2024-03-28T18:21:48.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55811148d11b] 2024-03-28T18:21:48.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55811148d6bc] 2024-03-28T18:21:48.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5581114771ff] 2024-03-28T18:21:48.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f01a4901488] 2024-03-28T18:21:48.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 16: (DispatchQueue::entry()+0x51f) [0x7f01a48fe60f] 2024-03-28T18:21:48.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f01a49d1e81] 2024-03-28T18:21:48.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f01a33ec1ca] 2024-03-28T18:21:48.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 19: clone() 2024-03-28T18:21:48.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:21:48.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2024-03-28T18:21:48.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: -1> 2024-03-28T18:21:47.951+0000 7f019a045700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f019a045700 time 2024-03-28T18:21:47.950473+0000 2024-03-28T18:21:48.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:21:48.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2024-03-28T18:21:48.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:48.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f01a466f4a1] 2024-03-28T18:21:48.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f01a466f667] 2024-03-28T18:21:48.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5581116bf223] 2024-03-28T18:21:48.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5581115395d6] 2024-03-28T18:21:48.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558111539da1] 2024-03-28T18:21:48.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55811153c2d5] 2024-03-28T18:21:48.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55811158c233] 2024-03-28T18:21:48.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55811158e197] 2024-03-28T18:21:48.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55811148d11b] 2024-03-28T18:21:48.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55811148d6bc] 2024-03-28T18:21:48.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5581114771ff] 2024-03-28T18:21:48.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f01a4901488] 2024-03-28T18:21:48.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 13: (DispatchQueue::entry()+0x51f) [0x7f01a48fe60f] 2024-03-28T18:21:48.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f01a49d1e81] 2024-03-28T18:21:48.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f01a33ec1ca] 2024-03-28T18:21:48.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 16: clone() 2024-03-28T18:21:48.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2024-03-28T18:21:48.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 0> 2024-03-28T18:21:47.953+0000 7f019a045700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:21:48.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: in thread 7f019a045700 thread_name:ms_dispatch 2024-03-28T18:21:48.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2024-03-28T18:21:48.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:48.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f01a33f6d20] 2024-03-28T18:21:48.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2: gsignal() 2024-03-28T18:21:48.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 3: abort() 2024-03-28T18:21:48.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f01a466f4fb] 2024-03-28T18:21:48.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f01a466f667] 2024-03-28T18:21:48.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5581116bf223] 2024-03-28T18:21:48.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5581115395d6] 2024-03-28T18:21:48.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558111539da1] 2024-03-28T18:21:48.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55811153c2d5] 2024-03-28T18:21:48.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55811158c233] 2024-03-28T18:21:48.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55811158e197] 2024-03-28T18:21:48.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55811148d11b] 2024-03-28T18:21:48.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55811148d6bc] 2024-03-28T18:21:48.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5581114771ff] 2024-03-28T18:21:48.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f01a4901488] 2024-03-28T18:21:48.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 16: (DispatchQueue::entry()+0x51f) [0x7f01a48fe60f] 2024-03-28T18:21:48.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f01a49d1e81] 2024-03-28T18:21:48.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f01a33ec1ca] 2024-03-28T18:21:48.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 19: clone() 2024-03-28T18:21:48.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:21:48.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2024-03-28T18:21:48.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: -1> 2024-03-28T18:21:47.951+0000 7f019a045700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f019a045700 time 2024-03-28T18:21:47.950473+0000 2024-03-28T18:21:48.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:21:48.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2024-03-28T18:21:48.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:48.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f01a466f4a1] 2024-03-28T18:21:48.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f01a466f667] 2024-03-28T18:21:48.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5581116bf223] 2024-03-28T18:21:48.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5581115395d6] 2024-03-28T18:21:48.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558111539da1] 2024-03-28T18:21:48.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55811153c2d5] 2024-03-28T18:21:48.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55811158c233] 2024-03-28T18:21:48.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55811158e197] 2024-03-28T18:21:48.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55811148d11b] 2024-03-28T18:21:48.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55811148d6bc] 2024-03-28T18:21:48.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5581114771ff] 2024-03-28T18:21:48.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f01a4901488] 2024-03-28T18:21:48.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 13: (DispatchQueue::entry()+0x51f) [0x7f01a48fe60f] 2024-03-28T18:21:48.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f01a49d1e81] 2024-03-28T18:21:48.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f01a33ec1ca] 2024-03-28T18:21:48.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 16: clone() 2024-03-28T18:21:48.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2024-03-28T18:21:48.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 0> 2024-03-28T18:21:47.953+0000 7f019a045700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:21:48.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: in thread 7f019a045700 thread_name:ms_dispatch 2024-03-28T18:21:48.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2024-03-28T18:21:48.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:21:48.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f01a33f6d20] 2024-03-28T18:21:48.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2: gsignal() 2024-03-28T18:21:48.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 3: abort() 2024-03-28T18:21:48.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f01a466f4fb] 2024-03-28T18:21:48.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f01a466f667] 2024-03-28T18:21:48.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5581116bf223] 2024-03-28T18:21:48.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5581115395d6] 2024-03-28T18:21:48.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558111539da1] 2024-03-28T18:21:48.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55811153c2d5] 2024-03-28T18:21:48.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55811158c233] 2024-03-28T18:21:48.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55811158e197] 2024-03-28T18:21:48.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55811148d11b] 2024-03-28T18:21:48.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55811148d6bc] 2024-03-28T18:21:48.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5581114771ff] 2024-03-28T18:21:48.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f01a4901488] 2024-03-28T18:21:48.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 16: (DispatchQueue::entry()+0x51f) [0x7f01a48fe60f] 2024-03-28T18:21:48.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f01a49d1e81] 2024-03-28T18:21:48.301 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f01a33ec1ca] 2024-03-28T18:21:48.301 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 19: clone() 2024-03-28T18:21:48.301 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:21:48.301 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:47 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73210]: 2024-03-28T18:21:48.301 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:48 smithi059 podman[73836]: 2024-03-28 18:21:48.020135275 +0000 UTC m=+0.021598904 container died f2576b6e31b2ce8a98518c39c3209a9b7f51984b1c23a1803decabbaa1587c49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:21:48.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:47 smithi037 ceph-mon[31460]: pgmap v358: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 124 B/s wr, 7 op/s 2024-03-28T18:21:48.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:47 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/10766106,v1:172.21.15.59:6833/10766106] up:rejoin 2024-03-28T18:21:48.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:47 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:21:48.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:47 smithi037 ceph-mon[31460]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-28T18:21:49.054 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:48 smithi059 podman[73836]: 2024-03-28 18:21:48.798408053 +0000 UTC m=+0.799871677 container remove f2576b6e31b2ce8a98518c39c3209a9b7f51984b1c23a1803decabbaa1587c49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-28T18:21:49.055 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:48 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:21:49.055 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:48 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:21:49.055 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:48 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/10766106,v1:172.21.15.59:6833/10766106] up:active 2024-03-28T18:21:49.055 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:48 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:21:49.055 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:48 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T18:21:49.055 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:48 smithi059 ceph-mon[36190]: osdmap e125: 12 total, 12 up, 12 in 2024-03-28T18:21:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:48 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:21:49.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:48 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/10766106,v1:172.21.15.59:6833/10766106] up:active 2024-03-28T18:21:49.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:48 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:21:49.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:48 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T18:21:49.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:48 smithi170 ceph-mon[36280]: osdmap e125: 12 total, 12 up, 12 in 2024-03-28T18:21:49.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:48 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:21:49.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:48 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/10766106,v1:172.21.15.59:6833/10766106] up:active 2024-03-28T18:21:49.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:48 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:21:49.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:48 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T18:21:49.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:48 smithi037 ceph-mon[31460]: osdmap e125: 12 total, 12 up, 12 in 2024-03-28T18:21:49.535 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:49 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-28T18:21:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:49 smithi059 ceph-mon[36190]: pgmap v359: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:21:50.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:49 smithi170 ceph-mon[36280]: pgmap v359: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:21:50.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:49 smithi037 ceph-mon[31460]: pgmap v359: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:21:51.625 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:21:51 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:21:51.441+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:21:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:51 smithi037 ceph-mon[31460]: pgmap v361: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:21:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:51 smithi059 ceph-mon[36190]: pgmap v361: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:21:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:51 smithi170 ceph-mon[36280]: pgmap v361: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:21:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:52 smithi059 ceph-mon[36190]: pgmap v362: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 228 B/s wr, 18 op/s 2024-03-28T18:21:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:52 smithi170 ceph-mon[36280]: pgmap v362: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 228 B/s wr, 18 op/s 2024-03-28T18:21:53.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:52 smithi037 ceph-mon[31460]: pgmap v362: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 228 B/s wr, 18 op/s 2024-03-28T18:21:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:55 smithi059 ceph-mon[36190]: pgmap v363: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:21:56.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:55 smithi170 ceph-mon[36280]: pgmap v363: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:21:56.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:55 smithi037 ceph-mon[31460]: pgmap v363: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:21:58.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:57 smithi059 ceph-mon[36190]: pgmap v364: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:21:58.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:57 smithi170 ceph-mon[36280]: pgmap v364: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:21:58.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:57 smithi037 ceph-mon[31460]: pgmap v364: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:21:59.557 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:59 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Scheduled restart job, restart counter is at 3. 2024-03-28T18:21:59.557 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:59 smithi059 systemd[1]: Stopped Ceph mds.k for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:21:59.557 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:59 smithi059 systemd[1]: Starting Ceph mds.k for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:21:59.557 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:59 smithi059 podman[73929]: 2024-03-28 18:21:59.432287652 +0000 UTC m=+0.019487031 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:21:59.557 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:59 smithi059 podman[73929]: 2024-03-28 18:21:59.557803282 +0000 UTC m=+0.145002727 container create f60ae69d07cad91385da0187e6e90f541d4b8bbcac577d5993325c3153607d2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-28T18:22:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:59 smithi059 ceph-mon[36190]: pgmap v365: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:22:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:59 smithi059 ceph-mon[36190]: daemon mds.k restarted 2024-03-28T18:22:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:21:59 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:22:00.285 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:59 smithi059 podman[73929]: 2024-03-28 18:21:59.828577679 +0000 UTC m=+0.415777058 container init f60ae69d07cad91385da0187e6e90f541d4b8bbcac577d5993325c3153607d2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:22:00.285 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:59 smithi059 podman[73929]: 2024-03-28 18:21:59.831945895 +0000 UTC m=+0.419145264 container start f60ae69d07cad91385da0187e6e90f541d4b8bbcac577d5993325c3153607d2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:22:00.285 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:59 smithi059 ceph-mds[73948]: -- 172.21.15.59:0/3122125518 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564404dfba40 con 0x564404dcec00 2024-03-28T18:22:00.285 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:21:59 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: starting mds.k at 2024-03-28T18:22:00.286 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:00 smithi059 bash[73929]: f60ae69d07cad91385da0187e6e90f541d4b8bbcac577d5993325c3153607d2c 2024-03-28T18:22:00.286 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:00 smithi059 systemd[1]: Started Ceph mds.k for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:22:00.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:59 smithi170 ceph-mon[36280]: pgmap v365: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:22:00.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:59 smithi170 ceph-mon[36280]: daemon mds.k restarted 2024-03-28T18:22:00.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:21:59 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:22:00.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:59 smithi037 ceph-mon[31460]: pgmap v365: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:22:00.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:59 smithi037 ceph-mon[31460]: daemon mds.k restarted 2024-03-28T18:22:00.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:21:59 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:22:01.279 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:00 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:22:01.279 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:00 smithi059 ceph-mon[36190]: osdmap e126: 12 total, 12 up, 12 in 2024-03-28T18:22:01.279 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:00 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/3114955897,v1:172.21.15.59:6833/3114955897] up:boot 2024-03-28T18:22:01.279 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:00 smithi059 ceph-mon[36190]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-28T18:22:01.279 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:00 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T18:22:01.279 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:00 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:00 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:22:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:00 smithi170 ceph-mon[36280]: osdmap e126: 12 total, 12 up, 12 in 2024-03-28T18:22:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:00 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/3114955897,v1:172.21.15.59:6833/3114955897] up:boot 2024-03-28T18:22:01.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:00 smithi170 ceph-mon[36280]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-28T18:22:01.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:00 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T18:22:01.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:00 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:00 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:22:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:00 smithi037 ceph-mon[31460]: osdmap e126: 12 total, 12 up, 12 in 2024-03-28T18:22:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:00 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/3114955897,v1:172.21.15.59:6833/3114955897] up:boot 2024-03-28T18:22:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:00 smithi037 ceph-mon[31460]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-28T18:22:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:00 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T18:22:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:00 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:02.277 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:01 smithi037 ceph-mon[31460]: pgmap v367: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:22:02.277 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:01 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/192494457,v1:172.21.15.37:6837/192494457] up:resolve 2024-03-28T18:22:02.277 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:01 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:02.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:01 smithi059 ceph-mon[36190]: pgmap v367: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:22:02.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:01 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/192494457,v1:172.21.15.37:6837/192494457] up:resolve 2024-03-28T18:22:02.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:01 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:02.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:01 smithi170 ceph-mon[36280]: pgmap v367: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:22:02.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:01 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/192494457,v1:172.21.15.37:6837/192494457] up:resolve 2024-03-28T18:22:02.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:01 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:02 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:22:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:02 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/192494457,v1:172.21.15.37:6837/192494457] up:reconnect 2024-03-28T18:22:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:02 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:02 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999987 2024-03-28T18:22:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:02 smithi059 ceph-mon[36190]: pgmap v368: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:22:03.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:02 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:22:03.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:02 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/192494457,v1:172.21.15.37:6837/192494457] up:reconnect 2024-03-28T18:22:03.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:02 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:03.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:02 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999987 2024-03-28T18:22:03.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:02 smithi170 ceph-mon[36280]: pgmap v368: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:22:03.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:02 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:22:03.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:02 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/192494457,v1:172.21.15.37:6837/192494457] up:reconnect 2024-03-28T18:22:03.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:02 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:03.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:02 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999987 2024-03-28T18:22:03.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:02 smithi037 ceph-mon[31460]: pgmap v368: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:22:04.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/192494457,v1:172.21.15.37:6837/192494457] up:rejoin 2024-03-28T18:22:04.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:04.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-mon[31460]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-28T18:22:04.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:22:04.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/192494457,v1:172.21.15.37:6837/192494457] up:active 2024-03-28T18:22:04.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T18:22:04.126 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f451a400700 time 2024-03-28T18:22:03.913712+0000 2024-03-28T18:22:04.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:22:04.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:04.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f4525c934a1] 2024-03-28T18:22:04.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4525c93667] 2024-03-28T18:22:04.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56120276a223] 2024-03-28T18:22:04.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5612025e45d6] 2024-03-28T18:22:04.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5612025e4da1] 2024-03-28T18:22:04.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5612025e72d5] 2024-03-28T18:22:04.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561202637233] 2024-03-28T18:22:04.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561202639197] 2024-03-28T18:22:04.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56120253811b] 2024-03-28T18:22:04.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5612025386bc] 2024-03-28T18:22:04.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5612025221ff] 2024-03-28T18:22:04.127 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4525f25488] 2024-03-28T18:22:04.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 13: (DispatchQueue::entry()+0x51f) [0x7f4525f2260f] 2024-03-28T18:22:04.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4525ff5e81] 2024-03-28T18:22:04.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4524a101ca] 2024-03-28T18:22:04.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 16: clone() 2024-03-28T18:22:04.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2024-03-28T18:22:03.914+0000 7f451a400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f451a400700 time 2024-03-28T18:22:03.913712+0000 2024-03-28T18:22:04.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:22:04.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2024-03-28T18:22:04.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:04.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f4525c934a1] 2024-03-28T18:22:04.128 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4525c93667] 2024-03-28T18:22:04.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56120276a223] 2024-03-28T18:22:04.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5612025e45d6] 2024-03-28T18:22:04.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5612025e4da1] 2024-03-28T18:22:04.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5612025e72d5] 2024-03-28T18:22:04.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561202637233] 2024-03-28T18:22:04.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561202639197] 2024-03-28T18:22:04.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56120253811b] 2024-03-28T18:22:04.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5612025386bc] 2024-03-28T18:22:04.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5612025221ff] 2024-03-28T18:22:04.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4525f25488] 2024-03-28T18:22:04.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 13: (DispatchQueue::entry()+0x51f) [0x7f4525f2260f] 2024-03-28T18:22:04.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4525ff5e81] 2024-03-28T18:22:04.129 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4524a101ca] 2024-03-28T18:22:04.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 16: clone() 2024-03-28T18:22:04.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2024-03-28T18:22:04.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: *** Caught signal (Aborted) ** 2024-03-28T18:22:04.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: in thread 7f451a400700 thread_name:ms_dispatch 2024-03-28T18:22:04.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:04.130 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4524a1ad20] 2024-03-28T18:22:04.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2: gsignal() 2024-03-28T18:22:04.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 3: abort() 2024-03-28T18:22:04.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4525c934fb] 2024-03-28T18:22:04.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4525c93667] 2024-03-28T18:22:04.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56120276a223] 2024-03-28T18:22:04.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5612025e45d6] 2024-03-28T18:22:04.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5612025e4da1] 2024-03-28T18:22:04.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5612025e72d5] 2024-03-28T18:22:04.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561202637233] 2024-03-28T18:22:04.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561202639197] 2024-03-28T18:22:04.131 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56120253811b] 2024-03-28T18:22:04.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5612025386bc] 2024-03-28T18:22:04.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5612025221ff] 2024-03-28T18:22:04.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4525f25488] 2024-03-28T18:22:04.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 16: (DispatchQueue::entry()+0x51f) [0x7f4525f2260f] 2024-03-28T18:22:04.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4525ff5e81] 2024-03-28T18:22:04.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4524a101ca] 2024-03-28T18:22:04.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 19: clone() 2024-03-28T18:22:04.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2024-03-28T18:22:03.917+0000 7f451a400700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:22:04.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: in thread 7f451a400700 thread_name:ms_dispatch 2024-03-28T18:22:04.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2024-03-28T18:22:04.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:04.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4524a1ad20] 2024-03-28T18:22:04.132 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2: gsignal() 2024-03-28T18:22:04.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 3: abort() 2024-03-28T18:22:04.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4525c934fb] 2024-03-28T18:22:04.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4525c93667] 2024-03-28T18:22:04.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56120276a223] 2024-03-28T18:22:04.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5612025e45d6] 2024-03-28T18:22:04.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5612025e4da1] 2024-03-28T18:22:04.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5612025e72d5] 2024-03-28T18:22:04.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561202637233] 2024-03-28T18:22:04.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561202639197] 2024-03-28T18:22:04.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56120253811b] 2024-03-28T18:22:04.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5612025386bc] 2024-03-28T18:22:04.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5612025221ff] 2024-03-28T18:22:04.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4525f25488] 2024-03-28T18:22:04.133 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 16: (DispatchQueue::entry()+0x51f) [0x7f4525f2260f] 2024-03-28T18:22:04.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4525ff5e81] 2024-03-28T18:22:04.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4524a101ca] 2024-03-28T18:22:04.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 19: clone() 2024-03-28T18:22:04.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:22:04.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2024-03-28T18:22:04.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: -1> 2024-03-28T18:22:03.914+0000 7f451a400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f451a400700 time 2024-03-28T18:22:03.913712+0000 2024-03-28T18:22:04.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:22:04.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2024-03-28T18:22:04.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:04.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f4525c934a1] 2024-03-28T18:22:04.134 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4525c93667] 2024-03-28T18:22:04.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56120276a223] 2024-03-28T18:22:04.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5612025e45d6] 2024-03-28T18:22:04.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5612025e4da1] 2024-03-28T18:22:04.135 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5612025e72d5] 2024-03-28T18:22:04.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561202637233] 2024-03-28T18:22:04.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561202639197] 2024-03-28T18:22:04.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56120253811b] 2024-03-28T18:22:04.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5612025386bc] 2024-03-28T18:22:04.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5612025221ff] 2024-03-28T18:22:04.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4525f25488] 2024-03-28T18:22:04.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 13: (DispatchQueue::entry()+0x51f) [0x7f4525f2260f] 2024-03-28T18:22:04.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4525ff5e81] 2024-03-28T18:22:04.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4524a101ca] 2024-03-28T18:22:04.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 16: clone() 2024-03-28T18:22:04.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2024-03-28T18:22:04.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 0> 2024-03-28T18:22:03.917+0000 7f451a400700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:22:04.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: in thread 7f451a400700 thread_name:ms_dispatch 2024-03-28T18:22:04.136 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2024-03-28T18:22:04.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:04.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4524a1ad20] 2024-03-28T18:22:04.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2: gsignal() 2024-03-28T18:22:04.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 3: abort() 2024-03-28T18:22:04.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4525c934fb] 2024-03-28T18:22:04.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4525c93667] 2024-03-28T18:22:04.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56120276a223] 2024-03-28T18:22:04.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5612025e45d6] 2024-03-28T18:22:04.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5612025e4da1] 2024-03-28T18:22:04.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5612025e72d5] 2024-03-28T18:22:04.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561202637233] 2024-03-28T18:22:04.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561202639197] 2024-03-28T18:22:04.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56120253811b] 2024-03-28T18:22:04.137 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5612025386bc] 2024-03-28T18:22:04.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5612025221ff] 2024-03-28T18:22:04.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4525f25488] 2024-03-28T18:22:04.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 16: (DispatchQueue::entry()+0x51f) [0x7f4525f2260f] 2024-03-28T18:22:04.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4525ff5e81] 2024-03-28T18:22:04.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4524a101ca] 2024-03-28T18:22:04.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 19: clone() 2024-03-28T18:22:04.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:22:04.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2024-03-28T18:22:04.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: -1> 2024-03-28T18:22:03.914+0000 7f451a400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f451a400700 time 2024-03-28T18:22:03.913712+0000 2024-03-28T18:22:04.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:22:04.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2024-03-28T18:22:04.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:04.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f4525c934a1] 2024-03-28T18:22:04.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4525c93667] 2024-03-28T18:22:04.138 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56120276a223] 2024-03-28T18:22:04.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5612025e45d6] 2024-03-28T18:22:04.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5612025e4da1] 2024-03-28T18:22:04.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5612025e72d5] 2024-03-28T18:22:04.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561202637233] 2024-03-28T18:22:04.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561202639197] 2024-03-28T18:22:04.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56120253811b] 2024-03-28T18:22:04.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5612025386bc] 2024-03-28T18:22:04.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5612025221ff] 2024-03-28T18:22:04.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4525f25488] 2024-03-28T18:22:04.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 13: (DispatchQueue::entry()+0x51f) [0x7f4525f2260f] 2024-03-28T18:22:04.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4525ff5e81] 2024-03-28T18:22:04.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4524a101ca] 2024-03-28T18:22:04.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 16: clone() 2024-03-28T18:22:04.139 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2024-03-28T18:22:04.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 0> 2024-03-28T18:22:03.917+0000 7f451a400700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:22:04.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: in thread 7f451a400700 thread_name:ms_dispatch 2024-03-28T18:22:04.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2024-03-28T18:22:04.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:04.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4524a1ad20] 2024-03-28T18:22:04.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2: gsignal() 2024-03-28T18:22:04.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 3: abort() 2024-03-28T18:22:04.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4525c934fb] 2024-03-28T18:22:04.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4525c93667] 2024-03-28T18:22:04.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56120276a223] 2024-03-28T18:22:04.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5612025e45d6] 2024-03-28T18:22:04.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5612025e4da1] 2024-03-28T18:22:04.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5612025e72d5] 2024-03-28T18:22:04.140 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561202637233] 2024-03-28T18:22:04.141 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561202639197] 2024-03-28T18:22:04.141 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56120253811b] 2024-03-28T18:22:04.141 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5612025386bc] 2024-03-28T18:22:04.141 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5612025221ff] 2024-03-28T18:22:04.141 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4525f25488] 2024-03-28T18:22:04.141 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 16: (DispatchQueue::entry()+0x51f) [0x7f4525f2260f] 2024-03-28T18:22:04.141 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4525ff5e81] 2024-03-28T18:22:04.141 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4524a101ca] 2024-03-28T18:22:04.141 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 19: clone() 2024-03-28T18:22:04.141 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:22:04.141 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[88131]: 2024-03-28T18:22:04.141 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:04 smithi037 podman[89135]: 2024-03-28 18:22:04.040545244 +0000 UTC m=+0.034018210 container died 248bbd8bbca0ae6c4fe75c54403268debd1f6eec8fd695c2e7a8ea61e75a966e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:22:04.186 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:03 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/192494457,v1:172.21.15.37:6837/192494457] up:rejoin 2024-03-28T18:22:04.186 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:03 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:04.186 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:03 smithi059 ceph-mon[36190]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-28T18:22:04.187 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:03 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:22:04.187 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:03 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/192494457,v1:172.21.15.37:6837/192494457] up:active 2024-03-28T18:22:04.187 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:03 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T18:22:04.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:03 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/192494457,v1:172.21.15.37:6837/192494457] up:rejoin 2024-03-28T18:22:04.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:03 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:04.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:03 smithi170 ceph-mon[36280]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-28T18:22:04.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:03 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:22:04.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:03 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/192494457,v1:172.21.15.37:6837/192494457] up:active 2024-03-28T18:22:04.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:03 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T18:22:05.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:04 smithi059 ceph-mon[36190]: pgmap v369: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:22:05.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:04 smithi170 ceph-mon[36280]: pgmap v369: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:22:05.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:04 smithi037 ceph-mon[31460]: pgmap v369: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:22:05.375 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:04 smithi037 podman[89135]: 2024-03-28 18:22:04.991342814 +0000 UTC m=+0.984815801 container remove 248bbd8bbca0ae6c4fe75c54403268debd1f6eec8fd695c2e7a8ea61e75a966e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-28T18:22:05.375 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:05 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:22:05.875 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:05 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-28T18:22:05.875 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:05 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.896s CPU time. 2024-03-28T18:22:06.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:22:06 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:22:06.443+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:22:08.260 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:07 smithi037 ceph-mon[31460]: pgmap v370: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:22:08.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:07 smithi059 ceph-mon[36190]: pgmap v370: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:22:08.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:07 smithi170 ceph-mon[36280]: pgmap v370: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:22:10.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:09 smithi059 ceph-mon[36190]: pgmap v371: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:22:10.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:09 smithi170 ceph-mon[36280]: pgmap v371: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:22:10.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:09 smithi037 ceph-mon[31460]: pgmap v371: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:22:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:11 smithi059 ceph-mon[36190]: pgmap v372: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 25 KiB/s wr, 31 op/s 2024-03-28T18:22:12.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:11 smithi170 ceph-mon[36280]: pgmap v372: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 25 KiB/s wr, 31 op/s 2024-03-28T18:22:12.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:11 smithi037 ceph-mon[31460]: pgmap v372: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 25 KiB/s wr, 31 op/s 2024-03-28T18:22:14.249 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:13 smithi059 ceph-mon[36190]: pgmap v373: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 31 op/s 2024-03-28T18:22:14.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:13 smithi170 ceph-mon[36280]: pgmap v373: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 31 op/s 2024-03-28T18:22:14.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:13 smithi037 ceph-mon[31460]: pgmap v373: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 31 op/s 2024-03-28T18:22:15.874 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:15 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 5. 2024-03-28T18:22:15.875 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:15 smithi037 systemd[1]: Stopped Ceph mds.d for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:22:15.875 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:15 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.896s CPU time. 2024-03-28T18:22:15.875 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:15 smithi037 systemd[1]: Starting Ceph mds.d for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:22:16.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:15 smithi059 ceph-mon[36190]: pgmap v374: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:22:16.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:15 smithi170 ceph-mon[36280]: pgmap v374: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:22:16.356 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:15 smithi037 ceph-mon[31460]: pgmap v374: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:22:16.357 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:16 smithi037 podman[89223]: 2024-03-28 18:22:15.929222848 +0000 UTC m=+0.048961094 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:22:16.357 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:16 smithi037 podman[89223]: 2024-03-28 18:22:16.075999384 +0000 UTC m=+0.195737657 container create c97a43ca5460e65a8b6a58c0bff99b98927fa87a94fd1f539c52d9e8a83bde39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:22:16.875 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:16 smithi037 podman[89223]: 2024-03-28 18:22:16.481402119 +0000 UTC m=+0.601140399 container init c97a43ca5460e65a8b6a58c0bff99b98927fa87a94fd1f539c52d9e8a83bde39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:22:16.875 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:16 smithi037 podman[89223]: 2024-03-28 18:22:16.492984245 +0000 UTC m=+0.612722501 container start c97a43ca5460e65a8b6a58c0bff99b98927fa87a94fd1f539c52d9e8a83bde39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:22:16.875 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:16 smithi037 ceph-mds[89242]: -- 172.21.15.37:0/4188942044 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560bdb437a40 con 0x560bdb40b000 2024-03-28T18:22:16.875 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:16 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: starting mds.d at 2024-03-28T18:22:16.875 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:16 smithi037 bash[89223]: c97a43ca5460e65a8b6a58c0bff99b98927fa87a94fd1f539c52d9e8a83bde39 2024-03-28T18:22:16.876 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:16 smithi037 systemd[1]: Started Ceph mds.d for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:22:17.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:16 smithi059 ceph-mon[36190]: daemon mds.d restarted 2024-03-28T18:22:17.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:16 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:22:17.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:16 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:22:17.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:16 smithi059 ceph-mon[36190]: osdmap e127: 12 total, 12 up, 12 in 2024-03-28T18:22:17.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:16 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/1553180221,v1:172.21.15.37:6837/1553180221] up:boot 2024-03-28T18:22:17.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:16 smithi059 ceph-mon[36190]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-28T18:22:17.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:16 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T18:22:17.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:16 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:17.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:16 smithi170 ceph-mon[36280]: daemon mds.d restarted 2024-03-28T18:22:17.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:16 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:22:17.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:16 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:22:17.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:16 smithi170 ceph-mon[36280]: osdmap e127: 12 total, 12 up, 12 in 2024-03-28T18:22:17.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:16 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/1553180221,v1:172.21.15.37:6837/1553180221] up:boot 2024-03-28T18:22:17.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:16 smithi170 ceph-mon[36280]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-28T18:22:17.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:16 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T18:22:17.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:16 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:17.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:16 smithi037 ceph-mon[31460]: daemon mds.d restarted 2024-03-28T18:22:17.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:16 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:22:17.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:16 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:22:17.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:16 smithi037 ceph-mon[31460]: osdmap e127: 12 total, 12 up, 12 in 2024-03-28T18:22:17.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:16 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/1553180221,v1:172.21.15.37:6837/1553180221] up:boot 2024-03-28T18:22:17.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:16 smithi037 ceph-mon[31460]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-28T18:22:17.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:16 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T18:22:17.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:16 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:18.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:17 smithi059 ceph-mon[36190]: pgmap v376: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:22:18.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:17 smithi170 ceph-mon[36280]: pgmap v376: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:22:18.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:17 smithi037 ceph-mon[31460]: pgmap v376: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:22:19.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:18 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/3114955897,v1:172.21.15.59:6833/3114955897] up:resolve 2024-03-28T18:22:19.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:18 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:19.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:18 smithi059 ceph-mon[36190]: pgmap v377: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:22:19.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:18 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/3114955897,v1:172.21.15.59:6833/3114955897] up:resolve 2024-03-28T18:22:19.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:18 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:19.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:18 smithi170 ceph-mon[36280]: pgmap v377: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:22:19.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:18 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/3114955897,v1:172.21.15.59:6833/3114955897] up:resolve 2024-03-28T18:22:19.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:18 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:19.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:18 smithi037 ceph-mon[31460]: pgmap v377: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:22:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:19 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199998 2024-03-28T18:22:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:19 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/3114955897,v1:172.21.15.59:6833/3114955897] up:reconnect 2024-03-28T18:22:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:19 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:19 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899989 2024-03-28T18:22:20.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:19 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199998 2024-03-28T18:22:20.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:19 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/3114955897,v1:172.21.15.59:6833/3114955897] up:reconnect 2024-03-28T18:22:20.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:19 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:20.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:19 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899989 2024-03-28T18:22:20.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:19 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199998 2024-03-28T18:22:20.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:19 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/3114955897,v1:172.21.15.59:6833/3114955897] up:reconnect 2024-03-28T18:22:20.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:19 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:20.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:19 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899989 2024-03-28T18:22:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/3114955897,v1:172.21.15.59:6833/3114955897] up:rejoin 2024-03-28T18:22:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-mon[36190]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-28T18:22:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-mon[36190]: pgmap v378: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:22:21.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:20 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/3114955897,v1:172.21.15.59:6833/3114955897] up:rejoin 2024-03-28T18:22:21.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:20 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:21.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:20 smithi170 ceph-mon[36280]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-28T18:22:21.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:20 smithi170 ceph-mon[36280]: pgmap v378: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:22:21.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe0ab0eb700 time 2024-03-28T18:22:20.991860+0000 2024-03-28T18:22:21.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:22:21.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:21.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe0b57154a1] 2024-03-28T18:22:21.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe0b5715667] 2024-03-28T18:22:21.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56440280b223] 2024-03-28T18:22:21.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5644026855d6] 2024-03-28T18:22:21.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x564402685da1] 2024-03-28T18:22:21.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5644026882d5] 2024-03-28T18:22:21.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5644026d8233] 2024-03-28T18:22:21.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5644026da197] 2024-03-28T18:22:21.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5644025d911b] 2024-03-28T18:22:21.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5644025d96bc] 2024-03-28T18:22:21.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5644025c31ff] 2024-03-28T18:22:21.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe0b59a7488] 2024-03-28T18:22:21.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 13: (DispatchQueue::entry()+0x51f) [0x7fe0b59a460f] 2024-03-28T18:22:21.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe0b5a77e81] 2024-03-28T18:22:21.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe0b44921ca] 2024-03-28T18:22:21.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 16: clone() 2024-03-28T18:22:21.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2024-03-28T18:22:20.993+0000 7fe0ab0eb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe0ab0eb700 time 2024-03-28T18:22:20.991860+0000 2024-03-28T18:22:21.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:22:21.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2024-03-28T18:22:21.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:21.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe0b57154a1] 2024-03-28T18:22:21.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe0b5715667] 2024-03-28T18:22:21.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56440280b223] 2024-03-28T18:22:21.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5644026855d6] 2024-03-28T18:22:21.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x564402685da1] 2024-03-28T18:22:21.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5644026882d5] 2024-03-28T18:22:21.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5644026d8233] 2024-03-28T18:22:21.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5644026da197] 2024-03-28T18:22:21.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5644025d911b] 2024-03-28T18:22:21.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5644025d96bc] 2024-03-28T18:22:21.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5644025c31ff] 2024-03-28T18:22:21.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe0b59a7488] 2024-03-28T18:22:21.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 13: (DispatchQueue::entry()+0x51f) [0x7fe0b59a460f] 2024-03-28T18:22:21.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe0b5a77e81] 2024-03-28T18:22:21.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe0b44921ca] 2024-03-28T18:22:21.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 16: clone() 2024-03-28T18:22:21.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2024-03-28T18:22:21.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: *** Caught signal (Aborted) ** 2024-03-28T18:22:21.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: in thread 7fe0ab0eb700 thread_name:ms_dispatch 2024-03-28T18:22:21.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:21.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe0b449cd20] 2024-03-28T18:22:21.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2: gsignal() 2024-03-28T18:22:21.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 3: abort() 2024-03-28T18:22:21.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe0b57154fb] 2024-03-28T18:22:21.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe0b5715667] 2024-03-28T18:22:21.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56440280b223] 2024-03-28T18:22:21.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5644026855d6] 2024-03-28T18:22:21.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x564402685da1] 2024-03-28T18:22:21.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5644026882d5] 2024-03-28T18:22:21.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5644026d8233] 2024-03-28T18:22:21.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5644026da197] 2024-03-28T18:22:21.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5644025d911b] 2024-03-28T18:22:21.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5644025d96bc] 2024-03-28T18:22:21.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5644025c31ff] 2024-03-28T18:22:21.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe0b59a7488] 2024-03-28T18:22:21.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 16: (DispatchQueue::entry()+0x51f) [0x7fe0b59a460f] 2024-03-28T18:22:21.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe0b5a77e81] 2024-03-28T18:22:21.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe0b44921ca] 2024-03-28T18:22:21.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 19: clone() 2024-03-28T18:22:21.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2024-03-28T18:22:20.995+0000 7fe0ab0eb700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:22:21.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: in thread 7fe0ab0eb700 thread_name:ms_dispatch 2024-03-28T18:22:21.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2024-03-28T18:22:21.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:21.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe0b449cd20] 2024-03-28T18:22:21.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2: gsignal() 2024-03-28T18:22:21.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 3: abort() 2024-03-28T18:22:21.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe0b57154fb] 2024-03-28T18:22:21.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe0b5715667] 2024-03-28T18:22:21.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56440280b223] 2024-03-28T18:22:21.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5644026855d6] 2024-03-28T18:22:21.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x564402685da1] 2024-03-28T18:22:21.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5644026882d5] 2024-03-28T18:22:21.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5644026d8233] 2024-03-28T18:22:21.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5644026da197] 2024-03-28T18:22:21.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5644025d911b] 2024-03-28T18:22:21.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5644025d96bc] 2024-03-28T18:22:21.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5644025c31ff] 2024-03-28T18:22:21.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe0b59a7488] 2024-03-28T18:22:21.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 16: (DispatchQueue::entry()+0x51f) [0x7fe0b59a460f] 2024-03-28T18:22:21.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe0b5a77e81] 2024-03-28T18:22:21.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe0b44921ca] 2024-03-28T18:22:21.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 19: clone() 2024-03-28T18:22:21.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:22:21.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:20 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2024-03-28T18:22:21.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: -1> 2024-03-28T18:22:20.993+0000 7fe0ab0eb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe0ab0eb700 time 2024-03-28T18:22:20.991860+0000 2024-03-28T18:22:21.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:22:21.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2024-03-28T18:22:21.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:21.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe0b57154a1] 2024-03-28T18:22:21.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe0b5715667] 2024-03-28T18:22:21.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56440280b223] 2024-03-28T18:22:21.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5644026855d6] 2024-03-28T18:22:21.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x564402685da1] 2024-03-28T18:22:21.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5644026882d5] 2024-03-28T18:22:21.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5644026d8233] 2024-03-28T18:22:21.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5644026da197] 2024-03-28T18:22:21.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5644025d911b] 2024-03-28T18:22:21.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5644025d96bc] 2024-03-28T18:22:21.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5644025c31ff] 2024-03-28T18:22:21.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe0b59a7488] 2024-03-28T18:22:21.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 13: (DispatchQueue::entry()+0x51f) [0x7fe0b59a460f] 2024-03-28T18:22:21.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe0b5a77e81] 2024-03-28T18:22:21.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe0b44921ca] 2024-03-28T18:22:21.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 16: clone() 2024-03-28T18:22:21.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2024-03-28T18:22:21.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 0> 2024-03-28T18:22:20.995+0000 7fe0ab0eb700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:22:21.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: in thread 7fe0ab0eb700 thread_name:ms_dispatch 2024-03-28T18:22:21.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2024-03-28T18:22:21.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:21.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe0b449cd20] 2024-03-28T18:22:21.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2: gsignal() 2024-03-28T18:22:21.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 3: abort() 2024-03-28T18:22:21.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe0b57154fb] 2024-03-28T18:22:21.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe0b5715667] 2024-03-28T18:22:21.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56440280b223] 2024-03-28T18:22:21.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5644026855d6] 2024-03-28T18:22:21.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x564402685da1] 2024-03-28T18:22:21.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5644026882d5] 2024-03-28T18:22:21.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5644026d8233] 2024-03-28T18:22:21.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5644026da197] 2024-03-28T18:22:21.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5644025d911b] 2024-03-28T18:22:21.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5644025d96bc] 2024-03-28T18:22:21.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5644025c31ff] 2024-03-28T18:22:21.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe0b59a7488] 2024-03-28T18:22:21.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 16: (DispatchQueue::entry()+0x51f) [0x7fe0b59a460f] 2024-03-28T18:22:21.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe0b5a77e81] 2024-03-28T18:22:21.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe0b44921ca] 2024-03-28T18:22:21.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 19: clone() 2024-03-28T18:22:21.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:22:21.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2024-03-28T18:22:21.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: -1> 2024-03-28T18:22:20.993+0000 7fe0ab0eb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe0ab0eb700 time 2024-03-28T18:22:20.991860+0000 2024-03-28T18:22:21.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:22:21.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2024-03-28T18:22:21.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:21.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe0b57154a1] 2024-03-28T18:22:21.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe0b5715667] 2024-03-28T18:22:21.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56440280b223] 2024-03-28T18:22:21.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5644026855d6] 2024-03-28T18:22:21.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x564402685da1] 2024-03-28T18:22:21.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5644026882d5] 2024-03-28T18:22:21.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5644026d8233] 2024-03-28T18:22:21.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5644026da197] 2024-03-28T18:22:21.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5644025d911b] 2024-03-28T18:22:21.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5644025d96bc] 2024-03-28T18:22:21.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5644025c31ff] 2024-03-28T18:22:21.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe0b59a7488] 2024-03-28T18:22:21.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 13: (DispatchQueue::entry()+0x51f) [0x7fe0b59a460f] 2024-03-28T18:22:21.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe0b5a77e81] 2024-03-28T18:22:21.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe0b44921ca] 2024-03-28T18:22:21.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 16: clone() 2024-03-28T18:22:21.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2024-03-28T18:22:21.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 0> 2024-03-28T18:22:20.995+0000 7fe0ab0eb700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:22:21.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: in thread 7fe0ab0eb700 thread_name:ms_dispatch 2024-03-28T18:22:21.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2024-03-28T18:22:21.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:21.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe0b449cd20] 2024-03-28T18:22:21.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2: gsignal() 2024-03-28T18:22:21.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 3: abort() 2024-03-28T18:22:21.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe0b57154fb] 2024-03-28T18:22:21.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe0b5715667] 2024-03-28T18:22:21.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56440280b223] 2024-03-28T18:22:21.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5644026855d6] 2024-03-28T18:22:21.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x564402685da1] 2024-03-28T18:22:21.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5644026882d5] 2024-03-28T18:22:21.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5644026d8233] 2024-03-28T18:22:21.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5644026da197] 2024-03-28T18:22:21.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5644025d911b] 2024-03-28T18:22:21.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5644025d96bc] 2024-03-28T18:22:21.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5644025c31ff] 2024-03-28T18:22:21.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe0b59a7488] 2024-03-28T18:22:21.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 16: (DispatchQueue::entry()+0x51f) [0x7fe0b59a460f] 2024-03-28T18:22:21.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe0b5a77e81] 2024-03-28T18:22:21.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe0b44921ca] 2024-03-28T18:22:21.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 19: clone() 2024-03-28T18:22:21.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:22:21.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[73944]: 2024-03-28T18:22:21.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 podman[73995]: 2024-03-28 18:22:21.056132274 +0000 UTC m=+0.020426222 container died f60ae69d07cad91385da0187e6e90f541d4b8bbcac577d5993325c3153607d2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-28T18:22:21.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:20 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/3114955897,v1:172.21.15.59:6833/3114955897] up:rejoin 2024-03-28T18:22:21.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:20 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:21.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:20 smithi037 ceph-mon[31460]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-28T18:22:21.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:20 smithi037 ceph-mon[31460]: pgmap v378: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:22:21.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:22:21 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:22:21.444+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:22:22.018 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:22:22.018 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/3114955897,v1:172.21.15.59:6833/3114955897] up:active 2024-03-28T18:22:22.018 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:22.018 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:21 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T18:22:22.019 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 podman[73995]: 2024-03-28 18:22:21.768621268 +0000 UTC m=+0.732915206 container remove f60ae69d07cad91385da0187e6e90f541d4b8bbcac577d5993325c3153607d2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-28T18:22:22.019 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:21 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:22:22.285 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:22 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-28T18:22:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:21 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:22:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:21 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/3114955897,v1:172.21.15.59:6833/3114955897] up:active 2024-03-28T18:22:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:21 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:21 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T18:22:22.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:21 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:22:22.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:21 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/3114955897,v1:172.21.15.59:6833/3114955897] up:active 2024-03-28T18:22:22.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:21 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:22.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:21 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T18:22:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:22 smithi059 ceph-mon[36190]: pgmap v379: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:22:23.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:22 smithi170 ceph-mon[36280]: pgmap v379: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:22:23.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:22 smithi037 ceph-mon[31460]: pgmap v379: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:22:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:25 smithi059 ceph-mon[36190]: pgmap v380: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:22:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:25 smithi170 ceph-mon[36280]: pgmap v380: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:22:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:25 smithi037 ceph-mon[31460]: pgmap v380: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:22:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:27 smithi059 ceph-mon[36190]: pgmap v381: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 26 KiB/s wr, 18 op/s 2024-03-28T18:22:28.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:27 smithi170 ceph-mon[36280]: pgmap v381: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 26 KiB/s wr, 18 op/s 2024-03-28T18:22:28.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:27 smithi037 ceph-mon[31460]: pgmap v381: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 26 KiB/s wr, 18 op/s 2024-03-28T18:22:30.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:29 smithi059 ceph-mon[36190]: pgmap v382: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-28T18:22:30.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:29 smithi170 ceph-mon[36280]: pgmap v382: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-28T18:22:30.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:29 smithi037 ceph-mon[31460]: pgmap v382: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-28T18:22:32.271 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:31 smithi059 ceph-mon[36190]: pgmap v383: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-28T18:22:32.271 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:31 smithi059 ceph-mon[36190]: osdmap e128: 12 total, 12 up, 12 in 2024-03-28T18:22:32.272 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:32 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Scheduled restart job, restart counter is at 4. 2024-03-28T18:22:32.272 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:32 smithi059 systemd[1]: Stopped Ceph mds.k for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:22:32.272 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:32 smithi059 systemd[1]: Starting Ceph mds.k for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:22:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:31 smithi170 ceph-mon[36280]: pgmap v383: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-28T18:22:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:31 smithi170 ceph-mon[36280]: osdmap e128: 12 total, 12 up, 12 in 2024-03-28T18:22:32.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:31 smithi037 ceph-mon[31460]: pgmap v383: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-28T18:22:32.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:31 smithi037 ceph-mon[31460]: osdmap e128: 12 total, 12 up, 12 in 2024-03-28T18:22:32.535 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:32 smithi059 podman[74088]: 2024-03-28 18:22:32.175016378 +0000 UTC m=+0.017970295 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:22:32.536 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:32 smithi059 podman[74088]: 2024-03-28 18:22:32.461702689 +0000 UTC m=+0.304656580 container create 5bcbcb8431401318058bd04ed550c3c255197f34a893c8ee44cd32be42a841d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-28T18:22:32.911 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:32 smithi059 podman[74088]: 2024-03-28 18:22:32.689416151 +0000 UTC m=+0.532370056 container init 5bcbcb8431401318058bd04ed550c3c255197f34a893c8ee44cd32be42a841d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-28T18:22:32.911 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:32 smithi059 podman[74088]: 2024-03-28 18:22:32.692191286 +0000 UTC m=+0.535145186 container start 5bcbcb8431401318058bd04ed550c3c255197f34a893c8ee44cd32be42a841d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:22:32.911 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:32 smithi059 ceph-mds[74107]: -- 172.21.15.59:0/3305697238 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55efcd277a40 con 0x55efcd24a800 2024-03-28T18:22:32.911 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:32 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: starting mds.k at 2024-03-28T18:22:32.912 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:32 smithi059 bash[74088]: 5bcbcb8431401318058bd04ed550c3c255197f34a893c8ee44cd32be42a841d6 2024-03-28T18:22:32.912 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:32 smithi059 systemd[1]: Started Ceph mds.k for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:22:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:32 smithi059 ceph-mon[36190]: daemon mds.k restarted 2024-03-28T18:22:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:32 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:22:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:32 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:22:33.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:32 smithi059 ceph-mon[36190]: osdmap e129: 12 total, 12 up, 12 in 2024-03-28T18:22:33.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:32 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/2898798947,v1:172.21.15.59:6833/2898798947] up:boot 2024-03-28T18:22:33.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:32 smithi059 ceph-mon[36190]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-28T18:22:33.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:32 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T18:22:33.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:32 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:33.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:32 smithi170 ceph-mon[36280]: daemon mds.k restarted 2024-03-28T18:22:33.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:32 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:22:33.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:32 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:22:33.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:32 smithi170 ceph-mon[36280]: osdmap e129: 12 total, 12 up, 12 in 2024-03-28T18:22:33.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:32 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/2898798947,v1:172.21.15.59:6833/2898798947] up:boot 2024-03-28T18:22:33.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:32 smithi170 ceph-mon[36280]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-28T18:22:33.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:32 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T18:22:33.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:32 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:33.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:32 smithi037 ceph-mon[31460]: daemon mds.k restarted 2024-03-28T18:22:33.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:32 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:22:33.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:32 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:22:33.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:32 smithi037 ceph-mon[31460]: osdmap e129: 12 total, 12 up, 12 in 2024-03-28T18:22:33.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:32 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/2898798947,v1:172.21.15.59:6833/2898798947] up:boot 2024-03-28T18:22:33.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:32 smithi037 ceph-mon[31460]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-28T18:22:33.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:32 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T18:22:33.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:32 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:34.186 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:33 smithi059 ceph-mon[36190]: pgmap v386: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T18:22:34.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:33 smithi170 ceph-mon[36280]: pgmap v386: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T18:22:34.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:33 smithi037 ceph-mon[31460]: pgmap v386: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T18:22:35.187 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:34 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/1553180221,v1:172.21.15.37:6837/1553180221] up:resolve 2024-03-28T18:22:35.187 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:34 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:35.187 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:34 smithi170 ceph-mon[36280]: pgmap v387: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T18:22:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:34 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/1553180221,v1:172.21.15.37:6837/1553180221] up:resolve 2024-03-28T18:22:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:34 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:34 smithi059 ceph-mon[36190]: pgmap v387: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T18:22:35.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:34 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/1553180221,v1:172.21.15.37:6837/1553180221] up:resolve 2024-03-28T18:22:35.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:34 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:35.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:34 smithi037 ceph-mon[31460]: pgmap v387: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T18:22:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:35 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:22:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:35 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/1553180221,v1:172.21.15.37:6837/1553180221] up:reconnect 2024-03-28T18:22:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:35 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:35 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899988 2024-03-28T18:22:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:35 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:22:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:35 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/1553180221,v1:172.21.15.37:6837/1553180221] up:reconnect 2024-03-28T18:22:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:35 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:35 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899988 2024-03-28T18:22:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:35 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:22:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:35 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/1553180221,v1:172.21.15.37:6837/1553180221] up:reconnect 2024-03-28T18:22:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:35 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:35 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899988 2024-03-28T18:22:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:36 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/1553180221,v1:172.21.15.37:6837/1553180221] up:rejoin 2024-03-28T18:22:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:36 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:36 smithi059 ceph-mon[36190]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-28T18:22:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:36 smithi059 ceph-mon[36190]: pgmap v388: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T18:22:37.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:36 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/1553180221,v1:172.21.15.37:6837/1553180221] up:rejoin 2024-03-28T18:22:37.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:36 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:37.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:36 smithi170 ceph-mon[36280]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-28T18:22:37.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:36 smithi170 ceph-mon[36280]: pgmap v388: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T18:22:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:36 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/1553180221,v1:172.21.15.37:6837/1553180221] up:rejoin 2024-03-28T18:22:37.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:36 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:37.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:36 smithi037 ceph-mon[31460]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-28T18:22:37.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:36 smithi037 ceph-mon[31460]: pgmap v388: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T18:22:37.376 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc97a200700 time 2024-03-28T18:22:37.018401+0000 2024-03-28T18:22:37.376 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:22:37.376 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:37.376 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc985a224a1] 2024-03-28T18:22:37.376 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc985a22667] 2024-03-28T18:22:37.376 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560bda101223] 2024-03-28T18:22:37.376 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560bd9f7b5d6] 2024-03-28T18:22:37.376 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560bd9f7bda1] 2024-03-28T18:22:37.376 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560bd9f7e2d5] 2024-03-28T18:22:37.376 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560bd9fce233] 2024-03-28T18:22:37.376 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560bd9fd0197] 2024-03-28T18:22:37.376 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560bd9ecf11b] 2024-03-28T18:22:37.376 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560bd9ecf6bc] 2024-03-28T18:22:37.376 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560bd9eb91ff] 2024-03-28T18:22:37.377 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc985cb4488] 2024-03-28T18:22:37.377 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 13: (DispatchQueue::entry()+0x51f) [0x7fc985cb160f] 2024-03-28T18:22:37.377 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc985d84e81] 2024-03-28T18:22:37.377 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc98479f1ca] 2024-03-28T18:22:37.377 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 16: clone() 2024-03-28T18:22:37.377 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: *** Caught signal (Aborted) ** 2024-03-28T18:22:37.377 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: in thread 7fc97a200700 thread_name:ms_dispatch 2024-03-28T18:22:37.377 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2024-03-28T18:22:37.020+0000 7fc97a200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc97a200700 time 2024-03-28T18:22:37.018401+0000 2024-03-28T18:22:37.377 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:22:37.377 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2024-03-28T18:22:37.377 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:37.378 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc985a224a1] 2024-03-28T18:22:37.378 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc985a22667] 2024-03-28T18:22:37.378 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560bda101223] 2024-03-28T18:22:37.378 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560bd9f7b5d6] 2024-03-28T18:22:37.378 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560bd9f7bda1] 2024-03-28T18:22:37.378 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560bd9f7e2d5] 2024-03-28T18:22:37.378 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560bd9fce233] 2024-03-28T18:22:37.378 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560bd9fd0197] 2024-03-28T18:22:37.378 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560bd9ecf11b] 2024-03-28T18:22:37.378 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560bd9ecf6bc] 2024-03-28T18:22:37.378 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560bd9eb91ff] 2024-03-28T18:22:37.378 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc985cb4488] 2024-03-28T18:22:37.378 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 13: (DispatchQueue::entry()+0x51f) [0x7fc985cb160f] 2024-03-28T18:22:37.378 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc985d84e81] 2024-03-28T18:22:37.379 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc98479f1ca] 2024-03-28T18:22:37.379 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 16: clone() 2024-03-28T18:22:37.379 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2024-03-28T18:22:37.379 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:37.379 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc9847a9d20] 2024-03-28T18:22:37.379 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2: gsignal() 2024-03-28T18:22:37.379 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 3: abort() 2024-03-28T18:22:37.379 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc985a224fb] 2024-03-28T18:22:37.379 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc985a22667] 2024-03-28T18:22:37.379 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560bda101223] 2024-03-28T18:22:37.380 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560bd9f7b5d6] 2024-03-28T18:22:37.380 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560bd9f7bda1] 2024-03-28T18:22:37.380 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560bd9f7e2d5] 2024-03-28T18:22:37.380 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560bd9fce233] 2024-03-28T18:22:37.380 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560bd9fd0197] 2024-03-28T18:22:37.380 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560bd9ecf11b] 2024-03-28T18:22:37.380 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560bd9ecf6bc] 2024-03-28T18:22:37.380 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560bd9eb91ff] 2024-03-28T18:22:37.380 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc985cb4488] 2024-03-28T18:22:37.380 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 16: (DispatchQueue::entry()+0x51f) [0x7fc985cb160f] 2024-03-28T18:22:37.380 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc985d84e81] 2024-03-28T18:22:37.380 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc98479f1ca] 2024-03-28T18:22:37.380 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 19: clone() 2024-03-28T18:22:37.380 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2024-03-28T18:22:37.024+0000 7fc97a200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:22:37.380 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: in thread 7fc97a200700 thread_name:ms_dispatch 2024-03-28T18:22:37.380 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2024-03-28T18:22:37.381 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:37.381 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc9847a9d20] 2024-03-28T18:22:37.381 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2: gsignal() 2024-03-28T18:22:37.381 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 3: abort() 2024-03-28T18:22:37.381 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc985a224fb] 2024-03-28T18:22:37.381 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc985a22667] 2024-03-28T18:22:37.381 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560bda101223] 2024-03-28T18:22:37.381 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560bd9f7b5d6] 2024-03-28T18:22:37.381 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560bd9f7bda1] 2024-03-28T18:22:37.381 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560bd9f7e2d5] 2024-03-28T18:22:37.382 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560bd9fce233] 2024-03-28T18:22:37.382 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560bd9fd0197] 2024-03-28T18:22:37.382 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560bd9ecf11b] 2024-03-28T18:22:37.382 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560bd9ecf6bc] 2024-03-28T18:22:37.382 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560bd9eb91ff] 2024-03-28T18:22:37.382 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc985cb4488] 2024-03-28T18:22:37.382 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 16: (DispatchQueue::entry()+0x51f) [0x7fc985cb160f] 2024-03-28T18:22:37.382 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc985d84e81] 2024-03-28T18:22:37.382 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc98479f1ca] 2024-03-28T18:22:37.382 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 19: clone() 2024-03-28T18:22:37.382 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:22:37.382 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2024-03-28T18:22:37.382 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: -1> 2024-03-28T18:22:37.020+0000 7fc97a200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc97a200700 time 2024-03-28T18:22:37.018401+0000 2024-03-28T18:22:37.382 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:22:37.382 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2024-03-28T18:22:37.383 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:37.383 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc985a224a1] 2024-03-28T18:22:37.383 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc985a22667] 2024-03-28T18:22:37.383 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560bda101223] 2024-03-28T18:22:37.383 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560bd9f7b5d6] 2024-03-28T18:22:37.383 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560bd9f7bda1] 2024-03-28T18:22:37.384 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560bd9f7e2d5] 2024-03-28T18:22:37.384 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560bd9fce233] 2024-03-28T18:22:37.384 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560bd9fd0197] 2024-03-28T18:22:37.384 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560bd9ecf11b] 2024-03-28T18:22:37.384 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560bd9ecf6bc] 2024-03-28T18:22:37.384 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560bd9eb91ff] 2024-03-28T18:22:37.384 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc985cb4488] 2024-03-28T18:22:37.384 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 13: (DispatchQueue::entry()+0x51f) [0x7fc985cb160f] 2024-03-28T18:22:37.384 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc985d84e81] 2024-03-28T18:22:37.384 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc98479f1ca] 2024-03-28T18:22:37.384 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 16: clone() 2024-03-28T18:22:37.384 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2024-03-28T18:22:37.384 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 0> 2024-03-28T18:22:37.024+0000 7fc97a200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:22:37.384 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: in thread 7fc97a200700 thread_name:ms_dispatch 2024-03-28T18:22:37.385 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2024-03-28T18:22:37.385 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:37.385 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc9847a9d20] 2024-03-28T18:22:37.385 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2: gsignal() 2024-03-28T18:22:37.385 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 3: abort() 2024-03-28T18:22:37.385 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc985a224fb] 2024-03-28T18:22:37.385 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc985a22667] 2024-03-28T18:22:37.385 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560bda101223] 2024-03-28T18:22:37.385 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560bd9f7b5d6] 2024-03-28T18:22:37.385 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560bd9f7bda1] 2024-03-28T18:22:37.385 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560bd9f7e2d5] 2024-03-28T18:22:37.385 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560bd9fce233] 2024-03-28T18:22:37.385 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560bd9fd0197] 2024-03-28T18:22:37.385 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560bd9ecf11b] 2024-03-28T18:22:37.385 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560bd9ecf6bc] 2024-03-28T18:22:37.386 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560bd9eb91ff] 2024-03-28T18:22:37.386 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc985cb4488] 2024-03-28T18:22:37.386 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 16: (DispatchQueue::entry()+0x51f) [0x7fc985cb160f] 2024-03-28T18:22:37.386 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc985d84e81] 2024-03-28T18:22:37.386 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc98479f1ca] 2024-03-28T18:22:37.386 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 19: clone() 2024-03-28T18:22:37.386 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:22:37.386 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2024-03-28T18:22:37.386 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: -1> 2024-03-28T18:22:37.020+0000 7fc97a200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc97a200700 time 2024-03-28T18:22:37.018401+0000 2024-03-28T18:22:37.386 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:22:37.386 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2024-03-28T18:22:37.386 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:37.386 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc985a224a1] 2024-03-28T18:22:37.386 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc985a22667] 2024-03-28T18:22:37.387 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560bda101223] 2024-03-28T18:22:37.387 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560bd9f7b5d6] 2024-03-28T18:22:37.387 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560bd9f7bda1] 2024-03-28T18:22:37.387 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560bd9f7e2d5] 2024-03-28T18:22:37.387 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560bd9fce233] 2024-03-28T18:22:37.387 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560bd9fd0197] 2024-03-28T18:22:37.388 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560bd9ecf11b] 2024-03-28T18:22:37.388 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560bd9ecf6bc] 2024-03-28T18:22:37.388 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560bd9eb91ff] 2024-03-28T18:22:37.388 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc985cb4488] 2024-03-28T18:22:37.388 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 13: (DispatchQueue::entry()+0x51f) [0x7fc985cb160f] 2024-03-28T18:22:37.388 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc985d84e81] 2024-03-28T18:22:37.388 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc98479f1ca] 2024-03-28T18:22:37.388 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 16: clone() 2024-03-28T18:22:37.388 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2024-03-28T18:22:37.388 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 0> 2024-03-28T18:22:37.024+0000 7fc97a200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:22:37.388 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: in thread 7fc97a200700 thread_name:ms_dispatch 2024-03-28T18:22:37.388 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2024-03-28T18:22:37.389 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:37.389 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc9847a9d20] 2024-03-28T18:22:37.389 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2: gsignal() 2024-03-28T18:22:37.389 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 3: abort() 2024-03-28T18:22:37.389 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc985a224fb] 2024-03-28T18:22:37.389 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc985a22667] 2024-03-28T18:22:37.389 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560bda101223] 2024-03-28T18:22:37.389 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560bd9f7b5d6] 2024-03-28T18:22:37.389 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560bd9f7bda1] 2024-03-28T18:22:37.389 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560bd9f7e2d5] 2024-03-28T18:22:37.389 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560bd9fce233] 2024-03-28T18:22:37.389 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560bd9fd0197] 2024-03-28T18:22:37.389 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560bd9ecf11b] 2024-03-28T18:22:37.389 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560bd9ecf6bc] 2024-03-28T18:22:37.389 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560bd9eb91ff] 2024-03-28T18:22:37.389 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc985cb4488] 2024-03-28T18:22:37.390 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 16: (DispatchQueue::entry()+0x51f) [0x7fc985cb160f] 2024-03-28T18:22:37.390 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc985d84e81] 2024-03-28T18:22:37.390 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc98479f1ca] 2024-03-28T18:22:37.390 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 19: clone() 2024-03-28T18:22:37.390 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:22:37.390 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d[89238]: 2024-03-28T18:22:37.390 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 podman[89276]: 2024-03-28 18:22:37.170004653 +0000 UTC m=+0.041881791 container died c97a43ca5460e65a8b6a58c0bff99b98927fa87a94fd1f539c52d9e8a83bde39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-28T18:22:38.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:37 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:22:38.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:37 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/1553180221,v1:172.21.15.37:6837/1553180221] up:active 2024-03-28T18:22:38.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:37 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:38.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:37 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T18:22:38.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:37 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:22:38.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:37 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/1553180221,v1:172.21.15.37:6837/1553180221] up:active 2024-03-28T18:22:38.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:37 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:38.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:37 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T18:22:38.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:22:38.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/1553180221,v1:172.21.15.37:6837/1553180221] up:active 2024-03-28T18:22:38.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T18:22:38.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:37 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T18:22:38.375 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 podman[89276]: 2024-03-28 18:22:37.960482768 +0000 UTC m=+0.832359939 container remove c97a43ca5460e65a8b6a58c0bff99b98927fa87a94fd1f539c52d9e8a83bde39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-28T18:22:38.375 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:37 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:22:38.875 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:38 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-28T18:22:38.875 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:38 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.848s CPU time. 2024-03-28T18:22:40.005 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:39 smithi037 ceph-mon[31460]: pgmap v389: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T18:22:40.005 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:39 smithi037 ceph-mon[31460]: osdmap e130: 12 total, 12 up, 12 in 2024-03-28T18:22:40.005 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:39 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:22:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:39 smithi059 ceph-mon[36190]: pgmap v389: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T18:22:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:39 smithi059 ceph-mon[36190]: osdmap e130: 12 total, 12 up, 12 in 2024-03-28T18:22:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:39 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:22:40.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:39 smithi170 ceph-mon[36280]: pgmap v389: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T18:22:40.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:39 smithi170 ceph-mon[36280]: osdmap e130: 12 total, 12 up, 12 in 2024-03-28T18:22:40.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:39 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:22:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:40 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:22:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:40 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:22:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:40 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:22:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:40 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:22:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:40 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:22:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:40 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:22:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:40 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:22:41.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:40 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:22:41.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:40 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:22:41.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:22:41 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:22:41.446+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:22:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:41 smithi059 ceph-mon[36190]: pgmap v391: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 404 B/s wr, 4 op/s 2024-03-28T18:22:42.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:41 smithi059 ceph-mon[36190]: pgmap v392: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 410 B/s wr, 4 op/s 2024-03-28T18:22:42.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:41 smithi170 ceph-mon[36280]: pgmap v391: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 404 B/s wr, 4 op/s 2024-03-28T18:22:42.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:41 smithi170 ceph-mon[36280]: pgmap v392: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 410 B/s wr, 4 op/s 2024-03-28T18:22:42.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:41 smithi037 ceph-mon[31460]: pgmap v391: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 404 B/s wr, 4 op/s 2024-03-28T18:22:42.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:41 smithi037 ceph-mon[31460]: pgmap v392: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 410 B/s wr, 4 op/s 2024-03-28T18:22:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:43 smithi059 ceph-mon[36190]: pgmap v393: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 956 B/s wr, 22 op/s 2024-03-28T18:22:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:43 smithi170 ceph-mon[36280]: pgmap v393: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 956 B/s wr, 22 op/s 2024-03-28T18:22:44.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:43 smithi037 ceph-mon[31460]: pgmap v393: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 956 B/s wr, 22 op/s 2024-03-28T18:22:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:45 smithi059 ceph-mon[36190]: pgmap v394: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 546 B/s wr, 18 op/s 2024-03-28T18:22:46.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:45 smithi170 ceph-mon[36280]: pgmap v394: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 546 B/s wr, 18 op/s 2024-03-28T18:22:46.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:45 smithi037 ceph-mon[31460]: pgmap v394: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 546 B/s wr, 18 op/s 2024-03-28T18:22:48.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:47 smithi059 ceph-mon[36190]: pgmap v395: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 546 B/s wr, 18 op/s 2024-03-28T18:22:48.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:47 smithi170 ceph-mon[36280]: pgmap v395: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 546 B/s wr, 18 op/s 2024-03-28T18:22:48.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:47 smithi037 ceph-mon[31460]: pgmap v395: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 546 B/s wr, 18 op/s 2024-03-28T18:22:49.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:48 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 6. 2024-03-28T18:22:49.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:48 smithi037 systemd[1]: Stopped Ceph mds.d for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:22:49.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:48 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.848s CPU time. 2024-03-28T18:22:49.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:48 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Start request repeated too quickly. 2024-03-28T18:22:49.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:48 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-28T18:22:49.125 INFO:journalctl@ceph.mds.d.smithi037.stdout:Mar 28 18:22:48 smithi037 systemd[1]: Failed to start Ceph mds.d for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:22:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:49 smithi059 ceph-mon[36190]: pgmap v396: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 433 B/s wr, 17 op/s 2024-03-28T18:22:50.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:49 smithi170 ceph-mon[36280]: pgmap v396: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 433 B/s wr, 17 op/s 2024-03-28T18:22:50.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:49 smithi037 ceph-mon[31460]: pgmap v396: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 433 B/s wr, 17 op/s 2024-03-28T18:22:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:51 smithi059 ceph-mon[36190]: pgmap v397: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 409 B/s wr, 16 op/s 2024-03-28T18:22:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:51 smithi170 ceph-mon[36280]: pgmap v397: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 409 B/s wr, 16 op/s 2024-03-28T18:22:52.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:51 smithi037 ceph-mon[31460]: pgmap v397: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 409 B/s wr, 16 op/s 2024-03-28T18:22:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:53 smithi059 ceph-mon[36190]: pgmap v398: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 341 B/s wr, 16 op/s 2024-03-28T18:22:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:53 smithi059 ceph-mon[36190]: Replacing daemon mds.d as rank 1 with standby daemon mds.k 2024-03-28T18:22:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:53 smithi059 ceph-mon[36190]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-03-28T18:22:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:53 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:22:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:53 smithi059 ceph-mon[36190]: osdmap e131: 12 total, 12 up, 12 in 2024-03-28T18:22:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:53 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:22:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:53 smithi170 ceph-mon[36280]: pgmap v398: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 341 B/s wr, 16 op/s 2024-03-28T18:22:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:53 smithi170 ceph-mon[36280]: Replacing daemon mds.d as rank 1 with standby daemon mds.k 2024-03-28T18:22:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:53 smithi170 ceph-mon[36280]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-03-28T18:22:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:53 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:22:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:53 smithi170 ceph-mon[36280]: osdmap e131: 12 total, 12 up, 12 in 2024-03-28T18:22:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:53 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:22:54.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:53 smithi037 ceph-mon[31460]: pgmap v398: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 341 B/s wr, 16 op/s 2024-03-28T18:22:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:53 smithi037 ceph-mon[31460]: Replacing daemon mds.d as rank 1 with standby daemon mds.k 2024-03-28T18:22:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:53 smithi037 ceph-mon[31460]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-03-28T18:22:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:53 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:22:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:53 smithi037 ceph-mon[31460]: osdmap e131: 12 total, 12 up, 12 in 2024-03-28T18:22:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:53 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:22:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:55 smithi059 ceph-mon[36190]: pgmap v400: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:22:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:55 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/2898798947,v1:172.21.15.59:6833/2898798947] up:resolve 2024-03-28T18:22:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:55 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:22:56.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:55 smithi170 ceph-mon[36280]: pgmap v400: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:22:56.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:55 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/2898798947,v1:172.21.15.59:6833/2898798947] up:resolve 2024-03-28T18:22:56.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:55 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:22:56.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:55 smithi037 ceph-mon[31460]: pgmap v400: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:22:56.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:55 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/2898798947,v1:172.21.15.59:6833/2898798947] up:resolve 2024-03-28T18:22:56.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:55 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:22:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:56 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/2898798947,v1:172.21.15.59:6833/2898798947] up:reconnect 2024-03-28T18:22:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:56 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:22:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:56 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.0079999 2024-03-28T18:22:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:56 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0079999 2024-03-28T18:22:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:56 smithi059 ceph-mon[36190]: pgmap v401: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:22:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:56 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/2898798947,v1:172.21.15.59:6833/2898798947] up:reconnect 2024-03-28T18:22:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:56 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:22:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:56 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.0079999 2024-03-28T18:22:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:56 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0079999 2024-03-28T18:22:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:56 smithi170 ceph-mon[36280]: pgmap v401: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:22:57.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:56 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/2898798947,v1:172.21.15.59:6833/2898798947] up:reconnect 2024-03-28T18:22:57.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:56 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:22:57.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:56 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.0079999 2024-03-28T18:22:57.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:56 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0079999 2024-03-28T18:22:57.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:56 smithi037 ceph-mon[31460]: pgmap v401: 97 pgs: 97 active+clean; 45 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:22:58.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/2898798947,v1:172.21.15.59:6833/2898798947] up:rejoin 2024-03-28T18:22:58.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:22:58.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-mon[36190]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-28T18:22:58.286 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2440b97700 time 2024-03-28T18:22:57.977421+0000 2024-03-28T18:22:58.286 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:22:58.286 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:58.286 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f244b1c14a1] 2024-03-28T18:22:58.286 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f244b1c1667] 2024-03-28T18:22:58.286 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55efca867223] 2024-03-28T18:22:58.286 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55efca6e15d6] 2024-03-28T18:22:58.286 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55efca6e1da1] 2024-03-28T18:22:58.286 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55efca6e42d5] 2024-03-28T18:22:58.286 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55efca734233] 2024-03-28T18:22:58.286 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55efca736197] 2024-03-28T18:22:58.286 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55efca63511b] 2024-03-28T18:22:58.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55efca6356bc] 2024-03-28T18:22:58.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55efca61f1ff] 2024-03-28T18:22:58.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f244b453488] 2024-03-28T18:22:58.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 13: (DispatchQueue::entry()+0x51f) [0x7f244b45060f] 2024-03-28T18:22:58.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f244b523e81] 2024-03-28T18:22:58.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2449f3e1ca] 2024-03-28T18:22:58.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 16: clone() 2024-03-28T18:22:58.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2024-03-28T18:22:57.977+0000 7f2440b97700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2440b97700 time 2024-03-28T18:22:57.977421+0000 2024-03-28T18:22:58.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:22:58.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2024-03-28T18:22:58.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:58.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f244b1c14a1] 2024-03-28T18:22:58.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f244b1c1667] 2024-03-28T18:22:58.287 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55efca867223] 2024-03-28T18:22:58.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55efca6e15d6] 2024-03-28T18:22:58.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55efca6e1da1] 2024-03-28T18:22:58.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55efca6e42d5] 2024-03-28T18:22:58.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55efca734233] 2024-03-28T18:22:58.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55efca736197] 2024-03-28T18:22:58.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55efca63511b] 2024-03-28T18:22:58.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55efca6356bc] 2024-03-28T18:22:58.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55efca61f1ff] 2024-03-28T18:22:58.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f244b453488] 2024-03-28T18:22:58.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 13: (DispatchQueue::entry()+0x51f) [0x7f244b45060f] 2024-03-28T18:22:58.288 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f244b523e81] 2024-03-28T18:22:58.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2449f3e1ca] 2024-03-28T18:22:58.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 16: clone() 2024-03-28T18:22:58.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2024-03-28T18:22:58.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: *** Caught signal (Aborted) ** 2024-03-28T18:22:58.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: in thread 7f2440b97700 thread_name:ms_dispatch 2024-03-28T18:22:58.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:58.289 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2449f48d20] 2024-03-28T18:22:58.290 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:57 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/2898798947,v1:172.21.15.59:6833/2898798947] up:rejoin 2024-03-28T18:22:58.290 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:57 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:22:58.290 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:57 smithi170 ceph-mon[36280]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-28T18:22:58.290 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2: gsignal() 2024-03-28T18:22:58.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 3: abort() 2024-03-28T18:22:58.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f244b1c14fb] 2024-03-28T18:22:58.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f244b1c1667] 2024-03-28T18:22:58.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55efca867223] 2024-03-28T18:22:58.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55efca6e15d6] 2024-03-28T18:22:58.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55efca6e1da1] 2024-03-28T18:22:58.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55efca6e42d5] 2024-03-28T18:22:58.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55efca734233] 2024-03-28T18:22:58.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55efca736197] 2024-03-28T18:22:58.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55efca63511b] 2024-03-28T18:22:58.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55efca6356bc] 2024-03-28T18:22:58.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55efca61f1ff] 2024-03-28T18:22:58.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f244b453488] 2024-03-28T18:22:58.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 16: (DispatchQueue::entry()+0x51f) [0x7f244b45060f] 2024-03-28T18:22:58.291 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f244b523e81] 2024-03-28T18:22:58.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2449f3e1ca] 2024-03-28T18:22:58.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 19: clone() 2024-03-28T18:22:58.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2024-03-28T18:22:57.979+0000 7f2440b97700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:22:58.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: in thread 7f2440b97700 thread_name:ms_dispatch 2024-03-28T18:22:58.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2024-03-28T18:22:58.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:58.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2449f48d20] 2024-03-28T18:22:58.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2: gsignal() 2024-03-28T18:22:58.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 3: abort() 2024-03-28T18:22:58.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f244b1c14fb] 2024-03-28T18:22:58.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f244b1c1667] 2024-03-28T18:22:58.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55efca867223] 2024-03-28T18:22:58.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55efca6e15d6] 2024-03-28T18:22:58.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55efca6e1da1] 2024-03-28T18:22:58.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55efca6e42d5] 2024-03-28T18:22:58.292 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55efca734233] 2024-03-28T18:22:58.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55efca736197] 2024-03-28T18:22:58.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55efca63511b] 2024-03-28T18:22:58.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55efca6356bc] 2024-03-28T18:22:58.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55efca61f1ff] 2024-03-28T18:22:58.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f244b453488] 2024-03-28T18:22:58.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 16: (DispatchQueue::entry()+0x51f) [0x7f244b45060f] 2024-03-28T18:22:58.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f244b523e81] 2024-03-28T18:22:58.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2449f3e1ca] 2024-03-28T18:22:58.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 19: clone() 2024-03-28T18:22:58.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:22:58.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2024-03-28T18:22:58.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: -1> 2024-03-28T18:22:57.977+0000 7f2440b97700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2440b97700 time 2024-03-28T18:22:57.977421+0000 2024-03-28T18:22:58.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:22:58.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2024-03-28T18:22:58.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:58.293 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f244b1c14a1] 2024-03-28T18:22:58.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f244b1c1667] 2024-03-28T18:22:58.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55efca867223] 2024-03-28T18:22:58.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55efca6e15d6] 2024-03-28T18:22:58.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55efca6e1da1] 2024-03-28T18:22:58.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55efca6e42d5] 2024-03-28T18:22:58.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55efca734233] 2024-03-28T18:22:58.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55efca736197] 2024-03-28T18:22:58.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55efca63511b] 2024-03-28T18:22:58.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55efca6356bc] 2024-03-28T18:22:58.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55efca61f1ff] 2024-03-28T18:22:58.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f244b453488] 2024-03-28T18:22:58.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 13: (DispatchQueue::entry()+0x51f) [0x7f244b45060f] 2024-03-28T18:22:58.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f244b523e81] 2024-03-28T18:22:58.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2449f3e1ca] 2024-03-28T18:22:58.294 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 16: clone() 2024-03-28T18:22:58.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2024-03-28T18:22:58.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 0> 2024-03-28T18:22:57.979+0000 7f2440b97700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:22:58.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: in thread 7f2440b97700 thread_name:ms_dispatch 2024-03-28T18:22:58.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2024-03-28T18:22:58.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:58.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2449f48d20] 2024-03-28T18:22:58.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2: gsignal() 2024-03-28T18:22:58.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 3: abort() 2024-03-28T18:22:58.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f244b1c14fb] 2024-03-28T18:22:58.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f244b1c1667] 2024-03-28T18:22:58.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55efca867223] 2024-03-28T18:22:58.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55efca6e15d6] 2024-03-28T18:22:58.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55efca6e1da1] 2024-03-28T18:22:58.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55efca6e42d5] 2024-03-28T18:22:58.295 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55efca734233] 2024-03-28T18:22:58.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55efca736197] 2024-03-28T18:22:58.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55efca63511b] 2024-03-28T18:22:58.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55efca6356bc] 2024-03-28T18:22:58.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55efca61f1ff] 2024-03-28T18:22:58.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f244b453488] 2024-03-28T18:22:58.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 16: (DispatchQueue::entry()+0x51f) [0x7f244b45060f] 2024-03-28T18:22:58.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f244b523e81] 2024-03-28T18:22:58.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2449f3e1ca] 2024-03-28T18:22:58.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 19: clone() 2024-03-28T18:22:58.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:22:58.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:57 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2024-03-28T18:22:58.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: -1> 2024-03-28T18:22:57.977+0000 7f2440b97700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2440b97700 time 2024-03-28T18:22:57.977421+0000 2024-03-28T18:22:58.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:22:58.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2024-03-28T18:22:58.296 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:58.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f244b1c14a1] 2024-03-28T18:22:58.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f244b1c1667] 2024-03-28T18:22:58.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55efca867223] 2024-03-28T18:22:58.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55efca6e15d6] 2024-03-28T18:22:58.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55efca6e1da1] 2024-03-28T18:22:58.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55efca6e42d5] 2024-03-28T18:22:58.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55efca734233] 2024-03-28T18:22:58.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55efca736197] 2024-03-28T18:22:58.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55efca63511b] 2024-03-28T18:22:58.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55efca6356bc] 2024-03-28T18:22:58.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55efca61f1ff] 2024-03-28T18:22:58.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f244b453488] 2024-03-28T18:22:58.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 13: (DispatchQueue::entry()+0x51f) [0x7f244b45060f] 2024-03-28T18:22:58.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f244b523e81] 2024-03-28T18:22:58.297 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2449f3e1ca] 2024-03-28T18:22:58.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 16: clone() 2024-03-28T18:22:58.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2024-03-28T18:22:58.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 0> 2024-03-28T18:22:57.979+0000 7f2440b97700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:22:58.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: in thread 7f2440b97700 thread_name:ms_dispatch 2024-03-28T18:22:58.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2024-03-28T18:22:58.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:22:58.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2449f48d20] 2024-03-28T18:22:58.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2: gsignal() 2024-03-28T18:22:58.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 3: abort() 2024-03-28T18:22:58.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f244b1c14fb] 2024-03-28T18:22:58.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f244b1c1667] 2024-03-28T18:22:58.298 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55efca867223] 2024-03-28T18:22:58.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55efca6e15d6] 2024-03-28T18:22:58.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55efca6e1da1] 2024-03-28T18:22:58.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55efca6e42d5] 2024-03-28T18:22:58.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55efca734233] 2024-03-28T18:22:58.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55efca736197] 2024-03-28T18:22:58.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55efca63511b] 2024-03-28T18:22:58.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55efca6356bc] 2024-03-28T18:22:58.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55efca61f1ff] 2024-03-28T18:22:58.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f244b453488] 2024-03-28T18:22:58.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 16: (DispatchQueue::entry()+0x51f) [0x7f244b45060f] 2024-03-28T18:22:58.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f244b523e81] 2024-03-28T18:22:58.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2449f3e1ca] 2024-03-28T18:22:58.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 19: clone() 2024-03-28T18:22:58.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:22:58.299 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74103]: 2024-03-28T18:22:58.300 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 podman[74271]: 2024-03-28 18:22:58.040396831 +0000 UTC m=+0.019797315 container died 5bcbcb8431401318058bd04ed550c3c255197f34a893c8ee44cd32be42a841d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-28T18:22:58.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:57 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/2898798947,v1:172.21.15.59:6833/2898798947] up:rejoin 2024-03-28T18:22:58.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:57 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:22:58.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:57 smithi037 ceph-mon[31460]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-28T18:22:59.128 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:22:59.128 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/2898798947,v1:172.21.15.59:6833/2898798947] up:active 2024-03-28T18:22:59.128 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:22:59.128 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-03-28T18:22:59.128 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:22:58 smithi059 ceph-mon[36190]: pgmap v402: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T18:22:59.129 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 podman[74271]: 2024-03-28 18:22:58.878447988 +0000 UTC m=+0.857848465 container remove 5bcbcb8431401318058bd04ed550c3c255197f34a893c8ee44cd32be42a841d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-28T18:22:59.129 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:58 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:22:59.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:58 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:22:59.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:58 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/2898798947,v1:172.21.15.59:6833/2898798947] up:active 2024-03-28T18:22:59.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:58 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:22:59.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:58 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-03-28T18:22:59.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:22:58 smithi170 ceph-mon[36280]: pgmap v402: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T18:22:59.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:58 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:22:59.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:58 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/2898798947,v1:172.21.15.59:6833/2898798947] up:active 2024-03-28T18:22:59.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:58 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:22:59.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:58 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-03-28T18:22:59.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:22:58 smithi037 ceph-mon[31460]: pgmap v402: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T18:22:59.535 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:22:59 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-28T18:23:01.758 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:01 smithi037 ceph-mon[31460]: pgmap v403: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T18:23:01.759 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:23:01 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:23:01.448+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:23:01.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:01 smithi059 ceph-mon[36190]: pgmap v403: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T18:23:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:01 smithi170 ceph-mon[36280]: pgmap v403: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T18:23:03.734 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:03 smithi037 ceph-mon[31460]: pgmap v404: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T18:23:03.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:03 smithi059 ceph-mon[36190]: pgmap v404: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T18:23:03.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:03 smithi170 ceph-mon[36280]: pgmap v404: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T18:23:05.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:05 smithi059 ceph-mon[36190]: pgmap v405: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 195 B/s wr, 17 op/s 2024-03-28T18:23:05.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:05 smithi170 ceph-mon[36280]: pgmap v405: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 195 B/s wr, 17 op/s 2024-03-28T18:23:05.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:05 smithi037 ceph-mon[31460]: pgmap v405: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 195 B/s wr, 17 op/s 2024-03-28T18:23:07.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:07 smithi059 ceph-mon[36190]: pgmap v406: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 15 op/s 2024-03-28T18:23:07.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:07 smithi170 ceph-mon[36280]: pgmap v406: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 15 op/s 2024-03-28T18:23:07.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:07 smithi037 ceph-mon[31460]: pgmap v406: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 15 op/s 2024-03-28T18:23:09.554 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:09 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Scheduled restart job, restart counter is at 5. 2024-03-28T18:23:09.554 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:09 smithi059 systemd[1]: Stopped Ceph mds.k for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:23:09.554 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:09 smithi059 systemd[1]: Starting Ceph mds.k for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:23:09.554 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:09 smithi059 podman[74368]: 2024-03-28 18:23:09.426397754 +0000 UTC m=+0.018728820 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:23:09.554 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:09 smithi059 podman[74368]: 2024-03-28 18:23:09.55442104 +0000 UTC m=+0.146752125 container create 1bacb84202758f01a5c6819835be1184eb7009fe33e08eb3682645c1d106944f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-28T18:23:10.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:09 smithi059 ceph-mon[36190]: pgmap v407: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 17 op/s 2024-03-28T18:23:10.285 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:09 smithi059 podman[74368]: 2024-03-28 18:23:09.907512753 +0000 UTC m=+0.499843837 container init 1bacb84202758f01a5c6819835be1184eb7009fe33e08eb3682645c1d106944f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-28T18:23:10.285 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:09 smithi059 podman[74368]: 2024-03-28 18:23:09.910510857 +0000 UTC m=+0.502841930 container start 1bacb84202758f01a5c6819835be1184eb7009fe33e08eb3682645c1d106944f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:23:10.285 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:09 smithi059 ceph-mds[74387]: -- 172.21.15.59:0/3146015653 <== mon.2 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561a02477a40 con 0x561a0244ac00 2024-03-28T18:23:10.285 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:09 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: starting mds.k at 2024-03-28T18:23:10.285 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:10 smithi059 bash[74368]: 1bacb84202758f01a5c6819835be1184eb7009fe33e08eb3682645c1d106944f 2024-03-28T18:23:10.286 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:10 smithi059 systemd[1]: Started Ceph mds.k for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:23:10.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:09 smithi170 ceph-mon[36280]: pgmap v407: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 17 op/s 2024-03-28T18:23:10.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:09 smithi037 ceph-mon[31460]: pgmap v407: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 17 op/s 2024-03-28T18:23:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:10 smithi059 ceph-mon[36190]: daemon mds.k restarted 2024-03-28T18:23:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:10 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:23:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:23:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:10 smithi059 ceph-mon[36190]: osdmap e132: 12 total, 12 up, 12 in 2024-03-28T18:23:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:10 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/2519390530,v1:172.21.15.59:6833/2519390530] up:boot 2024-03-28T18:23:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:10 smithi059 ceph-mon[36190]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-28T18:23:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:10 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-03-28T18:23:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:10 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:11.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:10 smithi170 ceph-mon[36280]: daemon mds.k restarted 2024-03-28T18:23:11.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:10 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:23:11.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:23:11.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:10 smithi170 ceph-mon[36280]: osdmap e132: 12 total, 12 up, 12 in 2024-03-28T18:23:11.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:10 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/2519390530,v1:172.21.15.59:6833/2519390530] up:boot 2024-03-28T18:23:11.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:10 smithi170 ceph-mon[36280]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-28T18:23:11.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:10 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-03-28T18:23:11.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:10 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:10 smithi037 ceph-mon[31460]: daemon mds.k restarted 2024-03-28T18:23:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:10 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:23:11.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:23:11.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:10 smithi037 ceph-mon[31460]: osdmap e132: 12 total, 12 up, 12 in 2024-03-28T18:23:11.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:10 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/2519390530,v1:172.21.15.59:6833/2519390530] up:boot 2024-03-28T18:23:11.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:10 smithi037 ceph-mon[31460]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-28T18:23:11.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:10 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-03-28T18:23:11.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:10 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:11 smithi059 ceph-mon[36190]: pgmap v409: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T18:23:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:11 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6838/1926270371,v1:172.21.15.37:6839/1926270371] up:resolve 2024-03-28T18:23:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:11 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:12.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:11 smithi170 ceph-mon[36280]: pgmap v409: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T18:23:12.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:11 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6838/1926270371,v1:172.21.15.37:6839/1926270371] up:resolve 2024-03-28T18:23:12.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:11 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:12.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:11 smithi037 ceph-mon[31460]: pgmap v409: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T18:23:12.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:11 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6838/1926270371,v1:172.21.15.37:6839/1926270371] up:resolve 2024-03-28T18:23:12.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:11 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:12 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:23:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:12 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6838/1926270371,v1:172.21.15.37:6839/1926270371] up:reconnect 2024-03-28T18:23:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:12 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:12 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899988 2024-03-28T18:23:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:12 smithi059 ceph-mon[36190]: pgmap v410: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-28T18:23:13.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:12 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:23:13.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:12 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6838/1926270371,v1:172.21.15.37:6839/1926270371] up:reconnect 2024-03-28T18:23:13.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:12 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:13.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:12 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899988 2024-03-28T18:23:13.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:12 smithi170 ceph-mon[36280]: pgmap v410: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-28T18:23:13.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:12 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:23:13.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:12 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6838/1926270371,v1:172.21.15.37:6839/1926270371] up:reconnect 2024-03-28T18:23:13.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:12 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:13.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:12 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899988 2024-03-28T18:23:13.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:12 smithi037 ceph-mon[31460]: pgmap v410: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-28T18:23:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:13 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6838/1926270371,v1:172.21.15.37:6839/1926270371] up:rejoin 2024-03-28T18:23:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:13 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:13 smithi059 ceph-mon[36190]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-28T18:23:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:13 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:23:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:13 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6838/1926270371,v1:172.21.15.37:6839/1926270371] up:active 2024-03-28T18:23:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:13 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-03-28T18:23:14.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:13 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6838/1926270371,v1:172.21.15.37:6839/1926270371] up:rejoin 2024-03-28T18:23:14.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:13 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:14.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:13 smithi170 ceph-mon[36280]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-28T18:23:14.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:13 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:23:14.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:13 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6838/1926270371,v1:172.21.15.37:6839/1926270371] up:active 2024-03-28T18:23:14.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:13 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-03-28T18:23:14.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6838/1926270371,v1:172.21.15.37:6839/1926270371] up:rejoin 2024-03-28T18:23:14.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:14.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-mon[31460]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-28T18:23:14.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:23:14.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6838/1926270371,v1:172.21.15.37:6839/1926270371] up:active 2024-03-28T18:23:14.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-03-28T18:23:14.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff663e00700 time 2024-03-28T18:23:13.957352+0000 2024-03-28T18:23:14.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:23:14.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:14.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff66f6694a1] 2024-03-28T18:23:14.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff66f669667] 2024-03-28T18:23:14.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558163186223] 2024-03-28T18:23:14.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5581630005d6] 2024-03-28T18:23:14.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558163000da1] 2024-03-28T18:23:14.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5581630032d5] 2024-03-28T18:23:14.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558163053233] 2024-03-28T18:23:14.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558163055197] 2024-03-28T18:23:14.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558162f5411b] 2024-03-28T18:23:14.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558162f546bc] 2024-03-28T18:23:14.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558162f3e1ff] 2024-03-28T18:23:14.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff66f8fb488] 2024-03-28T18:23:14.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 13: (DispatchQueue::entry()+0x51f) [0x7ff66f8f860f] 2024-03-28T18:23:14.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff66f9cbe81] 2024-03-28T18:23:14.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff66e3e61ca] 2024-03-28T18:23:14.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 16: clone() 2024-03-28T18:23:14.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: *** Caught signal (Aborted) ** 2024-03-28T18:23:14.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: in thread 7ff663e00700 thread_name:ms_dispatch 2024-03-28T18:23:14.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2024-03-28T18:23:13.958+0000 7ff663e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff663e00700 time 2024-03-28T18:23:13.957352+0000 2024-03-28T18:23:14.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:23:14.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2024-03-28T18:23:14.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:14.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff66f6694a1] 2024-03-28T18:23:14.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff66f669667] 2024-03-28T18:23:14.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558163186223] 2024-03-28T18:23:14.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5581630005d6] 2024-03-28T18:23:14.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558163000da1] 2024-03-28T18:23:14.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5581630032d5] 2024-03-28T18:23:14.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558163053233] 2024-03-28T18:23:14.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558163055197] 2024-03-28T18:23:14.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558162f5411b] 2024-03-28T18:23:14.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558162f546bc] 2024-03-28T18:23:14.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558162f3e1ff] 2024-03-28T18:23:14.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff66f8fb488] 2024-03-28T18:23:14.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 13: (DispatchQueue::entry()+0x51f) [0x7ff66f8f860f] 2024-03-28T18:23:14.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff66f9cbe81] 2024-03-28T18:23:14.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff66e3e61ca] 2024-03-28T18:23:14.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 16: clone() 2024-03-28T18:23:14.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2024-03-28T18:23:14.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:14.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff66e3f0d20] 2024-03-28T18:23:14.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2: gsignal() 2024-03-28T18:23:14.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 3: abort() 2024-03-28T18:23:14.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff66f6694fb] 2024-03-28T18:23:14.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff66f669667] 2024-03-28T18:23:14.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558163186223] 2024-03-28T18:23:14.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5581630005d6] 2024-03-28T18:23:14.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558163000da1] 2024-03-28T18:23:14.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5581630032d5] 2024-03-28T18:23:14.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558163053233] 2024-03-28T18:23:14.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558163055197] 2024-03-28T18:23:14.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558162f5411b] 2024-03-28T18:23:14.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558162f546bc] 2024-03-28T18:23:14.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558162f3e1ff] 2024-03-28T18:23:14.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff66f8fb488] 2024-03-28T18:23:14.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 16: (DispatchQueue::entry()+0x51f) [0x7ff66f8f860f] 2024-03-28T18:23:14.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff66f9cbe81] 2024-03-28T18:23:14.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff66e3e61ca] 2024-03-28T18:23:14.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 19: clone() 2024-03-28T18:23:14.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2024-03-28T18:23:13.961+0000 7ff663e00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:23:14.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: in thread 7ff663e00700 thread_name:ms_dispatch 2024-03-28T18:23:14.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2024-03-28T18:23:14.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:14.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff66e3f0d20] 2024-03-28T18:23:14.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2: gsignal() 2024-03-28T18:23:14.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 3: abort() 2024-03-28T18:23:14.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff66f6694fb] 2024-03-28T18:23:14.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff66f669667] 2024-03-28T18:23:14.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558163186223] 2024-03-28T18:23:14.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5581630005d6] 2024-03-28T18:23:14.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558163000da1] 2024-03-28T18:23:14.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5581630032d5] 2024-03-28T18:23:14.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558163053233] 2024-03-28T18:23:14.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558163055197] 2024-03-28T18:23:14.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558162f5411b] 2024-03-28T18:23:14.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558162f546bc] 2024-03-28T18:23:14.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558162f3e1ff] 2024-03-28T18:23:14.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff66f8fb488] 2024-03-28T18:23:14.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 16: (DispatchQueue::entry()+0x51f) [0x7ff66f8f860f] 2024-03-28T18:23:14.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff66f9cbe81] 2024-03-28T18:23:14.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff66e3e61ca] 2024-03-28T18:23:14.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 19: clone() 2024-03-28T18:23:14.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:23:14.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2024-03-28T18:23:14.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: -1> 2024-03-28T18:23:13.958+0000 7ff663e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff663e00700 time 2024-03-28T18:23:13.957352+0000 2024-03-28T18:23:14.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:23:14.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2024-03-28T18:23:14.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:14.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff66f6694a1] 2024-03-28T18:23:14.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff66f669667] 2024-03-28T18:23:14.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558163186223] 2024-03-28T18:23:14.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5581630005d6] 2024-03-28T18:23:14.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558163000da1] 2024-03-28T18:23:14.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5581630032d5] 2024-03-28T18:23:14.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558163053233] 2024-03-28T18:23:14.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558163055197] 2024-03-28T18:23:14.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558162f5411b] 2024-03-28T18:23:14.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558162f546bc] 2024-03-28T18:23:14.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558162f3e1ff] 2024-03-28T18:23:14.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff66f8fb488] 2024-03-28T18:23:14.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 13: (DispatchQueue::entry()+0x51f) [0x7ff66f8f860f] 2024-03-28T18:23:14.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff66f9cbe81] 2024-03-28T18:23:14.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff66e3e61ca] 2024-03-28T18:23:14.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 16: clone() 2024-03-28T18:23:14.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2024-03-28T18:23:14.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 0> 2024-03-28T18:23:13.961+0000 7ff663e00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:23:14.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: in thread 7ff663e00700 thread_name:ms_dispatch 2024-03-28T18:23:14.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2024-03-28T18:23:14.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:14.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff66e3f0d20] 2024-03-28T18:23:14.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2: gsignal() 2024-03-28T18:23:14.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 3: abort() 2024-03-28T18:23:14.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff66f6694fb] 2024-03-28T18:23:14.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff66f669667] 2024-03-28T18:23:14.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558163186223] 2024-03-28T18:23:14.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5581630005d6] 2024-03-28T18:23:14.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558163000da1] 2024-03-28T18:23:14.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5581630032d5] 2024-03-28T18:23:14.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558163053233] 2024-03-28T18:23:14.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558163055197] 2024-03-28T18:23:14.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558162f5411b] 2024-03-28T18:23:14.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558162f546bc] 2024-03-28T18:23:14.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558162f3e1ff] 2024-03-28T18:23:14.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff66f8fb488] 2024-03-28T18:23:14.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 16: (DispatchQueue::entry()+0x51f) [0x7ff66f8f860f] 2024-03-28T18:23:14.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff66f9cbe81] 2024-03-28T18:23:14.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff66e3e61ca] 2024-03-28T18:23:14.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 19: clone() 2024-03-28T18:23:14.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:23:14.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2024-03-28T18:23:14.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: -1> 2024-03-28T18:23:13.958+0000 7ff663e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff663e00700 time 2024-03-28T18:23:13.957352+0000 2024-03-28T18:23:14.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:23:14.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2024-03-28T18:23:14.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:14.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff66f6694a1] 2024-03-28T18:23:14.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff66f669667] 2024-03-28T18:23:14.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558163186223] 2024-03-28T18:23:14.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5581630005d6] 2024-03-28T18:23:14.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558163000da1] 2024-03-28T18:23:14.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5581630032d5] 2024-03-28T18:23:14.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558163053233] 2024-03-28T18:23:14.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558163055197] 2024-03-28T18:23:14.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558162f5411b] 2024-03-28T18:23:14.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558162f546bc] 2024-03-28T18:23:14.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558162f3e1ff] 2024-03-28T18:23:14.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff66f8fb488] 2024-03-28T18:23:14.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 13: (DispatchQueue::entry()+0x51f) [0x7ff66f8f860f] 2024-03-28T18:23:14.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff66f9cbe81] 2024-03-28T18:23:14.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff66e3e61ca] 2024-03-28T18:23:14.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 16: clone() 2024-03-28T18:23:14.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2024-03-28T18:23:14.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 0> 2024-03-28T18:23:13.961+0000 7ff663e00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:23:14.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: in thread 7ff663e00700 thread_name:ms_dispatch 2024-03-28T18:23:14.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2024-03-28T18:23:14.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:14.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff66e3f0d20] 2024-03-28T18:23:14.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2: gsignal() 2024-03-28T18:23:14.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 3: abort() 2024-03-28T18:23:14.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff66f6694fb] 2024-03-28T18:23:14.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff66f669667] 2024-03-28T18:23:14.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558163186223] 2024-03-28T18:23:14.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5581630005d6] 2024-03-28T18:23:14.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558163000da1] 2024-03-28T18:23:14.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5581630032d5] 2024-03-28T18:23:14.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558163053233] 2024-03-28T18:23:14.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558163055197] 2024-03-28T18:23:14.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558162f5411b] 2024-03-28T18:23:14.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558162f546bc] 2024-03-28T18:23:14.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558162f3e1ff] 2024-03-28T18:23:14.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff66f8fb488] 2024-03-28T18:23:14.391 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 16: (DispatchQueue::entry()+0x51f) [0x7ff66f8f860f] 2024-03-28T18:23:14.391 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff66f9cbe81] 2024-03-28T18:23:14.391 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff66e3e61ca] 2024-03-28T18:23:14.391 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 19: clone() 2024-03-28T18:23:14.391 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:23:14.391 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[70712]: 2024-03-28T18:23:14.391 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:14 smithi037 podman[89478]: 2024-03-28 18:23:14.096741339 +0000 UTC m=+0.032558819 container died 5c83e57ee4ad2444c7fd51e10067c69e1d1ab98690c22d751358d64408ce3a90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-28T18:23:15.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:14 smithi059 ceph-mon[36190]: pgmap v411: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-28T18:23:15.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:14 smithi170 ceph-mon[36280]: pgmap v411: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-28T18:23:15.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:14 smithi037 ceph-mon[31460]: pgmap v411: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-28T18:23:15.375 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:15 smithi037 podman[89478]: 2024-03-28 18:23:15.044452502 +0000 UTC m=+0.980270036 container remove 5c83e57ee4ad2444c7fd51e10067c69e1d1ab98690c22d751358d64408ce3a90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:23:15.375 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:15 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:23:15.875 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:15 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-28T18:23:15.875 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:15 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Consumed 2.173s CPU time. 2024-03-28T18:23:16.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:23:16 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:23:16.449+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:23:17.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:17 smithi059 ceph-mon[36190]: pgmap v412: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-28T18:23:17.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:17 smithi170 ceph-mon[36280]: pgmap v412: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-28T18:23:17.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:17 smithi037 ceph-mon[31460]: pgmap v412: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-28T18:23:19.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:19 smithi059 ceph-mon[36190]: pgmap v413: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 32 op/s 2024-03-28T18:23:19.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:19 smithi170 ceph-mon[36280]: pgmap v413: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 32 op/s 2024-03-28T18:23:19.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:19 smithi037 ceph-mon[31460]: pgmap v413: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 32 op/s 2024-03-28T18:23:21.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:21 smithi059 ceph-mon[36190]: pgmap v414: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 26 KiB/s wr, 31 op/s 2024-03-28T18:23:21.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:21 smithi170 ceph-mon[36280]: pgmap v414: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 26 KiB/s wr, 31 op/s 2024-03-28T18:23:21.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:21 smithi037 ceph-mon[31460]: pgmap v414: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 26 KiB/s wr, 31 op/s 2024-03-28T18:23:23.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:23 smithi059 ceph-mon[36190]: pgmap v415: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:23:23.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:23 smithi170 ceph-mon[36280]: pgmap v415: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:23:23.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:23 smithi037 ceph-mon[31460]: pgmap v415: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:23:25.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:25 smithi059 ceph-mon[36190]: pgmap v416: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:23:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:25 smithi170 ceph-mon[36280]: pgmap v416: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:23:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:25 smithi037 ceph-mon[31460]: pgmap v416: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:23:25.875 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:25 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 1. 2024-03-28T18:23:25.875 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:25 smithi037 systemd[1]: Stopped Ceph mds.g for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:23:25.875 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:25 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Consumed 2.173s CPU time. 2024-03-28T18:23:25.875 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:25 smithi037 systemd[1]: Starting Ceph mds.g for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:23:26.351 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:26 smithi037 podman[89568]: 2024-03-28 18:23:25.927494469 +0000 UTC m=+0.036921911 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:23:26.351 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:26 smithi037 podman[89568]: 2024-03-28 18:23:26.07583702 +0000 UTC m=+0.185264516 container create 8613456fd40aa7a49bf323d19a3b2a7a3679ede4f5b35aa1d4a481f17a411883 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-03-28T18:23:26.875 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:26 smithi037 podman[89568]: 2024-03-28 18:23:26.478212057 +0000 UTC m=+0.587639530 container init 8613456fd40aa7a49bf323d19a3b2a7a3679ede4f5b35aa1d4a481f17a411883 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-28T18:23:26.875 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:26 smithi037 podman[89568]: 2024-03-28 18:23:26.489125168 +0000 UTC m=+0.598552620 container start 8613456fd40aa7a49bf323d19a3b2a7a3679ede4f5b35aa1d4a481f17a411883 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:23:26.875 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:26 smithi037 ceph-mds[89586]: -- 172.21.15.37:0/519191040 <== mon.0 v2:172.21.15.37:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559e875cda40 con 0x559e875a0c00 2024-03-28T18:23:26.875 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:26 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: starting mds.g at 2024-03-28T18:23:26.875 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:26 smithi037 bash[89568]: 8613456fd40aa7a49bf323d19a3b2a7a3679ede4f5b35aa1d4a481f17a411883 2024-03-28T18:23:26.875 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:26 smithi037 systemd[1]: Started Ceph mds.g for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:23:27.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:27 smithi059 ceph-mon[36190]: pgmap v417: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:23:27.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:27 smithi059 ceph-mon[36190]: daemon mds.g restarted 2024-03-28T18:23:27.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:27 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:23:27.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:27 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:23:27.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:27 smithi059 ceph-mon[36190]: osdmap e133: 12 total, 12 up, 12 in 2024-03-28T18:23:27.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:27 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/2752961889,v1:172.21.15.37:6837/2752961889] up:boot 2024-03-28T18:23:27.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:27 smithi059 ceph-mon[36190]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-28T18:23:27.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:27 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-03-28T18:23:27.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:27 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:27.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:27 smithi170 ceph-mon[36280]: pgmap v417: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:23:27.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:27 smithi170 ceph-mon[36280]: daemon mds.g restarted 2024-03-28T18:23:27.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:27 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:23:27.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:27 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:23:27.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:27 smithi170 ceph-mon[36280]: osdmap e133: 12 total, 12 up, 12 in 2024-03-28T18:23:27.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:27 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/2752961889,v1:172.21.15.37:6837/2752961889] up:boot 2024-03-28T18:23:27.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:27 smithi170 ceph-mon[36280]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-28T18:23:27.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:27 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-03-28T18:23:27.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:27 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:27.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:27 smithi037 ceph-mon[31460]: pgmap v417: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:23:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:27 smithi037 ceph-mon[31460]: daemon mds.g restarted 2024-03-28T18:23:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:27 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:23:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:27 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:23:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:27 smithi037 ceph-mon[31460]: osdmap e133: 12 total, 12 up, 12 in 2024-03-28T18:23:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:27 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/2752961889,v1:172.21.15.37:6837/2752961889] up:boot 2024-03-28T18:23:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:27 smithi037 ceph-mon[31460]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-28T18:23:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:27 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-03-28T18:23:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:27 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:28.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:28 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/2519390530,v1:172.21.15.59:6833/2519390530] up:resolve 2024-03-28T18:23:28.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:28 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:29.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:28 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/2519390530,v1:172.21.15.59:6833/2519390530] up:resolve 2024-03-28T18:23:29.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:28 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:29.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:28 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/2519390530,v1:172.21.15.59:6833/2519390530] up:resolve 2024-03-28T18:23:29.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:28 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:29 smithi037 ceph-mon[31460]: pgmap v419: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:23:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:29 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199998 2024-03-28T18:23:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:29 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/2519390530,v1:172.21.15.59:6833/2519390530] up:reconnect 2024-03-28T18:23:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:29 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:29.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:29 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899989 2024-03-28T18:23:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:29 smithi059 ceph-mon[36190]: pgmap v419: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:23:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:29 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199998 2024-03-28T18:23:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:29 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/2519390530,v1:172.21.15.59:6833/2519390530] up:reconnect 2024-03-28T18:23:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:29 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:29 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899989 2024-03-28T18:23:30.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:29 smithi170 ceph-mon[36280]: pgmap v419: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:23:30.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:29 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199998 2024-03-28T18:23:30.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:29 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/2519390530,v1:172.21.15.59:6833/2519390530] up:reconnect 2024-03-28T18:23:30.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:29 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:30.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:29 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899989 2024-03-28T18:23:30.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:30 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/2519390530,v1:172.21.15.59:6833/2519390530] up:rejoin 2024-03-28T18:23:30.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:30 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:30.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:30 smithi037 ceph-mon[31460]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-28T18:23:31.002 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/2519390530,v1:172.21.15.59:6833/2519390530] up:rejoin 2024-03-28T18:23:31.003 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:31.003 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-mon[36190]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-28T18:23:31.003 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4955d15700 time 2024-03-28T18:23:30.647203+0000 2024-03-28T18:23:31.003 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:23:31.003 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:31.003 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f496033f4a1] 2024-03-28T18:23:31.003 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f496033f667] 2024-03-28T18:23:31.004 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561a0123b223] 2024-03-28T18:23:31.004 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561a010b55d6] 2024-03-28T18:23:31.004 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561a010b5da1] 2024-03-28T18:23:31.004 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561a010b82d5] 2024-03-28T18:23:31.004 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561a01108233] 2024-03-28T18:23:31.004 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561a0110a197] 2024-03-28T18:23:31.004 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561a0100911b] 2024-03-28T18:23:31.004 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561a010096bc] 2024-03-28T18:23:31.004 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561a00ff31ff] 2024-03-28T18:23:31.004 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f49605d1488] 2024-03-28T18:23:31.004 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 13: (DispatchQueue::entry()+0x51f) [0x7f49605ce60f] 2024-03-28T18:23:31.005 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f49606a1e81] 2024-03-28T18:23:31.005 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f495f0bc1ca] 2024-03-28T18:23:31.005 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 16: clone() 2024-03-28T18:23:31.006 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2024-03-28T18:23:30.647+0000 7f4955d15700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4955d15700 time 2024-03-28T18:23:30.647203+0000 2024-03-28T18:23:31.006 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:23:31.006 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2024-03-28T18:23:31.007 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:31.007 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f496033f4a1] 2024-03-28T18:23:31.007 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f496033f667] 2024-03-28T18:23:31.007 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561a0123b223] 2024-03-28T18:23:31.007 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561a010b55d6] 2024-03-28T18:23:31.007 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561a010b5da1] 2024-03-28T18:23:31.007 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561a010b82d5] 2024-03-28T18:23:31.007 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561a01108233] 2024-03-28T18:23:31.007 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561a0110a197] 2024-03-28T18:23:31.007 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561a0100911b] 2024-03-28T18:23:31.007 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561a010096bc] 2024-03-28T18:23:31.007 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561a00ff31ff] 2024-03-28T18:23:31.008 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f49605d1488] 2024-03-28T18:23:31.008 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 13: (DispatchQueue::entry()+0x51f) [0x7f49605ce60f] 2024-03-28T18:23:31.008 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f49606a1e81] 2024-03-28T18:23:31.008 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f495f0bc1ca] 2024-03-28T18:23:31.008 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 16: clone() 2024-03-28T18:23:31.008 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2024-03-28T18:23:31.008 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: *** Caught signal (Aborted) ** 2024-03-28T18:23:31.008 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: in thread 7f4955d15700 thread_name:ms_dispatch 2024-03-28T18:23:31.008 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:31.008 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f495f0c6d20] 2024-03-28T18:23:31.008 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2: gsignal() 2024-03-28T18:23:31.008 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 3: abort() 2024-03-28T18:23:31.008 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f496033f4fb] 2024-03-28T18:23:31.009 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f496033f667] 2024-03-28T18:23:31.009 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561a0123b223] 2024-03-28T18:23:31.009 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561a010b55d6] 2024-03-28T18:23:31.009 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561a010b5da1] 2024-03-28T18:23:31.009 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561a010b82d5] 2024-03-28T18:23:31.009 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561a01108233] 2024-03-28T18:23:31.009 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561a0110a197] 2024-03-28T18:23:31.009 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561a0100911b] 2024-03-28T18:23:31.009 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561a010096bc] 2024-03-28T18:23:31.009 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561a00ff31ff] 2024-03-28T18:23:31.009 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f49605d1488] 2024-03-28T18:23:31.009 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 16: (DispatchQueue::entry()+0x51f) [0x7f49605ce60f] 2024-03-28T18:23:31.009 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f49606a1e81] 2024-03-28T18:23:31.010 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f495f0bc1ca] 2024-03-28T18:23:31.010 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 19: clone() 2024-03-28T18:23:31.010 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2024-03-28T18:23:30.649+0000 7f4955d15700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:23:31.010 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: in thread 7f4955d15700 thread_name:ms_dispatch 2024-03-28T18:23:31.010 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2024-03-28T18:23:31.010 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:31.010 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f495f0c6d20] 2024-03-28T18:23:31.010 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2: gsignal() 2024-03-28T18:23:31.010 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 3: abort() 2024-03-28T18:23:31.010 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f496033f4fb] 2024-03-28T18:23:31.010 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f496033f667] 2024-03-28T18:23:31.010 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561a0123b223] 2024-03-28T18:23:31.010 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561a010b55d6] 2024-03-28T18:23:31.010 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561a010b5da1] 2024-03-28T18:23:31.011 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561a010b82d5] 2024-03-28T18:23:31.011 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561a01108233] 2024-03-28T18:23:31.011 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561a0110a197] 2024-03-28T18:23:31.011 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561a0100911b] 2024-03-28T18:23:31.011 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561a010096bc] 2024-03-28T18:23:31.011 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561a00ff31ff] 2024-03-28T18:23:31.011 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f49605d1488] 2024-03-28T18:23:31.011 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 16: (DispatchQueue::entry()+0x51f) [0x7f49605ce60f] 2024-03-28T18:23:31.011 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f49606a1e81] 2024-03-28T18:23:31.012 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f495f0bc1ca] 2024-03-28T18:23:31.012 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 19: clone() 2024-03-28T18:23:31.012 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:23:31.012 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2024-03-28T18:23:31.012 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: -1> 2024-03-28T18:23:30.647+0000 7f4955d15700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4955d15700 time 2024-03-28T18:23:30.647203+0000 2024-03-28T18:23:31.012 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:23:31.012 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2024-03-28T18:23:31.013 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:31.013 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f496033f4a1] 2024-03-28T18:23:31.013 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f496033f667] 2024-03-28T18:23:31.013 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561a0123b223] 2024-03-28T18:23:31.013 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561a010b55d6] 2024-03-28T18:23:31.013 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561a010b5da1] 2024-03-28T18:23:31.013 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561a010b82d5] 2024-03-28T18:23:31.013 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561a01108233] 2024-03-28T18:23:31.013 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561a0110a197] 2024-03-28T18:23:31.013 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561a0100911b] 2024-03-28T18:23:31.013 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561a010096bc] 2024-03-28T18:23:31.014 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561a00ff31ff] 2024-03-28T18:23:31.014 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f49605d1488] 2024-03-28T18:23:31.014 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 13: (DispatchQueue::entry()+0x51f) [0x7f49605ce60f] 2024-03-28T18:23:31.014 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f49606a1e81] 2024-03-28T18:23:31.014 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f495f0bc1ca] 2024-03-28T18:23:31.014 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 16: clone() 2024-03-28T18:23:31.014 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2024-03-28T18:23:31.014 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 0> 2024-03-28T18:23:30.649+0000 7f4955d15700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:23:31.014 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: in thread 7f4955d15700 thread_name:ms_dispatch 2024-03-28T18:23:31.014 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2024-03-28T18:23:31.014 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:31.014 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f495f0c6d20] 2024-03-28T18:23:31.014 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2: gsignal() 2024-03-28T18:23:31.015 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 3: abort() 2024-03-28T18:23:31.015 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f496033f4fb] 2024-03-28T18:23:31.015 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f496033f667] 2024-03-28T18:23:31.015 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561a0123b223] 2024-03-28T18:23:31.015 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561a010b55d6] 2024-03-28T18:23:31.015 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561a010b5da1] 2024-03-28T18:23:31.015 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561a010b82d5] 2024-03-28T18:23:31.015 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561a01108233] 2024-03-28T18:23:31.015 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561a0110a197] 2024-03-28T18:23:31.015 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561a0100911b] 2024-03-28T18:23:31.016 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561a010096bc] 2024-03-28T18:23:31.016 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561a00ff31ff] 2024-03-28T18:23:31.016 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f49605d1488] 2024-03-28T18:23:31.016 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 16: (DispatchQueue::entry()+0x51f) [0x7f49605ce60f] 2024-03-28T18:23:31.016 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f49606a1e81] 2024-03-28T18:23:31.016 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f495f0bc1ca] 2024-03-28T18:23:31.016 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 19: clone() 2024-03-28T18:23:31.016 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:23:31.016 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2024-03-28T18:23:31.016 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: -1> 2024-03-28T18:23:30.647+0000 7f4955d15700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4955d15700 time 2024-03-28T18:23:30.647203+0000 2024-03-28T18:23:31.016 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:23:31.016 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2024-03-28T18:23:31.016 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:31.016 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f496033f4a1] 2024-03-28T18:23:31.016 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f496033f667] 2024-03-28T18:23:31.017 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561a0123b223] 2024-03-28T18:23:31.017 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561a010b55d6] 2024-03-28T18:23:31.017 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561a010b5da1] 2024-03-28T18:23:31.017 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561a010b82d5] 2024-03-28T18:23:31.017 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561a01108233] 2024-03-28T18:23:31.017 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561a0110a197] 2024-03-28T18:23:31.017 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561a0100911b] 2024-03-28T18:23:31.017 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561a010096bc] 2024-03-28T18:23:31.017 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561a00ff31ff] 2024-03-28T18:23:31.017 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f49605d1488] 2024-03-28T18:23:31.017 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 13: (DispatchQueue::entry()+0x51f) [0x7f49605ce60f] 2024-03-28T18:23:31.017 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f49606a1e81] 2024-03-28T18:23:31.017 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f495f0bc1ca] 2024-03-28T18:23:31.017 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 16: clone() 2024-03-28T18:23:31.018 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2024-03-28T18:23:31.018 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 0> 2024-03-28T18:23:30.649+0000 7f4955d15700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:23:31.018 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: in thread 7f4955d15700 thread_name:ms_dispatch 2024-03-28T18:23:31.018 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2024-03-28T18:23:31.018 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:31.018 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f495f0c6d20] 2024-03-28T18:23:31.018 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2: gsignal() 2024-03-28T18:23:31.018 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 3: abort() 2024-03-28T18:23:31.018 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f496033f4fb] 2024-03-28T18:23:31.018 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f496033f667] 2024-03-28T18:23:31.018 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561a0123b223] 2024-03-28T18:23:31.018 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561a010b55d6] 2024-03-28T18:23:31.019 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561a010b5da1] 2024-03-28T18:23:31.019 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561a010b82d5] 2024-03-28T18:23:31.019 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561a01108233] 2024-03-28T18:23:31.019 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561a0110a197] 2024-03-28T18:23:31.019 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561a0100911b] 2024-03-28T18:23:31.019 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561a010096bc] 2024-03-28T18:23:31.019 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561a00ff31ff] 2024-03-28T18:23:31.019 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f49605d1488] 2024-03-28T18:23:31.019 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 16: (DispatchQueue::entry()+0x51f) [0x7f49605ce60f] 2024-03-28T18:23:31.019 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f49606a1e81] 2024-03-28T18:23:31.019 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f495f0bc1ca] 2024-03-28T18:23:31.019 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 19: clone() 2024-03-28T18:23:31.019 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:23:31.019 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k[74383]: 2024-03-28T18:23:31.019 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:30 smithi059 podman[74421]: 2024-03-28 18:23:30.711542752 +0000 UTC m=+0.019469033 container died 1bacb84202758f01a5c6819835be1184eb7009fe33e08eb3682645c1d106944f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-28T18:23:31.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:30 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/2519390530,v1:172.21.15.59:6833/2519390530] up:rejoin 2024-03-28T18:23:31.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:30 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:31.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:30 smithi170 ceph-mon[36280]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-28T18:23:31.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:23:31 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:23:31.451+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:23:31.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:31 smithi037 ceph-mon[31460]: pgmap v420: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:23:31.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:31 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:23:31.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:31 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.59:6832/2519390530,v1:172.21.15.59:6833/2519390530] up:active 2024-03-28T18:23:31.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:31 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:31.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:31 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-03-28T18:23:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:31 smithi059 ceph-mon[36190]: pgmap v420: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:23:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:31 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:23:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:31 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.59:6832/2519390530,v1:172.21.15.59:6833/2519390530] up:active 2024-03-28T18:23:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:31 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:31 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-03-28T18:23:32.035 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:31 smithi059 podman[74421]: 2024-03-28 18:23:31.580084166 +0000 UTC m=+0.888010438 container remove 1bacb84202758f01a5c6819835be1184eb7009fe33e08eb3682645c1d106944f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-k, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:23:32.035 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:31 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:23:32.035 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:31 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-28T18:23:32.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:31 smithi170 ceph-mon[36280]: pgmap v420: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:23:32.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:31 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:23:32.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:31 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.59:6832/2519390530,v1:172.21.15.59:6833/2519390530] up:active 2024-03-28T18:23:32.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:31 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T18:23:32.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:31 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-03-28T18:23:34.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:33 smithi059 ceph-mon[36190]: pgmap v421: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:23:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:33 smithi170 ceph-mon[36280]: pgmap v421: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:23:34.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:33 smithi037 ceph-mon[31460]: pgmap v421: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:23:35.815 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:35 smithi037 ceph-mon[31460]: pgmap v422: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:23:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:35 smithi059 ceph-mon[36190]: pgmap v422: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:23:36.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:35 smithi170 ceph-mon[36280]: pgmap v422: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:23:37.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:37 smithi037 ceph-mon[31460]: pgmap v423: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:23:38.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:37 smithi059 ceph-mon[36190]: pgmap v423: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:23:38.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:37 smithi170 ceph-mon[36280]: pgmap v423: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:23:39.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:39 smithi037 ceph-mon[31460]: pgmap v424: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-03-28T18:23:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:39 smithi059 ceph-mon[36190]: pgmap v424: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-03-28T18:23:40.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:39 smithi170 ceph-mon[36280]: pgmap v424: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-03-28T18:23:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:40 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:23:41.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:40 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:23:41.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:40 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:23:42.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:41 smithi059 ceph-mon[36190]: pgmap v425: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-28T18:23:42.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:41 smithi059 ceph-mon[36190]: osdmap e134: 12 total, 12 up, 12 in 2024-03-28T18:23:42.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:41 smithi170 ceph-mon[36280]: pgmap v425: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-28T18:23:42.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:41 smithi170 ceph-mon[36280]: osdmap e134: 12 total, 12 up, 12 in 2024-03-28T18:23:42.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:41 smithi037 ceph-mon[31460]: pgmap v425: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-28T18:23:42.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:41 smithi037 ceph-mon[31460]: osdmap e134: 12 total, 12 up, 12 in 2024-03-28T18:23:42.535 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:42 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Scheduled restart job, restart counter is at 6. 2024-03-28T18:23:42.535 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:42 smithi059 systemd[1]: Stopped Ceph mds.k for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:23:42.535 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:42 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Start request repeated too quickly. 2024-03-28T18:23:42.535 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:42 smithi059 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-28T18:23:42.535 INFO:journalctl@ceph.mds.k.smithi059.stdout:Mar 28 18:23:42 smithi059 systemd[1]: Failed to start Ceph mds.k for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:23:43.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:42 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:23:43.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:42 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:23:43.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:42 smithi059 ceph-mon[36190]: pgmap v427: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 215 B/s wr, 6 op/s 2024-03-28T18:23:43.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:42 smithi059 ceph-mon[36190]: pgmap v428: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 273 B/s wr, 8 op/s 2024-03-28T18:23:43.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:42 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:23:43.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:42 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:23:43.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:42 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:23:43.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:42 smithi170 ceph-mon[36280]: pgmap v427: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 215 B/s wr, 6 op/s 2024-03-28T18:23:43.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:42 smithi170 ceph-mon[36280]: pgmap v428: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 273 B/s wr, 8 op/s 2024-03-28T18:23:43.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:42 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:23:43.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:42 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:23:43.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:42 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:23:43.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:42 smithi037 ceph-mon[31460]: pgmap v427: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 215 B/s wr, 6 op/s 2024-03-28T18:23:43.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:42 smithi037 ceph-mon[31460]: pgmap v428: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 273 B/s wr, 8 op/s 2024-03-28T18:23:43.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:42 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:23:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:44 smithi059 ceph-mon[36190]: pgmap v429: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 273 B/s wr, 8 op/s 2024-03-28T18:23:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:44 smithi059 ceph-mon[36190]: osdmap e135: 12 total, 12 up, 12 in 2024-03-28T18:23:45.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:44 smithi170 ceph-mon[36280]: pgmap v429: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 273 B/s wr, 8 op/s 2024-03-28T18:23:45.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:44 smithi170 ceph-mon[36280]: osdmap e135: 12 total, 12 up, 12 in 2024-03-28T18:23:45.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:44 smithi037 ceph-mon[31460]: pgmap v429: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 273 B/s wr, 8 op/s 2024-03-28T18:23:45.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:44 smithi037 ceph-mon[31460]: osdmap e135: 12 total, 12 up, 12 in 2024-03-28T18:23:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:46 smithi059 ceph-mon[36190]: pgmap v431: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 186 B/s wr, 5 op/s 2024-03-28T18:23:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:46 smithi170 ceph-mon[36280]: pgmap v431: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 186 B/s wr, 5 op/s 2024-03-28T18:23:47.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:46 smithi037 ceph-mon[31460]: pgmap v431: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 186 B/s wr, 5 op/s 2024-03-28T18:23:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:48 smithi059 ceph-mon[36190]: pgmap v432: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 284 B/s wr, 8 op/s 2024-03-28T18:23:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:48 smithi059 ceph-mon[36190]: Replacing daemon mds.k as rank 1 with standby daemon mds.g 2024-03-28T18:23:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:48 smithi059 ceph-mon[36190]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-03-28T18:23:49.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:48 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:23:49.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:48 smithi170 ceph-mon[36280]: pgmap v432: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 284 B/s wr, 8 op/s 2024-03-28T18:23:49.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:48 smithi170 ceph-mon[36280]: Replacing daemon mds.k as rank 1 with standby daemon mds.g 2024-03-28T18:23:49.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:48 smithi170 ceph-mon[36280]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-03-28T18:23:49.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:48 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:23:49.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:48 smithi037 ceph-mon[31460]: pgmap v432: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 284 B/s wr, 8 op/s 2024-03-28T18:23:49.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:48 smithi037 ceph-mon[31460]: Replacing daemon mds.k as rank 1 with standby daemon mds.g 2024-03-28T18:23:49.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:48 smithi037 ceph-mon[31460]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-03-28T18:23:49.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:48 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:23:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:49 smithi059 ceph-mon[36190]: osdmap e136: 12 total, 12 up, 12 in 2024-03-28T18:23:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:49 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:23:50.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:49 smithi170 ceph-mon[36280]: osdmap e136: 12 total, 12 up, 12 in 2024-03-28T18:23:50.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:49 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:23:50.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:49 smithi037 ceph-mon[31460]: osdmap e136: 12 total, 12 up, 12 in 2024-03-28T18:23:50.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:49 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:23:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:50 smithi059 ceph-mon[36190]: pgmap v434: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T18:23:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:50 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/2752961889,v1:172.21.15.37:6837/2752961889] up:resolve 2024-03-28T18:23:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:50 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:23:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:50 smithi170 ceph-mon[36280]: pgmap v434: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T18:23:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:50 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/2752961889,v1:172.21.15.37:6837/2752961889] up:resolve 2024-03-28T18:23:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:50 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:23:51.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:50 smithi037 ceph-mon[31460]: pgmap v434: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T18:23:51.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:50 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/2752961889,v1:172.21.15.37:6837/2752961889] up:resolve 2024-03-28T18:23:51.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:50 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:23:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:51 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:23:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:51 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/2752961889,v1:172.21.15.37:6837/2752961889] up:reconnect 2024-03-28T18:23:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:51 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:23:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:51 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0079999 2024-03-28T18:23:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:51 smithi059 ceph-mon[36190]: pgmap v435: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 127 B/s wr, 8 op/s 2024-03-28T18:23:52.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:51 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:23:52.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:51 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/2752961889,v1:172.21.15.37:6837/2752961889] up:reconnect 2024-03-28T18:23:52.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:51 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:23:52.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:51 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0079999 2024-03-28T18:23:52.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:51 smithi170 ceph-mon[36280]: pgmap v435: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 127 B/s wr, 8 op/s 2024-03-28T18:23:52.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:51 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:23:52.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:51 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/2752961889,v1:172.21.15.37:6837/2752961889] up:reconnect 2024-03-28T18:23:52.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:51 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:23:52.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:51 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0079999 2024-03-28T18:23:52.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:51 smithi037 ceph-mon[31460]: pgmap v435: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 127 B/s wr, 8 op/s 2024-03-28T18:23:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:52 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/2752961889,v1:172.21.15.37:6837/2752961889] up:rejoin 2024-03-28T18:23:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:52 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:23:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:52 smithi059 ceph-mon[36190]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-28T18:23:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:52 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/2752961889,v1:172.21.15.37:6837/2752961889] up:rejoin 2024-03-28T18:23:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:52 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:23:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:52 smithi170 ceph-mon[36280]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-28T18:23:53.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:52 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/2752961889,v1:172.21.15.37:6837/2752961889] up:rejoin 2024-03-28T18:23:53.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:52 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:23:53.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:52 smithi037 ceph-mon[31460]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-28T18:23:53.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f5a96200700 time 2024-03-28T18:23:53.020219+0000 2024-03-28T18:23:53.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:23:53.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:53.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5aa1a204a1] 2024-03-28T18:23:53.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5aa1a20667] 2024-03-28T18:23:53.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559e85197223] 2024-03-28T18:23:53.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559e850115d6] 2024-03-28T18:23:53.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559e85011da1] 2024-03-28T18:23:53.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559e850142d5] 2024-03-28T18:23:53.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559e85064233] 2024-03-28T18:23:53.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559e85066197] 2024-03-28T18:23:53.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559e84f6511b] 2024-03-28T18:23:53.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559e84f656bc] 2024-03-28T18:23:53.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559e84f4f1ff] 2024-03-28T18:23:53.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5aa1cb2488] 2024-03-28T18:23:53.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 13: (DispatchQueue::entry()+0x51f) [0x7f5aa1caf60f] 2024-03-28T18:23:53.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5aa1d82e81] 2024-03-28T18:23:53.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5aa079d1ca] 2024-03-28T18:23:53.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 16: clone() 2024-03-28T18:23:53.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2024-03-28T18:23:53.021+0000 7f5a96200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f5a96200700 time 2024-03-28T18:23:53.020219+0000 2024-03-28T18:23:53.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:23:53.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2024-03-28T18:23:53.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:53.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5aa1a204a1] 2024-03-28T18:23:53.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5aa1a20667] 2024-03-28T18:23:53.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559e85197223] 2024-03-28T18:23:53.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559e850115d6] 2024-03-28T18:23:53.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559e85011da1] 2024-03-28T18:23:53.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559e850142d5] 2024-03-28T18:23:53.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559e85064233] 2024-03-28T18:23:53.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559e85066197] 2024-03-28T18:23:53.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559e84f6511b] 2024-03-28T18:23:53.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559e84f656bc] 2024-03-28T18:23:53.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559e84f4f1ff] 2024-03-28T18:23:53.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5aa1cb2488] 2024-03-28T18:23:53.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 13: (DispatchQueue::entry()+0x51f) [0x7f5aa1caf60f] 2024-03-28T18:23:53.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5aa1d82e81] 2024-03-28T18:23:53.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5aa079d1ca] 2024-03-28T18:23:53.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 16: clone() 2024-03-28T18:23:53.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2024-03-28T18:23:53.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: *** Caught signal (Aborted) ** 2024-03-28T18:23:53.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: in thread 7f5a96200700 thread_name:ms_dispatch 2024-03-28T18:23:53.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:53.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5aa07a7d20] 2024-03-28T18:23:53.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2: gsignal() 2024-03-28T18:23:53.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 3: abort() 2024-03-28T18:23:53.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5aa1a204fb] 2024-03-28T18:23:53.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5aa1a20667] 2024-03-28T18:23:53.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559e85197223] 2024-03-28T18:23:53.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559e850115d6] 2024-03-28T18:23:53.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559e85011da1] 2024-03-28T18:23:53.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559e850142d5] 2024-03-28T18:23:53.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559e85064233] 2024-03-28T18:23:53.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559e85066197] 2024-03-28T18:23:53.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559e84f6511b] 2024-03-28T18:23:53.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559e84f656bc] 2024-03-28T18:23:53.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559e84f4f1ff] 2024-03-28T18:23:53.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5aa1cb2488] 2024-03-28T18:23:53.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 16: (DispatchQueue::entry()+0x51f) [0x7f5aa1caf60f] 2024-03-28T18:23:53.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5aa1d82e81] 2024-03-28T18:23:53.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5aa079d1ca] 2024-03-28T18:23:53.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 19: clone() 2024-03-28T18:23:53.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2024-03-28T18:23:53.024+0000 7f5a96200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:23:53.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: in thread 7f5a96200700 thread_name:ms_dispatch 2024-03-28T18:23:53.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2024-03-28T18:23:53.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:53.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5aa07a7d20] 2024-03-28T18:23:53.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2: gsignal() 2024-03-28T18:23:53.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 3: abort() 2024-03-28T18:23:53.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5aa1a204fb] 2024-03-28T18:23:53.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5aa1a20667] 2024-03-28T18:23:53.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559e85197223] 2024-03-28T18:23:53.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559e850115d6] 2024-03-28T18:23:53.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559e85011da1] 2024-03-28T18:23:53.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559e850142d5] 2024-03-28T18:23:53.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559e85064233] 2024-03-28T18:23:53.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559e85066197] 2024-03-28T18:23:53.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559e84f6511b] 2024-03-28T18:23:53.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559e84f656bc] 2024-03-28T18:23:53.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559e84f4f1ff] 2024-03-28T18:23:53.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5aa1cb2488] 2024-03-28T18:23:53.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 16: (DispatchQueue::entry()+0x51f) [0x7f5aa1caf60f] 2024-03-28T18:23:53.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5aa1d82e81] 2024-03-28T18:23:53.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5aa079d1ca] 2024-03-28T18:23:53.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 19: clone() 2024-03-28T18:23:53.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:23:53.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2024-03-28T18:23:53.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: -1> 2024-03-28T18:23:53.021+0000 7f5a96200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f5a96200700 time 2024-03-28T18:23:53.020219+0000 2024-03-28T18:23:53.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:23:53.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2024-03-28T18:23:53.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:53.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5aa1a204a1] 2024-03-28T18:23:53.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5aa1a20667] 2024-03-28T18:23:53.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559e85197223] 2024-03-28T18:23:53.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559e850115d6] 2024-03-28T18:23:53.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559e85011da1] 2024-03-28T18:23:53.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559e850142d5] 2024-03-28T18:23:53.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559e85064233] 2024-03-28T18:23:53.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559e85066197] 2024-03-28T18:23:53.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559e84f6511b] 2024-03-28T18:23:53.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559e84f656bc] 2024-03-28T18:23:53.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559e84f4f1ff] 2024-03-28T18:23:53.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5aa1cb2488] 2024-03-28T18:23:53.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 13: (DispatchQueue::entry()+0x51f) [0x7f5aa1caf60f] 2024-03-28T18:23:53.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5aa1d82e81] 2024-03-28T18:23:53.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5aa079d1ca] 2024-03-28T18:23:53.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 16: clone() 2024-03-28T18:23:53.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2024-03-28T18:23:53.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 0> 2024-03-28T18:23:53.024+0000 7f5a96200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:23:53.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: in thread 7f5a96200700 thread_name:ms_dispatch 2024-03-28T18:23:53.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2024-03-28T18:23:53.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:53.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5aa07a7d20] 2024-03-28T18:23:53.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2: gsignal() 2024-03-28T18:23:53.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 3: abort() 2024-03-28T18:23:53.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5aa1a204fb] 2024-03-28T18:23:53.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5aa1a20667] 2024-03-28T18:23:53.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559e85197223] 2024-03-28T18:23:53.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559e850115d6] 2024-03-28T18:23:53.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559e85011da1] 2024-03-28T18:23:53.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559e850142d5] 2024-03-28T18:23:53.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559e85064233] 2024-03-28T18:23:53.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559e85066197] 2024-03-28T18:23:53.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559e84f6511b] 2024-03-28T18:23:53.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559e84f656bc] 2024-03-28T18:23:53.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559e84f4f1ff] 2024-03-28T18:23:53.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5aa1cb2488] 2024-03-28T18:23:53.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 16: (DispatchQueue::entry()+0x51f) [0x7f5aa1caf60f] 2024-03-28T18:23:53.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5aa1d82e81] 2024-03-28T18:23:53.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5aa079d1ca] 2024-03-28T18:23:53.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 19: clone() 2024-03-28T18:23:53.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:23:53.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2024-03-28T18:23:53.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: -1> 2024-03-28T18:23:53.021+0000 7f5a96200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f5a96200700 time 2024-03-28T18:23:53.020219+0000 2024-03-28T18:23:53.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:23:53.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2024-03-28T18:23:53.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:53.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5aa1a204a1] 2024-03-28T18:23:53.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5aa1a20667] 2024-03-28T18:23:53.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559e85197223] 2024-03-28T18:23:53.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559e850115d6] 2024-03-28T18:23:53.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559e85011da1] 2024-03-28T18:23:53.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559e850142d5] 2024-03-28T18:23:53.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559e85064233] 2024-03-28T18:23:53.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559e85066197] 2024-03-28T18:23:53.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559e84f6511b] 2024-03-28T18:23:53.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559e84f656bc] 2024-03-28T18:23:53.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559e84f4f1ff] 2024-03-28T18:23:53.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5aa1cb2488] 2024-03-28T18:23:53.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 13: (DispatchQueue::entry()+0x51f) [0x7f5aa1caf60f] 2024-03-28T18:23:53.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5aa1d82e81] 2024-03-28T18:23:53.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5aa079d1ca] 2024-03-28T18:23:53.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 16: clone() 2024-03-28T18:23:53.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2024-03-28T18:23:53.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 0> 2024-03-28T18:23:53.024+0000 7f5a96200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:23:53.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: in thread 7f5a96200700 thread_name:ms_dispatch 2024-03-28T18:23:53.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2024-03-28T18:23:53.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:23:53.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5aa07a7d20] 2024-03-28T18:23:53.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2: gsignal() 2024-03-28T18:23:53.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 3: abort() 2024-03-28T18:23:53.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5aa1a204fb] 2024-03-28T18:23:53.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5aa1a20667] 2024-03-28T18:23:53.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559e85197223] 2024-03-28T18:23:53.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559e850115d6] 2024-03-28T18:23:53.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559e85011da1] 2024-03-28T18:23:53.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559e850142d5] 2024-03-28T18:23:53.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559e85064233] 2024-03-28T18:23:53.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559e85066197] 2024-03-28T18:23:53.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559e84f6511b] 2024-03-28T18:23:53.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559e84f656bc] 2024-03-28T18:23:53.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559e84f4f1ff] 2024-03-28T18:23:53.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5aa1cb2488] 2024-03-28T18:23:53.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 16: (DispatchQueue::entry()+0x51f) [0x7f5aa1caf60f] 2024-03-28T18:23:53.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5aa1d82e81] 2024-03-28T18:23:53.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5aa079d1ca] 2024-03-28T18:23:53.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 19: clone() 2024-03-28T18:23:53.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:23:53.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89582]: 2024-03-28T18:23:53.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:53 smithi037 podman[89759]: 2024-03-28 18:23:53.160914525 +0000 UTC m=+0.039489929 container died 8613456fd40aa7a49bf323d19a3b2a7a3679ede4f5b35aa1d4a481f17a411883 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-28T18:23:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:53 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:23:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:53 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/2752961889,v1:172.21.15.37:6837/2752961889] up:active 2024-03-28T18:23:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:53 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:23:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:53 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:23:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:53 smithi059 ceph-mon[36190]: pgmap v436: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 103 B/s wr, 6 op/s 2024-03-28T18:23:54.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:53 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:23:54.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:53 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/2752961889,v1:172.21.15.37:6837/2752961889] up:active 2024-03-28T18:23:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:53 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:23:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:53 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:23:54.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:53 smithi170 ceph-mon[36280]: pgmap v436: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 103 B/s wr, 6 op/s 2024-03-28T18:23:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:23:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/2752961889,v1:172.21.15.37:6837/2752961889] up:active 2024-03-28T18:23:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:23:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:23:54.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:53 smithi037 ceph-mon[31460]: pgmap v436: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 103 B/s wr, 6 op/s 2024-03-28T18:23:54.375 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:54 smithi037 podman[89759]: 2024-03-28 18:23:54.057752895 +0000 UTC m=+0.936328305 container remove 8613456fd40aa7a49bf323d19a3b2a7a3679ede4f5b35aa1d4a481f17a411883 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-28T18:23:54.375 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:54 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:23:54.874 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:54 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-28T18:23:54.874 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:23:54 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.898s CPU time. 2024-03-28T18:23:56.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:23:56 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:23:56.453+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:23:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:56 smithi059 ceph-mon[36190]: pgmap v437: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:23:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:56 smithi170 ceph-mon[36280]: pgmap v437: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:23:57.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:56 smithi037 ceph-mon[31460]: pgmap v437: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:23:59.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:23:58 smithi059 ceph-mon[36190]: pgmap v438: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:23:59.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:23:58 smithi170 ceph-mon[36280]: pgmap v438: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:23:59.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:23:58 smithi037 ceph-mon[31460]: pgmap v438: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:24:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:00 smithi059 ceph-mon[36190]: pgmap v439: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 187 B/s wr, 15 op/s 2024-03-28T18:24:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:00 smithi170 ceph-mon[36280]: pgmap v439: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 187 B/s wr, 15 op/s 2024-03-28T18:24:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:00 smithi037 ceph-mon[31460]: pgmap v439: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 187 B/s wr, 15 op/s 2024-03-28T18:24:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:02 smithi059 ceph-mon[36190]: pgmap v440: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 17 op/s 2024-03-28T18:24:03.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:02 smithi170 ceph-mon[36280]: pgmap v440: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 17 op/s 2024-03-28T18:24:03.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:02 smithi037 ceph-mon[31460]: pgmap v440: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 17 op/s 2024-03-28T18:24:04.915 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:04 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 2. 2024-03-28T18:24:04.915 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:04 smithi037 systemd[1]: Stopped Ceph mds.g for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:24:04.916 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:04 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.898s CPU time. 2024-03-28T18:24:04.916 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:04 smithi037 systemd[1]: Starting Ceph mds.g for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:24:05.175 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:04 smithi037 ceph-mon[31460]: pgmap v441: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 15 op/s 2024-03-28T18:24:05.176 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:05 smithi037 podman[89852]: 2024-03-28 18:24:04.922740708 +0000 UTC m=+0.035963345 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:24:05.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:04 smithi059 ceph-mon[36190]: pgmap v441: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 15 op/s 2024-03-28T18:24:05.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:04 smithi170 ceph-mon[36280]: pgmap v441: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 15 op/s 2024-03-28T18:24:05.436 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:05 smithi037 podman[89852]: 2024-03-28 18:24:05.175953525 +0000 UTC m=+0.289176169 container create fe1fe5bd318c9a475e084304e7ac612c7a61c0642021b11ceb45d944380cb20f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-28T18:24:05.817 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:05 smithi037 podman[89852]: 2024-03-28 18:24:05.458112475 +0000 UTC m=+0.571335163 container init fe1fe5bd318c9a475e084304e7ac612c7a61c0642021b11ceb45d944380cb20f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-28T18:24:05.817 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:05 smithi037 podman[89852]: 2024-03-28 18:24:05.469466397 +0000 UTC m=+0.582689071 container start fe1fe5bd318c9a475e084304e7ac612c7a61c0642021b11ceb45d944380cb20f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:24:05.817 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:05 smithi037 ceph-mds[89871]: -- 172.21.15.37:0/2135643799 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5568bc52da40 con 0x5568bc500800 2024-03-28T18:24:05.817 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:05 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: starting mds.g at 2024-03-28T18:24:05.817 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:05 smithi037 bash[89852]: fe1fe5bd318c9a475e084304e7ac612c7a61c0642021b11ceb45d944380cb20f 2024-03-28T18:24:05.817 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:05 smithi037 systemd[1]: Started Ceph mds.g for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:24:06.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:05 smithi037 ceph-mon[31460]: daemon mds.g restarted 2024-03-28T18:24:06.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:05 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:24:06.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:05 smithi037 ceph-mon[31460]: osdmap e137: 12 total, 12 up, 12 in 2024-03-28T18:24:06.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:05 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3635811332,v1:172.21.15.37:6837/3635811332] up:boot 2024-03-28T18:24:06.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:05 smithi037 ceph-mon[31460]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-28T18:24:06.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:05 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:24:06.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:05 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:24:06.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:05 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:05 smithi059 ceph-mon[36190]: daemon mds.g restarted 2024-03-28T18:24:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:05 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:24:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:05 smithi059 ceph-mon[36190]: osdmap e137: 12 total, 12 up, 12 in 2024-03-28T18:24:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:05 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3635811332,v1:172.21.15.37:6837/3635811332] up:boot 2024-03-28T18:24:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:05 smithi059 ceph-mon[36190]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-28T18:24:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:05 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:24:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:05 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:24:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:05 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:06.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:05 smithi170 ceph-mon[36280]: daemon mds.g restarted 2024-03-28T18:24:06.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:05 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:24:06.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:05 smithi170 ceph-mon[36280]: osdmap e137: 12 total, 12 up, 12 in 2024-03-28T18:24:06.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:05 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3635811332,v1:172.21.15.37:6837/3635811332] up:boot 2024-03-28T18:24:06.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:05 smithi170 ceph-mon[36280]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-28T18:24:06.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:05 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:24:06.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:05 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:24:06.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:05 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:06 smithi059 ceph-mon[36190]: pgmap v443: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T18:24:07.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:06 smithi170 ceph-mon[36280]: pgmap v443: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T18:24:07.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:06 smithi037 ceph-mon[31460]: pgmap v443: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T18:24:08.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:07 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/967056034,v1:172.21.15.37:6835/967056034] up:resolve 2024-03-28T18:24:08.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:07 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:08.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:07 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/967056034,v1:172.21.15.37:6835/967056034] up:resolve 2024-03-28T18:24:08.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:07 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:08.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:07 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/967056034,v1:172.21.15.37:6835/967056034] up:resolve 2024-03-28T18:24:08.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:07 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:08 smithi059 ceph-mon[36190]: pgmap v444: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:24:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:08 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.000999987 2024-03-28T18:24:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:08 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/967056034,v1:172.21.15.37:6835/967056034] up:reconnect 2024-03-28T18:24:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:08 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:08 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899988 2024-03-28T18:24:09.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:08 smithi170 ceph-mon[36280]: pgmap v444: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:24:09.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:08 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.000999987 2024-03-28T18:24:09.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:08 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/967056034,v1:172.21.15.37:6835/967056034] up:reconnect 2024-03-28T18:24:09.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:08 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:09.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:08 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899988 2024-03-28T18:24:09.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:08 smithi037 ceph-mon[31460]: pgmap v444: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:24:09.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:08 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.000999987 2024-03-28T18:24:09.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:08 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/967056034,v1:172.21.15.37:6835/967056034] up:reconnect 2024-03-28T18:24:09.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:08 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:09.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:08 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899988 2024-03-28T18:24:10.276 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:09 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/967056034,v1:172.21.15.37:6835/967056034] up:rejoin 2024-03-28T18:24:10.277 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:09 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:10.277 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:09 smithi037 ceph-mon[31460]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-28T18:24:10.277 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:09 smithi037 ceph-mon[31460]: pgmap v445: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:24:10.279 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3a4ba00700 time 2024-03-28T18:24:10.023384+0000 2024-03-28T18:24:10.279 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:24:10.279 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:10.279 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3a572da4a1] 2024-03-28T18:24:10.279 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3a572da667] 2024-03-28T18:24:10.279 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bdd4597223] 2024-03-28T18:24:10.279 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bdd44115d6] 2024-03-28T18:24:10.279 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bdd4411da1] 2024-03-28T18:24:10.280 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bdd44142d5] 2024-03-28T18:24:10.280 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bdd4464233] 2024-03-28T18:24:10.280 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bdd4466197] 2024-03-28T18:24:10.280 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bdd436511b] 2024-03-28T18:24:10.280 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bdd43656bc] 2024-03-28T18:24:10.280 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bdd434f1ff] 2024-03-28T18:24:10.280 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3a5756c488] 2024-03-28T18:24:10.280 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 13: (DispatchQueue::entry()+0x51f) [0x7f3a5756960f] 2024-03-28T18:24:10.280 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3a5763ce81] 2024-03-28T18:24:10.280 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3a560571ca] 2024-03-28T18:24:10.280 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 16: clone() 2024-03-28T18:24:10.280 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: *** Caught signal (Aborted) ** 2024-03-28T18:24:10.280 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: in thread 7f3a4ba00700 thread_name:ms_dispatch 2024-03-28T18:24:10.281 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2024-03-28T18:24:10.025+0000 7f3a4ba00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3a4ba00700 time 2024-03-28T18:24:10.023384+0000 2024-03-28T18:24:10.281 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:24:10.282 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2024-03-28T18:24:10.282 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:10.282 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3a572da4a1] 2024-03-28T18:24:10.282 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3a572da667] 2024-03-28T18:24:10.282 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bdd4597223] 2024-03-28T18:24:10.282 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bdd44115d6] 2024-03-28T18:24:10.282 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bdd4411da1] 2024-03-28T18:24:10.282 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bdd44142d5] 2024-03-28T18:24:10.282 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bdd4464233] 2024-03-28T18:24:10.282 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bdd4466197] 2024-03-28T18:24:10.282 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bdd436511b] 2024-03-28T18:24:10.282 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bdd43656bc] 2024-03-28T18:24:10.282 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bdd434f1ff] 2024-03-28T18:24:10.283 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3a5756c488] 2024-03-28T18:24:10.283 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 13: (DispatchQueue::entry()+0x51f) [0x7f3a5756960f] 2024-03-28T18:24:10.283 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3a5763ce81] 2024-03-28T18:24:10.283 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3a560571ca] 2024-03-28T18:24:10.283 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 16: clone() 2024-03-28T18:24:10.283 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2024-03-28T18:24:10.283 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:10.283 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3a56061d20] 2024-03-28T18:24:10.283 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2: gsignal() 2024-03-28T18:24:10.284 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 3: abort() 2024-03-28T18:24:10.284 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3a572da4fb] 2024-03-28T18:24:10.284 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3a572da667] 2024-03-28T18:24:10.284 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bdd4597223] 2024-03-28T18:24:10.284 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bdd44115d6] 2024-03-28T18:24:10.284 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bdd4411da1] 2024-03-28T18:24:10.284 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bdd44142d5] 2024-03-28T18:24:10.284 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bdd4464233] 2024-03-28T18:24:10.284 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bdd4466197] 2024-03-28T18:24:10.284 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bdd436511b] 2024-03-28T18:24:10.284 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bdd43656bc] 2024-03-28T18:24:10.284 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bdd434f1ff] 2024-03-28T18:24:10.284 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3a5756c488] 2024-03-28T18:24:10.285 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 16: (DispatchQueue::entry()+0x51f) [0x7f3a5756960f] 2024-03-28T18:24:10.285 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3a5763ce81] 2024-03-28T18:24:10.285 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3a560571ca] 2024-03-28T18:24:10.285 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 19: clone() 2024-03-28T18:24:10.285 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2024-03-28T18:24:10.028+0000 7f3a4ba00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:24:10.285 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: in thread 7f3a4ba00700 thread_name:ms_dispatch 2024-03-28T18:24:10.285 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2024-03-28T18:24:10.285 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:10.285 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3a56061d20] 2024-03-28T18:24:10.285 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2: gsignal() 2024-03-28T18:24:10.285 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 3: abort() 2024-03-28T18:24:10.285 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3a572da4fb] 2024-03-28T18:24:10.285 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3a572da667] 2024-03-28T18:24:10.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:09 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/967056034,v1:172.21.15.37:6835/967056034] up:rejoin 2024-03-28T18:24:10.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:09 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:10.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:09 smithi170 ceph-mon[36280]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-28T18:24:10.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:09 smithi170 ceph-mon[36280]: pgmap v445: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:24:10.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:09 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/967056034,v1:172.21.15.37:6835/967056034] up:rejoin 2024-03-28T18:24:10.287 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:09 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:10.287 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:09 smithi059 ceph-mon[36190]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-28T18:24:10.287 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:09 smithi059 ceph-mon[36190]: pgmap v445: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:24:10.287 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bdd4597223] 2024-03-28T18:24:10.287 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bdd44115d6] 2024-03-28T18:24:10.287 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bdd4411da1] 2024-03-28T18:24:10.287 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bdd44142d5] 2024-03-28T18:24:10.287 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bdd4464233] 2024-03-28T18:24:10.287 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bdd4466197] 2024-03-28T18:24:10.287 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bdd436511b] 2024-03-28T18:24:10.287 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bdd43656bc] 2024-03-28T18:24:10.288 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bdd434f1ff] 2024-03-28T18:24:10.288 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3a5756c488] 2024-03-28T18:24:10.288 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 16: (DispatchQueue::entry()+0x51f) [0x7f3a5756960f] 2024-03-28T18:24:10.288 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3a5763ce81] 2024-03-28T18:24:10.288 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3a560571ca] 2024-03-28T18:24:10.288 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 19: clone() 2024-03-28T18:24:10.288 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:24:10.288 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2024-03-28T18:24:10.288 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: -1> 2024-03-28T18:24:10.025+0000 7f3a4ba00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3a4ba00700 time 2024-03-28T18:24:10.023384+0000 2024-03-28T18:24:10.288 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:24:10.288 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2024-03-28T18:24:10.288 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:10.288 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3a572da4a1] 2024-03-28T18:24:10.288 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3a572da667] 2024-03-28T18:24:10.289 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bdd4597223] 2024-03-28T18:24:10.289 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bdd44115d6] 2024-03-28T18:24:10.289 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bdd4411da1] 2024-03-28T18:24:10.289 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bdd44142d5] 2024-03-28T18:24:10.289 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bdd4464233] 2024-03-28T18:24:10.289 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bdd4466197] 2024-03-28T18:24:10.289 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bdd436511b] 2024-03-28T18:24:10.289 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bdd43656bc] 2024-03-28T18:24:10.289 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bdd434f1ff] 2024-03-28T18:24:10.289 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3a5756c488] 2024-03-28T18:24:10.289 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 13: (DispatchQueue::entry()+0x51f) [0x7f3a5756960f] 2024-03-28T18:24:10.289 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3a5763ce81] 2024-03-28T18:24:10.289 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3a560571ca] 2024-03-28T18:24:10.289 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 16: clone() 2024-03-28T18:24:10.289 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2024-03-28T18:24:10.290 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 0> 2024-03-28T18:24:10.028+0000 7f3a4ba00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:24:10.290 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: in thread 7f3a4ba00700 thread_name:ms_dispatch 2024-03-28T18:24:10.290 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2024-03-28T18:24:10.290 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:10.290 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3a56061d20] 2024-03-28T18:24:10.290 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2: gsignal() 2024-03-28T18:24:10.290 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 3: abort() 2024-03-28T18:24:10.290 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3a572da4fb] 2024-03-28T18:24:10.290 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3a572da667] 2024-03-28T18:24:10.290 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bdd4597223] 2024-03-28T18:24:10.290 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bdd44115d6] 2024-03-28T18:24:10.291 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bdd4411da1] 2024-03-28T18:24:10.291 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bdd44142d5] 2024-03-28T18:24:10.291 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bdd4464233] 2024-03-28T18:24:10.291 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bdd4466197] 2024-03-28T18:24:10.291 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bdd436511b] 2024-03-28T18:24:10.291 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bdd43656bc] 2024-03-28T18:24:10.291 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bdd434f1ff] 2024-03-28T18:24:10.291 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3a5756c488] 2024-03-28T18:24:10.291 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 16: (DispatchQueue::entry()+0x51f) [0x7f3a5756960f] 2024-03-28T18:24:10.291 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3a5763ce81] 2024-03-28T18:24:10.291 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3a560571ca] 2024-03-28T18:24:10.291 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 19: clone() 2024-03-28T18:24:10.291 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:24:10.291 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2024-03-28T18:24:10.291 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: -1> 2024-03-28T18:24:10.025+0000 7f3a4ba00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3a4ba00700 time 2024-03-28T18:24:10.023384+0000 2024-03-28T18:24:10.292 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:24:10.292 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2024-03-28T18:24:10.292 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:10.292 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3a572da4a1] 2024-03-28T18:24:10.292 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3a572da667] 2024-03-28T18:24:10.292 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bdd4597223] 2024-03-28T18:24:10.292 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bdd44115d6] 2024-03-28T18:24:10.292 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bdd4411da1] 2024-03-28T18:24:10.292 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bdd44142d5] 2024-03-28T18:24:10.292 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bdd4464233] 2024-03-28T18:24:10.292 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bdd4466197] 2024-03-28T18:24:10.293 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bdd436511b] 2024-03-28T18:24:10.293 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bdd43656bc] 2024-03-28T18:24:10.293 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bdd434f1ff] 2024-03-28T18:24:10.293 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3a5756c488] 2024-03-28T18:24:10.293 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 13: (DispatchQueue::entry()+0x51f) [0x7f3a5756960f] 2024-03-28T18:24:10.293 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3a5763ce81] 2024-03-28T18:24:10.293 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3a560571ca] 2024-03-28T18:24:10.293 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 16: clone() 2024-03-28T18:24:10.293 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2024-03-28T18:24:10.293 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 0> 2024-03-28T18:24:10.028+0000 7f3a4ba00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:24:10.293 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: in thread 7f3a4ba00700 thread_name:ms_dispatch 2024-03-28T18:24:10.293 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2024-03-28T18:24:10.293 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:10.293 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3a56061d20] 2024-03-28T18:24:10.293 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2: gsignal() 2024-03-28T18:24:10.293 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 3: abort() 2024-03-28T18:24:10.294 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3a572da4fb] 2024-03-28T18:24:10.294 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3a572da667] 2024-03-28T18:24:10.294 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bdd4597223] 2024-03-28T18:24:10.294 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bdd44115d6] 2024-03-28T18:24:10.294 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bdd4411da1] 2024-03-28T18:24:10.294 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bdd44142d5] 2024-03-28T18:24:10.294 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bdd4464233] 2024-03-28T18:24:10.294 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bdd4466197] 2024-03-28T18:24:10.294 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bdd436511b] 2024-03-28T18:24:10.294 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bdd43656bc] 2024-03-28T18:24:10.294 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bdd434f1ff] 2024-03-28T18:24:10.294 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3a5756c488] 2024-03-28T18:24:10.295 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 16: (DispatchQueue::entry()+0x51f) [0x7f3a5756960f] 2024-03-28T18:24:10.295 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3a5763ce81] 2024-03-28T18:24:10.295 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3a560571ca] 2024-03-28T18:24:10.295 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 19: clone() 2024-03-28T18:24:10.295 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:24:10.295 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[69734]: 2024-03-28T18:24:10.295 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 podman[89917]: 2024-03-28 18:24:10.166816017 +0000 UTC m=+0.035185116 container died feecadfcbd8b35fa662e58858bafb839c34dc4edcfd6840159af18c7c2da48c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:24:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:10 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:24:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:10 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/967056034,v1:172.21.15.37:6835/967056034] up:active 2024-03-28T18:24:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:10 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:10 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:24:11.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:10 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:24:11.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:10 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/967056034,v1:172.21.15.37:6835/967056034] up:active 2024-03-28T18:24:11.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:10 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:11.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:10 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:24:11.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:24:11.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/967056034,v1:172.21.15.37:6835/967056034] up:active 2024-03-28T18:24:11.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:11.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:10 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:24:11.375 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 podman[89917]: 2024-03-28 18:24:10.951582883 +0000 UTC m=+0.819951998 container remove feecadfcbd8b35fa662e58858bafb839c34dc4edcfd6840159af18c7c2da48c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-28T18:24:11.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:10 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:24:11.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:24:11 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:24:11.454+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:24:11.875 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:11 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-28T18:24:11.875 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:11 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Consumed 2.300s CPU time. 2024-03-28T18:24:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:11 smithi059 ceph-mon[36190]: pgmap v446: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:24:12.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:11 smithi170 ceph-mon[36280]: pgmap v446: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:24:12.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:11 smithi037 ceph-mon[31460]: pgmap v446: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:24:15.236 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:14 smithi037 ceph-mon[31460]: pgmap v447: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:24:15.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:14 smithi059 ceph-mon[36190]: pgmap v447: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:24:15.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:14 smithi170 ceph-mon[36280]: pgmap v447: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 33 op/s 2024-03-28T18:24:17.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:16 smithi059 ceph-mon[36190]: pgmap v448: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 26 KiB/s wr, 32 op/s 2024-03-28T18:24:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:16 smithi170 ceph-mon[36280]: pgmap v448: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 26 KiB/s wr, 32 op/s 2024-03-28T18:24:17.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:16 smithi037 ceph-mon[31460]: pgmap v448: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 26 KiB/s wr, 32 op/s 2024-03-28T18:24:19.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:18 smithi059 ceph-mon[36190]: pgmap v449: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:24:19.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:18 smithi170 ceph-mon[36280]: pgmap v449: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:24:19.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:18 smithi037 ceph-mon[31460]: pgmap v449: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:24:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:20 smithi059 ceph-mon[36190]: pgmap v450: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:24:21.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:20 smithi170 ceph-mon[36280]: pgmap v450: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:24:21.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:20 smithi037 ceph-mon[31460]: pgmap v450: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:24:22.004 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:21 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 1. 2024-03-28T18:24:22.004 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:21 smithi037 systemd[1]: Stopped Ceph mds.j for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:24:22.004 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:21 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Consumed 2.300s CPU time. 2024-03-28T18:24:22.004 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:21 smithi037 systemd[1]: Starting Ceph mds.j for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:24:22.333 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:22 smithi037 podman[90010]: 2024-03-28 18:24:21.908594994 +0000 UTC m=+0.031204267 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:24:22.333 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:22 smithi037 podman[90010]: 2024-03-28 18:24:22.053143913 +0000 UTC m=+0.175753223 container create 73f39a1abe38ae77dc9d70edd84b71bc4e3e3e0c86dc81b2d8b177cf7551e8fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:24:22.625 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:22 smithi037 podman[90010]: 2024-03-28 18:24:22.332282334 +0000 UTC m=+0.454891651 container init 73f39a1abe38ae77dc9d70edd84b71bc4e3e3e0c86dc81b2d8b177cf7551e8fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-28T18:24:22.625 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:22 smithi037 podman[90010]: 2024-03-28 18:24:22.34343287 +0000 UTC m=+0.466042160 container start 73f39a1abe38ae77dc9d70edd84b71bc4e3e3e0c86dc81b2d8b177cf7551e8fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:24:22.625 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:22 smithi037 ceph-mds[90029]: -- 172.21.15.37:0/348081337 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5580056f1a40 con 0x5580056c4c00 2024-03-28T18:24:22.625 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:22 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: starting mds.j at 2024-03-28T18:24:22.625 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:22 smithi037 bash[90010]: 73f39a1abe38ae77dc9d70edd84b71bc4e3e3e0c86dc81b2d8b177cf7551e8fb 2024-03-28T18:24:22.626 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:22 smithi037 systemd[1]: Started Ceph mds.j for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:24:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:22 smithi059 ceph-mon[36190]: pgmap v451: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:24:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:22 smithi059 ceph-mon[36190]: daemon mds.j restarted 2024-03-28T18:24:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:22 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:24:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:22 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:24:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:22 smithi059 ceph-mon[36190]: osdmap e138: 12 total, 12 up, 12 in 2024-03-28T18:24:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:22 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/1824250776,v1:172.21.15.37:6835/1824250776] up:boot 2024-03-28T18:24:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:22 smithi059 ceph-mon[36190]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-28T18:24:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:22 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:24:23.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:22 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:22 smithi170 ceph-mon[36280]: pgmap v451: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:24:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:22 smithi170 ceph-mon[36280]: daemon mds.j restarted 2024-03-28T18:24:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:22 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:24:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:22 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:24:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:22 smithi170 ceph-mon[36280]: osdmap e138: 12 total, 12 up, 12 in 2024-03-28T18:24:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:22 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/1824250776,v1:172.21.15.37:6835/1824250776] up:boot 2024-03-28T18:24:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:22 smithi170 ceph-mon[36280]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-28T18:24:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:22 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:24:23.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:22 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:23.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:22 smithi037 ceph-mon[31460]: pgmap v451: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:24:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:22 smithi037 ceph-mon[31460]: daemon mds.j restarted 2024-03-28T18:24:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:22 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:24:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:22 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:24:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:22 smithi037 ceph-mon[31460]: osdmap e138: 12 total, 12 up, 12 in 2024-03-28T18:24:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:22 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/1824250776,v1:172.21.15.37:6835/1824250776] up:boot 2024-03-28T18:24:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:22 smithi037 ceph-mon[31460]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-28T18:24:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:22 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:24:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:22 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:25.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:24 smithi059 ceph-mon[36190]: pgmap v453: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:24:25.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:24 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3635811332,v1:172.21.15.37:6837/3635811332] up:resolve 2024-03-28T18:24:25.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:24 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:25.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:24 smithi170 ceph-mon[36280]: pgmap v453: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:24:25.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:24 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3635811332,v1:172.21.15.37:6837/3635811332] up:resolve 2024-03-28T18:24:25.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:24 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:25.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:24 smithi037 ceph-mon[31460]: pgmap v453: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:24:25.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:24 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3635811332,v1:172.21.15.37:6837/3635811332] up:resolve 2024-03-28T18:24:25.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:24 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:25 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:24:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:25 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3635811332,v1:172.21.15.37:6837/3635811332] up:reconnect 2024-03-28T18:24:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:25 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:25 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0119998 2024-03-28T18:24:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:25 smithi059 ceph-mon[36190]: pgmap v454: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:24:26.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:25 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:24:26.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:25 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3635811332,v1:172.21.15.37:6837/3635811332] up:reconnect 2024-03-28T18:24:26.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:25 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:26.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:25 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0119998 2024-03-28T18:24:26.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:25 smithi170 ceph-mon[36280]: pgmap v454: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:24:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:25 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:24:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:25 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3635811332,v1:172.21.15.37:6837/3635811332] up:reconnect 2024-03-28T18:24:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:25 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:25 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0119998 2024-03-28T18:24:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:25 smithi037 ceph-mon[31460]: pgmap v454: 97 pgs: 97 active+clean; 45 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:24:27.074 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:26 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3635811332,v1:172.21.15.37:6837/3635811332] up:rejoin 2024-03-28T18:24:27.075 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:26 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:27.075 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:26 smithi037 ceph-mon[31460]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-28T18:24:27.075 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fef42000700 time 2024-03-28T18:24:27.008211+0000 2024-03-28T18:24:27.075 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:24:27.075 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:27.075 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fef4d96a4a1] 2024-03-28T18:24:27.075 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fef4d96a667] 2024-03-28T18:24:27.075 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5568ba68f223] 2024-03-28T18:24:27.075 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5568ba5095d6] 2024-03-28T18:24:27.075 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5568ba509da1] 2024-03-28T18:24:27.076 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5568ba50c2d5] 2024-03-28T18:24:27.076 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5568ba55c233] 2024-03-28T18:24:27.076 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5568ba55e197] 2024-03-28T18:24:27.076 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5568ba45d11b] 2024-03-28T18:24:27.076 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5568ba45d6bc] 2024-03-28T18:24:27.076 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5568ba4471ff] 2024-03-28T18:24:27.076 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fef4dbfc488] 2024-03-28T18:24:27.076 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 13: (DispatchQueue::entry()+0x51f) [0x7fef4dbf960f] 2024-03-28T18:24:27.076 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fef4dccce81] 2024-03-28T18:24:27.076 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fef4c6e71ca] 2024-03-28T18:24:27.076 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 16: clone() 2024-03-28T18:24:27.077 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2024-03-28T18:24:27.009+0000 7fef42000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fef42000700 time 2024-03-28T18:24:27.008211+0000 2024-03-28T18:24:27.077 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:24:27.077 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2024-03-28T18:24:27.077 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:27.077 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fef4d96a4a1] 2024-03-28T18:24:27.077 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fef4d96a667] 2024-03-28T18:24:27.077 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5568ba68f223] 2024-03-28T18:24:27.077 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5568ba5095d6] 2024-03-28T18:24:27.077 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5568ba509da1] 2024-03-28T18:24:27.077 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5568ba50c2d5] 2024-03-28T18:24:27.077 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5568ba55c233] 2024-03-28T18:24:27.077 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5568ba55e197] 2024-03-28T18:24:27.077 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5568ba45d11b] 2024-03-28T18:24:27.077 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5568ba45d6bc] 2024-03-28T18:24:27.077 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5568ba4471ff] 2024-03-28T18:24:27.078 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fef4dbfc488] 2024-03-28T18:24:27.078 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 13: (DispatchQueue::entry()+0x51f) [0x7fef4dbf960f] 2024-03-28T18:24:27.078 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fef4dccce81] 2024-03-28T18:24:27.078 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fef4c6e71ca] 2024-03-28T18:24:27.078 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 16: clone() 2024-03-28T18:24:27.078 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2024-03-28T18:24:27.078 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: *** Caught signal (Aborted) ** 2024-03-28T18:24:27.078 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: in thread 7fef42000700 thread_name:ms_dispatch 2024-03-28T18:24:27.078 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:27.078 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fef4c6f1d20] 2024-03-28T18:24:27.078 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2: gsignal() 2024-03-28T18:24:27.079 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 3: abort() 2024-03-28T18:24:27.079 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fef4d96a4fb] 2024-03-28T18:24:27.079 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fef4d96a667] 2024-03-28T18:24:27.079 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5568ba68f223] 2024-03-28T18:24:27.079 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5568ba5095d6] 2024-03-28T18:24:27.079 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5568ba509da1] 2024-03-28T18:24:27.079 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5568ba50c2d5] 2024-03-28T18:24:27.079 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5568ba55c233] 2024-03-28T18:24:27.079 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5568ba55e197] 2024-03-28T18:24:27.079 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5568ba45d11b] 2024-03-28T18:24:27.079 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5568ba45d6bc] 2024-03-28T18:24:27.079 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5568ba4471ff] 2024-03-28T18:24:27.079 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fef4dbfc488] 2024-03-28T18:24:27.080 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 16: (DispatchQueue::entry()+0x51f) [0x7fef4dbf960f] 2024-03-28T18:24:27.080 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fef4dccce81] 2024-03-28T18:24:27.080 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fef4c6e71ca] 2024-03-28T18:24:27.080 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 19: clone() 2024-03-28T18:24:27.080 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2024-03-28T18:24:27.012+0000 7fef42000700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:24:27.080 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: in thread 7fef42000700 thread_name:ms_dispatch 2024-03-28T18:24:27.080 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2024-03-28T18:24:27.080 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:27.080 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fef4c6f1d20] 2024-03-28T18:24:27.080 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2: gsignal() 2024-03-28T18:24:27.080 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 3: abort() 2024-03-28T18:24:27.080 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fef4d96a4fb] 2024-03-28T18:24:27.080 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fef4d96a667] 2024-03-28T18:24:27.081 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5568ba68f223] 2024-03-28T18:24:27.081 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5568ba5095d6] 2024-03-28T18:24:27.081 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5568ba509da1] 2024-03-28T18:24:27.081 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5568ba50c2d5] 2024-03-28T18:24:27.081 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5568ba55c233] 2024-03-28T18:24:27.081 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5568ba55e197] 2024-03-28T18:24:27.081 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5568ba45d11b] 2024-03-28T18:24:27.081 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5568ba45d6bc] 2024-03-28T18:24:27.081 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5568ba4471ff] 2024-03-28T18:24:27.081 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fef4dbfc488] 2024-03-28T18:24:27.081 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 16: (DispatchQueue::entry()+0x51f) [0x7fef4dbf960f] 2024-03-28T18:24:27.081 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fef4dccce81] 2024-03-28T18:24:27.082 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fef4c6e71ca] 2024-03-28T18:24:27.082 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 19: clone() 2024-03-28T18:24:27.082 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:24:27.082 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2024-03-28T18:24:27.082 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: -1> 2024-03-28T18:24:27.009+0000 7fef42000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fef42000700 time 2024-03-28T18:24:27.008211+0000 2024-03-28T18:24:27.082 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:24:27.082 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2024-03-28T18:24:27.082 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:27.082 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fef4d96a4a1] 2024-03-28T18:24:27.082 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fef4d96a667] 2024-03-28T18:24:27.082 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5568ba68f223] 2024-03-28T18:24:27.082 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5568ba5095d6] 2024-03-28T18:24:27.082 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5568ba509da1] 2024-03-28T18:24:27.082 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5568ba50c2d5] 2024-03-28T18:24:27.082 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5568ba55c233] 2024-03-28T18:24:27.083 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5568ba55e197] 2024-03-28T18:24:27.083 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5568ba45d11b] 2024-03-28T18:24:27.083 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5568ba45d6bc] 2024-03-28T18:24:27.083 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5568ba4471ff] 2024-03-28T18:24:27.083 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fef4dbfc488] 2024-03-28T18:24:27.083 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 13: (DispatchQueue::entry()+0x51f) [0x7fef4dbf960f] 2024-03-28T18:24:27.083 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fef4dccce81] 2024-03-28T18:24:27.083 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fef4c6e71ca] 2024-03-28T18:24:27.083 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 16: clone() 2024-03-28T18:24:27.083 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2024-03-28T18:24:27.083 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 0> 2024-03-28T18:24:27.012+0000 7fef42000700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:24:27.083 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: in thread 7fef42000700 thread_name:ms_dispatch 2024-03-28T18:24:27.083 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2024-03-28T18:24:27.083 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:27.083 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fef4c6f1d20] 2024-03-28T18:24:27.083 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2: gsignal() 2024-03-28T18:24:27.084 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 3: abort() 2024-03-28T18:24:27.084 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fef4d96a4fb] 2024-03-28T18:24:27.084 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fef4d96a667] 2024-03-28T18:24:27.084 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5568ba68f223] 2024-03-28T18:24:27.084 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5568ba5095d6] 2024-03-28T18:24:27.085 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5568ba509da1] 2024-03-28T18:24:27.085 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5568ba50c2d5] 2024-03-28T18:24:27.085 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5568ba55c233] 2024-03-28T18:24:27.085 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5568ba55e197] 2024-03-28T18:24:27.085 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5568ba45d11b] 2024-03-28T18:24:27.085 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5568ba45d6bc] 2024-03-28T18:24:27.085 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5568ba4471ff] 2024-03-28T18:24:27.085 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fef4dbfc488] 2024-03-28T18:24:27.085 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 16: (DispatchQueue::entry()+0x51f) [0x7fef4dbf960f] 2024-03-28T18:24:27.085 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fef4dccce81] 2024-03-28T18:24:27.085 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fef4c6e71ca] 2024-03-28T18:24:27.085 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 19: clone() 2024-03-28T18:24:27.085 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:24:27.085 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2024-03-28T18:24:27.086 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: -1> 2024-03-28T18:24:27.009+0000 7fef42000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fef42000700 time 2024-03-28T18:24:27.008211+0000 2024-03-28T18:24:27.086 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:24:27.086 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2024-03-28T18:24:27.086 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:27.086 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fef4d96a4a1] 2024-03-28T18:24:27.086 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fef4d96a667] 2024-03-28T18:24:27.086 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5568ba68f223] 2024-03-28T18:24:27.086 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5568ba5095d6] 2024-03-28T18:24:27.086 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5568ba509da1] 2024-03-28T18:24:27.086 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5568ba50c2d5] 2024-03-28T18:24:27.086 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5568ba55c233] 2024-03-28T18:24:27.086 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5568ba55e197] 2024-03-28T18:24:27.086 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5568ba45d11b] 2024-03-28T18:24:27.086 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5568ba45d6bc] 2024-03-28T18:24:27.086 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5568ba4471ff] 2024-03-28T18:24:27.086 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fef4dbfc488] 2024-03-28T18:24:27.087 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 13: (DispatchQueue::entry()+0x51f) [0x7fef4dbf960f] 2024-03-28T18:24:27.087 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fef4dccce81] 2024-03-28T18:24:27.087 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fef4c6e71ca] 2024-03-28T18:24:27.087 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 16: clone() 2024-03-28T18:24:27.087 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2024-03-28T18:24:27.087 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 0> 2024-03-28T18:24:27.012+0000 7fef42000700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:24:27.087 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: in thread 7fef42000700 thread_name:ms_dispatch 2024-03-28T18:24:27.087 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2024-03-28T18:24:27.087 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:27.087 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fef4c6f1d20] 2024-03-28T18:24:27.087 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2: gsignal() 2024-03-28T18:24:27.087 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 3: abort() 2024-03-28T18:24:27.087 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fef4d96a4fb] 2024-03-28T18:24:27.087 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fef4d96a667] 2024-03-28T18:24:27.087 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5568ba68f223] 2024-03-28T18:24:27.087 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5568ba5095d6] 2024-03-28T18:24:27.087 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5568ba509da1] 2024-03-28T18:24:27.088 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5568ba50c2d5] 2024-03-28T18:24:27.088 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5568ba55c233] 2024-03-28T18:24:27.088 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5568ba55e197] 2024-03-28T18:24:27.088 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5568ba45d11b] 2024-03-28T18:24:27.088 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5568ba45d6bc] 2024-03-28T18:24:27.088 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5568ba4471ff] 2024-03-28T18:24:27.088 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fef4dbfc488] 2024-03-28T18:24:27.088 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 16: (DispatchQueue::entry()+0x51f) [0x7fef4dbf960f] 2024-03-28T18:24:27.088 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fef4dccce81] 2024-03-28T18:24:27.088 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fef4c6e71ca] 2024-03-28T18:24:27.088 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 19: clone() 2024-03-28T18:24:27.088 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:24:27.088 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[89866]: 2024-03-28T18:24:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:26 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3635811332,v1:172.21.15.37:6837/3635811332] up:rejoin 2024-03-28T18:24:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:26 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:26 smithi059 ceph-mon[36190]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-28T18:24:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:26 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3635811332,v1:172.21.15.37:6837/3635811332] up:rejoin 2024-03-28T18:24:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:26 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:27.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:26 smithi170 ceph-mon[36280]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-28T18:24:27.374 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 podman[90063]: 2024-03-28 18:24:27.143642751 +0000 UTC m=+0.032382363 container died fe1fe5bd318c9a475e084304e7ac612c7a61c0642021b11ceb45d944380cb20f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:24:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:27 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:24:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:27 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3635811332,v1:172.21.15.37:6837/3635811332] up:active 2024-03-28T18:24:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:27 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:27 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:24:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:27 smithi059 ceph-mon[36190]: pgmap v455: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:24:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:27 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:24:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:27 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3635811332,v1:172.21.15.37:6837/3635811332] up:active 2024-03-28T18:24:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:27 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:27 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:24:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:27 smithi170 ceph-mon[36280]: pgmap v455: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:24:28.325 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 podman[90063]: 2024-03-28 18:24:27.90299455 +0000 UTC m=+0.791734205 container remove fe1fe5bd318c9a475e084304e7ac612c7a61c0642021b11ceb45d944380cb20f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:24:28.325 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:27 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:24:28.325 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:24:28.325 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3635811332,v1:172.21.15.37:6837/3635811332] up:active 2024-03-28T18:24:28.325 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:28.326 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:24:28.326 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:27 smithi037 ceph-mon[31460]: pgmap v455: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:24:28.624 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:28 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-28T18:24:28.624 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:28 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.743s CPU time. 2024-03-28T18:24:31.108 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:30 smithi037 ceph-mon[31460]: pgmap v456: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:24:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:30 smithi059 ceph-mon[36190]: pgmap v456: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:24:31.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:30 smithi170 ceph-mon[36280]: pgmap v456: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:24:31.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:24:31 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:24:31.456+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:24:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:32 smithi059 ceph-mon[36190]: pgmap v457: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:24:33.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:32 smithi170 ceph-mon[36280]: pgmap v457: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:24:33.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:32 smithi037 ceph-mon[31460]: pgmap v457: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:24:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:34 smithi059 ceph-mon[36190]: pgmap v458: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-03-28T18:24:35.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:34 smithi170 ceph-mon[36280]: pgmap v458: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-03-28T18:24:35.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:34 smithi037 ceph-mon[31460]: pgmap v458: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-03-28T18:24:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:36 smithi059 ceph-mon[36190]: pgmap v459: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-28T18:24:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:36 smithi059 ceph-mon[36190]: osdmap e139: 12 total, 12 up, 12 in 2024-03-28T18:24:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:36 smithi170 ceph-mon[36280]: pgmap v459: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-28T18:24:37.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:36 smithi170 ceph-mon[36280]: osdmap e139: 12 total, 12 up, 12 in 2024-03-28T18:24:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:36 smithi037 ceph-mon[31460]: pgmap v459: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-28T18:24:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:36 smithi037 ceph-mon[31460]: osdmap e139: 12 total, 12 up, 12 in 2024-03-28T18:24:38.748 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:38 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 3. 2024-03-28T18:24:38.748 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:38 smithi037 systemd[1]: Stopped Ceph mds.g for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:24:38.748 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:38 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.743s CPU time. 2024-03-28T18:24:38.748 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:38 smithi037 systemd[1]: Starting Ceph mds.g for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:24:39.059 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:38 smithi037 ceph-mon[31460]: pgmap v461: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:24:39.059 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:38 smithi037 podman[90158]: 2024-03-28 18:24:38.652783866 +0000 UTC m=+0.034270593 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:24:39.059 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:38 smithi037 podman[90158]: 2024-03-28 18:24:38.795623001 +0000 UTC m=+0.177109756 container create 7e8a59b31ded63f821c5c8bb89dc45669de0e99ad3571986aa97ba24740d52b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-28T18:24:39.059 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:39 smithi037 podman[90158]: 2024-03-28 18:24:39.058552961 +0000 UTC m=+0.440039719 container init 7e8a59b31ded63f821c5c8bb89dc45669de0e99ad3571986aa97ba24740d52b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:24:39.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:38 smithi059 ceph-mon[36190]: pgmap v461: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:24:39.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:38 smithi170 ceph-mon[36280]: pgmap v461: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:24:39.375 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:39 smithi037 podman[90158]: 2024-03-28 18:24:39.067534888 +0000 UTC m=+0.449021609 container start 7e8a59b31ded63f821c5c8bb89dc45669de0e99ad3571986aa97ba24740d52b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-28T18:24:39.375 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:39 smithi037 ceph-mds[90177]: -- 172.21.15.37:0/3354618945 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557d7f321a40 con 0x557d7f2f4800 2024-03-28T18:24:39.375 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:39 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: starting mds.g at 2024-03-28T18:24:39.375 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:39 smithi037 bash[90158]: 7e8a59b31ded63f821c5c8bb89dc45669de0e99ad3571986aa97ba24740d52b0 2024-03-28T18:24:39.375 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:24:39 smithi037 systemd[1]: Started Ceph mds.g for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:24:40.277 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:39 smithi037 ceph-mon[31460]: osdmap e140: 12 total, 12 up, 12 in 2024-03-28T18:24:40.278 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:39 smithi037 ceph-mon[31460]: daemon mds.g restarted 2024-03-28T18:24:40.278 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:39 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:24:40.278 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:39 smithi037 ceph-mon[31460]: osdmap e141: 12 total, 12 up, 12 in 2024-03-28T18:24:40.278 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:39 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:24:40.278 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:39 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3149561600,v1:172.21.15.37:6837/3149561600] up:boot 2024-03-28T18:24:40.278 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:39 smithi037 ceph-mon[31460]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-28T18:24:40.278 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:39 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:24:40.278 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:39 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:40.278 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:39 smithi037 ceph-mon[31460]: pgmap v464: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:24:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:39 smithi059 ceph-mon[36190]: osdmap e140: 12 total, 12 up, 12 in 2024-03-28T18:24:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:39 smithi059 ceph-mon[36190]: daemon mds.g restarted 2024-03-28T18:24:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:39 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:24:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:39 smithi059 ceph-mon[36190]: osdmap e141: 12 total, 12 up, 12 in 2024-03-28T18:24:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:39 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:24:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:39 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3149561600,v1:172.21.15.37:6837/3149561600] up:boot 2024-03-28T18:24:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:39 smithi059 ceph-mon[36190]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-28T18:24:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:39 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:24:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:39 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:39 smithi059 ceph-mon[36190]: pgmap v464: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:24:40.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:39 smithi170 ceph-mon[36280]: osdmap e140: 12 total, 12 up, 12 in 2024-03-28T18:24:40.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:39 smithi170 ceph-mon[36280]: daemon mds.g restarted 2024-03-28T18:24:40.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:39 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:24:40.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:39 smithi170 ceph-mon[36280]: osdmap e141: 12 total, 12 up, 12 in 2024-03-28T18:24:40.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:39 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:24:40.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:39 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3149561600,v1:172.21.15.37:6837/3149561600] up:boot 2024-03-28T18:24:40.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:39 smithi170 ceph-mon[36280]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-28T18:24:40.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:39 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:24:40.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:39 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:40.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:39 smithi170 ceph-mon[36280]: pgmap v464: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:24:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:41 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/1824250776,v1:172.21.15.37:6835/1824250776] up:resolve 2024-03-28T18:24:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:41 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:41 smithi059 ceph-mon[36190]: pgmap v465: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-28T18:24:42.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:41 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/1824250776,v1:172.21.15.37:6835/1824250776] up:resolve 2024-03-28T18:24:42.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:41 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:42.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:41 smithi170 ceph-mon[36280]: pgmap v465: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-28T18:24:42.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:41 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/1824250776,v1:172.21.15.37:6835/1824250776] up:resolve 2024-03-28T18:24:42.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:41 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:42.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:41 smithi037 ceph-mon[31460]: pgmap v465: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-28T18:24:43.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:42 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:24:43.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:42 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/1824250776,v1:172.21.15.37:6835/1824250776] up:reconnect 2024-03-28T18:24:43.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:42 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:43.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:42 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0119998 2024-03-28T18:24:43.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:42 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:24:43.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:42 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:24:43.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:42 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/1824250776,v1:172.21.15.37:6835/1824250776] up:reconnect 2024-03-28T18:24:43.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:42 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:43.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:42 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0119998 2024-03-28T18:24:43.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:42 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:24:43.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:42 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:24:43.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:42 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/1824250776,v1:172.21.15.37:6835/1824250776] up:reconnect 2024-03-28T18:24:43.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:42 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:43.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:42 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0119998 2024-03-28T18:24:43.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:42 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:24:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:43 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/1824250776,v1:172.21.15.37:6835/1824250776] up:rejoin 2024-03-28T18:24:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:43 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:43 smithi059 ceph-mon[36190]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-28T18:24:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:43 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:24:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:43 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:24:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:43 smithi059 ceph-mon[36190]: pgmap v466: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 272 B/s wr, 8 op/s 2024-03-28T18:24:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:43 smithi059 ceph-mon[36190]: pgmap v467: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 183 B/s wr, 6 op/s 2024-03-28T18:24:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:43 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:24:44.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:43 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:24:44.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:43 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/1824250776,v1:172.21.15.37:6835/1824250776] up:active 2024-03-28T18:24:44.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:43 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:24:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:43 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/1824250776,v1:172.21.15.37:6835/1824250776] up:rejoin 2024-03-28T18:24:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:43 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:43 smithi170 ceph-mon[36280]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-28T18:24:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:43 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:24:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:43 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:24:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:43 smithi170 ceph-mon[36280]: pgmap v466: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 272 B/s wr, 8 op/s 2024-03-28T18:24:44.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:43 smithi170 ceph-mon[36280]: pgmap v467: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 183 B/s wr, 6 op/s 2024-03-28T18:24:44.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:43 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:24:44.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:43 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:24:44.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:43 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/1824250776,v1:172.21.15.37:6835/1824250776] up:active 2024-03-28T18:24:44.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:43 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:24:44.375 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc5f8200700 time 2024-03-28T18:24:43.996925+0000 2024-03-28T18:24:44.375 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:24:44.375 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:44.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc603a424a1] 2024-03-28T18:24:44.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc603a42667] 2024-03-28T18:24:44.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558003cab223] 2024-03-28T18:24:44.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558003b255d6] 2024-03-28T18:24:44.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558003b25da1] 2024-03-28T18:24:44.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558003b282d5] 2024-03-28T18:24:44.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558003b78233] 2024-03-28T18:24:44.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558003b7a197] 2024-03-28T18:24:44.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558003a7911b] 2024-03-28T18:24:44.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558003a796bc] 2024-03-28T18:24:44.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558003a631ff] 2024-03-28T18:24:44.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc603cd4488] 2024-03-28T18:24:44.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 13: (DispatchQueue::entry()+0x51f) [0x7fc603cd160f] 2024-03-28T18:24:44.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc603da4e81] 2024-03-28T18:24:44.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc6027bf1ca] 2024-03-28T18:24:44.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 16: clone() 2024-03-28T18:24:44.377 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/1824250776,v1:172.21.15.37:6835/1824250776] up:rejoin 2024-03-28T18:24:44.377 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:44.377 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-mon[31460]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-28T18:24:44.377 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:24:44.377 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:24:44.377 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-mon[31460]: pgmap v466: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 272 B/s wr, 8 op/s 2024-03-28T18:24:44.377 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-mon[31460]: pgmap v467: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 183 B/s wr, 6 op/s 2024-03-28T18:24:44.377 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:24:44.377 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:24:44.377 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/1824250776,v1:172.21.15.37:6835/1824250776] up:active 2024-03-28T18:24:44.378 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:24:44.378 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2024-03-28T18:24:43.998+0000 7fc5f8200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc5f8200700 time 2024-03-28T18:24:43.996925+0000 2024-03-28T18:24:44.378 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:24:44.378 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2024-03-28T18:24:44.378 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:44.379 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc603a424a1] 2024-03-28T18:24:44.379 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc603a42667] 2024-03-28T18:24:44.379 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558003cab223] 2024-03-28T18:24:44.379 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558003b255d6] 2024-03-28T18:24:44.379 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558003b25da1] 2024-03-28T18:24:44.379 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558003b282d5] 2024-03-28T18:24:44.379 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558003b78233] 2024-03-28T18:24:44.379 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558003b7a197] 2024-03-28T18:24:44.379 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558003a7911b] 2024-03-28T18:24:44.379 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558003a796bc] 2024-03-28T18:24:44.379 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558003a631ff] 2024-03-28T18:24:44.379 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc603cd4488] 2024-03-28T18:24:44.379 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 13: (DispatchQueue::entry()+0x51f) [0x7fc603cd160f] 2024-03-28T18:24:44.379 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc603da4e81] 2024-03-28T18:24:44.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc6027bf1ca] 2024-03-28T18:24:44.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 16: clone() 2024-03-28T18:24:44.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2024-03-28T18:24:44.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: *** Caught signal (Aborted) ** 2024-03-28T18:24:44.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:43 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: in thread 7fc5f8200700 thread_name:ms_dispatch 2024-03-28T18:24:44.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:44.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc6027c9d20] 2024-03-28T18:24:44.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2: gsignal() 2024-03-28T18:24:44.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 3: abort() 2024-03-28T18:24:44.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc603a424fb] 2024-03-28T18:24:44.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc603a42667] 2024-03-28T18:24:44.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558003cab223] 2024-03-28T18:24:44.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558003b255d6] 2024-03-28T18:24:44.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558003b25da1] 2024-03-28T18:24:44.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558003b282d5] 2024-03-28T18:24:44.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558003b78233] 2024-03-28T18:24:44.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558003b7a197] 2024-03-28T18:24:44.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558003a7911b] 2024-03-28T18:24:44.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558003a796bc] 2024-03-28T18:24:44.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558003a631ff] 2024-03-28T18:24:44.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc603cd4488] 2024-03-28T18:24:44.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 16: (DispatchQueue::entry()+0x51f) [0x7fc603cd160f] 2024-03-28T18:24:44.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc603da4e81] 2024-03-28T18:24:44.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc6027bf1ca] 2024-03-28T18:24:44.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 19: clone() 2024-03-28T18:24:44.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2024-03-28T18:24:44.001+0000 7fc5f8200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:24:44.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: in thread 7fc5f8200700 thread_name:ms_dispatch 2024-03-28T18:24:44.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2024-03-28T18:24:44.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:44.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc6027c9d20] 2024-03-28T18:24:44.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2: gsignal() 2024-03-28T18:24:44.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 3: abort() 2024-03-28T18:24:44.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc603a424fb] 2024-03-28T18:24:44.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc603a42667] 2024-03-28T18:24:44.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558003cab223] 2024-03-28T18:24:44.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558003b255d6] 2024-03-28T18:24:44.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558003b25da1] 2024-03-28T18:24:44.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558003b282d5] 2024-03-28T18:24:44.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558003b78233] 2024-03-28T18:24:44.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558003b7a197] 2024-03-28T18:24:44.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558003a7911b] 2024-03-28T18:24:44.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558003a796bc] 2024-03-28T18:24:44.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558003a631ff] 2024-03-28T18:24:44.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc603cd4488] 2024-03-28T18:24:44.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 16: (DispatchQueue::entry()+0x51f) [0x7fc603cd160f] 2024-03-28T18:24:44.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc603da4e81] 2024-03-28T18:24:44.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc6027bf1ca] 2024-03-28T18:24:44.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 19: clone() 2024-03-28T18:24:44.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:24:44.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2024-03-28T18:24:44.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: -1> 2024-03-28T18:24:43.998+0000 7fc5f8200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc5f8200700 time 2024-03-28T18:24:43.996925+0000 2024-03-28T18:24:44.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:24:44.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2024-03-28T18:24:44.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:44.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc603a424a1] 2024-03-28T18:24:44.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc603a42667] 2024-03-28T18:24:44.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558003cab223] 2024-03-28T18:24:44.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558003b255d6] 2024-03-28T18:24:44.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558003b25da1] 2024-03-28T18:24:44.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558003b282d5] 2024-03-28T18:24:44.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558003b78233] 2024-03-28T18:24:44.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558003b7a197] 2024-03-28T18:24:44.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558003a7911b] 2024-03-28T18:24:44.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558003a796bc] 2024-03-28T18:24:44.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558003a631ff] 2024-03-28T18:24:44.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc603cd4488] 2024-03-28T18:24:44.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 13: (DispatchQueue::entry()+0x51f) [0x7fc603cd160f] 2024-03-28T18:24:44.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc603da4e81] 2024-03-28T18:24:44.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc6027bf1ca] 2024-03-28T18:24:44.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 16: clone() 2024-03-28T18:24:44.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2024-03-28T18:24:44.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 0> 2024-03-28T18:24:44.001+0000 7fc5f8200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:24:44.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: in thread 7fc5f8200700 thread_name:ms_dispatch 2024-03-28T18:24:44.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2024-03-28T18:24:44.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:44.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc6027c9d20] 2024-03-28T18:24:44.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2: gsignal() 2024-03-28T18:24:44.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 3: abort() 2024-03-28T18:24:44.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc603a424fb] 2024-03-28T18:24:44.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc603a42667] 2024-03-28T18:24:44.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558003cab223] 2024-03-28T18:24:44.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558003b255d6] 2024-03-28T18:24:44.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558003b25da1] 2024-03-28T18:24:44.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558003b282d5] 2024-03-28T18:24:44.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558003b78233] 2024-03-28T18:24:44.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558003b7a197] 2024-03-28T18:24:44.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558003a7911b] 2024-03-28T18:24:44.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558003a796bc] 2024-03-28T18:24:44.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558003a631ff] 2024-03-28T18:24:44.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc603cd4488] 2024-03-28T18:24:44.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 16: (DispatchQueue::entry()+0x51f) [0x7fc603cd160f] 2024-03-28T18:24:44.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc603da4e81] 2024-03-28T18:24:44.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc6027bf1ca] 2024-03-28T18:24:44.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 19: clone() 2024-03-28T18:24:44.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:24:44.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2024-03-28T18:24:44.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: -1> 2024-03-28T18:24:43.998+0000 7fc5f8200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc5f8200700 time 2024-03-28T18:24:43.996925+0000 2024-03-28T18:24:44.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:24:44.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2024-03-28T18:24:44.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:44.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc603a424a1] 2024-03-28T18:24:44.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc603a42667] 2024-03-28T18:24:44.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558003cab223] 2024-03-28T18:24:44.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558003b255d6] 2024-03-28T18:24:44.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558003b25da1] 2024-03-28T18:24:44.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558003b282d5] 2024-03-28T18:24:44.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558003b78233] 2024-03-28T18:24:44.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558003b7a197] 2024-03-28T18:24:44.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558003a7911b] 2024-03-28T18:24:44.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558003a796bc] 2024-03-28T18:24:44.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558003a631ff] 2024-03-28T18:24:44.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc603cd4488] 2024-03-28T18:24:44.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 13: (DispatchQueue::entry()+0x51f) [0x7fc603cd160f] 2024-03-28T18:24:44.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc603da4e81] 2024-03-28T18:24:44.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc6027bf1ca] 2024-03-28T18:24:44.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 16: clone() 2024-03-28T18:24:44.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2024-03-28T18:24:44.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 0> 2024-03-28T18:24:44.001+0000 7fc5f8200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:24:44.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: in thread 7fc5f8200700 thread_name:ms_dispatch 2024-03-28T18:24:44.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2024-03-28T18:24:44.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:24:44.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc6027c9d20] 2024-03-28T18:24:44.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2: gsignal() 2024-03-28T18:24:44.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 3: abort() 2024-03-28T18:24:44.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc603a424fb] 2024-03-28T18:24:44.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc603a42667] 2024-03-28T18:24:44.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558003cab223] 2024-03-28T18:24:44.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558003b255d6] 2024-03-28T18:24:44.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558003b25da1] 2024-03-28T18:24:44.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558003b282d5] 2024-03-28T18:24:44.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558003b78233] 2024-03-28T18:24:44.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558003b7a197] 2024-03-28T18:24:44.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558003a7911b] 2024-03-28T18:24:44.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558003a796bc] 2024-03-28T18:24:44.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558003a631ff] 2024-03-28T18:24:44.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc603cd4488] 2024-03-28T18:24:44.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 16: (DispatchQueue::entry()+0x51f) [0x7fc603cd160f] 2024-03-28T18:24:44.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc603da4e81] 2024-03-28T18:24:44.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc6027bf1ca] 2024-03-28T18:24:44.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 19: clone() 2024-03-28T18:24:44.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:24:44.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90025]: 2024-03-28T18:24:44.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 podman[90342]: 2024-03-28 18:24:44.118474803 +0000 UTC m=+0.033028253 container died 73f39a1abe38ae77dc9d70edd84b71bc4e3e3e0c86dc81b2d8b177cf7551e8fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:24:45.125 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 podman[90342]: 2024-03-28 18:24:44.863340473 +0000 UTC m=+0.777893960 container remove 73f39a1abe38ae77dc9d70edd84b71bc4e3e3e0c86dc81b2d8b177cf7551e8fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-28T18:24:45.125 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:44 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:24:45.625 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:45 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-28T18:24:45.625 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:45 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Consumed 1.773s CPU time. 2024-03-28T18:24:46.716 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:46 smithi037 ceph-mon[31460]: pgmap v468: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 157 B/s wr, 5 op/s 2024-03-28T18:24:46.717 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:24:46 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:24:46.457+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:24:46.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:46 smithi059 ceph-mon[36190]: pgmap v468: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 157 B/s wr, 5 op/s 2024-03-28T18:24:46.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:46 smithi170 ceph-mon[36280]: pgmap v468: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 157 B/s wr, 5 op/s 2024-03-28T18:24:48.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:48 smithi059 ceph-mon[36190]: pgmap v469: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 369 B/s wr, 20 op/s 2024-03-28T18:24:48.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:48 smithi170 ceph-mon[36280]: pgmap v469: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 369 B/s wr, 20 op/s 2024-03-28T18:24:48.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:48 smithi037 ceph-mon[31460]: pgmap v469: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 369 B/s wr, 20 op/s 2024-03-28T18:24:50.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:50 smithi059 ceph-mon[36190]: pgmap v470: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 320 B/s wr, 17 op/s 2024-03-28T18:24:50.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:50 smithi170 ceph-mon[36280]: pgmap v470: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 320 B/s wr, 17 op/s 2024-03-28T18:24:50.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:50 smithi037 ceph-mon[31460]: pgmap v470: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 320 B/s wr, 17 op/s 2024-03-28T18:24:52.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:52 smithi059 ceph-mon[36190]: pgmap v471: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 213 B/s wr, 14 op/s 2024-03-28T18:24:52.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:52 smithi170 ceph-mon[36280]: pgmap v471: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 213 B/s wr, 14 op/s 2024-03-28T18:24:52.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:52 smithi037 ceph-mon[31460]: pgmap v471: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 213 B/s wr, 14 op/s 2024-03-28T18:24:54.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:54 smithi059 ceph-mon[36190]: pgmap v472: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:24:54.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:54 smithi170 ceph-mon[36280]: pgmap v472: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:24:54.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:54 smithi037 ceph-mon[31460]: pgmap v472: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:24:55.781 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:55 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 2. 2024-03-28T18:24:55.781 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:55 smithi037 systemd[1]: Stopped Ceph mds.j for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:24:55.781 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:55 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Consumed 1.773s CPU time. 2024-03-28T18:24:55.781 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:55 smithi037 systemd[1]: Starting Ceph mds.j for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:24:56.125 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:55 smithi037 podman[90434]: 2024-03-28 18:24:55.689827643 +0000 UTC m=+0.041872523 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:24:56.125 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:55 smithi037 podman[90434]: 2024-03-28 18:24:55.989298619 +0000 UTC m=+0.341343502 container create bba2cae7a408b54e438692c77084a1e4921ca4ac50caffe19345b95d99d4e121 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-28T18:24:56.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:56 smithi037 ceph-mon[31460]: pgmap v473: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:24:56.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:56 smithi037 ceph-mon[31460]: daemon mds.j restarted 2024-03-28T18:24:56.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:56 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:24:56.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:56 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:24:56.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:56 smithi037 ceph-mon[31460]: osdmap e142: 12 total, 12 up, 12 in 2024-03-28T18:24:56.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:56 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/961897107,v1:172.21.15.37:6835/961897107] up:boot 2024-03-28T18:24:56.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:56 smithi037 ceph-mon[31460]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-28T18:24:56.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:56 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:24:56.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:56 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:56.626 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:56 smithi037 podman[90434]: 2024-03-28 18:24:56.285444667 +0000 UTC m=+0.637489587 container init bba2cae7a408b54e438692c77084a1e4921ca4ac50caffe19345b95d99d4e121 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-28T18:24:56.626 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:56 smithi037 podman[90434]: 2024-03-28 18:24:56.297852213 +0000 UTC m=+0.649897141 container start bba2cae7a408b54e438692c77084a1e4921ca4ac50caffe19345b95d99d4e121 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-28T18:24:56.626 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:56 smithi037 ceph-mds[90453]: -- 172.21.15.37:0/199408533 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5640d4929a40 con 0x5640d48fd000 2024-03-28T18:24:56.627 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:56 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: starting mds.j at 2024-03-28T18:24:56.627 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:56 smithi037 bash[90434]: bba2cae7a408b54e438692c77084a1e4921ca4ac50caffe19345b95d99d4e121 2024-03-28T18:24:56.627 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:24:56 smithi037 systemd[1]: Started Ceph mds.j for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:24:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:56 smithi059 ceph-mon[36190]: pgmap v473: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:24:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:56 smithi059 ceph-mon[36190]: daemon mds.j restarted 2024-03-28T18:24:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:56 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:24:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:56 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:24:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:56 smithi059 ceph-mon[36190]: osdmap e142: 12 total, 12 up, 12 in 2024-03-28T18:24:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:56 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/961897107,v1:172.21.15.37:6835/961897107] up:boot 2024-03-28T18:24:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:56 smithi059 ceph-mon[36190]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-28T18:24:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:56 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:24:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:56 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:56 smithi170 ceph-mon[36280]: pgmap v473: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:24:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:56 smithi170 ceph-mon[36280]: daemon mds.j restarted 2024-03-28T18:24:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:56 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:24:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:56 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:24:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:56 smithi170 ceph-mon[36280]: osdmap e142: 12 total, 12 up, 12 in 2024-03-28T18:24:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:56 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/961897107,v1:172.21.15.37:6835/961897107] up:boot 2024-03-28T18:24:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:56 smithi170 ceph-mon[36280]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-28T18:24:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:56 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:24:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:56 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:58.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:58 smithi037 ceph-mon[31460]: pgmap v475: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:24:58.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:58 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3149561600,v1:172.21.15.37:6837/3149561600] up:resolve 2024-03-28T18:24:58.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:58 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:58 smithi059 ceph-mon[36190]: pgmap v475: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:24:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:58 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3149561600,v1:172.21.15.37:6837/3149561600] up:resolve 2024-03-28T18:24:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:58 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:59.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:58 smithi170 ceph-mon[36280]: pgmap v475: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:24:59.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:58 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3149561600,v1:172.21.15.37:6837/3149561600] up:resolve 2024-03-28T18:24:59.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:58 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:59.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:59 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00399995 2024-03-28T18:24:59.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:59 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3149561600,v1:172.21.15.37:6837/3149561600] up:reconnect 2024-03-28T18:24:59.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:59 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:24:59.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:24:59 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999987 2024-03-28T18:25:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:59 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00399995 2024-03-28T18:25:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:59 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3149561600,v1:172.21.15.37:6837/3149561600] up:reconnect 2024-03-28T18:25:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:59 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:24:59 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999987 2024-03-28T18:25:00.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:59 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00399995 2024-03-28T18:25:00.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:59 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3149561600,v1:172.21.15.37:6837/3149561600] up:reconnect 2024-03-28T18:25:00.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:59 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:00.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:24:59 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999987 2024-03-28T18:25:00.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-mon[31460]: pgmap v476: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:25:00.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3149561600,v1:172.21.15.37:6837/3149561600] up:rejoin 2024-03-28T18:25:00.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:00.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-mon[31460]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-28T18:25:00.875 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0e3f200700 time 2024-03-28T18:25:00.617545+0000 2024-03-28T18:25:00.876 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:25:00.876 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:00.876 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0e4aad24a1] 2024-03-28T18:25:00.876 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0e4aad2667] 2024-03-28T18:25:00.876 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557d7da2b223] 2024-03-28T18:25:00.876 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557d7d8a55d6] 2024-03-28T18:25:00.876 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557d7d8a5da1] 2024-03-28T18:25:00.876 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557d7d8a82d5] 2024-03-28T18:25:00.876 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557d7d8f8233] 2024-03-28T18:25:00.876 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557d7d8fa197] 2024-03-28T18:25:00.876 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557d7d7f911b] 2024-03-28T18:25:00.876 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557d7d7f96bc] 2024-03-28T18:25:00.876 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557d7d7e31ff] 2024-03-28T18:25:00.876 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0e4ad64488] 2024-03-28T18:25:00.877 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 13: (DispatchQueue::entry()+0x51f) [0x7f0e4ad6160f] 2024-03-28T18:25:00.877 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0e4ae34e81] 2024-03-28T18:25:00.877 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0e4984f1ca] 2024-03-28T18:25:00.877 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 16: clone() 2024-03-28T18:25:00.877 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2024-03-28T18:25:00.617+0000 7f0e3f200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0e3f200700 time 2024-03-28T18:25:00.617545+0000 2024-03-28T18:25:00.877 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:25:00.877 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2024-03-28T18:25:00.878 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:00.878 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0e4aad24a1] 2024-03-28T18:25:00.878 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0e4aad2667] 2024-03-28T18:25:00.878 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557d7da2b223] 2024-03-28T18:25:00.878 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557d7d8a55d6] 2024-03-28T18:25:00.878 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557d7d8a5da1] 2024-03-28T18:25:00.878 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557d7d8a82d5] 2024-03-28T18:25:00.878 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557d7d8f8233] 2024-03-28T18:25:00.878 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557d7d8fa197] 2024-03-28T18:25:00.878 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557d7d7f911b] 2024-03-28T18:25:00.878 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557d7d7f96bc] 2024-03-28T18:25:00.878 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557d7d7e31ff] 2024-03-28T18:25:00.879 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0e4ad64488] 2024-03-28T18:25:00.879 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 13: (DispatchQueue::entry()+0x51f) [0x7f0e4ad6160f] 2024-03-28T18:25:00.879 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0e4ae34e81] 2024-03-28T18:25:00.879 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0e4984f1ca] 2024-03-28T18:25:00.879 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 16: clone() 2024-03-28T18:25:00.879 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2024-03-28T18:25:00.879 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: *** Caught signal (Aborted) ** 2024-03-28T18:25:00.879 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: in thread 7f0e3f200700 thread_name:ms_dispatch 2024-03-28T18:25:00.879 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:00.879 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0e49859d20] 2024-03-28T18:25:00.879 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2: gsignal() 2024-03-28T18:25:00.879 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 3: abort() 2024-03-28T18:25:00.879 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0e4aad24fb] 2024-03-28T18:25:00.879 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0e4aad2667] 2024-03-28T18:25:00.880 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557d7da2b223] 2024-03-28T18:25:00.880 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557d7d8a55d6] 2024-03-28T18:25:00.880 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557d7d8a5da1] 2024-03-28T18:25:00.880 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557d7d8a82d5] 2024-03-28T18:25:00.880 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557d7d8f8233] 2024-03-28T18:25:00.880 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557d7d8fa197] 2024-03-28T18:25:00.880 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557d7d7f911b] 2024-03-28T18:25:00.880 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557d7d7f96bc] 2024-03-28T18:25:00.880 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557d7d7e31ff] 2024-03-28T18:25:00.880 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0e4ad64488] 2024-03-28T18:25:00.880 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 16: (DispatchQueue::entry()+0x51f) [0x7f0e4ad6160f] 2024-03-28T18:25:00.880 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0e4ae34e81] 2024-03-28T18:25:00.880 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0e4984f1ca] 2024-03-28T18:25:00.880 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 19: clone() 2024-03-28T18:25:00.881 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2024-03-28T18:25:00.620+0000 7f0e3f200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:25:00.881 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: in thread 7f0e3f200700 thread_name:ms_dispatch 2024-03-28T18:25:00.881 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2024-03-28T18:25:00.881 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:00.881 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0e49859d20] 2024-03-28T18:25:00.881 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2: gsignal() 2024-03-28T18:25:00.881 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 3: abort() 2024-03-28T18:25:00.881 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0e4aad24fb] 2024-03-28T18:25:00.881 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0e4aad2667] 2024-03-28T18:25:00.882 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557d7da2b223] 2024-03-28T18:25:00.882 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557d7d8a55d6] 2024-03-28T18:25:00.882 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557d7d8a5da1] 2024-03-28T18:25:00.882 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557d7d8a82d5] 2024-03-28T18:25:00.882 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557d7d8f8233] 2024-03-28T18:25:00.883 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557d7d8fa197] 2024-03-28T18:25:00.883 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557d7d7f911b] 2024-03-28T18:25:00.883 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557d7d7f96bc] 2024-03-28T18:25:00.883 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557d7d7e31ff] 2024-03-28T18:25:00.883 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0e4ad64488] 2024-03-28T18:25:00.883 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 16: (DispatchQueue::entry()+0x51f) [0x7f0e4ad6160f] 2024-03-28T18:25:00.883 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0e4ae34e81] 2024-03-28T18:25:00.883 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0e4984f1ca] 2024-03-28T18:25:00.883 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 19: clone() 2024-03-28T18:25:00.883 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:25:00.883 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2024-03-28T18:25:00.883 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: -1> 2024-03-28T18:25:00.617+0000 7f0e3f200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0e3f200700 time 2024-03-28T18:25:00.617545+0000 2024-03-28T18:25:00.883 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:25:00.883 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2024-03-28T18:25:00.884 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:00.884 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0e4aad24a1] 2024-03-28T18:25:00.884 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0e4aad2667] 2024-03-28T18:25:00.884 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557d7da2b223] 2024-03-28T18:25:00.884 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557d7d8a55d6] 2024-03-28T18:25:00.884 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557d7d8a5da1] 2024-03-28T18:25:00.884 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557d7d8a82d5] 2024-03-28T18:25:00.884 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557d7d8f8233] 2024-03-28T18:25:00.884 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557d7d8fa197] 2024-03-28T18:25:00.884 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557d7d7f911b] 2024-03-28T18:25:00.884 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557d7d7f96bc] 2024-03-28T18:25:00.884 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557d7d7e31ff] 2024-03-28T18:25:00.884 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0e4ad64488] 2024-03-28T18:25:00.884 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 13: (DispatchQueue::entry()+0x51f) [0x7f0e4ad6160f] 2024-03-28T18:25:00.885 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0e4ae34e81] 2024-03-28T18:25:00.885 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0e4984f1ca] 2024-03-28T18:25:00.885 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 16: clone() 2024-03-28T18:25:00.885 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2024-03-28T18:25:00.885 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 0> 2024-03-28T18:25:00.620+0000 7f0e3f200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:25:00.885 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: in thread 7f0e3f200700 thread_name:ms_dispatch 2024-03-28T18:25:00.885 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2024-03-28T18:25:00.885 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:00.885 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0e49859d20] 2024-03-28T18:25:00.885 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2: gsignal() 2024-03-28T18:25:00.885 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 3: abort() 2024-03-28T18:25:00.885 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0e4aad24fb] 2024-03-28T18:25:00.885 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0e4aad2667] 2024-03-28T18:25:00.885 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557d7da2b223] 2024-03-28T18:25:00.885 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557d7d8a55d6] 2024-03-28T18:25:00.885 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557d7d8a5da1] 2024-03-28T18:25:00.886 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557d7d8a82d5] 2024-03-28T18:25:00.886 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557d7d8f8233] 2024-03-28T18:25:00.886 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557d7d8fa197] 2024-03-28T18:25:00.886 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557d7d7f911b] 2024-03-28T18:25:00.886 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557d7d7f96bc] 2024-03-28T18:25:00.886 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557d7d7e31ff] 2024-03-28T18:25:00.886 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0e4ad64488] 2024-03-28T18:25:00.886 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 16: (DispatchQueue::entry()+0x51f) [0x7f0e4ad6160f] 2024-03-28T18:25:00.886 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0e4ae34e81] 2024-03-28T18:25:00.886 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0e4984f1ca] 2024-03-28T18:25:00.886 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 19: clone() 2024-03-28T18:25:00.886 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:25:00.886 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2024-03-28T18:25:00.886 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: -1> 2024-03-28T18:25:00.617+0000 7f0e3f200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0e3f200700 time 2024-03-28T18:25:00.617545+0000 2024-03-28T18:25:00.887 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:25:00.887 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2024-03-28T18:25:00.887 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:00.887 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0e4aad24a1] 2024-03-28T18:25:00.887 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0e4aad2667] 2024-03-28T18:25:00.887 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557d7da2b223] 2024-03-28T18:25:00.887 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557d7d8a55d6] 2024-03-28T18:25:00.887 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557d7d8a5da1] 2024-03-28T18:25:00.887 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557d7d8a82d5] 2024-03-28T18:25:00.887 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557d7d8f8233] 2024-03-28T18:25:00.887 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557d7d8fa197] 2024-03-28T18:25:00.887 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557d7d7f911b] 2024-03-28T18:25:00.887 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557d7d7f96bc] 2024-03-28T18:25:00.887 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557d7d7e31ff] 2024-03-28T18:25:00.887 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0e4ad64488] 2024-03-28T18:25:00.887 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 13: (DispatchQueue::entry()+0x51f) [0x7f0e4ad6160f] 2024-03-28T18:25:00.888 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0e4ae34e81] 2024-03-28T18:25:00.888 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0e4984f1ca] 2024-03-28T18:25:00.888 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 16: clone() 2024-03-28T18:25:00.888 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2024-03-28T18:25:00.888 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 0> 2024-03-28T18:25:00.620+0000 7f0e3f200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:25:00.888 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: in thread 7f0e3f200700 thread_name:ms_dispatch 2024-03-28T18:25:00.888 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2024-03-28T18:25:00.888 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:00.888 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0e49859d20] 2024-03-28T18:25:00.888 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2: gsignal() 2024-03-28T18:25:00.888 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 3: abort() 2024-03-28T18:25:00.888 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0e4aad24fb] 2024-03-28T18:25:00.888 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0e4aad2667] 2024-03-28T18:25:00.888 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557d7da2b223] 2024-03-28T18:25:00.888 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557d7d8a55d6] 2024-03-28T18:25:00.888 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557d7d8a5da1] 2024-03-28T18:25:00.889 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557d7d8a82d5] 2024-03-28T18:25:00.889 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557d7d8f8233] 2024-03-28T18:25:00.889 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557d7d8fa197] 2024-03-28T18:25:00.889 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557d7d7f911b] 2024-03-28T18:25:00.889 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557d7d7f96bc] 2024-03-28T18:25:00.889 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557d7d7e31ff] 2024-03-28T18:25:00.889 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0e4ad64488] 2024-03-28T18:25:00.889 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 16: (DispatchQueue::entry()+0x51f) [0x7f0e4ad6160f] 2024-03-28T18:25:00.889 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0e4ae34e81] 2024-03-28T18:25:00.889 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0e4984f1ca] 2024-03-28T18:25:00.889 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 19: clone() 2024-03-28T18:25:00.889 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:25:00.889 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90173]: 2024-03-28T18:25:00.889 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:00 smithi037 podman[90488]: 2024-03-28 18:25:00.756679435 +0000 UTC m=+0.032933888 container died 7e8a59b31ded63f821c5c8bb89dc45669de0e99ad3571986aa97ba24740d52b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-28T18:25:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:00 smithi059 ceph-mon[36190]: pgmap v476: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:25:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:00 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3149561600,v1:172.21.15.37:6837/3149561600] up:rejoin 2024-03-28T18:25:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:00 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:00 smithi059 ceph-mon[36190]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-28T18:25:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:00 smithi170 ceph-mon[36280]: pgmap v476: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T18:25:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:00 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3149561600,v1:172.21.15.37:6837/3149561600] up:rejoin 2024-03-28T18:25:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:00 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:00 smithi170 ceph-mon[36280]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-28T18:25:01.569 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:25:01 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:25:01.458+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:25:01.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:01 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:25:01.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:01 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/3149561600,v1:172.21.15.37:6837/3149561600] up:active 2024-03-28T18:25:01.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:01 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:01.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:01 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:25:01.875 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:01 smithi037 podman[90488]: 2024-03-28 18:25:01.632346072 +0000 UTC m=+0.908600565 container remove 7e8a59b31ded63f821c5c8bb89dc45669de0e99ad3571986aa97ba24740d52b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-28T18:25:01.875 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:01 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:25:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:01 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:25:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:01 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/3149561600,v1:172.21.15.37:6837/3149561600] up:active 2024-03-28T18:25:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:01 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:01 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:25:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:01 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:25:02.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:01 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/3149561600,v1:172.21.15.37:6837/3149561600] up:active 2024-03-28T18:25:02.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:01 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:02.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:01 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:25:02.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:02 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-28T18:25:02.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:02 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.749s CPU time. 2024-03-28T18:25:02.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:02 smithi037 ceph-mon[31460]: pgmap v477: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-03-28T18:25:03.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:02 smithi059 ceph-mon[36190]: pgmap v477: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-03-28T18:25:03.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:02 smithi170 ceph-mon[36280]: pgmap v477: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-03-28T18:25:04.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:04 smithi037 ceph-mon[31460]: pgmap v478: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:25:05.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:04 smithi059 ceph-mon[36190]: pgmap v478: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:25:05.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:04 smithi170 ceph-mon[36280]: pgmap v478: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:25:06.838 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:06 smithi170 ceph-mon[36280]: pgmap v479: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:25:06.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:06 smithi037 ceph-mon[31460]: pgmap v479: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:25:07.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:06 smithi059 ceph-mon[36190]: pgmap v479: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 27 KiB/s wr, 34 op/s 2024-03-28T18:25:08.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:08 smithi037 ceph-mon[31460]: pgmap v480: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 24 KiB/s wr, 34 op/s 2024-03-28T18:25:09.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:08 smithi059 ceph-mon[36190]: pgmap v480: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 24 KiB/s wr, 34 op/s 2024-03-28T18:25:09.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:08 smithi170 ceph-mon[36280]: pgmap v480: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 24 KiB/s wr, 34 op/s 2024-03-28T18:25:10.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:10 smithi037 ceph-mon[31460]: pgmap v481: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 29 op/s 2024-03-28T18:25:11.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:10 smithi059 ceph-mon[36190]: pgmap v481: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 29 op/s 2024-03-28T18:25:11.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:10 smithi170 ceph-mon[36280]: pgmap v481: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 29 op/s 2024-03-28T18:25:12.336 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:12 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 4. 2024-03-28T18:25:12.337 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:12 smithi037 systemd[1]: Stopped Ceph mds.g for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:25:12.337 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:12 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.749s CPU time. 2024-03-28T18:25:12.337 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:12 smithi037 systemd[1]: Starting Ceph mds.g for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:25:12.601 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:12 smithi037 podman[90587]: 2024-03-28 18:25:12.419679102 +0000 UTC m=+0.037861674 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:25:12.602 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:12 smithi037 podman[90587]: 2024-03-28 18:25:12.566789238 +0000 UTC m=+0.184971792 container create 10e315ce6e24ebbe10e79b338df65a06c2d60a900376af6c777ffc2151877c30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-28T18:25:12.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:12 smithi037 ceph-mon[31460]: pgmap v482: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 29 op/s 2024-03-28T18:25:13.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:12 smithi059 ceph-mon[36190]: pgmap v482: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 29 op/s 2024-03-28T18:25:13.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:12 smithi170 ceph-mon[36280]: pgmap v482: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 29 op/s 2024-03-28T18:25:13.274 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:12 smithi037 podman[90587]: 2024-03-28 18:25:12.947432789 +0000 UTC m=+0.565615343 container init 10e315ce6e24ebbe10e79b338df65a06c2d60a900376af6c777ffc2151877c30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:25:13.274 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:12 smithi037 podman[90587]: 2024-03-28 18:25:12.959488365 +0000 UTC m=+0.577670910 container start 10e315ce6e24ebbe10e79b338df65a06c2d60a900376af6c777ffc2151877c30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-28T18:25:13.274 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:12 smithi037 ceph-mds[90607]: -- 172.21.15.37:0/3706398809 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56202bae9a40 con 0x56202babcc00 2024-03-28T18:25:13.275 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:13 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: starting mds.g at 2024-03-28T18:25:13.275 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:13 smithi037 bash[90587]: 10e315ce6e24ebbe10e79b338df65a06c2d60a900376af6c777ffc2151877c30 2024-03-28T18:25:13.275 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:13 smithi037 systemd[1]: Started Ceph mds.g for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:25:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:13 smithi037 ceph-mon[31460]: daemon mds.g restarted 2024-03-28T18:25:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:13 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:25:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:13 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:25:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:13 smithi037 ceph-mon[31460]: osdmap e143: 12 total, 12 up, 12 in 2024-03-28T18:25:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:13 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/505326814,v1:172.21.15.37:6837/505326814] up:boot 2024-03-28T18:25:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:13 smithi037 ceph-mon[31460]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-28T18:25:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:13 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:25:13.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:13 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:14.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:13 smithi059 ceph-mon[36190]: daemon mds.g restarted 2024-03-28T18:25:14.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:13 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:25:14.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:13 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:25:14.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:13 smithi059 ceph-mon[36190]: osdmap e143: 12 total, 12 up, 12 in 2024-03-28T18:25:14.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:13 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/505326814,v1:172.21.15.37:6837/505326814] up:boot 2024-03-28T18:25:14.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:13 smithi059 ceph-mon[36190]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-28T18:25:14.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:13 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:25:14.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:13 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:14.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:13 smithi170 ceph-mon[36280]: daemon mds.g restarted 2024-03-28T18:25:14.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:13 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:25:14.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:13 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:25:14.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:13 smithi170 ceph-mon[36280]: osdmap e143: 12 total, 12 up, 12 in 2024-03-28T18:25:14.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:13 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/505326814,v1:172.21.15.37:6837/505326814] up:boot 2024-03-28T18:25:14.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:13 smithi170 ceph-mon[36280]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-28T18:25:14.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:13 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:25:14.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:13 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:14.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:14 smithi037 ceph-mon[31460]: pgmap v484: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:25:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:14 smithi059 ceph-mon[36190]: pgmap v484: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:25:15.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:14 smithi170 ceph-mon[36280]: pgmap v484: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:25:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:15 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/961897107,v1:172.21.15.37:6835/961897107] up:resolve 2024-03-28T18:25:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:15 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:15 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/961897107,v1:172.21.15.37:6835/961897107] up:resolve 2024-03-28T18:25:16.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:15 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:16.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:15 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/961897107,v1:172.21.15.37:6835/961897107] up:resolve 2024-03-28T18:25:16.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:15 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:17.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:16 smithi059 ceph-mon[36190]: pgmap v485: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:25:17.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:16 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:25:17.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:16 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/961897107,v1:172.21.15.37:6835/961897107] up:reconnect 2024-03-28T18:25:17.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:16 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:17.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:16 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999987 2024-03-28T18:25:17.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:16 smithi170 ceph-mon[36280]: pgmap v485: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:25:17.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:16 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:25:17.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:16 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/961897107,v1:172.21.15.37:6835/961897107] up:reconnect 2024-03-28T18:25:17.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:16 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:17.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:16 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999987 2024-03-28T18:25:17.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:16 smithi037 ceph-mon[31460]: pgmap v485: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:25:17.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:16 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:25:17.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:16 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/961897107,v1:172.21.15.37:6835/961897107] up:reconnect 2024-03-28T18:25:17.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:16 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:17.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:16 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999987 2024-03-28T18:25:18.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:17 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/961897107,v1:172.21.15.37:6835/961897107] up:rejoin 2024-03-28T18:25:18.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:17 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:18.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:17 smithi059 ceph-mon[36190]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-28T18:25:18.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:17 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/961897107,v1:172.21.15.37:6835/961897107] up:rejoin 2024-03-28T18:25:18.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:17 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:18.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:17 smithi170 ceph-mon[36280]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-28T18:25:18.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/961897107,v1:172.21.15.37:6835/961897107] up:rejoin 2024-03-28T18:25:18.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:18.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-mon[31460]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-28T18:25:18.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd283e00700 time 2024-03-28T18:25:17.716181+0000 2024-03-28T18:25:18.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:25:18.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:18.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd28f7654a1] 2024-03-28T18:25:18.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd28f765667] 2024-03-28T18:25:18.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5640d1e88223] 2024-03-28T18:25:18.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5640d1d025d6] 2024-03-28T18:25:18.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5640d1d02da1] 2024-03-28T18:25:18.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5640d1d052d5] 2024-03-28T18:25:18.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5640d1d55233] 2024-03-28T18:25:18.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5640d1d57197] 2024-03-28T18:25:18.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5640d1c5611b] 2024-03-28T18:25:18.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5640d1c566bc] 2024-03-28T18:25:18.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5640d1c401ff] 2024-03-28T18:25:18.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd28f9f7488] 2024-03-28T18:25:18.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 13: (DispatchQueue::entry()+0x51f) [0x7fd28f9f460f] 2024-03-28T18:25:18.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd28fac7e81] 2024-03-28T18:25:18.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd28e4e21ca] 2024-03-28T18:25:18.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 16: clone() 2024-03-28T18:25:18.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2024-03-28T18:25:17.717+0000 7fd283e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd283e00700 time 2024-03-28T18:25:17.716181+0000 2024-03-28T18:25:18.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:25:18.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2024-03-28T18:25:18.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:18.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd28f7654a1] 2024-03-28T18:25:18.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd28f765667] 2024-03-28T18:25:18.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5640d1e88223] 2024-03-28T18:25:18.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5640d1d025d6] 2024-03-28T18:25:18.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5640d1d02da1] 2024-03-28T18:25:18.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5640d1d052d5] 2024-03-28T18:25:18.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5640d1d55233] 2024-03-28T18:25:18.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5640d1d57197] 2024-03-28T18:25:18.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5640d1c5611b] 2024-03-28T18:25:18.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5640d1c566bc] 2024-03-28T18:25:18.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5640d1c401ff] 2024-03-28T18:25:18.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd28f9f7488] 2024-03-28T18:25:18.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 13: (DispatchQueue::entry()+0x51f) [0x7fd28f9f460f] 2024-03-28T18:25:18.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd28fac7e81] 2024-03-28T18:25:18.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd28e4e21ca] 2024-03-28T18:25:18.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 16: clone() 2024-03-28T18:25:18.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2024-03-28T18:25:18.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: *** Caught signal (Aborted) ** 2024-03-28T18:25:18.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: in thread 7fd283e00700 thread_name:ms_dispatch 2024-03-28T18:25:18.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:18.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd28e4ecd20] 2024-03-28T18:25:18.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2: gsignal() 2024-03-28T18:25:18.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 3: abort() 2024-03-28T18:25:18.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd28f7654fb] 2024-03-28T18:25:18.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd28f765667] 2024-03-28T18:25:18.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5640d1e88223] 2024-03-28T18:25:18.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5640d1d025d6] 2024-03-28T18:25:18.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5640d1d02da1] 2024-03-28T18:25:18.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5640d1d052d5] 2024-03-28T18:25:18.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5640d1d55233] 2024-03-28T18:25:18.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5640d1d57197] 2024-03-28T18:25:18.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5640d1c5611b] 2024-03-28T18:25:18.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5640d1c566bc] 2024-03-28T18:25:18.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5640d1c401ff] 2024-03-28T18:25:18.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd28f9f7488] 2024-03-28T18:25:18.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 16: (DispatchQueue::entry()+0x51f) [0x7fd28f9f460f] 2024-03-28T18:25:18.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd28fac7e81] 2024-03-28T18:25:18.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd28e4e21ca] 2024-03-28T18:25:18.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 19: clone() 2024-03-28T18:25:18.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2024-03-28T18:25:17.719+0000 7fd283e00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:25:18.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: in thread 7fd283e00700 thread_name:ms_dispatch 2024-03-28T18:25:18.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2024-03-28T18:25:18.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:18.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd28e4ecd20] 2024-03-28T18:25:18.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2: gsignal() 2024-03-28T18:25:18.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 3: abort() 2024-03-28T18:25:18.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd28f7654fb] 2024-03-28T18:25:18.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd28f765667] 2024-03-28T18:25:18.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5640d1e88223] 2024-03-28T18:25:18.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5640d1d025d6] 2024-03-28T18:25:18.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5640d1d02da1] 2024-03-28T18:25:18.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5640d1d052d5] 2024-03-28T18:25:18.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5640d1d55233] 2024-03-28T18:25:18.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5640d1d57197] 2024-03-28T18:25:18.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5640d1c5611b] 2024-03-28T18:25:18.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5640d1c566bc] 2024-03-28T18:25:18.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5640d1c401ff] 2024-03-28T18:25:18.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd28f9f7488] 2024-03-28T18:25:18.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 16: (DispatchQueue::entry()+0x51f) [0x7fd28f9f460f] 2024-03-28T18:25:18.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd28fac7e81] 2024-03-28T18:25:18.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd28e4e21ca] 2024-03-28T18:25:18.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 19: clone() 2024-03-28T18:25:18.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:25:18.133 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2024-03-28T18:25:18.133 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: -1> 2024-03-28T18:25:17.717+0000 7fd283e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd283e00700 time 2024-03-28T18:25:17.716181+0000 2024-03-28T18:25:18.133 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:25:18.133 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2024-03-28T18:25:18.133 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:18.133 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd28f7654a1] 2024-03-28T18:25:18.133 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd28f765667] 2024-03-28T18:25:18.133 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5640d1e88223] 2024-03-28T18:25:18.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5640d1d025d6] 2024-03-28T18:25:18.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5640d1d02da1] 2024-03-28T18:25:18.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5640d1d052d5] 2024-03-28T18:25:18.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5640d1d55233] 2024-03-28T18:25:18.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5640d1d57197] 2024-03-28T18:25:18.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5640d1c5611b] 2024-03-28T18:25:18.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5640d1c566bc] 2024-03-28T18:25:18.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5640d1c401ff] 2024-03-28T18:25:18.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd28f9f7488] 2024-03-28T18:25:18.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 13: (DispatchQueue::entry()+0x51f) [0x7fd28f9f460f] 2024-03-28T18:25:18.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd28fac7e81] 2024-03-28T18:25:18.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd28e4e21ca] 2024-03-28T18:25:18.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 16: clone() 2024-03-28T18:25:18.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2024-03-28T18:25:18.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 0> 2024-03-28T18:25:17.719+0000 7fd283e00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:25:18.135 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: in thread 7fd283e00700 thread_name:ms_dispatch 2024-03-28T18:25:18.135 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2024-03-28T18:25:18.135 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:18.135 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd28e4ecd20] 2024-03-28T18:25:18.135 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2: gsignal() 2024-03-28T18:25:18.135 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 3: abort() 2024-03-28T18:25:18.135 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd28f7654fb] 2024-03-28T18:25:18.135 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd28f765667] 2024-03-28T18:25:18.135 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5640d1e88223] 2024-03-28T18:25:18.135 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5640d1d025d6] 2024-03-28T18:25:18.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5640d1d02da1] 2024-03-28T18:25:18.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5640d1d052d5] 2024-03-28T18:25:18.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5640d1d55233] 2024-03-28T18:25:18.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5640d1d57197] 2024-03-28T18:25:18.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5640d1c5611b] 2024-03-28T18:25:18.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5640d1c566bc] 2024-03-28T18:25:18.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5640d1c401ff] 2024-03-28T18:25:18.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd28f9f7488] 2024-03-28T18:25:18.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 16: (DispatchQueue::entry()+0x51f) [0x7fd28f9f460f] 2024-03-28T18:25:18.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd28fac7e81] 2024-03-28T18:25:18.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd28e4e21ca] 2024-03-28T18:25:18.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 19: clone() 2024-03-28T18:25:18.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:25:18.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2024-03-28T18:25:18.137 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: -1> 2024-03-28T18:25:17.717+0000 7fd283e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd283e00700 time 2024-03-28T18:25:17.716181+0000 2024-03-28T18:25:18.137 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:25:18.137 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2024-03-28T18:25:18.137 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:18.137 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd28f7654a1] 2024-03-28T18:25:18.137 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd28f765667] 2024-03-28T18:25:18.137 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5640d1e88223] 2024-03-28T18:25:18.137 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5640d1d025d6] 2024-03-28T18:25:18.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5640d1d02da1] 2024-03-28T18:25:18.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5640d1d052d5] 2024-03-28T18:25:18.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5640d1d55233] 2024-03-28T18:25:18.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5640d1d57197] 2024-03-28T18:25:18.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5640d1c5611b] 2024-03-28T18:25:18.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5640d1c566bc] 2024-03-28T18:25:18.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5640d1c401ff] 2024-03-28T18:25:18.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd28f9f7488] 2024-03-28T18:25:18.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 13: (DispatchQueue::entry()+0x51f) [0x7fd28f9f460f] 2024-03-28T18:25:18.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd28fac7e81] 2024-03-28T18:25:18.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd28e4e21ca] 2024-03-28T18:25:18.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 16: clone() 2024-03-28T18:25:18.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2024-03-28T18:25:18.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 0> 2024-03-28T18:25:17.719+0000 7fd283e00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:25:18.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: in thread 7fd283e00700 thread_name:ms_dispatch 2024-03-28T18:25:18.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2024-03-28T18:25:18.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:18.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd28e4ecd20] 2024-03-28T18:25:18.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2: gsignal() 2024-03-28T18:25:18.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 3: abort() 2024-03-28T18:25:18.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd28f7654fb] 2024-03-28T18:25:18.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd28f765667] 2024-03-28T18:25:18.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5640d1e88223] 2024-03-28T18:25:18.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5640d1d025d6] 2024-03-28T18:25:18.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5640d1d02da1] 2024-03-28T18:25:18.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5640d1d052d5] 2024-03-28T18:25:18.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5640d1d55233] 2024-03-28T18:25:18.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5640d1d57197] 2024-03-28T18:25:18.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5640d1c5611b] 2024-03-28T18:25:18.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5640d1c566bc] 2024-03-28T18:25:18.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5640d1c401ff] 2024-03-28T18:25:18.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd28f9f7488] 2024-03-28T18:25:18.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 16: (DispatchQueue::entry()+0x51f) [0x7fd28f9f460f] 2024-03-28T18:25:18.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd28fac7e81] 2024-03-28T18:25:18.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd28e4e21ca] 2024-03-28T18:25:18.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 19: clone() 2024-03-28T18:25:18.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:25:18.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90449]: 2024-03-28T18:25:18.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:17 smithi037 podman[90642]: 2024-03-28 18:25:17.833140271 +0000 UTC m=+0.034204486 container died bba2cae7a408b54e438692c77084a1e4921ca4ac50caffe19345b95d99d4e121 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-28T18:25:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:18 smithi059 ceph-mon[36190]: pgmap v486: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:25:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:18 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:25:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:18 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/961897107,v1:172.21.15.37:6835/961897107] up:active 2024-03-28T18:25:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:18 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:18 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:25:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:18 smithi170 ceph-mon[36280]: pgmap v486: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:25:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:18 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:25:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:18 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/961897107,v1:172.21.15.37:6835/961897107] up:active 2024-03-28T18:25:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:18 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:18 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:25:19.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:18 smithi037 ceph-mon[31460]: pgmap v486: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:25:19.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:18 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:25:19.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:18 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/961897107,v1:172.21.15.37:6835/961897107] up:active 2024-03-28T18:25:19.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:18 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:19.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:18 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:25:19.125 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:18 smithi037 podman[90642]: 2024-03-28 18:25:18.717985057 +0000 UTC m=+0.919049272 container remove bba2cae7a408b54e438692c77084a1e4921ca4ac50caffe19345b95d99d4e121 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-28T18:25:19.125 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:18 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:25:19.624 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:19 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-28T18:25:19.625 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:19 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Consumed 1.914s CPU time. 2024-03-28T18:25:20.772 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:20 smithi037 ceph-mon[31460]: pgmap v487: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:25:21.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:20 smithi059 ceph-mon[36190]: pgmap v487: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:25:21.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:20 smithi170 ceph-mon[36280]: pgmap v487: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:25:21.625 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:25:21 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:25:21.460+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:25:22.705 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:22 smithi037 ceph-mon[31460]: pgmap v488: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-03-28T18:25:23.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:22 smithi059 ceph-mon[36190]: pgmap v488: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-03-28T18:25:23.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:22 smithi170 ceph-mon[36280]: pgmap v488: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-03-28T18:25:25.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:24 smithi059 ceph-mon[36190]: pgmap v489: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 26 KiB/s wr, 18 op/s 2024-03-28T18:25:25.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:24 smithi170 ceph-mon[36280]: pgmap v489: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 26 KiB/s wr, 18 op/s 2024-03-28T18:25:25.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:24 smithi037 ceph-mon[31460]: pgmap v489: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 26 KiB/s wr, 18 op/s 2024-03-28T18:25:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:26 smithi059 ceph-mon[36190]: pgmap v490: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-28T18:25:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:26 smithi059 ceph-mon[36190]: osdmap e144: 12 total, 12 up, 12 in 2024-03-28T18:25:27.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:26 smithi170 ceph-mon[36280]: pgmap v490: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-28T18:25:27.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:26 smithi170 ceph-mon[36280]: osdmap e144: 12 total, 12 up, 12 in 2024-03-28T18:25:27.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:26 smithi037 ceph-mon[31460]: pgmap v490: 97 pgs: 97 active+clean; 45 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-28T18:25:27.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:26 smithi037 ceph-mon[31460]: osdmap e144: 12 total, 12 up, 12 in 2024-03-28T18:25:28.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:28 smithi037 ceph-mon[31460]: pgmap v492: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:25:29.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:28 smithi059 ceph-mon[36190]: pgmap v492: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:25:29.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:28 smithi170 ceph-mon[36280]: pgmap v492: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:25:29.775 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:29 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 3. 2024-03-28T18:25:29.775 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:29 smithi037 systemd[1]: Stopped Ceph mds.j for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:25:29.775 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:29 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Consumed 1.914s CPU time. 2024-03-28T18:25:29.775 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:29 smithi037 systemd[1]: Starting Ceph mds.j for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:25:30.107 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:29 smithi037 podman[90732]: 2024-03-28 18:25:29.680807488 +0000 UTC m=+0.031577349 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:25:30.107 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:29 smithi037 podman[90732]: 2024-03-28 18:25:29.828095485 +0000 UTC m=+0.178865394 container create ef6a6c5175e8a5a14c61d8f3226775c471b3b922b7064c27fa2984c090f02da1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-28T18:25:30.375 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:30 smithi037 podman[90732]: 2024-03-28 18:25:30.106903057 +0000 UTC m=+0.457673016 container init ef6a6c5175e8a5a14c61d8f3226775c471b3b922b7064c27fa2984c090f02da1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:25:30.375 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:30 smithi037 podman[90732]: 2024-03-28 18:25:30.116445876 +0000 UTC m=+0.467215732 container start ef6a6c5175e8a5a14c61d8f3226775c471b3b922b7064c27fa2984c090f02da1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-28T18:25:30.375 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:30 smithi037 ceph-mds[90750]: -- 172.21.15.37:0/1232460388 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bfcc355a40 con 0x55bfcc328c00 2024-03-28T18:25:30.375 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:30 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: starting mds.j at 2024-03-28T18:25:30.375 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:30 smithi037 bash[90732]: ef6a6c5175e8a5a14c61d8f3226775c471b3b922b7064c27fa2984c090f02da1 2024-03-28T18:25:30.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:30 smithi037 systemd[1]: Started Ceph mds.j for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:25:31.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:30 smithi059 ceph-mon[36190]: pgmap v493: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:25:31.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:30 smithi059 ceph-mon[36190]: daemon mds.j restarted 2024-03-28T18:25:31.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:30 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:25:31.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:30 smithi059 ceph-mon[36190]: osdmap e145: 12 total, 12 up, 12 in 2024-03-28T18:25:31.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:30 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/2295994436,v1:172.21.15.37:6835/2295994436] up:boot 2024-03-28T18:25:31.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:30 smithi059 ceph-mon[36190]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-28T18:25:31.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:30 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:25:31.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:30 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:25:31.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:30 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:31.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:30 smithi170 ceph-mon[36280]: pgmap v493: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:25:31.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:30 smithi170 ceph-mon[36280]: daemon mds.j restarted 2024-03-28T18:25:31.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:30 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:25:31.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:30 smithi170 ceph-mon[36280]: osdmap e145: 12 total, 12 up, 12 in 2024-03-28T18:25:31.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:30 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/2295994436,v1:172.21.15.37:6835/2295994436] up:boot 2024-03-28T18:25:31.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:30 smithi170 ceph-mon[36280]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-28T18:25:31.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:30 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:25:31.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:30 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:25:31.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:30 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:31.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:30 smithi037 ceph-mon[31460]: pgmap v493: 97 pgs: 97 active+clean; 45 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 27 KiB/s wr, 19 op/s 2024-03-28T18:25:31.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:30 smithi037 ceph-mon[31460]: daemon mds.j restarted 2024-03-28T18:25:31.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:30 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:25:31.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:30 smithi037 ceph-mon[31460]: osdmap e145: 12 total, 12 up, 12 in 2024-03-28T18:25:31.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:30 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/2295994436,v1:172.21.15.37:6835/2295994436] up:boot 2024-03-28T18:25:31.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:30 smithi037 ceph-mon[31460]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-28T18:25:31.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:30 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:25:31.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:30 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:25:31.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:30 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:33.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:32 smithi059 ceph-mon[36190]: pgmap v495: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:25:33.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:32 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/505326814,v1:172.21.15.37:6837/505326814] up:resolve 2024-03-28T18:25:33.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:32 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:33.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:32 smithi170 ceph-mon[36280]: pgmap v495: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:25:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:32 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/505326814,v1:172.21.15.37:6837/505326814] up:resolve 2024-03-28T18:25:33.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:32 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:33.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:32 smithi037 ceph-mon[31460]: pgmap v495: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:25:33.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:32 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/505326814,v1:172.21.15.37:6837/505326814] up:resolve 2024-03-28T18:25:33.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:32 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:33.982 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:33 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/505326814,v1:172.21.15.37:6837/505326814] up:reconnect 2024-03-28T18:25:33.982 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:33 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:33.983 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:33 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0 2024-03-28T18:25:33.983 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:33 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.000999987 2024-03-28T18:25:34.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:33 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/505326814,v1:172.21.15.37:6837/505326814] up:reconnect 2024-03-28T18:25:34.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:33 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:34.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:33 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0 2024-03-28T18:25:34.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:33 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.000999987 2024-03-28T18:25:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:33 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/505326814,v1:172.21.15.37:6837/505326814] up:reconnect 2024-03-28T18:25:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:33 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:33 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0 2024-03-28T18:25:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:33 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.000999987 2024-03-28T18:25:34.375 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f67dce00700 time 2024-03-28T18:25:34.031198+0000 2024-03-28T18:25:34.375 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:25:34.375 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:34.375 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f67e86b14a1] 2024-03-28T18:25:34.375 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f67e86b1667] 2024-03-28T18:25:34.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5620293f8223] 2024-03-28T18:25:34.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5620292725d6] 2024-03-28T18:25:34.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x562029272da1] 2024-03-28T18:25:34.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5620292752d5] 2024-03-28T18:25:34.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5620292c5233] 2024-03-28T18:25:34.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5620292c7197] 2024-03-28T18:25:34.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5620291c611b] 2024-03-28T18:25:34.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5620291c66bc] 2024-03-28T18:25:34.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5620291b01ff] 2024-03-28T18:25:34.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f67e8943488] 2024-03-28T18:25:34.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 13: (DispatchQueue::entry()+0x51f) [0x7f67e894060f] 2024-03-28T18:25:34.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f67e8a13e81] 2024-03-28T18:25:34.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f67e742e1ca] 2024-03-28T18:25:34.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 16: clone() 2024-03-28T18:25:34.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2024-03-28T18:25:34.032+0000 7f67dce00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f67dce00700 time 2024-03-28T18:25:34.031198+0000 2024-03-28T18:25:34.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:25:34.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2024-03-28T18:25:34.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:34.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f67e86b14a1] 2024-03-28T18:25:34.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f67e86b1667] 2024-03-28T18:25:34.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5620293f8223] 2024-03-28T18:25:34.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5620292725d6] 2024-03-28T18:25:34.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x562029272da1] 2024-03-28T18:25:34.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5620292752d5] 2024-03-28T18:25:34.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5620292c5233] 2024-03-28T18:25:34.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5620292c7197] 2024-03-28T18:25:34.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5620291c611b] 2024-03-28T18:25:34.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5620291c66bc] 2024-03-28T18:25:34.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5620291b01ff] 2024-03-28T18:25:34.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f67e8943488] 2024-03-28T18:25:34.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 13: (DispatchQueue::entry()+0x51f) [0x7f67e894060f] 2024-03-28T18:25:34.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f67e8a13e81] 2024-03-28T18:25:34.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f67e742e1ca] 2024-03-28T18:25:34.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 16: clone() 2024-03-28T18:25:34.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2024-03-28T18:25:34.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: *** Caught signal (Aborted) ** 2024-03-28T18:25:34.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: in thread 7f67dce00700 thread_name:ms_dispatch 2024-03-28T18:25:34.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:34.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f67e7438d20] 2024-03-28T18:25:34.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2: gsignal() 2024-03-28T18:25:34.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 3: abort() 2024-03-28T18:25:34.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f67e86b14fb] 2024-03-28T18:25:34.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f67e86b1667] 2024-03-28T18:25:34.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5620293f8223] 2024-03-28T18:25:34.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5620292725d6] 2024-03-28T18:25:34.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x562029272da1] 2024-03-28T18:25:34.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5620292752d5] 2024-03-28T18:25:34.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5620292c5233] 2024-03-28T18:25:34.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5620292c7197] 2024-03-28T18:25:34.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5620291c611b] 2024-03-28T18:25:34.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5620291c66bc] 2024-03-28T18:25:34.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5620291b01ff] 2024-03-28T18:25:34.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f67e8943488] 2024-03-28T18:25:34.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 16: (DispatchQueue::entry()+0x51f) [0x7f67e894060f] 2024-03-28T18:25:34.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f67e8a13e81] 2024-03-28T18:25:34.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f67e742e1ca] 2024-03-28T18:25:34.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 19: clone() 2024-03-28T18:25:34.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2024-03-28T18:25:34.035+0000 7f67dce00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:25:34.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: in thread 7f67dce00700 thread_name:ms_dispatch 2024-03-28T18:25:34.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2024-03-28T18:25:34.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:34.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f67e7438d20] 2024-03-28T18:25:34.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2: gsignal() 2024-03-28T18:25:34.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 3: abort() 2024-03-28T18:25:34.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f67e86b14fb] 2024-03-28T18:25:34.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f67e86b1667] 2024-03-28T18:25:34.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5620293f8223] 2024-03-28T18:25:34.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5620292725d6] 2024-03-28T18:25:34.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x562029272da1] 2024-03-28T18:25:34.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5620292752d5] 2024-03-28T18:25:34.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5620292c5233] 2024-03-28T18:25:34.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5620292c7197] 2024-03-28T18:25:34.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5620291c611b] 2024-03-28T18:25:34.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5620291c66bc] 2024-03-28T18:25:34.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5620291b01ff] 2024-03-28T18:25:34.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f67e8943488] 2024-03-28T18:25:34.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 16: (DispatchQueue::entry()+0x51f) [0x7f67e894060f] 2024-03-28T18:25:34.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f67e8a13e81] 2024-03-28T18:25:34.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f67e742e1ca] 2024-03-28T18:25:34.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 19: clone() 2024-03-28T18:25:34.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:25:34.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2024-03-28T18:25:34.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: -1> 2024-03-28T18:25:34.032+0000 7f67dce00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f67dce00700 time 2024-03-28T18:25:34.031198+0000 2024-03-28T18:25:34.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:25:34.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2024-03-28T18:25:34.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:34.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f67e86b14a1] 2024-03-28T18:25:34.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f67e86b1667] 2024-03-28T18:25:34.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5620293f8223] 2024-03-28T18:25:34.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5620292725d6] 2024-03-28T18:25:34.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x562029272da1] 2024-03-28T18:25:34.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5620292752d5] 2024-03-28T18:25:34.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5620292c5233] 2024-03-28T18:25:34.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5620292c7197] 2024-03-28T18:25:34.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5620291c611b] 2024-03-28T18:25:34.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5620291c66bc] 2024-03-28T18:25:34.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5620291b01ff] 2024-03-28T18:25:34.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f67e8943488] 2024-03-28T18:25:34.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 13: (DispatchQueue::entry()+0x51f) [0x7f67e894060f] 2024-03-28T18:25:34.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f67e8a13e81] 2024-03-28T18:25:34.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f67e742e1ca] 2024-03-28T18:25:34.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 16: clone() 2024-03-28T18:25:34.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2024-03-28T18:25:34.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 0> 2024-03-28T18:25:34.035+0000 7f67dce00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:25:34.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: in thread 7f67dce00700 thread_name:ms_dispatch 2024-03-28T18:25:34.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2024-03-28T18:25:34.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:34.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f67e7438d20] 2024-03-28T18:25:34.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2: gsignal() 2024-03-28T18:25:34.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 3: abort() 2024-03-28T18:25:34.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f67e86b14fb] 2024-03-28T18:25:34.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f67e86b1667] 2024-03-28T18:25:34.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5620293f8223] 2024-03-28T18:25:34.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5620292725d6] 2024-03-28T18:25:34.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x562029272da1] 2024-03-28T18:25:34.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5620292752d5] 2024-03-28T18:25:34.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5620292c5233] 2024-03-28T18:25:34.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5620292c7197] 2024-03-28T18:25:34.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5620291c611b] 2024-03-28T18:25:34.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5620291c66bc] 2024-03-28T18:25:34.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5620291b01ff] 2024-03-28T18:25:34.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f67e8943488] 2024-03-28T18:25:34.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 16: (DispatchQueue::entry()+0x51f) [0x7f67e894060f] 2024-03-28T18:25:34.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f67e8a13e81] 2024-03-28T18:25:34.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f67e742e1ca] 2024-03-28T18:25:34.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 19: clone() 2024-03-28T18:25:34.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:25:34.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2024-03-28T18:25:34.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: -1> 2024-03-28T18:25:34.032+0000 7f67dce00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f67dce00700 time 2024-03-28T18:25:34.031198+0000 2024-03-28T18:25:34.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:25:34.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2024-03-28T18:25:34.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:34.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f67e86b14a1] 2024-03-28T18:25:34.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f67e86b1667] 2024-03-28T18:25:34.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5620293f8223] 2024-03-28T18:25:34.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5620292725d6] 2024-03-28T18:25:34.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x562029272da1] 2024-03-28T18:25:34.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5620292752d5] 2024-03-28T18:25:34.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5620292c5233] 2024-03-28T18:25:34.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5620292c7197] 2024-03-28T18:25:34.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5620291c611b] 2024-03-28T18:25:34.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5620291c66bc] 2024-03-28T18:25:34.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5620291b01ff] 2024-03-28T18:25:34.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f67e8943488] 2024-03-28T18:25:34.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 13: (DispatchQueue::entry()+0x51f) [0x7f67e894060f] 2024-03-28T18:25:34.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f67e8a13e81] 2024-03-28T18:25:34.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f67e742e1ca] 2024-03-28T18:25:34.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 16: clone() 2024-03-28T18:25:34.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2024-03-28T18:25:34.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 0> 2024-03-28T18:25:34.035+0000 7f67dce00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:25:34.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: in thread 7f67dce00700 thread_name:ms_dispatch 2024-03-28T18:25:34.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2024-03-28T18:25:34.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:34.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f67e7438d20] 2024-03-28T18:25:34.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2: gsignal() 2024-03-28T18:25:34.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 3: abort() 2024-03-28T18:25:34.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f67e86b14fb] 2024-03-28T18:25:34.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f67e86b1667] 2024-03-28T18:25:34.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5620293f8223] 2024-03-28T18:25:34.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5620292725d6] 2024-03-28T18:25:34.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x562029272da1] 2024-03-28T18:25:34.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5620292752d5] 2024-03-28T18:25:34.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5620292c5233] 2024-03-28T18:25:34.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5620292c7197] 2024-03-28T18:25:34.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5620291c611b] 2024-03-28T18:25:34.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5620291c66bc] 2024-03-28T18:25:34.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5620291b01ff] 2024-03-28T18:25:34.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f67e8943488] 2024-03-28T18:25:34.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 16: (DispatchQueue::entry()+0x51f) [0x7f67e894060f] 2024-03-28T18:25:34.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f67e8a13e81] 2024-03-28T18:25:34.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f67e742e1ca] 2024-03-28T18:25:34.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 19: clone() 2024-03-28T18:25:34.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:25:34.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[90603]: 2024-03-28T18:25:34.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:34 smithi037 podman[90784]: 2024-03-28 18:25:34.174047876 +0000 UTC m=+0.033604570 container died 10e315ce6e24ebbe10e79b338df65a06c2d60a900376af6c777ffc2151877c30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-28T18:25:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:34 smithi059 ceph-mon[36190]: pgmap v496: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T18:25:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:34 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/505326814,v1:172.21.15.37:6837/505326814] up:rejoin 2024-03-28T18:25:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:34 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:35.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:34 smithi059 ceph-mon[36190]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-28T18:25:35.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:34 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:25:35.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:34 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/505326814,v1:172.21.15.37:6837/505326814] up:active 2024-03-28T18:25:35.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:34 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:25:35.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:34 smithi059 ceph-mon[36190]: osdmap e146: 12 total, 12 up, 12 in 2024-03-28T18:25:35.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:34 smithi170 ceph-mon[36280]: pgmap v496: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T18:25:35.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:34 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/505326814,v1:172.21.15.37:6837/505326814] up:rejoin 2024-03-28T18:25:35.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:34 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:35.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:34 smithi170 ceph-mon[36280]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-28T18:25:35.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:34 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:25:35.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:34 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/505326814,v1:172.21.15.37:6837/505326814] up:active 2024-03-28T18:25:35.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:34 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:25:35.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:34 smithi170 ceph-mon[36280]: osdmap e146: 12 total, 12 up, 12 in 2024-03-28T18:25:35.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-mon[31460]: pgmap v496: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T18:25:35.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/505326814,v1:172.21.15.37:6837/505326814] up:rejoin 2024-03-28T18:25:35.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:35.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-mon[31460]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-28T18:25:35.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:25:35.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/505326814,v1:172.21.15.37:6837/505326814] up:active 2024-03-28T18:25:35.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:25:35.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:34 smithi037 ceph-mon[31460]: osdmap e146: 12 total, 12 up, 12 in 2024-03-28T18:25:35.625 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:35 smithi037 podman[90784]: 2024-03-28 18:25:35.237437119 +0000 UTC m=+1.096993861 container remove 10e315ce6e24ebbe10e79b338df65a06c2d60a900376af6c777ffc2151877c30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-28T18:25:35.625 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:35 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:25:36.125 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:35 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-28T18:25:36.125 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:35 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.878s CPU time. 2024-03-28T18:25:36.722 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:25:36 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:25:36.462+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:25:37.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:36 smithi059 ceph-mon[36190]: pgmap v498: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:25:37.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:36 smithi170 ceph-mon[36280]: pgmap v498: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:25:37.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:36 smithi037 ceph-mon[31460]: pgmap v498: 97 pgs: 97 active+clean; 45 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:25:39.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:38 smithi059 ceph-mon[36190]: pgmap v499: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 383 B/s wr, 20 op/s 2024-03-28T18:25:39.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:38 smithi170 ceph-mon[36280]: pgmap v499: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 383 B/s wr, 20 op/s 2024-03-28T18:25:39.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:38 smithi037 ceph-mon[31460]: pgmap v499: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 383 B/s wr, 20 op/s 2024-03-28T18:25:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:40 smithi059 ceph-mon[36190]: pgmap v500: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 330 B/s wr, 18 op/s 2024-03-28T18:25:41.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:40 smithi170 ceph-mon[36280]: pgmap v500: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 330 B/s wr, 18 op/s 2024-03-28T18:25:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:40 smithi037 ceph-mon[31460]: pgmap v500: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 330 B/s wr, 18 op/s 2024-03-28T18:25:43.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:42 smithi059 ceph-mon[36190]: pgmap v501: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 307 B/s wr, 17 op/s 2024-03-28T18:25:43.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:42 smithi170 ceph-mon[36280]: pgmap v501: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 307 B/s wr, 17 op/s 2024-03-28T18:25:43.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:42 smithi037 ceph-mon[31460]: pgmap v501: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 307 B/s wr, 17 op/s 2024-03-28T18:25:45.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:44 smithi059 ceph-mon[36190]: pgmap v502: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:25:45.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:44 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:25:45.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:44 smithi170 ceph-mon[36280]: pgmap v502: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:25:45.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:44 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:25:45.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:44 smithi037 ceph-mon[31460]: pgmap v502: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:25:45.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:44 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:25:46.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:45 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:25:46.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:45 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:25:46.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:45 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:25:46.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:45 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:25:46.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:45 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:25:46.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:45 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:25:46.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:45 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:25:46.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:45 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:25:46.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:45 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:25:46.125 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:45 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 5. 2024-03-28T18:25:46.125 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:45 smithi037 systemd[1]: Stopped Ceph mds.g for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:25:46.125 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:45 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.878s CPU time. 2024-03-28T18:25:46.125 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:45 smithi037 systemd[1]: Starting Ceph mds.g for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:25:46.606 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:46 smithi037 podman[91005]: 2024-03-28 18:25:46.185053643 +0000 UTC m=+0.033947077 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:25:46.606 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:46 smithi037 podman[91005]: 2024-03-28 18:25:46.430500331 +0000 UTC m=+0.279393782 container create 507edadc1766f820c619aaed0189d72543bb87be3ca1298f64d8beba6caf67f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-28T18:25:47.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:46 smithi059 ceph-mon[36190]: pgmap v503: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 184 B/s wr, 14 op/s 2024-03-28T18:25:47.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:46 smithi059 ceph-mon[36190]: pgmap v504: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 213 B/s wr, 16 op/s 2024-03-28T18:25:47.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:46 smithi170 ceph-mon[36280]: pgmap v503: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 184 B/s wr, 14 op/s 2024-03-28T18:25:47.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:46 smithi170 ceph-mon[36280]: pgmap v504: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 213 B/s wr, 16 op/s 2024-03-28T18:25:47.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:46 smithi037 ceph-mon[31460]: pgmap v503: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 184 B/s wr, 14 op/s 2024-03-28T18:25:47.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:46 smithi037 ceph-mon[31460]: pgmap v504: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 213 B/s wr, 16 op/s 2024-03-28T18:25:47.125 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:46 smithi037 podman[91005]: 2024-03-28 18:25:46.718292453 +0000 UTC m=+0.567185911 container init 507edadc1766f820c619aaed0189d72543bb87be3ca1298f64d8beba6caf67f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-28T18:25:47.125 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:46 smithi037 podman[91005]: 2024-03-28 18:25:46.732403468 +0000 UTC m=+0.581296953 container start 507edadc1766f820c619aaed0189d72543bb87be3ca1298f64d8beba6caf67f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-28T18:25:47.126 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:46 smithi037 ceph-mds[91024]: -- 172.21.15.37:0/57859189 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e2ea0d7a40 con 0x55e2ea0ab000 2024-03-28T18:25:47.126 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:46 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: starting mds.g at 2024-03-28T18:25:47.126 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:46 smithi037 bash[91005]: 507edadc1766f820c619aaed0189d72543bb87be3ca1298f64d8beba6caf67f1 2024-03-28T18:25:47.126 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:25:46 smithi037 systemd[1]: Started Ceph mds.g for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:25:48.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:47 smithi059 ceph-mon[36190]: daemon mds.g restarted 2024-03-28T18:25:48.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:47 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:25:48.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:47 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:25:48.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:47 smithi059 ceph-mon[36190]: osdmap e147: 12 total, 12 up, 12 in 2024-03-28T18:25:48.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:47 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/838985367,v1:172.21.15.37:6837/838985367] up:boot 2024-03-28T18:25:48.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:47 smithi059 ceph-mon[36190]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-28T18:25:48.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:47 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:25:48.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:47 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:48.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:47 smithi170 ceph-mon[36280]: daemon mds.g restarted 2024-03-28T18:25:48.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:47 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:25:48.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:47 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:25:48.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:47 smithi170 ceph-mon[36280]: osdmap e147: 12 total, 12 up, 12 in 2024-03-28T18:25:48.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:47 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/838985367,v1:172.21.15.37:6837/838985367] up:boot 2024-03-28T18:25:48.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:47 smithi170 ceph-mon[36280]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-28T18:25:48.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:47 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:25:48.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:47 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:48.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:47 smithi037 ceph-mon[31460]: daemon mds.g restarted 2024-03-28T18:25:48.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:47 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:25:48.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:47 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:25:48.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:47 smithi037 ceph-mon[31460]: osdmap e147: 12 total, 12 up, 12 in 2024-03-28T18:25:48.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:47 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/838985367,v1:172.21.15.37:6837/838985367] up:boot 2024-03-28T18:25:48.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:47 smithi037 ceph-mon[31460]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-28T18:25:48.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:47 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:25:48.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:47 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:49.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:48 smithi037 ceph-mon[31460]: pgmap v506: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T18:25:49.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:48 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/2295994436,v1:172.21.15.37:6835/2295994436] up:resolve 2024-03-28T18:25:49.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:48 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:48 smithi059 ceph-mon[36190]: pgmap v506: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T18:25:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:48 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/2295994436,v1:172.21.15.37:6835/2295994436] up:resolve 2024-03-28T18:25:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:48 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:48 smithi170 ceph-mon[36280]: pgmap v506: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T18:25:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:48 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/2295994436,v1:172.21.15.37:6835/2295994436] up:resolve 2024-03-28T18:25:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:48 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:49 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:25:50.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:49 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/2295994436,v1:172.21.15.37:6835/2295994436] up:reconnect 2024-03-28T18:25:50.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:49 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:50.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:49 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999987 2024-03-28T18:25:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:49 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:25:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:49 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/2295994436,v1:172.21.15.37:6835/2295994436] up:reconnect 2024-03-28T18:25:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:49 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:49 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999987 2024-03-28T18:25:50.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:49 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:25:50.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:49 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/2295994436,v1:172.21.15.37:6835/2295994436] up:reconnect 2024-03-28T18:25:50.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:49 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:50.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:49 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999987 2024-03-28T18:25:51.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-mon[31460]: pgmap v507: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:25:51.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/2295994436,v1:172.21.15.37:6835/2295994436] up:rejoin 2024-03-28T18:25:51.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:51.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-mon[31460]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-28T18:25:51.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb47ac00700 time 2024-03-28T18:25:50.898130+0000 2024-03-28T18:25:51.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:25:51.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:51.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb48653b4a1] 2024-03-28T18:25:51.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb48653b667] 2024-03-28T18:25:51.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bfca5ff223] 2024-03-28T18:25:51.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bfca4795d6] 2024-03-28T18:25:51.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bfca479da1] 2024-03-28T18:25:51.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bfca47c2d5] 2024-03-28T18:25:51.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bfca4cc233] 2024-03-28T18:25:51.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bfca4ce197] 2024-03-28T18:25:51.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bfca3cd11b] 2024-03-28T18:25:51.126 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bfca3cd6bc] 2024-03-28T18:25:51.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bfca3b71ff] 2024-03-28T18:25:51.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb4867cd488] 2024-03-28T18:25:51.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 13: (DispatchQueue::entry()+0x51f) [0x7fb4867ca60f] 2024-03-28T18:25:51.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb48689de81] 2024-03-28T18:25:51.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb4852b81ca] 2024-03-28T18:25:51.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 16: clone() 2024-03-28T18:25:51.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2024-03-28T18:25:50.899+0000 7fb47ac00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb47ac00700 time 2024-03-28T18:25:50.898130+0000 2024-03-28T18:25:51.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:25:51.127 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2024-03-28T18:25:51.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:51.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb48653b4a1] 2024-03-28T18:25:51.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb48653b667] 2024-03-28T18:25:51.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bfca5ff223] 2024-03-28T18:25:51.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bfca4795d6] 2024-03-28T18:25:51.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bfca479da1] 2024-03-28T18:25:51.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bfca47c2d5] 2024-03-28T18:25:51.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bfca4cc233] 2024-03-28T18:25:51.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bfca4ce197] 2024-03-28T18:25:51.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bfca3cd11b] 2024-03-28T18:25:51.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bfca3cd6bc] 2024-03-28T18:25:51.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bfca3b71ff] 2024-03-28T18:25:51.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb4867cd488] 2024-03-28T18:25:51.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 13: (DispatchQueue::entry()+0x51f) [0x7fb4867ca60f] 2024-03-28T18:25:51.128 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb48689de81] 2024-03-28T18:25:51.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb4852b81ca] 2024-03-28T18:25:51.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 16: clone() 2024-03-28T18:25:51.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2024-03-28T18:25:51.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: *** Caught signal (Aborted) ** 2024-03-28T18:25:51.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: in thread 7fb47ac00700 thread_name:ms_dispatch 2024-03-28T18:25:51.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:51.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb4852c2d20] 2024-03-28T18:25:51.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2: gsignal() 2024-03-28T18:25:51.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 3: abort() 2024-03-28T18:25:51.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb48653b4fb] 2024-03-28T18:25:51.129 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb48653b667] 2024-03-28T18:25:51.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bfca5ff223] 2024-03-28T18:25:51.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bfca4795d6] 2024-03-28T18:25:51.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bfca479da1] 2024-03-28T18:25:51.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bfca47c2d5] 2024-03-28T18:25:51.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bfca4cc233] 2024-03-28T18:25:51.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bfca4ce197] 2024-03-28T18:25:51.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bfca3cd11b] 2024-03-28T18:25:51.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bfca3cd6bc] 2024-03-28T18:25:51.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bfca3b71ff] 2024-03-28T18:25:51.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb4867cd488] 2024-03-28T18:25:51.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 16: (DispatchQueue::entry()+0x51f) [0x7fb4867ca60f] 2024-03-28T18:25:51.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb48689de81] 2024-03-28T18:25:51.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb4852b81ca] 2024-03-28T18:25:51.130 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 19: clone() 2024-03-28T18:25:51.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2024-03-28T18:25:50.902+0000 7fb47ac00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:25:51.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: in thread 7fb47ac00700 thread_name:ms_dispatch 2024-03-28T18:25:51.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2024-03-28T18:25:51.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:51.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb4852c2d20] 2024-03-28T18:25:51.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2: gsignal() 2024-03-28T18:25:51.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 3: abort() 2024-03-28T18:25:51.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb48653b4fb] 2024-03-28T18:25:51.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb48653b667] 2024-03-28T18:25:51.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bfca5ff223] 2024-03-28T18:25:51.131 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bfca4795d6] 2024-03-28T18:25:51.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bfca479da1] 2024-03-28T18:25:51.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bfca47c2d5] 2024-03-28T18:25:51.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bfca4cc233] 2024-03-28T18:25:51.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bfca4ce197] 2024-03-28T18:25:51.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bfca3cd11b] 2024-03-28T18:25:51.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bfca3cd6bc] 2024-03-28T18:25:51.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bfca3b71ff] 2024-03-28T18:25:51.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb4867cd488] 2024-03-28T18:25:51.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 16: (DispatchQueue::entry()+0x51f) [0x7fb4867ca60f] 2024-03-28T18:25:51.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb48689de81] 2024-03-28T18:25:51.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb4852b81ca] 2024-03-28T18:25:51.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 19: clone() 2024-03-28T18:25:51.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:25:51.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2024-03-28T18:25:51.132 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: -1> 2024-03-28T18:25:50.899+0000 7fb47ac00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb47ac00700 time 2024-03-28T18:25:50.898130+0000 2024-03-28T18:25:51.133 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:25:51.133 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2024-03-28T18:25:51.133 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:51.133 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb48653b4a1] 2024-03-28T18:25:51.133 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb48653b667] 2024-03-28T18:25:51.133 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bfca5ff223] 2024-03-28T18:25:51.133 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bfca4795d6] 2024-03-28T18:25:51.133 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bfca479da1] 2024-03-28T18:25:51.133 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bfca47c2d5] 2024-03-28T18:25:51.133 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bfca4cc233] 2024-03-28T18:25:51.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bfca4ce197] 2024-03-28T18:25:51.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bfca3cd11b] 2024-03-28T18:25:51.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bfca3cd6bc] 2024-03-28T18:25:51.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bfca3b71ff] 2024-03-28T18:25:51.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb4867cd488] 2024-03-28T18:25:51.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 13: (DispatchQueue::entry()+0x51f) [0x7fb4867ca60f] 2024-03-28T18:25:51.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb48689de81] 2024-03-28T18:25:51.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb4852b81ca] 2024-03-28T18:25:51.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 16: clone() 2024-03-28T18:25:51.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2024-03-28T18:25:51.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 0> 2024-03-28T18:25:50.902+0000 7fb47ac00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:25:51.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: in thread 7fb47ac00700 thread_name:ms_dispatch 2024-03-28T18:25:51.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2024-03-28T18:25:51.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:51.134 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb4852c2d20] 2024-03-28T18:25:51.135 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2: gsignal() 2024-03-28T18:25:51.135 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 3: abort() 2024-03-28T18:25:51.135 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb48653b4fb] 2024-03-28T18:25:51.135 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb48653b667] 2024-03-28T18:25:51.135 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bfca5ff223] 2024-03-28T18:25:51.135 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bfca4795d6] 2024-03-28T18:25:51.135 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bfca479da1] 2024-03-28T18:25:51.135 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bfca47c2d5] 2024-03-28T18:25:51.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bfca4cc233] 2024-03-28T18:25:51.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bfca4ce197] 2024-03-28T18:25:51.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bfca3cd11b] 2024-03-28T18:25:51.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bfca3cd6bc] 2024-03-28T18:25:51.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bfca3b71ff] 2024-03-28T18:25:51.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb4867cd488] 2024-03-28T18:25:51.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 16: (DispatchQueue::entry()+0x51f) [0x7fb4867ca60f] 2024-03-28T18:25:51.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb48689de81] 2024-03-28T18:25:51.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb4852b81ca] 2024-03-28T18:25:51.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 19: clone() 2024-03-28T18:25:51.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:25:51.136 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2024-03-28T18:25:51.137 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: -1> 2024-03-28T18:25:50.899+0000 7fb47ac00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb47ac00700 time 2024-03-28T18:25:50.898130+0000 2024-03-28T18:25:51.137 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:25:51.137 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2024-03-28T18:25:51.137 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:51.137 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb48653b4a1] 2024-03-28T18:25:51.137 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb48653b667] 2024-03-28T18:25:51.137 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bfca5ff223] 2024-03-28T18:25:51.137 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bfca4795d6] 2024-03-28T18:25:51.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bfca479da1] 2024-03-28T18:25:51.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bfca47c2d5] 2024-03-28T18:25:51.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bfca4cc233] 2024-03-28T18:25:51.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bfca4ce197] 2024-03-28T18:25:51.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bfca3cd11b] 2024-03-28T18:25:51.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bfca3cd6bc] 2024-03-28T18:25:51.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bfca3b71ff] 2024-03-28T18:25:51.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb4867cd488] 2024-03-28T18:25:51.138 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 13: (DispatchQueue::entry()+0x51f) [0x7fb4867ca60f] 2024-03-28T18:25:51.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb48689de81] 2024-03-28T18:25:51.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb4852b81ca] 2024-03-28T18:25:51.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 16: clone() 2024-03-28T18:25:51.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2024-03-28T18:25:51.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 0> 2024-03-28T18:25:50.902+0000 7fb47ac00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:25:51.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: in thread 7fb47ac00700 thread_name:ms_dispatch 2024-03-28T18:25:51.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2024-03-28T18:25:51.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:25:51.139 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb4852c2d20] 2024-03-28T18:25:51.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2: gsignal() 2024-03-28T18:25:51.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 3: abort() 2024-03-28T18:25:51.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb48653b4fb] 2024-03-28T18:25:51.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb48653b667] 2024-03-28T18:25:51.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bfca5ff223] 2024-03-28T18:25:51.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bfca4795d6] 2024-03-28T18:25:51.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bfca479da1] 2024-03-28T18:25:51.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bfca47c2d5] 2024-03-28T18:25:51.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bfca4cc233] 2024-03-28T18:25:51.140 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bfca4ce197] 2024-03-28T18:25:51.141 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bfca3cd11b] 2024-03-28T18:25:51.141 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bfca3cd6bc] 2024-03-28T18:25:51.141 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bfca3b71ff] 2024-03-28T18:25:51.141 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb4867cd488] 2024-03-28T18:25:51.141 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 16: (DispatchQueue::entry()+0x51f) [0x7fb4867ca60f] 2024-03-28T18:25:51.141 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb48689de81] 2024-03-28T18:25:51.141 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb4852b81ca] 2024-03-28T18:25:51.141 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 19: clone() 2024-03-28T18:25:51.141 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:25:51.141 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:50 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[90746]: 2024-03-28T18:25:51.141 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:51 smithi037 podman[91058]: 2024-03-28 18:25:51.030374682 +0000 UTC m=+0.033419853 container died ef6a6c5175e8a5a14c61d8f3226775c471b3b922b7064c27fa2984c090f02da1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:25:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:50 smithi059 ceph-mon[36190]: pgmap v507: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:25:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:50 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/2295994436,v1:172.21.15.37:6835/2295994436] up:rejoin 2024-03-28T18:25:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:50 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:50 smithi059 ceph-mon[36190]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-28T18:25:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:50 smithi170 ceph-mon[36280]: pgmap v507: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:25:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:50 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/2295994436,v1:172.21.15.37:6835/2295994436] up:rejoin 2024-03-28T18:25:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:50 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:50 smithi170 ceph-mon[36280]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-28T18:25:51.838 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:25:51 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:25:51.463+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:25:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:51 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:25:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:51 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/2295994436,v1:172.21.15.37:6835/2295994436] up:active 2024-03-28T18:25:52.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:51 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:52.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:51 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:25:52.125 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:51 smithi037 podman[91058]: 2024-03-28 18:25:51.857984114 +0000 UTC m=+0.861029264 container remove ef6a6c5175e8a5a14c61d8f3226775c471b3b922b7064c27fa2984c090f02da1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-28T18:25:52.125 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:51 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:25:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:51 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:25:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:51 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/2295994436,v1:172.21.15.37:6835/2295994436] up:active 2024-03-28T18:25:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:51 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:51 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:25:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:51 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:25:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:51 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/2295994436,v1:172.21.15.37:6835/2295994436] up:active 2024-03-28T18:25:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:51 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:25:52.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:51 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:25:52.624 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:52 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-28T18:25:52.625 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:25:52 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Consumed 1.650s CPU time. 2024-03-28T18:25:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:52 smithi037 ceph-mon[31460]: pgmap v508: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:25:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:52 smithi059 ceph-mon[36190]: pgmap v508: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:25:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:52 smithi170 ceph-mon[36280]: pgmap v508: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:25:55.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:54 smithi037 ceph-mon[31460]: pgmap v509: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 255 B/s wr, 20 op/s 2024-03-28T18:25:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:54 smithi059 ceph-mon[36190]: pgmap v509: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 255 B/s wr, 20 op/s 2024-03-28T18:25:55.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:54 smithi170 ceph-mon[36280]: pgmap v509: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 255 B/s wr, 20 op/s 2024-03-28T18:25:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:56 smithi037 ceph-mon[31460]: pgmap v510: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:25:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:56 smithi059 ceph-mon[36190]: pgmap v510: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:25:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:56 smithi170 ceph-mon[36280]: pgmap v510: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 16 op/s 2024-03-28T18:25:59.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:25:58 smithi037 ceph-mon[31460]: pgmap v511: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 199 B/s wr, 15 op/s 2024-03-28T18:25:59.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:25:58 smithi059 ceph-mon[36190]: pgmap v511: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 199 B/s wr, 15 op/s 2024-03-28T18:25:59.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:25:58 smithi170 ceph-mon[36280]: pgmap v511: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 199 B/s wr, 15 op/s 2024-03-28T18:26:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:00 smithi059 ceph-mon[36190]: pgmap v512: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:26:01.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:00 smithi170 ceph-mon[36280]: pgmap v512: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:26:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:00 smithi037 ceph-mon[31460]: pgmap v512: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:26:02.744 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:02 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 4. 2024-03-28T18:26:02.744 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:02 smithi037 systemd[1]: Stopped Ceph mds.j for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:26:02.744 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:02 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Consumed 1.650s CPU time. 2024-03-28T18:26:02.744 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:02 smithi037 systemd[1]: Starting Ceph mds.j for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:26:03.077 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:02 smithi037 ceph-mon[31460]: pgmap v513: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:26:03.077 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:02 smithi037 podman[91144]: 2024-03-28 18:26:02.648949243 +0000 UTC m=+0.032198821 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:26:03.077 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:02 smithi037 podman[91144]: 2024-03-28 18:26:02.789046078 +0000 UTC m=+0.172295717 container create c6abd22ff4453b6c35b84ed2c4ea7420f1c4023bab0142faf08dba5a1ab08fbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-28T18:26:03.077 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:03 smithi037 podman[91144]: 2024-03-28 18:26:03.077150333 +0000 UTC m=+0.460399956 container init c6abd22ff4453b6c35b84ed2c4ea7420f1c4023bab0142faf08dba5a1ab08fbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:26:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:02 smithi059 ceph-mon[36190]: pgmap v513: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:26:03.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:02 smithi170 ceph-mon[36280]: pgmap v513: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-28T18:26:03.375 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:03 smithi037 podman[91144]: 2024-03-28 18:26:03.089819995 +0000 UTC m=+0.473069592 container start c6abd22ff4453b6c35b84ed2c4ea7420f1c4023bab0142faf08dba5a1ab08fbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:26:03.375 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:03 smithi037 ceph-mds[91163]: -- 172.21.15.37:0/3790758059 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55768a5ffa40 con 0x55768a5d3000 2024-03-28T18:26:03.375 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:03 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: starting mds.j at 2024-03-28T18:26:03.375 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:03 smithi037 bash[91144]: c6abd22ff4453b6c35b84ed2c4ea7420f1c4023bab0142faf08dba5a1ab08fbd 2024-03-28T18:26:03.375 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:03 smithi037 systemd[1]: Started Ceph mds.j for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:26:04.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:03 smithi059 ceph-mon[36190]: daemon mds.j restarted 2024-03-28T18:26:04.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:03 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:26:04.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:03 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:26:04.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:03 smithi059 ceph-mon[36190]: osdmap e148: 12 total, 12 up, 12 in 2024-03-28T18:26:04.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:03 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/257262966,v1:172.21.15.37:6835/257262966] up:boot 2024-03-28T18:26:04.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:03 smithi059 ceph-mon[36190]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-28T18:26:04.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:03 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:26:04.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:03 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:26:04.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:03 smithi170 ceph-mon[36280]: daemon mds.j restarted 2024-03-28T18:26:04.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:03 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:26:04.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:03 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:26:04.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:03 smithi170 ceph-mon[36280]: osdmap e148: 12 total, 12 up, 12 in 2024-03-28T18:26:04.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:03 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/257262966,v1:172.21.15.37:6835/257262966] up:boot 2024-03-28T18:26:04.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:03 smithi170 ceph-mon[36280]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-28T18:26:04.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:03 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:26:04.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:03 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:26:04.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:03 smithi037 ceph-mon[31460]: daemon mds.j restarted 2024-03-28T18:26:04.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:03 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:26:04.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:03 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:26:04.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:03 smithi037 ceph-mon[31460]: osdmap e148: 12 total, 12 up, 12 in 2024-03-28T18:26:04.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:03 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/257262966,v1:172.21.15.37:6835/257262966] up:boot 2024-03-28T18:26:04.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:03 smithi037 ceph-mon[31460]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-28T18:26:04.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:03 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:26:04.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:03 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:26:05.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:04 smithi059 ceph-mon[36190]: pgmap v514: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 17 op/s 2024-03-28T18:26:05.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:04 smithi170 ceph-mon[36280]: pgmap v514: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 17 op/s 2024-03-28T18:26:05.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:04 smithi037 ceph-mon[31460]: pgmap v514: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 17 op/s 2024-03-28T18:26:06.186 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:05 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/838985367,v1:172.21.15.37:6837/838985367] up:resolve 2024-03-28T18:26:06.186 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:05 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:26:06.187 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:05 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/838985367,v1:172.21.15.37:6837/838985367] up:resolve 2024-03-28T18:26:06.187 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:05 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:26:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:05 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/838985367,v1:172.21.15.37:6837/838985367] up:resolve 2024-03-28T18:26:06.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:05 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:26:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:06 smithi059 ceph-mon[36190]: pgmap v516: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:26:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:06 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:26:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:06 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/838985367,v1:172.21.15.37:6837/838985367] up:reconnect 2024-03-28T18:26:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:06 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:26:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:06 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899988 2024-03-28T18:26:07.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:06 smithi170 ceph-mon[36280]: pgmap v516: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:26:07.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:06 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:26:07.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:06 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/838985367,v1:172.21.15.37:6837/838985367] up:reconnect 2024-03-28T18:26:07.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:06 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:26:07.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:06 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899988 2024-03-28T18:26:07.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:06 smithi037 ceph-mon[31460]: pgmap v516: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:26:07.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:06 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:26:07.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:06 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/838985367,v1:172.21.15.37:6837/838985367] up:reconnect 2024-03-28T18:26:07.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:06 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:26:07.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:06 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00899988 2024-03-28T18:26:08.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:07 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/838985367,v1:172.21.15.37:6837/838985367] up:rejoin 2024-03-28T18:26:08.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:07 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:26:08.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:07 smithi059 ceph-mon[36190]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-28T18:26:08.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:07 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/838985367,v1:172.21.15.37:6837/838985367] up:rejoin 2024-03-28T18:26:08.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:07 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:26:08.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:07 smithi170 ceph-mon[36280]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-28T18:26:08.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:07 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/838985367,v1:172.21.15.37:6837/838985367] up:rejoin 2024-03-28T18:26:08.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:07 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:26:08.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:07 smithi037 ceph-mon[31460]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-28T18:26:08.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3d00800700 time 2024-03-28T18:26:08.006063+0000 2024-03-28T18:26:08.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:26:08.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:08.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3d0c1bc4a1] 2024-03-28T18:26:08.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3d0c1bc667] 2024-03-28T18:26:08.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e2e83b2223] 2024-03-28T18:26:08.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e2e822c5d6] 2024-03-28T18:26:08.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e2e822cda1] 2024-03-28T18:26:08.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e2e822f2d5] 2024-03-28T18:26:08.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e2e827f233] 2024-03-28T18:26:08.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e2e8281197] 2024-03-28T18:26:08.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e2e818011b] 2024-03-28T18:26:08.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e2e81806bc] 2024-03-28T18:26:08.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e2e816a1ff] 2024-03-28T18:26:08.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3d0c44e488] 2024-03-28T18:26:08.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 13: (DispatchQueue::entry()+0x51f) [0x7f3d0c44b60f] 2024-03-28T18:26:08.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3d0c51ee81] 2024-03-28T18:26:08.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3d0af391ca] 2024-03-28T18:26:08.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 16: clone() 2024-03-28T18:26:08.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: *** Caught signal (Aborted) ** 2024-03-28T18:26:08.377 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: in thread 7f3d00800700 thread_name:ms_dispatch 2024-03-28T18:26:08.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2024-03-28T18:26:08.007+0000 7f3d00800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3d00800700 time 2024-03-28T18:26:08.006063+0000 2024-03-28T18:26:08.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:26:08.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2024-03-28T18:26:08.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:08.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3d0c1bc4a1] 2024-03-28T18:26:08.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3d0c1bc667] 2024-03-28T18:26:08.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e2e83b2223] 2024-03-28T18:26:08.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e2e822c5d6] 2024-03-28T18:26:08.378 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e2e822cda1] 2024-03-28T18:26:08.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e2e822f2d5] 2024-03-28T18:26:08.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e2e827f233] 2024-03-28T18:26:08.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e2e8281197] 2024-03-28T18:26:08.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e2e818011b] 2024-03-28T18:26:08.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e2e81806bc] 2024-03-28T18:26:08.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e2e816a1ff] 2024-03-28T18:26:08.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3d0c44e488] 2024-03-28T18:26:08.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 13: (DispatchQueue::entry()+0x51f) [0x7f3d0c44b60f] 2024-03-28T18:26:08.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3d0c51ee81] 2024-03-28T18:26:08.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3d0af391ca] 2024-03-28T18:26:08.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 16: clone() 2024-03-28T18:26:08.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2024-03-28T18:26:08.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:08.379 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3d0af43d20] 2024-03-28T18:26:08.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2: gsignal() 2024-03-28T18:26:08.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 3: abort() 2024-03-28T18:26:08.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3d0c1bc4fb] 2024-03-28T18:26:08.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3d0c1bc667] 2024-03-28T18:26:08.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e2e83b2223] 2024-03-28T18:26:08.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e2e822c5d6] 2024-03-28T18:26:08.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e2e822cda1] 2024-03-28T18:26:08.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e2e822f2d5] 2024-03-28T18:26:08.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e2e827f233] 2024-03-28T18:26:08.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e2e8281197] 2024-03-28T18:26:08.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e2e818011b] 2024-03-28T18:26:08.380 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e2e81806bc] 2024-03-28T18:26:08.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e2e816a1ff] 2024-03-28T18:26:08.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3d0c44e488] 2024-03-28T18:26:08.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 16: (DispatchQueue::entry()+0x51f) [0x7f3d0c44b60f] 2024-03-28T18:26:08.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3d0c51ee81] 2024-03-28T18:26:08.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3d0af391ca] 2024-03-28T18:26:08.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 19: clone() 2024-03-28T18:26:08.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2024-03-28T18:26:08.010+0000 7f3d00800700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:26:08.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: in thread 7f3d00800700 thread_name:ms_dispatch 2024-03-28T18:26:08.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2024-03-28T18:26:08.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:08.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3d0af43d20] 2024-03-28T18:26:08.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2: gsignal() 2024-03-28T18:26:08.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 3: abort() 2024-03-28T18:26:08.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3d0c1bc4fb] 2024-03-28T18:26:08.381 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3d0c1bc667] 2024-03-28T18:26:08.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e2e83b2223] 2024-03-28T18:26:08.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e2e822c5d6] 2024-03-28T18:26:08.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e2e822cda1] 2024-03-28T18:26:08.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e2e822f2d5] 2024-03-28T18:26:08.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e2e827f233] 2024-03-28T18:26:08.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e2e8281197] 2024-03-28T18:26:08.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e2e818011b] 2024-03-28T18:26:08.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e2e81806bc] 2024-03-28T18:26:08.382 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e2e816a1ff] 2024-03-28T18:26:08.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3d0c44e488] 2024-03-28T18:26:08.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 16: (DispatchQueue::entry()+0x51f) [0x7f3d0c44b60f] 2024-03-28T18:26:08.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3d0c51ee81] 2024-03-28T18:26:08.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3d0af391ca] 2024-03-28T18:26:08.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 19: clone() 2024-03-28T18:26:08.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:26:08.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2024-03-28T18:26:08.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: -1> 2024-03-28T18:26:08.007+0000 7f3d00800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3d00800700 time 2024-03-28T18:26:08.006063+0000 2024-03-28T18:26:08.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:26:08.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2024-03-28T18:26:08.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:08.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3d0c1bc4a1] 2024-03-28T18:26:08.383 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3d0c1bc667] 2024-03-28T18:26:08.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e2e83b2223] 2024-03-28T18:26:08.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e2e822c5d6] 2024-03-28T18:26:08.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e2e822cda1] 2024-03-28T18:26:08.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e2e822f2d5] 2024-03-28T18:26:08.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e2e827f233] 2024-03-28T18:26:08.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e2e8281197] 2024-03-28T18:26:08.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e2e818011b] 2024-03-28T18:26:08.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e2e81806bc] 2024-03-28T18:26:08.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e2e816a1ff] 2024-03-28T18:26:08.384 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3d0c44e488] 2024-03-28T18:26:08.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 13: (DispatchQueue::entry()+0x51f) [0x7f3d0c44b60f] 2024-03-28T18:26:08.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3d0c51ee81] 2024-03-28T18:26:08.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3d0af391ca] 2024-03-28T18:26:08.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 16: clone() 2024-03-28T18:26:08.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2024-03-28T18:26:08.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 0> 2024-03-28T18:26:08.010+0000 7f3d00800700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:26:08.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: in thread 7f3d00800700 thread_name:ms_dispatch 2024-03-28T18:26:08.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2024-03-28T18:26:08.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:08.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3d0af43d20] 2024-03-28T18:26:08.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2: gsignal() 2024-03-28T18:26:08.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 3: abort() 2024-03-28T18:26:08.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3d0c1bc4fb] 2024-03-28T18:26:08.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3d0c1bc667] 2024-03-28T18:26:08.385 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e2e83b2223] 2024-03-28T18:26:08.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e2e822c5d6] 2024-03-28T18:26:08.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e2e822cda1] 2024-03-28T18:26:08.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e2e822f2d5] 2024-03-28T18:26:08.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e2e827f233] 2024-03-28T18:26:08.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e2e8281197] 2024-03-28T18:26:08.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e2e818011b] 2024-03-28T18:26:08.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e2e81806bc] 2024-03-28T18:26:08.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e2e816a1ff] 2024-03-28T18:26:08.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3d0c44e488] 2024-03-28T18:26:08.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 16: (DispatchQueue::entry()+0x51f) [0x7f3d0c44b60f] 2024-03-28T18:26:08.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3d0c51ee81] 2024-03-28T18:26:08.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3d0af391ca] 2024-03-28T18:26:08.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 19: clone() 2024-03-28T18:26:08.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:26:08.386 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2024-03-28T18:26:08.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: -1> 2024-03-28T18:26:08.007+0000 7f3d00800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3d00800700 time 2024-03-28T18:26:08.006063+0000 2024-03-28T18:26:08.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:26:08.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2024-03-28T18:26:08.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:08.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3d0c1bc4a1] 2024-03-28T18:26:08.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3d0c1bc667] 2024-03-28T18:26:08.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e2e83b2223] 2024-03-28T18:26:08.387 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e2e822c5d6] 2024-03-28T18:26:08.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e2e822cda1] 2024-03-28T18:26:08.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e2e822f2d5] 2024-03-28T18:26:08.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e2e827f233] 2024-03-28T18:26:08.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e2e8281197] 2024-03-28T18:26:08.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e2e818011b] 2024-03-28T18:26:08.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e2e81806bc] 2024-03-28T18:26:08.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e2e816a1ff] 2024-03-28T18:26:08.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3d0c44e488] 2024-03-28T18:26:08.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 13: (DispatchQueue::entry()+0x51f) [0x7f3d0c44b60f] 2024-03-28T18:26:08.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3d0c51ee81] 2024-03-28T18:26:08.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3d0af391ca] 2024-03-28T18:26:08.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 16: clone() 2024-03-28T18:26:08.388 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2024-03-28T18:26:08.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 0> 2024-03-28T18:26:08.010+0000 7f3d00800700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:26:08.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: in thread 7f3d00800700 thread_name:ms_dispatch 2024-03-28T18:26:08.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2024-03-28T18:26:08.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:08.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3d0af43d20] 2024-03-28T18:26:08.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2: gsignal() 2024-03-28T18:26:08.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 3: abort() 2024-03-28T18:26:08.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3d0c1bc4fb] 2024-03-28T18:26:08.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3d0c1bc667] 2024-03-28T18:26:08.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e2e83b2223] 2024-03-28T18:26:08.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e2e822c5d6] 2024-03-28T18:26:08.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e2e822cda1] 2024-03-28T18:26:08.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e2e822f2d5] 2024-03-28T18:26:08.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e2e827f233] 2024-03-28T18:26:08.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e2e8281197] 2024-03-28T18:26:08.389 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e2e818011b] 2024-03-28T18:26:08.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e2e81806bc] 2024-03-28T18:26:08.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e2e816a1ff] 2024-03-28T18:26:08.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3d0c44e488] 2024-03-28T18:26:08.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 16: (DispatchQueue::entry()+0x51f) [0x7f3d0c44b60f] 2024-03-28T18:26:08.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3d0c51ee81] 2024-03-28T18:26:08.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3d0af391ca] 2024-03-28T18:26:08.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 19: clone() 2024-03-28T18:26:08.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:26:08.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g[91020]: 2024-03-28T18:26:08.390 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 podman[91198]: 2024-03-28 18:26:08.146220909 +0000 UTC m=+0.032701431 container died 507edadc1766f820c619aaed0189d72543bb87be3ca1298f64d8beba6caf67f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:26:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:08 smithi059 ceph-mon[36190]: pgmap v517: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:26:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:08 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:26:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:08 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6836/838985367,v1:172.21.15.37:6837/838985367] up:active 2024-03-28T18:26:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:08 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:26:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:08 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:26:09.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:08 smithi170 ceph-mon[36280]: pgmap v517: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:26:09.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:08 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:26:09.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:08 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6836/838985367,v1:172.21.15.37:6837/838985367] up:active 2024-03-28T18:26:09.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:08 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:26:09.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:08 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:26:09.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-mon[31460]: pgmap v517: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:26:09.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:26:09.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6836/838985367,v1:172.21.15.37:6837/838985367] up:active 2024-03-28T18:26:09.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:26:09.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:08 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-03-28T18:26:09.375 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 podman[91198]: 2024-03-28 18:26:08.97318467 +0000 UTC m=+0.859665221 container remove 507edadc1766f820c619aaed0189d72543bb87be3ca1298f64d8beba6caf67f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-g, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-28T18:26:09.376 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:08 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:26:09.875 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:09 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-28T18:26:09.875 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:09 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.798s CPU time. 2024-03-28T18:26:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:10 smithi059 ceph-mon[36190]: pgmap v518: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:26:11.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:10 smithi170 ceph-mon[36280]: pgmap v518: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:26:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:10 smithi037 ceph-mon[31460]: pgmap v518: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:26:11.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:26:11 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:26:11.465+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:26:13.057 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:12 smithi037 ceph-mon[31460]: pgmap v519: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:26:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:12 smithi059 ceph-mon[36190]: pgmap v519: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:26:13.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:12 smithi170 ceph-mon[36280]: pgmap v519: 97 pgs: 97 active+clean; 45 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:26:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:13 smithi059 ceph-mon[36190]: pgmap v520: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 32 op/s 2024-03-28T18:26:14.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:13 smithi170 ceph-mon[36280]: pgmap v520: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 32 op/s 2024-03-28T18:26:14.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:13 smithi037 ceph-mon[31460]: pgmap v520: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 27 KiB/s wr, 32 op/s 2024-03-28T18:26:16.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:16 smithi059 ceph-mon[36190]: pgmap v521: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-28T18:26:16.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:16 smithi170 ceph-mon[36280]: pgmap v521: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-28T18:26:16.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:16 smithi037 ceph-mon[31460]: pgmap v521: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-28T18:26:18.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:18 smithi059 ceph-mon[36190]: pgmap v522: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:26:18.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:18 smithi170 ceph-mon[36280]: pgmap v522: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:26:18.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:18 smithi037 ceph-mon[31460]: pgmap v522: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:26:20.125 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:19 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 6. 2024-03-28T18:26:20.125 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:19 smithi037 systemd[1]: Stopped Ceph mds.g for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:26:20.125 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:19 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.798s CPU time. 2024-03-28T18:26:20.125 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:19 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Start request repeated too quickly. 2024-03-28T18:26:20.126 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:19 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-28T18:26:20.126 INFO:journalctl@ceph.mds.g.smithi037.stdout:Mar 28 18:26:19 smithi037 systemd[1]: Failed to start Ceph mds.g for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:26:20.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:20 smithi059 ceph-mon[36190]: pgmap v523: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:26:20.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:20 smithi170 ceph-mon[36280]: pgmap v523: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:26:20.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:20 smithi037 ceph-mon[31460]: pgmap v523: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:26:22.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:22 smithi059 ceph-mon[36190]: pgmap v524: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:26:22.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:22 smithi170 ceph-mon[36280]: pgmap v524: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:26:22.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:22 smithi037 ceph-mon[31460]: pgmap v524: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:26:24.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:24 smithi037 ceph-mon[31460]: pgmap v525: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:26:24.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:24 smithi037 ceph-mon[31460]: Replacing daemon mds.g as rank 1 with standby daemon mds.j 2024-03-28T18:26:24.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:24 smithi037 ceph-mon[31460]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-03-28T18:26:24.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:24 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:26:24.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:24 smithi037 ceph-mon[31460]: osdmap e149: 12 total, 12 up, 12 in 2024-03-28T18:26:24.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:24 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:24.537 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:24 smithi059 ceph-mon[36190]: pgmap v525: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:26:24.537 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:24 smithi059 ceph-mon[36190]: Replacing daemon mds.g as rank 1 with standby daemon mds.j 2024-03-28T18:26:24.537 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:24 smithi059 ceph-mon[36190]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-03-28T18:26:24.537 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:24 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:26:24.537 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:24 smithi059 ceph-mon[36190]: osdmap e149: 12 total, 12 up, 12 in 2024-03-28T18:26:24.537 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:24 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:24.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:24 smithi170 ceph-mon[36280]: pgmap v525: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 22 KiB/s wr, 30 op/s 2024-03-28T18:26:24.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:24 smithi170 ceph-mon[36280]: Replacing daemon mds.g as rank 1 with standby daemon mds.j 2024-03-28T18:26:24.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:24 smithi170 ceph-mon[36280]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-03-28T18:26:24.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:24 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:26:24.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:24 smithi170 ceph-mon[36280]: osdmap e149: 12 total, 12 up, 12 in 2024-03-28T18:26:24.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:24 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:26 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/257262966,v1:172.21.15.37:6835/257262966] up:resolve 2024-03-28T18:26:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:26 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:26 smithi059 ceph-mon[36190]: pgmap v527: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:26:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:26 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/257262966,v1:172.21.15.37:6835/257262966] up:resolve 2024-03-28T18:26:26.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:26 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:26.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:26 smithi170 ceph-mon[36280]: pgmap v527: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:26:26.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:26 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/257262966,v1:172.21.15.37:6835/257262966] up:resolve 2024-03-28T18:26:26.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:26 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:26.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:26 smithi037 ceph-mon[31460]: pgmap v527: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:26:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:27 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:26:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:27 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/257262966,v1:172.21.15.37:6835/257262966] up:reconnect 2024-03-28T18:26:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:27 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:27 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999987 2024-03-28T18:26:27.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:27 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:26:27.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:27 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/257262966,v1:172.21.15.37:6835/257262966] up:reconnect 2024-03-28T18:26:27.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:27 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:27.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:27 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999987 2024-03-28T18:26:27.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:27 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00299996 2024-03-28T18:26:27.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:27 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/257262966,v1:172.21.15.37:6835/257262966] up:reconnect 2024-03-28T18:26:27.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:27 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:27.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:27 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.00999987 2024-03-28T18:26:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:28 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/257262966,v1:172.21.15.37:6835/257262966] up:rejoin 2024-03-28T18:26:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:28 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:28 smithi059 ceph-mon[36190]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-28T18:26:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:28 smithi059 ceph-mon[36190]: pgmap v528: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:26:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:28 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/257262966,v1:172.21.15.37:6835/257262966] up:rejoin 2024-03-28T18:26:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:28 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:28 smithi170 ceph-mon[36280]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-28T18:26:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:28 smithi170 ceph-mon[36280]: pgmap v528: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:26:28.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/257262966,v1:172.21.15.37:6835/257262966] up:rejoin 2024-03-28T18:26:28.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:28.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-mon[31460]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-28T18:26:28.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-mon[31460]: pgmap v528: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:26:28.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe021200700 time 2024-03-28T18:26:28.117041+0000 2024-03-28T18:26:28.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:26:28.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:28.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe02cab74a1] 2024-03-28T18:26:28.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe02cab7667] 2024-03-28T18:26:28.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557688f60223] 2024-03-28T18:26:28.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557688dda5d6] 2024-03-28T18:26:28.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557688ddada1] 2024-03-28T18:26:28.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557688ddd2d5] 2024-03-28T18:26:28.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557688e2d233] 2024-03-28T18:26:28.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557688e2f197] 2024-03-28T18:26:28.376 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557688d2e11b] 2024-03-28T18:26:28.377 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557688d2e6bc] 2024-03-28T18:26:28.377 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557688d181ff] 2024-03-28T18:26:28.377 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe02cd49488] 2024-03-28T18:26:28.377 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 13: (DispatchQueue::entry()+0x51f) [0x7fe02cd4660f] 2024-03-28T18:26:28.377 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe02ce19e81] 2024-03-28T18:26:28.377 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe02b8341ca] 2024-03-28T18:26:28.377 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 16: clone() 2024-03-28T18:26:28.377 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2024-03-28T18:26:28.118+0000 7fe021200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe021200700 time 2024-03-28T18:26:28.117041+0000 2024-03-28T18:26:28.377 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:26:28.377 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2024-03-28T18:26:28.377 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:28.377 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe02cab74a1] 2024-03-28T18:26:28.377 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe02cab7667] 2024-03-28T18:26:28.378 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557688f60223] 2024-03-28T18:26:28.378 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557688dda5d6] 2024-03-28T18:26:28.378 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557688ddada1] 2024-03-28T18:26:28.378 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557688ddd2d5] 2024-03-28T18:26:28.378 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557688e2d233] 2024-03-28T18:26:28.378 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557688e2f197] 2024-03-28T18:26:28.378 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557688d2e11b] 2024-03-28T18:26:28.378 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557688d2e6bc] 2024-03-28T18:26:28.378 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557688d181ff] 2024-03-28T18:26:28.378 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe02cd49488] 2024-03-28T18:26:28.378 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 13: (DispatchQueue::entry()+0x51f) [0x7fe02cd4660f] 2024-03-28T18:26:28.378 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe02ce19e81] 2024-03-28T18:26:28.378 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe02b8341ca] 2024-03-28T18:26:28.378 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 16: clone() 2024-03-28T18:26:28.379 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2024-03-28T18:26:28.379 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: *** Caught signal (Aborted) ** 2024-03-28T18:26:28.379 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: in thread 7fe021200700 thread_name:ms_dispatch 2024-03-28T18:26:28.379 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:28.379 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe02b83ed20] 2024-03-28T18:26:28.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2: gsignal() 2024-03-28T18:26:28.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 3: abort() 2024-03-28T18:26:28.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe02cab74fb] 2024-03-28T18:26:28.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe02cab7667] 2024-03-28T18:26:28.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557688f60223] 2024-03-28T18:26:28.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557688dda5d6] 2024-03-28T18:26:28.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557688ddada1] 2024-03-28T18:26:28.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557688ddd2d5] 2024-03-28T18:26:28.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557688e2d233] 2024-03-28T18:26:28.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557688e2f197] 2024-03-28T18:26:28.380 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557688d2e11b] 2024-03-28T18:26:28.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557688d2e6bc] 2024-03-28T18:26:28.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557688d181ff] 2024-03-28T18:26:28.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe02cd49488] 2024-03-28T18:26:28.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 16: (DispatchQueue::entry()+0x51f) [0x7fe02cd4660f] 2024-03-28T18:26:28.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe02ce19e81] 2024-03-28T18:26:28.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe02b8341ca] 2024-03-28T18:26:28.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 19: clone() 2024-03-28T18:26:28.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2024-03-28T18:26:28.120+0000 7fe021200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:26:28.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: in thread 7fe021200700 thread_name:ms_dispatch 2024-03-28T18:26:28.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2024-03-28T18:26:28.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:28.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe02b83ed20] 2024-03-28T18:26:28.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2: gsignal() 2024-03-28T18:26:28.381 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 3: abort() 2024-03-28T18:26:28.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe02cab74fb] 2024-03-28T18:26:28.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe02cab7667] 2024-03-28T18:26:28.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557688f60223] 2024-03-28T18:26:28.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557688dda5d6] 2024-03-28T18:26:28.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557688ddada1] 2024-03-28T18:26:28.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557688ddd2d5] 2024-03-28T18:26:28.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557688e2d233] 2024-03-28T18:26:28.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557688e2f197] 2024-03-28T18:26:28.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557688d2e11b] 2024-03-28T18:26:28.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557688d2e6bc] 2024-03-28T18:26:28.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557688d181ff] 2024-03-28T18:26:28.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe02cd49488] 2024-03-28T18:26:28.382 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 16: (DispatchQueue::entry()+0x51f) [0x7fe02cd4660f] 2024-03-28T18:26:28.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe02ce19e81] 2024-03-28T18:26:28.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe02b8341ca] 2024-03-28T18:26:28.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 19: clone() 2024-03-28T18:26:28.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:26:28.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2024-03-28T18:26:28.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: -1> 2024-03-28T18:26:28.118+0000 7fe021200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe021200700 time 2024-03-28T18:26:28.117041+0000 2024-03-28T18:26:28.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:26:28.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2024-03-28T18:26:28.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:28.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe02cab74a1] 2024-03-28T18:26:28.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe02cab7667] 2024-03-28T18:26:28.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557688f60223] 2024-03-28T18:26:28.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557688dda5d6] 2024-03-28T18:26:28.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557688ddada1] 2024-03-28T18:26:28.383 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557688ddd2d5] 2024-03-28T18:26:28.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557688e2d233] 2024-03-28T18:26:28.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557688e2f197] 2024-03-28T18:26:28.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557688d2e11b] 2024-03-28T18:26:28.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557688d2e6bc] 2024-03-28T18:26:28.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557688d181ff] 2024-03-28T18:26:28.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe02cd49488] 2024-03-28T18:26:28.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 13: (DispatchQueue::entry()+0x51f) [0x7fe02cd4660f] 2024-03-28T18:26:28.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe02ce19e81] 2024-03-28T18:26:28.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe02b8341ca] 2024-03-28T18:26:28.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 16: clone() 2024-03-28T18:26:28.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2024-03-28T18:26:28.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 0> 2024-03-28T18:26:28.120+0000 7fe021200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:26:28.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: in thread 7fe021200700 thread_name:ms_dispatch 2024-03-28T18:26:28.384 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2024-03-28T18:26:28.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:28.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe02b83ed20] 2024-03-28T18:26:28.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2: gsignal() 2024-03-28T18:26:28.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 3: abort() 2024-03-28T18:26:28.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe02cab74fb] 2024-03-28T18:26:28.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe02cab7667] 2024-03-28T18:26:28.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557688f60223] 2024-03-28T18:26:28.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557688dda5d6] 2024-03-28T18:26:28.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557688ddada1] 2024-03-28T18:26:28.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557688ddd2d5] 2024-03-28T18:26:28.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557688e2d233] 2024-03-28T18:26:28.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557688e2f197] 2024-03-28T18:26:28.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557688d2e11b] 2024-03-28T18:26:28.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557688d2e6bc] 2024-03-28T18:26:28.385 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557688d181ff] 2024-03-28T18:26:28.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe02cd49488] 2024-03-28T18:26:28.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 16: (DispatchQueue::entry()+0x51f) [0x7fe02cd4660f] 2024-03-28T18:26:28.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe02ce19e81] 2024-03-28T18:26:28.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe02b8341ca] 2024-03-28T18:26:28.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 19: clone() 2024-03-28T18:26:28.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:26:28.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2024-03-28T18:26:28.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: -4> 2024-03-28T18:26:28.118+0000 7fe021200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe021200700 time 2024-03-28T18:26:28.117041+0000 2024-03-28T18:26:28.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:26:28.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2024-03-28T18:26:28.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:28.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe02cab74a1] 2024-03-28T18:26:28.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe02cab7667] 2024-03-28T18:26:28.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557688f60223] 2024-03-28T18:26:28.386 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557688dda5d6] 2024-03-28T18:26:28.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557688ddada1] 2024-03-28T18:26:28.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557688ddd2d5] 2024-03-28T18:26:28.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557688e2d233] 2024-03-28T18:26:28.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557688e2f197] 2024-03-28T18:26:28.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557688d2e11b] 2024-03-28T18:26:28.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557688d2e6bc] 2024-03-28T18:26:28.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557688d181ff] 2024-03-28T18:26:28.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe02cd49488] 2024-03-28T18:26:28.387 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 13: (DispatchQueue::entry()+0x51f) [0x7fe02cd4660f] 2024-03-28T18:26:28.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe02ce19e81] 2024-03-28T18:26:28.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe02b8341ca] 2024-03-28T18:26:28.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 16: clone() 2024-03-28T18:26:28.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2024-03-28T18:26:28.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: -3> 2024-03-28T18:26:28.120+0000 7fe021200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:26:28.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: in thread 7fe021200700 thread_name:ms_dispatch 2024-03-28T18:26:28.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2024-03-28T18:26:28.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:28.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe02b83ed20] 2024-03-28T18:26:28.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2: gsignal() 2024-03-28T18:26:28.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 3: abort() 2024-03-28T18:26:28.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe02cab74fb] 2024-03-28T18:26:28.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe02cab7667] 2024-03-28T18:26:28.388 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557688f60223] 2024-03-28T18:26:28.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557688dda5d6] 2024-03-28T18:26:28.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557688ddada1] 2024-03-28T18:26:28.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557688ddd2d5] 2024-03-28T18:26:28.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557688e2d233] 2024-03-28T18:26:28.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557688e2f197] 2024-03-28T18:26:28.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557688d2e11b] 2024-03-28T18:26:28.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557688d2e6bc] 2024-03-28T18:26:28.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557688d181ff] 2024-03-28T18:26:28.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe02cd49488] 2024-03-28T18:26:28.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 16: (DispatchQueue::entry()+0x51f) [0x7fe02cd4660f] 2024-03-28T18:26:28.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe02ce19e81] 2024-03-28T18:26:28.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe02b8341ca] 2024-03-28T18:26:28.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 19: clone() 2024-03-28T18:26:28.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:26:28.389 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91159]: 2024-03-28T18:26:28.390 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 podman[91255]: 2024-03-28 18:26:28.235239771 +0000 UTC m=+0.034226792 container died c6abd22ff4453b6c35b84ed2c4ea7420f1c4023bab0142faf08dba5a1ab08fbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-28T18:26:29.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:29 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:26:29.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:29 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/257262966,v1:172.21.15.37:6835/257262966] up:active 2024-03-28T18:26:29.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:29 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:29.375 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:28 smithi037 podman[91255]: 2024-03-28 18:26:28.995449281 +0000 UTC m=+0.794436346 container remove c6abd22ff4453b6c35b84ed2c4ea7420f1c4023bab0142faf08dba5a1ab08fbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True) 2024-03-28T18:26:29.375 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:29 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:26:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:29 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:26:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:29 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/257262966,v1:172.21.15.37:6835/257262966] up:active 2024-03-28T18:26:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:29 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:29.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:29 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:26:29.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:29 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/257262966,v1:172.21.15.37:6835/257262966] up:active 2024-03-28T18:26:29.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:29 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:29.875 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:29 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-28T18:26:29.875 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:29 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Consumed 1.815s CPU time. 2024-03-28T18:26:30.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:30 smithi037 ceph-mon[31460]: pgmap v529: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:26:30.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:30 smithi059 ceph-mon[36190]: pgmap v529: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:26:30.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:30 smithi170 ceph-mon[36280]: pgmap v529: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:26:31.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:26:31 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:26:31.467+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:26:32.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:32 smithi037 ceph-mon[31460]: pgmap v530: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:26:32.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:32 smithi059 ceph-mon[36190]: pgmap v530: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:26:32.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:32 smithi170 ceph-mon[36280]: pgmap v530: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T18:26:34.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:34 smithi059 ceph-mon[36190]: pgmap v531: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 27 KiB/s wr, 18 op/s 2024-03-28T18:26:34.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:34 smithi170 ceph-mon[36280]: pgmap v531: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 27 KiB/s wr, 18 op/s 2024-03-28T18:26:34.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:34 smithi037 ceph-mon[31460]: pgmap v531: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 27 KiB/s wr, 18 op/s 2024-03-28T18:26:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:36 smithi059 ceph-mon[36190]: pgmap v532: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 24 KiB/s wr, 16 op/s 2024-03-28T18:26:36.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:36 smithi170 ceph-mon[36280]: pgmap v532: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 24 KiB/s wr, 16 op/s 2024-03-28T18:26:36.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:36 smithi037 ceph-mon[31460]: pgmap v532: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 24 KiB/s wr, 16 op/s 2024-03-28T18:26:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:37 smithi059 ceph-mon[36190]: osdmap e150: 12 total, 12 up, 12 in 2024-03-28T18:26:37.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:37 smithi170 ceph-mon[36280]: osdmap e150: 12 total, 12 up, 12 in 2024-03-28T18:26:37.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:37 smithi037 ceph-mon[31460]: osdmap e150: 12 total, 12 up, 12 in 2024-03-28T18:26:38.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:38 smithi059 ceph-mon[36190]: pgmap v534: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 27 KiB/s wr, 17 op/s 2024-03-28T18:26:38.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:38 smithi170 ceph-mon[36280]: pgmap v534: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 27 KiB/s wr, 17 op/s 2024-03-28T18:26:38.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:38 smithi037 ceph-mon[31460]: pgmap v534: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 27 KiB/s wr, 17 op/s 2024-03-28T18:26:39.874 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:39 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 5. 2024-03-28T18:26:39.874 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:39 smithi037 systemd[1]: Stopped Ceph mds.j for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:26:39.875 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:39 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Consumed 1.815s CPU time. 2024-03-28T18:26:39.875 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:39 smithi037 systemd[1]: Starting Ceph mds.j for b5ad257c-ed2c-11ee-95d0-87774f69a715... 2024-03-28T18:26:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:40 smithi059 ceph-mon[36190]: osdmap e151: 12 total, 12 up, 12 in 2024-03-28T18:26:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:40 smithi059 ceph-mon[36190]: pgmap v536: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 34 KiB/s wr, 21 op/s 2024-03-28T18:26:40.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:40 smithi170 ceph-mon[36280]: osdmap e151: 12 total, 12 up, 12 in 2024-03-28T18:26:40.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:40 smithi170 ceph-mon[36280]: pgmap v536: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 34 KiB/s wr, 21 op/s 2024-03-28T18:26:40.344 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:40 smithi037 ceph-mon[31460]: osdmap e151: 12 total, 12 up, 12 in 2024-03-28T18:26:40.344 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:40 smithi037 ceph-mon[31460]: pgmap v536: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 34 KiB/s wr, 21 op/s 2024-03-28T18:26:40.345 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:40 smithi037 podman[91343]: 2024-03-28 18:26:39.931678785 +0000 UTC m=+0.035133504 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:26:40.345 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:40 smithi037 podman[91343]: 2024-03-28 18:26:40.063441 +0000 UTC m=+0.166895708 container create 1a3260a4518140092c5ad60fa8d0e5644f9182a1c1a582855462965ae847b747 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:26:40.345 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:40 smithi037 podman[91343]: 2024-03-28 18:26:40.344413201 +0000 UTC m=+0.447867916 container init 1a3260a4518140092c5ad60fa8d0e5644f9182a1c1a582855462965ae847b747 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:26:40.624 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:40 smithi037 podman[91343]: 2024-03-28 18:26:40.358343818 +0000 UTC m=+0.461798525 container start 1a3260a4518140092c5ad60fa8d0e5644f9182a1c1a582855462965ae847b747 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-28T18:26:40.625 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:40 smithi037 ceph-mds[91361]: -- 172.21.15.37:0/1213279664 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5603b84e9a40 con 0x5603b84bcc00 2024-03-28T18:26:40.625 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:40 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: starting mds.j at 2024-03-28T18:26:40.625 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:40 smithi037 bash[91343]: 1a3260a4518140092c5ad60fa8d0e5644f9182a1c1a582855462965ae847b747 2024-03-28T18:26:40.625 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:40 smithi037 systemd[1]: Started Ceph mds.j for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:26:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:41 smithi059 ceph-mon[36190]: daemon mds.j restarted 2024-03-28T18:26:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:41 smithi059 ceph-mon[36190]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:26:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:41 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:26:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:41 smithi059 ceph-mon[36190]: osdmap e152: 12 total, 12 up, 12 in 2024-03-28T18:26:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:41 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/1875501487,v1:172.21.15.37:6835/1875501487] up:boot 2024-03-28T18:26:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:41 smithi059 ceph-mon[36190]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-28T18:26:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:41 smithi059 ceph-mon[36190]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:26:41.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:41 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:41 smithi170 ceph-mon[36280]: daemon mds.j restarted 2024-03-28T18:26:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:41 smithi170 ceph-mon[36280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:26:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:41 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:26:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:41 smithi170 ceph-mon[36280]: osdmap e152: 12 total, 12 up, 12 in 2024-03-28T18:26:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:41 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/1875501487,v1:172.21.15.37:6835/1875501487] up:boot 2024-03-28T18:26:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:41 smithi170 ceph-mon[36280]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-28T18:26:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:41 smithi170 ceph-mon[36280]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:26:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:41 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:41.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:41 smithi037 ceph-mon[31460]: daemon mds.j restarted 2024-03-28T18:26:41.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:41 smithi037 ceph-mon[31460]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:26:41.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:41 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:26:41.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:41 smithi037 ceph-mon[31460]: osdmap e152: 12 total, 12 up, 12 in 2024-03-28T18:26:41.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:41 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/1875501487,v1:172.21.15.37:6835/1875501487] up:boot 2024-03-28T18:26:41.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:41 smithi037 ceph-mon[31460]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-28T18:26:41.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:41 smithi037 ceph-mon[31460]: fsmap cephfs:2/3 {0=a=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:26:41.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:41 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:42.776 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:42 smithi037 ceph-mon[31460]: pgmap v538: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:26:42.776 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:42 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/1875501487,v1:172.21.15.37:6835/1875501487] up:resolve 2024-03-28T18:26:42.776 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:42 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:42.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:42 smithi059 ceph-mon[36190]: pgmap v538: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:26:42.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:42 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/1875501487,v1:172.21.15.37:6835/1875501487] up:resolve 2024-03-28T18:26:42.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:42 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:42.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:42 smithi170 ceph-mon[36280]: pgmap v538: 97 pgs: 97 active+clean; 45 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:26:42.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:42 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/1875501487,v1:172.21.15.37:6835/1875501487] up:resolve 2024-03-28T18:26:42.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:42 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:43.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:43 smithi059 ceph-mon[36190]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199997 2024-03-28T18:26:43.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:43 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/1875501487,v1:172.21.15.37:6835/1875501487] up:reconnect 2024-03-28T18:26:43.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:43 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:43.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:43 smithi059 ceph-mon[36190]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0109999 2024-03-28T18:26:43.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:43 smithi170 ceph-mon[36280]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199997 2024-03-28T18:26:43.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:43 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/1875501487,v1:172.21.15.37:6835/1875501487] up:reconnect 2024-03-28T18:26:43.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:43 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:43.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:43 smithi170 ceph-mon[36280]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0109999 2024-03-28T18:26:43.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:43 smithi037 ceph-mon[31460]: reconnect by client.25553 172.21.15.37:0/4218791617 after 0.00199997 2024-03-28T18:26:43.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:43 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/1875501487,v1:172.21.15.37:6835/1875501487] up:reconnect 2024-03-28T18:26:43.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:43 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:43.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:43 smithi037 ceph-mon[31460]: reconnect by client.25216 v1:172.21.15.37:0/167536383 after 0.0109999 2024-03-28T18:26:44.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:44 smithi059 ceph-mon[36190]: pgmap v539: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 295 B/s wr, 7 op/s 2024-03-28T18:26:44.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:44 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/1875501487,v1:172.21.15.37:6835/1875501487] up:rejoin 2024-03-28T18:26:44.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:44 smithi059 ceph-mon[36190]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:44.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:44 smithi059 ceph-mon[36190]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-28T18:26:44.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:44 smithi170 ceph-mon[36280]: pgmap v539: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 295 B/s wr, 7 op/s 2024-03-28T18:26:44.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:44 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/1875501487,v1:172.21.15.37:6835/1875501487] up:rejoin 2024-03-28T18:26:44.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:44 smithi170 ceph-mon[36280]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:44.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:44 smithi170 ceph-mon[36280]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-28T18:26:44.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-mon[31460]: pgmap v539: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 295 B/s wr, 7 op/s 2024-03-28T18:26:44.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/1875501487,v1:172.21.15.37:6835/1875501487] up:rejoin 2024-03-28T18:26:44.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-mon[31460]: fsmap cephfs:3/3 {0=a=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:44.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-mon[31460]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-28T18:26:44.876 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f38e5600700 time 2024-03-28T18:26:44.520108+0000 2024-03-28T18:26:44.876 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:26:44.876 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:44.876 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f38f0f5a4a1] 2024-03-28T18:26:44.876 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f38f0f5a667] 2024-03-28T18:26:44.876 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5603b6098223] 2024-03-28T18:26:44.876 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5603b5f125d6] 2024-03-28T18:26:44.876 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5603b5f12da1] 2024-03-28T18:26:44.876 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5603b5f152d5] 2024-03-28T18:26:44.876 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5603b5f65233] 2024-03-28T18:26:44.876 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5603b5f67197] 2024-03-28T18:26:44.876 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5603b5e6611b] 2024-03-28T18:26:44.877 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5603b5e666bc] 2024-03-28T18:26:44.877 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5603b5e501ff] 2024-03-28T18:26:44.877 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f38f11ec488] 2024-03-28T18:26:44.877 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 13: (DispatchQueue::entry()+0x51f) [0x7f38f11e960f] 2024-03-28T18:26:44.877 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f38f12bce81] 2024-03-28T18:26:44.877 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f38efcd71ca] 2024-03-28T18:26:44.877 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 16: clone() 2024-03-28T18:26:44.877 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2024-03-28T18:26:44.522+0000 7f38e5600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f38e5600700 time 2024-03-28T18:26:44.520108+0000 2024-03-28T18:26:44.878 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:26:44.878 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2024-03-28T18:26:44.878 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:44.878 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f38f0f5a4a1] 2024-03-28T18:26:44.878 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f38f0f5a667] 2024-03-28T18:26:44.878 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5603b6098223] 2024-03-28T18:26:44.878 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5603b5f125d6] 2024-03-28T18:26:44.878 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5603b5f12da1] 2024-03-28T18:26:44.878 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5603b5f152d5] 2024-03-28T18:26:44.878 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5603b5f65233] 2024-03-28T18:26:44.878 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5603b5f67197] 2024-03-28T18:26:44.878 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5603b5e6611b] 2024-03-28T18:26:44.878 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5603b5e666bc] 2024-03-28T18:26:44.879 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5603b5e501ff] 2024-03-28T18:26:44.879 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f38f11ec488] 2024-03-28T18:26:44.879 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 13: (DispatchQueue::entry()+0x51f) [0x7f38f11e960f] 2024-03-28T18:26:44.879 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f38f12bce81] 2024-03-28T18:26:44.879 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f38efcd71ca] 2024-03-28T18:26:44.879 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 16: clone() 2024-03-28T18:26:44.879 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2024-03-28T18:26:44.879 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: *** Caught signal (Aborted) ** 2024-03-28T18:26:44.879 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: in thread 7f38e5600700 thread_name:ms_dispatch 2024-03-28T18:26:44.879 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:44.879 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f38efce1d20] 2024-03-28T18:26:44.880 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2: gsignal() 2024-03-28T18:26:44.880 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 3: abort() 2024-03-28T18:26:44.880 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f38f0f5a4fb] 2024-03-28T18:26:44.880 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f38f0f5a667] 2024-03-28T18:26:44.880 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5603b6098223] 2024-03-28T18:26:44.880 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5603b5f125d6] 2024-03-28T18:26:44.880 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5603b5f12da1] 2024-03-28T18:26:44.880 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5603b5f152d5] 2024-03-28T18:26:44.880 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5603b5f65233] 2024-03-28T18:26:44.880 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5603b5f67197] 2024-03-28T18:26:44.880 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5603b5e6611b] 2024-03-28T18:26:44.880 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5603b5e666bc] 2024-03-28T18:26:44.880 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5603b5e501ff] 2024-03-28T18:26:44.881 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f38f11ec488] 2024-03-28T18:26:44.881 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 16: (DispatchQueue::entry()+0x51f) [0x7f38f11e960f] 2024-03-28T18:26:44.881 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f38f12bce81] 2024-03-28T18:26:44.881 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f38efcd71ca] 2024-03-28T18:26:44.881 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 19: clone() 2024-03-28T18:26:44.881 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2024-03-28T18:26:44.526+0000 7f38e5600700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:26:44.881 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: in thread 7f38e5600700 thread_name:ms_dispatch 2024-03-28T18:26:44.881 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2024-03-28T18:26:44.881 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:44.881 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f38efce1d20] 2024-03-28T18:26:44.881 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2: gsignal() 2024-03-28T18:26:44.881 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 3: abort() 2024-03-28T18:26:44.881 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f38f0f5a4fb] 2024-03-28T18:26:44.881 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f38f0f5a667] 2024-03-28T18:26:44.882 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5603b6098223] 2024-03-28T18:26:44.882 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5603b5f125d6] 2024-03-28T18:26:44.882 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5603b5f12da1] 2024-03-28T18:26:44.882 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5603b5f152d5] 2024-03-28T18:26:44.882 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5603b5f65233] 2024-03-28T18:26:44.882 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5603b5f67197] 2024-03-28T18:26:44.882 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5603b5e6611b] 2024-03-28T18:26:44.883 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5603b5e666bc] 2024-03-28T18:26:44.883 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5603b5e501ff] 2024-03-28T18:26:44.883 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f38f11ec488] 2024-03-28T18:26:44.883 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 16: (DispatchQueue::entry()+0x51f) [0x7f38f11e960f] 2024-03-28T18:26:44.883 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f38f12bce81] 2024-03-28T18:26:44.883 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f38efcd71ca] 2024-03-28T18:26:44.883 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 19: clone() 2024-03-28T18:26:44.883 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:26:44.883 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2024-03-28T18:26:44.883 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: -1> 2024-03-28T18:26:44.522+0000 7f38e5600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f38e5600700 time 2024-03-28T18:26:44.520108+0000 2024-03-28T18:26:44.883 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:26:44.883 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2024-03-28T18:26:44.884 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:44.884 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f38f0f5a4a1] 2024-03-28T18:26:44.884 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f38f0f5a667] 2024-03-28T18:26:44.884 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5603b6098223] 2024-03-28T18:26:44.884 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5603b5f125d6] 2024-03-28T18:26:44.884 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5603b5f12da1] 2024-03-28T18:26:44.884 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5603b5f152d5] 2024-03-28T18:26:44.884 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5603b5f65233] 2024-03-28T18:26:44.884 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5603b5f67197] 2024-03-28T18:26:44.884 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5603b5e6611b] 2024-03-28T18:26:44.884 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5603b5e666bc] 2024-03-28T18:26:44.884 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5603b5e501ff] 2024-03-28T18:26:44.884 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f38f11ec488] 2024-03-28T18:26:44.884 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 13: (DispatchQueue::entry()+0x51f) [0x7f38f11e960f] 2024-03-28T18:26:44.884 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f38f12bce81] 2024-03-28T18:26:44.885 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f38efcd71ca] 2024-03-28T18:26:44.885 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 16: clone() 2024-03-28T18:26:44.885 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2024-03-28T18:26:44.885 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 0> 2024-03-28T18:26:44.526+0000 7f38e5600700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:26:44.885 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: in thread 7f38e5600700 thread_name:ms_dispatch 2024-03-28T18:26:44.885 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2024-03-28T18:26:44.885 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:44.885 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f38efce1d20] 2024-03-28T18:26:44.885 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2: gsignal() 2024-03-28T18:26:44.885 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 3: abort() 2024-03-28T18:26:44.885 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f38f0f5a4fb] 2024-03-28T18:26:44.885 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f38f0f5a667] 2024-03-28T18:26:44.885 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5603b6098223] 2024-03-28T18:26:44.885 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5603b5f125d6] 2024-03-28T18:26:44.886 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5603b5f12da1] 2024-03-28T18:26:44.886 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5603b5f152d5] 2024-03-28T18:26:44.887 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5603b5f65233] 2024-03-28T18:26:44.887 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5603b5f67197] 2024-03-28T18:26:44.887 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5603b5e6611b] 2024-03-28T18:26:44.887 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5603b5e666bc] 2024-03-28T18:26:44.887 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5603b5e501ff] 2024-03-28T18:26:44.887 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f38f11ec488] 2024-03-28T18:26:44.887 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 16: (DispatchQueue::entry()+0x51f) [0x7f38f11e960f] 2024-03-28T18:26:44.887 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f38f12bce81] 2024-03-28T18:26:44.887 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f38efcd71ca] 2024-03-28T18:26:44.887 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 19: clone() 2024-03-28T18:26:44.887 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:26:44.887 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2024-03-28T18:26:44.887 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: -1> 2024-03-28T18:26:44.522+0000 7f38e5600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f38e5600700 time 2024-03-28T18:26:44.520108+0000 2024-03-28T18:26:44.887 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:26:44.888 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2024-03-28T18:26:44.888 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:44.888 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f38f0f5a4a1] 2024-03-28T18:26:44.888 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f38f0f5a667] 2024-03-28T18:26:44.888 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5603b6098223] 2024-03-28T18:26:44.888 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5603b5f125d6] 2024-03-28T18:26:44.888 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5603b5f12da1] 2024-03-28T18:26:44.888 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5603b5f152d5] 2024-03-28T18:26:44.888 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5603b5f65233] 2024-03-28T18:26:44.888 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5603b5f67197] 2024-03-28T18:26:44.888 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5603b5e6611b] 2024-03-28T18:26:44.888 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5603b5e666bc] 2024-03-28T18:26:44.888 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5603b5e501ff] 2024-03-28T18:26:44.888 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f38f11ec488] 2024-03-28T18:26:44.888 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 13: (DispatchQueue::entry()+0x51f) [0x7f38f11e960f] 2024-03-28T18:26:44.888 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f38f12bce81] 2024-03-28T18:26:44.889 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f38efcd71ca] 2024-03-28T18:26:44.889 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 16: clone() 2024-03-28T18:26:44.889 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2024-03-28T18:26:44.889 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 0> 2024-03-28T18:26:44.526+0000 7f38e5600700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:26:44.889 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: in thread 7f38e5600700 thread_name:ms_dispatch 2024-03-28T18:26:44.889 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2024-03-28T18:26:44.889 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:26:44.889 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f38efce1d20] 2024-03-28T18:26:44.889 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2: gsignal() 2024-03-28T18:26:44.889 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 3: abort() 2024-03-28T18:26:44.889 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f38f0f5a4fb] 2024-03-28T18:26:44.889 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f38f0f5a667] 2024-03-28T18:26:44.889 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5603b6098223] 2024-03-28T18:26:44.889 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5603b5f125d6] 2024-03-28T18:26:44.889 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5603b5f12da1] 2024-03-28T18:26:44.890 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5603b5f152d5] 2024-03-28T18:26:44.890 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5603b5f65233] 2024-03-28T18:26:44.890 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5603b5f67197] 2024-03-28T18:26:44.890 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5603b5e6611b] 2024-03-28T18:26:44.890 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5603b5e666bc] 2024-03-28T18:26:44.890 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5603b5e501ff] 2024-03-28T18:26:44.890 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f38f11ec488] 2024-03-28T18:26:44.890 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 16: (DispatchQueue::entry()+0x51f) [0x7f38f11e960f] 2024-03-28T18:26:44.890 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f38f12bce81] 2024-03-28T18:26:44.890 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f38efcd71ca] 2024-03-28T18:26:44.890 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 19: clone() 2024-03-28T18:26:44.890 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:26:44.890 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j[91357]: 2024-03-28T18:26:44.890 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:44 smithi037 podman[91395]: 2024-03-28 18:26:44.660523547 +0000 UTC m=+0.031186986 container died 1a3260a4518140092c5ad60fa8d0e5644f9182a1c1a582855462965ae847b747 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-28T18:26:45.769 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:45 smithi037 ceph-mon[31460]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:26:45.769 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:45 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6834/1875501487,v1:172.21.15.37:6835/1875501487] up:active 2024-03-28T18:26:45.769 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:45 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:45.769 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:45 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:26:45.769 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:45 smithi037 podman[91395]: 2024-03-28 18:26:45.388805302 +0000 UTC m=+0.759468808 container remove 1a3260a4518140092c5ad60fa8d0e5644f9182a1c1a582855462965ae847b747 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-j, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-28T18:26:45.769 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:45 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:26:45.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:45 smithi059 ceph-mon[36190]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:26:45.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:45 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6834/1875501487,v1:172.21.15.37:6835/1875501487] up:active 2024-03-28T18:26:45.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:45 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:45.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:45 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:26:45.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:45 smithi170 ceph-mon[36280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:26:45.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:45 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6834/1875501487,v1:172.21.15.37:6835/1875501487] up:active 2024-03-28T18:26:45.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:45 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 2024-03-28T18:26:45.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:45 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:26:46.124 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:45 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-28T18:26:46.125 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:45 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Consumed 1.709s CPU time. 2024-03-28T18:26:46.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:46 smithi059 ceph-mon[36190]: pgmap v540: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:26:46.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:46 smithi170 ceph-mon[36280]: pgmap v540: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:26:46.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:46 smithi037 ceph-mon[31460]: pgmap v540: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:26:46.875 INFO:journalctl@ceph.mds.a.smithi037.stdout:Mar 28 18:26:46 smithi037 ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715-mds-a[71061]: 2024-03-28T18:26:46.468+0000 7f1e66800700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:26:47.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:47 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:26:47.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:47 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:26:47.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:47 smithi059 ceph-mon[36190]: pgmap v541: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 405 B/s wr, 20 op/s 2024-03-28T18:26:47.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:47 smithi059 ceph-mon[36190]: pgmap v542: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 409 B/s wr, 20 op/s 2024-03-28T18:26:47.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:47 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:26:47.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:47 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:26:47.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:47 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:26:47.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:47 smithi170 ceph-mon[36280]: pgmap v541: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 405 B/s wr, 20 op/s 2024-03-28T18:26:47.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:47 smithi170 ceph-mon[36280]: pgmap v542: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 409 B/s wr, 20 op/s 2024-03-28T18:26:47.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:47 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:26:47.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:47 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:26:47.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:47 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:26:47.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:47 smithi037 ceph-mon[31460]: pgmap v541: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 405 B/s wr, 20 op/s 2024-03-28T18:26:47.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:47 smithi037 ceph-mon[31460]: pgmap v542: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 409 B/s wr, 20 op/s 2024-03-28T18:26:47.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:47 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:26:50.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:49 smithi059 ceph-mon[36190]: pgmap v543: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 376 B/s wr, 18 op/s 2024-03-28T18:26:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:49 smithi170 ceph-mon[36280]: pgmap v543: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 376 B/s wr, 18 op/s 2024-03-28T18:26:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:49 smithi037 ceph-mon[31460]: pgmap v543: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 376 B/s wr, 18 op/s 2024-03-28T18:26:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:51 smithi059 ceph-mon[36190]: pgmap v544: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 323 B/s wr, 16 op/s 2024-03-28T18:26:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:51 smithi170 ceph-mon[36280]: pgmap v544: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 323 B/s wr, 16 op/s 2024-03-28T18:26:52.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:51 smithi037 ceph-mon[31460]: pgmap v544: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 323 B/s wr, 16 op/s 2024-03-28T18:26:54.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:53 smithi059 ceph-mon[36190]: pgmap v545: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 215 B/s wr, 14 op/s 2024-03-28T18:26:54.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:53 smithi170 ceph-mon[36280]: pgmap v545: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 215 B/s wr, 14 op/s 2024-03-28T18:26:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:53 smithi037 ceph-mon[31460]: pgmap v545: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 215 B/s wr, 14 op/s 2024-03-28T18:26:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:55 smithi059 ceph-mon[36190]: pgmap v546: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 215 B/s wr, 14 op/s 2024-03-28T18:26:56.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:55 smithi170 ceph-mon[36280]: pgmap v546: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 215 B/s wr, 14 op/s 2024-03-28T18:26:56.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:55 smithi037 ceph-mon[31460]: pgmap v546: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 215 B/s wr, 14 op/s 2024-03-28T18:26:56.125 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:55 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 6. 2024-03-28T18:26:56.125 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:55 smithi037 systemd[1]: Stopped Ceph mds.j for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:26:56.125 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:55 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Consumed 1.709s CPU time. 2024-03-28T18:26:56.125 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:55 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Start request repeated too quickly. 2024-03-28T18:26:56.125 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:55 smithi037 systemd[1]: ceph-b5ad257c-ed2c-11ee-95d0-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-28T18:26:56.125 INFO:journalctl@ceph.mds.j.smithi037.stdout:Mar 28 18:26:55 smithi037 systemd[1]: Failed to start Ceph mds.j for b5ad257c-ed2c-11ee-95d0-87774f69a715. 2024-03-28T18:26:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:57 smithi059 ceph-mon[36190]: pgmap v547: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:26:58.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:57 smithi170 ceph-mon[36280]: pgmap v547: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:26:58.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:57 smithi037 ceph-mon[31460]: pgmap v547: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:27:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:59 smithi059 ceph-mon[36190]: pgmap v548: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:26:59 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-03-28T18:27:00.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:59 smithi170 ceph-mon[36280]: pgmap v548: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:00.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:26:59 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-03-28T18:27:00.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:59 smithi037 ceph-mon[31460]: pgmap v548: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:00.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:26:59 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-03-28T18:27:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:01 smithi059 ceph-mon[36190]: pgmap v549: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:01 smithi170 ceph-mon[36280]: pgmap v549: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:01 smithi037 ceph-mon[31460]: pgmap v549: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:03 smithi059 ceph-mon[36190]: pgmap v550: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:27:04.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:03 smithi170 ceph-mon[36280]: pgmap v550: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:27:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:03 smithi037 ceph-mon[31460]: pgmap v550: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:27:06.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:05 smithi059 ceph-mon[36190]: pgmap v551: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:27:06.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:05 smithi170 ceph-mon[36280]: pgmap v551: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:27:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:05 smithi037 ceph-mon[31460]: pgmap v551: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:27:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:07 smithi059 ceph-mon[36190]: pgmap v552: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:27:08.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:07 smithi170 ceph-mon[36280]: pgmap v552: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:27:08.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:07 smithi037 ceph-mon[31460]: pgmap v552: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:27:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:09 smithi059 ceph-mon[36190]: pgmap v553: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:27:10.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:09 smithi170 ceph-mon[36280]: pgmap v553: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:27:10.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:09 smithi037 ceph-mon[31460]: pgmap v553: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:27:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:11 smithi059 ceph-mon[36190]: pgmap v554: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:27:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:11 smithi170 ceph-mon[36280]: pgmap v554: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:27:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:11 smithi037 ceph-mon[31460]: pgmap v554: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:27:14.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:13 smithi059 ceph-mon[36190]: pgmap v555: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:27:14.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:13 smithi170 ceph-mon[36280]: pgmap v555: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:27:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:13 smithi037 ceph-mon[31460]: pgmap v555: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:27:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:15 smithi059 ceph-mon[36190]: pgmap v556: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:16.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:15 smithi170 ceph-mon[36280]: pgmap v556: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:16.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:15 smithi037 ceph-mon[31460]: pgmap v556: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:18.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:17 smithi059 ceph-mon[36190]: pgmap v557: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:18.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:17 smithi170 ceph-mon[36280]: pgmap v557: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:18.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:17 smithi037 ceph-mon[31460]: pgmap v557: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:19 smithi059 ceph-mon[36190]: pgmap v558: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:20.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:19 smithi170 ceph-mon[36280]: pgmap v558: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:20.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:19 smithi037 ceph-mon[31460]: pgmap v558: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:21 smithi059 ceph-mon[36190]: pgmap v559: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:21 smithi170 ceph-mon[36280]: pgmap v559: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:22.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:21 smithi037 ceph-mon[31460]: pgmap v559: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:23 smithi059 ceph-mon[36190]: pgmap v560: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:23 smithi170 ceph-mon[36280]: pgmap v560: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:24.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:23 smithi037 ceph-mon[31460]: pgmap v560: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:25 smithi059 ceph-mon[36190]: pgmap v561: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:26.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:25 smithi170 ceph-mon[36280]: pgmap v561: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:26.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:25 smithi037 ceph-mon[31460]: pgmap v561: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:27 smithi059 ceph-mon[36190]: pgmap v562: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:28.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:27 smithi170 ceph-mon[36280]: pgmap v562: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:28.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:27 smithi037 ceph-mon[31460]: pgmap v562: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:29 smithi059 ceph-mon[36190]: pgmap v563: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:30.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:29 smithi170 ceph-mon[36280]: pgmap v563: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:30.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:29 smithi037 ceph-mon[31460]: pgmap v563: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:31 smithi059 ceph-mon[36190]: pgmap v564: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:32.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:31 smithi170 ceph-mon[36280]: pgmap v564: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:32.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:31 smithi037 ceph-mon[31460]: pgmap v564: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:34.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:33 smithi059 ceph-mon[36190]: pgmap v565: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:33 smithi170 ceph-mon[36280]: pgmap v565: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:34.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:33 smithi037 ceph-mon[31460]: pgmap v565: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:35 smithi059 ceph-mon[36190]: pgmap v566: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:36.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:35 smithi170 ceph-mon[36280]: pgmap v566: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:36.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:35 smithi037 ceph-mon[31460]: pgmap v566: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:38.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:37 smithi059 ceph-mon[36190]: pgmap v567: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:38.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:37 smithi170 ceph-mon[36280]: pgmap v567: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:38.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:37 smithi037 ceph-mon[31460]: pgmap v567: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:39 smithi059 ceph-mon[36190]: pgmap v568: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:40.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:39 smithi170 ceph-mon[36280]: pgmap v568: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:40.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:39 smithi037 ceph-mon[31460]: pgmap v568: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:42.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:41 smithi059 ceph-mon[36190]: pgmap v569: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:42.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:41 smithi170 ceph-mon[36280]: pgmap v569: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:42.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:41 smithi037 ceph-mon[31460]: pgmap v569: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:44.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:43 smithi059 ceph-mon[36190]: pgmap v570: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:44.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:43 smithi170 ceph-mon[36280]: pgmap v570: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:44.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:43 smithi037 ceph-mon[31460]: pgmap v570: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:46.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:45 smithi059 ceph-mon[36190]: pgmap v571: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:46.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:45 smithi170 ceph-mon[36280]: pgmap v571: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:46.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:45 smithi037 ceph-mon[31460]: pgmap v571: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:48.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:47 smithi059 ceph-mon[36190]: pgmap v572: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:48.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:47 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:27:48.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:47 smithi170 ceph-mon[36280]: pgmap v572: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:48.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:47 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:27:48.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:47 smithi037 ceph-mon[31460]: pgmap v572: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:48.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:47 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:27:49.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:48 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:27:49.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:48 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:27:49.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:48 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:27:49.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:48 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:27:49.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:48 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:27:49.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:48 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:27:49.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:48 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:27:49.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:48 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:27:49.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:48 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:27:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:49 smithi170 ceph-mon[36280]: pgmap v573: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:49 smithi170 ceph-mon[36280]: pgmap v574: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:49 smithi037 ceph-mon[31460]: pgmap v573: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:49 smithi037 ceph-mon[31460]: pgmap v574: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:49 smithi059 ceph-mon[36190]: pgmap v573: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:49 smithi059 ceph-mon[36190]: pgmap v574: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:51.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:51 smithi037 ceph-mon[31460]: pgmap v575: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:51 smithi059 ceph-mon[36190]: pgmap v575: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:51 smithi170 ceph-mon[36280]: pgmap v575: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:54.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:53 smithi037 ceph-mon[31460]: pgmap v576: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:27:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:53 smithi059 ceph-mon[36190]: pgmap v576: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:27:54.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:53 smithi170 ceph-mon[36280]: pgmap v576: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:27:56.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:55 smithi037 ceph-mon[31460]: pgmap v577: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:27:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:55 smithi059 ceph-mon[36190]: pgmap v577: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:27:56.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:55 smithi170 ceph-mon[36280]: pgmap v577: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:27:58.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:57 smithi037 ceph-mon[31460]: pgmap v578: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 976 B/s rd, 1 op/s 2024-03-28T18:27:58.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:57 smithi059 ceph-mon[36190]: pgmap v578: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 976 B/s rd, 1 op/s 2024-03-28T18:27:58.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:57 smithi170 ceph-mon[36280]: pgmap v578: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 976 B/s rd, 1 op/s 2024-03-28T18:28:00.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:27:59 smithi037 ceph-mon[31460]: pgmap v579: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:28:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:27:59 smithi059 ceph-mon[36190]: pgmap v579: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:28:00.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:27:59 smithi170 ceph-mon[36280]: pgmap v579: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:28:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:01 smithi037 ceph-mon[31460]: pgmap v580: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:28:02.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:01 smithi059 ceph-mon[36190]: pgmap v580: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:28:02.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:01 smithi170 ceph-mon[36280]: pgmap v580: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:28:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:03 smithi037 ceph-mon[31460]: pgmap v581: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:28:04.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:03 smithi059 ceph-mon[36190]: pgmap v581: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:28:04.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:03 smithi170 ceph-mon[36280]: pgmap v581: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:28:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:05 smithi037 ceph-mon[31460]: pgmap v582: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:05 smithi059 ceph-mon[36190]: pgmap v582: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:06.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:05 smithi170 ceph-mon[36280]: pgmap v582: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:08.087 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:07 smithi059 ceph-mon[36190]: pgmap v583: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:08.088 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:07 smithi170 ceph-mon[36280]: pgmap v583: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:08.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:07 smithi037 ceph-mon[31460]: pgmap v583: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:10.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:09 smithi037 ceph-mon[31460]: pgmap v584: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:10.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:09 smithi059 ceph-mon[36190]: pgmap v584: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:10.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:09 smithi170 ceph-mon[36280]: pgmap v584: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:11 smithi037 ceph-mon[31460]: pgmap v585: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:11 smithi059 ceph-mon[36190]: pgmap v585: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:12.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:11 smithi170 ceph-mon[36280]: pgmap v585: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:13 smithi037 ceph-mon[31460]: pgmap v586: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:13 smithi059 ceph-mon[36190]: pgmap v586: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:14.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:13 smithi170 ceph-mon[36280]: pgmap v586: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:16.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:15 smithi037 ceph-mon[31460]: pgmap v587: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:16.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:15 smithi059 ceph-mon[36190]: pgmap v587: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:16.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:15 smithi170 ceph-mon[36280]: pgmap v587: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:18.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:17 smithi037 ceph-mon[31460]: pgmap v588: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:18.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:17 smithi059 ceph-mon[36190]: pgmap v588: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:18.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:17 smithi170 ceph-mon[36280]: pgmap v588: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:20.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:19 smithi037 ceph-mon[31460]: pgmap v589: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:19 smithi059 ceph-mon[36190]: pgmap v589: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:20.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:19 smithi170 ceph-mon[36280]: pgmap v589: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:22.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:21 smithi037 ceph-mon[31460]: pgmap v590: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:22.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:21 smithi059 ceph-mon[36190]: pgmap v590: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:21 smithi170 ceph-mon[36280]: pgmap v590: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:24.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:23 smithi037 ceph-mon[31460]: pgmap v591: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:24.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:23 smithi059 ceph-mon[36190]: pgmap v591: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:24.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:23 smithi170 ceph-mon[36280]: pgmap v591: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:26.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:25 smithi037 ceph-mon[31460]: pgmap v592: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:25 smithi059 ceph-mon[36190]: pgmap v592: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:25 smithi170 ceph-mon[36280]: pgmap v592: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:28.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:27 smithi037 ceph-mon[31460]: pgmap v593: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:27 smithi059 ceph-mon[36190]: pgmap v593: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:28.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:27 smithi170 ceph-mon[36280]: pgmap v593: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:30.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:29 smithi037 ceph-mon[31460]: pgmap v594: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:30.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:29 smithi059 ceph-mon[36190]: pgmap v594: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:30.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:29 smithi170 ceph-mon[36280]: pgmap v594: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:32.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:31 smithi037 ceph-mon[31460]: pgmap v595: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:32.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:31 smithi059 ceph-mon[36190]: pgmap v595: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:31 smithi170 ceph-mon[36280]: pgmap v595: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:34.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:33 smithi037 ceph-mon[31460]: pgmap v596: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:33 smithi059 ceph-mon[36190]: pgmap v596: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:34.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:33 smithi170 ceph-mon[36280]: pgmap v596: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:35 smithi059 ceph-mon[36190]: pgmap v597: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:36.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:35 smithi170 ceph-mon[36280]: pgmap v597: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:36.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:35 smithi037 ceph-mon[31460]: pgmap v597: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:38.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:37 smithi059 ceph-mon[36190]: pgmap v598: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:38.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:37 smithi170 ceph-mon[36280]: pgmap v598: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:38.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:37 smithi037 ceph-mon[31460]: pgmap v598: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:39 smithi059 ceph-mon[36190]: pgmap v599: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:40.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:39 smithi170 ceph-mon[36280]: pgmap v599: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:40.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:39 smithi037 ceph-mon[31460]: pgmap v599: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:42.177 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:41 smithi037 ceph-mon[31460]: pgmap v600: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:41 smithi059 ceph-mon[36190]: pgmap v600: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:42.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:41 smithi170 ceph-mon[36280]: pgmap v600: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:43 smithi059 ceph-mon[36190]: pgmap v601: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:44.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:43 smithi170 ceph-mon[36280]: pgmap v601: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:44.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:43 smithi037 ceph-mon[31460]: pgmap v601: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:45 smithi059 ceph-mon[36190]: pgmap v602: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:45 smithi170 ceph-mon[36280]: pgmap v602: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:46.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:45 smithi037 ceph-mon[31460]: pgmap v602: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:48.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:47 smithi059 ceph-mon[36190]: pgmap v603: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:48.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:47 smithi170 ceph-mon[36280]: pgmap v603: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:48.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:47 smithi037 ceph-mon[31460]: pgmap v603: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:48 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:28:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:48 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:28:49.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:48 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:28:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:49 smithi059 ceph-mon[36190]: pgmap v604: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:49 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:28:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:49 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:28:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:49 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:28:50.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:49 smithi170 ceph-mon[36280]: pgmap v604: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:50.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:49 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:28:50.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:49 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:28:50.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:49 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:28:50.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:49 smithi037 ceph-mon[31460]: pgmap v604: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:50.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:49 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:28:50.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:49 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:28:50.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:49 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:28:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:50 smithi059 ceph-mon[36190]: pgmap v605: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:50 smithi059 ceph-mon[36190]: pgmap v606: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:50 smithi170 ceph-mon[36280]: pgmap v605: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:50 smithi170 ceph-mon[36280]: pgmap v606: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:50 smithi037 ceph-mon[31460]: pgmap v605: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:50 smithi037 ceph-mon[31460]: pgmap v606: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:52 smithi059 ceph-mon[36190]: pgmap v607: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:28:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:52 smithi170 ceph-mon[36280]: pgmap v607: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:28:53.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:52 smithi037 ceph-mon[31460]: pgmap v607: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:28:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:54 smithi059 ceph-mon[36190]: pgmap v608: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:55.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:54 smithi170 ceph-mon[36280]: pgmap v608: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:55.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:54 smithi037 ceph-mon[31460]: pgmap v608: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:56 smithi059 ceph-mon[36190]: pgmap v609: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:56 smithi170 ceph-mon[36280]: pgmap v609: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:57.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:56 smithi037 ceph-mon[31460]: pgmap v609: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:59.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:28:58 smithi059 ceph-mon[36190]: pgmap v610: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:59.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:28:58 smithi170 ceph-mon[36280]: pgmap v610: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:59.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:28:58 smithi037 ceph-mon[31460]: pgmap v610: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:00 smithi059 ceph-mon[36190]: pgmap v611: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:29:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:00 smithi170 ceph-mon[36280]: pgmap v611: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:29:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:00 smithi037 ceph-mon[31460]: pgmap v611: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:29:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:02 smithi059 ceph-mon[36190]: pgmap v612: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:03.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:02 smithi170 ceph-mon[36280]: pgmap v612: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:03.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:02 smithi037 ceph-mon[31460]: pgmap v612: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:05.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:04 smithi059 ceph-mon[36190]: pgmap v613: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:29:05.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:04 smithi170 ceph-mon[36280]: pgmap v613: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:29:05.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:04 smithi037 ceph-mon[31460]: pgmap v613: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:29:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:06 smithi059 ceph-mon[36190]: pgmap v614: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:07.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:06 smithi170 ceph-mon[36280]: pgmap v614: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:07.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:06 smithi037 ceph-mon[31460]: pgmap v614: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:08 smithi059 ceph-mon[36190]: pgmap v615: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:09.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:08 smithi170 ceph-mon[36280]: pgmap v615: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:09.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:08 smithi037 ceph-mon[31460]: pgmap v615: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:10 smithi059 ceph-mon[36190]: pgmap v616: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:11.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:10 smithi170 ceph-mon[36280]: pgmap v616: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:10 smithi037 ceph-mon[31460]: pgmap v616: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:12 smithi059 ceph-mon[36190]: pgmap v617: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:13.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:12 smithi170 ceph-mon[36280]: pgmap v617: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:13.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:12 smithi037 ceph-mon[31460]: pgmap v617: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:15.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:14 smithi059 ceph-mon[36190]: pgmap v618: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:29:15.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:14 smithi170 ceph-mon[36280]: pgmap v618: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:29:15.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:14 smithi037 ceph-mon[31460]: pgmap v618: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:29:16.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:15 smithi059 ceph-mon[36190]: pgmap v619: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:16.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:15 smithi170 ceph-mon[36280]: pgmap v619: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:16.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:15 smithi037 ceph-mon[31460]: pgmap v619: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:18.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:18 smithi037 ceph-mon[31460]: pgmap v620: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:18 smithi059 ceph-mon[36190]: pgmap v620: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:19.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:18 smithi170 ceph-mon[36280]: pgmap v620: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:20.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:20 smithi037 ceph-mon[31460]: pgmap v621: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:21.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:20 smithi059 ceph-mon[36190]: pgmap v621: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:21.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:20 smithi170 ceph-mon[36280]: pgmap v621: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:22.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:22 smithi037 ceph-mon[31460]: pgmap v622: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:23.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:22 smithi059 ceph-mon[36190]: pgmap v622: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:23.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:22 smithi170 ceph-mon[36280]: pgmap v622: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:24.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:24 smithi037 ceph-mon[31460]: pgmap v623: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:29:25.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:24 smithi059 ceph-mon[36190]: pgmap v623: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:29:25.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:24 smithi170 ceph-mon[36280]: pgmap v623: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:29:26.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:26 smithi037 ceph-mon[31460]: pgmap v624: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:26 smithi059 ceph-mon[36190]: pgmap v624: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:27.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:26 smithi170 ceph-mon[36280]: pgmap v624: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:28.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:28 smithi037 ceph-mon[31460]: pgmap v625: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:29.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:28 smithi059 ceph-mon[36190]: pgmap v625: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:29.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:28 smithi170 ceph-mon[36280]: pgmap v625: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:30.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:30 smithi037 ceph-mon[31460]: pgmap v626: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:31.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:30 smithi059 ceph-mon[36190]: pgmap v626: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:31.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:30 smithi170 ceph-mon[36280]: pgmap v626: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:32.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:32 smithi037 ceph-mon[31460]: pgmap v627: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:33.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:32 smithi059 ceph-mon[36190]: pgmap v627: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:33.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:32 smithi170 ceph-mon[36280]: pgmap v627: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:34.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:34 smithi037 ceph-mon[31460]: pgmap v628: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:29:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:34 smithi059 ceph-mon[36190]: pgmap v628: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:29:35.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:34 smithi170 ceph-mon[36280]: pgmap v628: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:29:36.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:36 smithi037 ceph-mon[31460]: pgmap v629: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:37.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:36 smithi059 ceph-mon[36190]: pgmap v629: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:37.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:36 smithi170 ceph-mon[36280]: pgmap v629: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:38.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:38 smithi037 ceph-mon[31460]: pgmap v630: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:39.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:38 smithi059 ceph-mon[36190]: pgmap v630: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:39.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:38 smithi170 ceph-mon[36280]: pgmap v630: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:40.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:40 smithi037 ceph-mon[31460]: pgmap v631: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:40 smithi059 ceph-mon[36190]: pgmap v631: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:41.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:40 smithi170 ceph-mon[36280]: pgmap v631: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:42.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:42 smithi037 ceph-mon[31460]: pgmap v632: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:43.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:42 smithi059 ceph-mon[36190]: pgmap v632: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:43.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:42 smithi170 ceph-mon[36280]: pgmap v632: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:45.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:44 smithi059 ceph-mon[36190]: pgmap v633: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:29:45.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:44 smithi170 ceph-mon[36280]: pgmap v633: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:29:45.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:44 smithi037 ceph-mon[31460]: pgmap v633: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:29:47.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:46 smithi059 ceph-mon[36190]: pgmap v634: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:47.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:46 smithi059 ceph-mon[36190]: 1 slow requests, 1 included below; oldest blocked for > 181.096790 secs 2024-03-28T18:29:47.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:46 smithi059 ceph-mon[36190]: slow request 181.096790 seconds old, received at 2024-03-28T18:26:44.515856+0000: peer_request:client.25216:13687 currently acquired locks 2024-03-28T18:29:47.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:46 smithi170 ceph-mon[36280]: pgmap v634: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:47.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:46 smithi170 ceph-mon[36280]: 1 slow requests, 1 included below; oldest blocked for > 181.096790 secs 2024-03-28T18:29:47.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:46 smithi170 ceph-mon[36280]: slow request 181.096790 seconds old, received at 2024-03-28T18:26:44.515856+0000: peer_request:client.25216:13687 currently acquired locks 2024-03-28T18:29:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:46 smithi037 ceph-mon[31460]: pgmap v634: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:46 smithi037 ceph-mon[31460]: 1 slow requests, 1 included below; oldest blocked for > 181.096790 secs 2024-03-28T18:29:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:46 smithi037 ceph-mon[31460]: slow request 181.096790 seconds old, received at 2024-03-28T18:26:44.515856+0000: peer_request:client.25216:13687 currently acquired locks 2024-03-28T18:29:49.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:48 smithi059 ceph-mon[36190]: pgmap v635: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:49.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:48 smithi170 ceph-mon[36280]: pgmap v635: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:49.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:48 smithi037 ceph-mon[31460]: pgmap v635: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:50.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:49 smithi059 ceph-mon[36190]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-28T18:29:50.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:49 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] up:active 2024-03-28T18:29:50.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:49 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-03-28T18:29:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:49 smithi170 ceph-mon[36280]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-28T18:29:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:49 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] up:active 2024-03-28T18:29:50.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:49 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-03-28T18:29:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:49 smithi037 ceph-mon[31460]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-28T18:29:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:49 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] up:active 2024-03-28T18:29:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:49 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-03-28T18:29:51.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:50 smithi059 ceph-mon[36190]: pgmap v636: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:51.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:50 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:51.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:50 smithi170 ceph-mon[36280]: pgmap v636: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:51.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:50 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:51.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:50 smithi037 ceph-mon[31460]: pgmap v636: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:51.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:50 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:51 smithi059 ceph-mon[36190]: 1 slow requests, 0 included below; oldest blocked for > 186.096865 secs 2024-03-28T18:29:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:51 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:51 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:51 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:29:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:51 smithi170 ceph-mon[36280]: 1 slow requests, 0 included below; oldest blocked for > 186.096865 secs 2024-03-28T18:29:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:51 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:51 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:52.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:51 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:29:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:51 smithi037 ceph-mon[31460]: 1 slow requests, 0 included below; oldest blocked for > 186.096865 secs 2024-03-28T18:29:52.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:51 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:52.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:51 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:52.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:51 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:29:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:52 smithi059 ceph-mon[36190]: pgmap v637: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:52 smithi059 ceph-mon[36190]: pgmap v638: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-03-28T18:29:53.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:52 smithi170 ceph-mon[36280]: pgmap v637: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:53.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:52 smithi170 ceph-mon[36280]: pgmap v638: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-03-28T18:29:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:52 smithi037 ceph-mon[31460]: pgmap v637: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:52 smithi037 ceph-mon[31460]: pgmap v638: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-03-28T18:29:54.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:54 smithi037 ceph-mon[31460]: pgmap v639: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:55.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:54 smithi059 ceph-mon[36190]: pgmap v639: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:55.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:54 smithi170 ceph-mon[36280]: pgmap v639: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:57.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:56 smithi059 ceph-mon[36190]: pgmap v640: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:57.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:56 smithi059 ceph-mon[36190]: 1 slow requests, 0 included below; oldest blocked for > 191.096985 secs 2024-03-28T18:29:57.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:56 smithi170 ceph-mon[36280]: pgmap v640: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:57.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:56 smithi170 ceph-mon[36280]: 1 slow requests, 0 included below; oldest blocked for > 191.096985 secs 2024-03-28T18:29:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:56 smithi037 ceph-mon[31460]: pgmap v640: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:56 smithi037 ceph-mon[31460]: 1 slow requests, 0 included below; oldest blocked for > 191.096985 secs 2024-03-28T18:29:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:29:58 smithi059 ceph-mon[36190]: pgmap v641: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:59.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:29:58 smithi170 ceph-mon[36280]: pgmap v641: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:59.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:29:58 smithi037 ceph-mon[31460]: pgmap v641: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:30:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:00 smithi059 ceph-mon[36190]: pgmap v642: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:30:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:00 smithi059 ceph-mon[36190]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T18:30:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:00 smithi059 ceph-mon[36190]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-03-28T18:30:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:00 smithi059 ceph-mon[36190]: daemon mds.e on smithi059 is in error state 2024-03-28T18:30:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:00 smithi059 ceph-mon[36190]: daemon mds.b on smithi059 is in error state 2024-03-28T18:30:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:00 smithi059 ceph-mon[36190]: daemon mds.l on smithi170 is in unknown state 2024-03-28T18:30:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:00 smithi059 ceph-mon[36190]: daemon mds.f on smithi170 is in error state 2024-03-28T18:30:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:00 smithi059 ceph-mon[36190]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-03-28T18:30:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:00 smithi059 ceph-mon[36190]: mds.a(mds.0): 1 slow requests are blocked > 180 secs 2024-03-28T18:30:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:00 smithi170 ceph-mon[36280]: pgmap v642: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:30:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:00 smithi170 ceph-mon[36280]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T18:30:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:00 smithi170 ceph-mon[36280]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-03-28T18:30:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:00 smithi170 ceph-mon[36280]: daemon mds.e on smithi059 is in error state 2024-03-28T18:30:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:00 smithi170 ceph-mon[36280]: daemon mds.b on smithi059 is in error state 2024-03-28T18:30:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:00 smithi170 ceph-mon[36280]: daemon mds.l on smithi170 is in unknown state 2024-03-28T18:30:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:00 smithi170 ceph-mon[36280]: daemon mds.f on smithi170 is in error state 2024-03-28T18:30:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:00 smithi170 ceph-mon[36280]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-03-28T18:30:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:00 smithi170 ceph-mon[36280]: mds.a(mds.0): 1 slow requests are blocked > 180 secs 2024-03-28T18:30:01.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:00 smithi037 ceph-mon[31460]: pgmap v642: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:30:01.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:00 smithi037 ceph-mon[31460]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T18:30:01.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:00 smithi037 ceph-mon[31460]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-03-28T18:30:01.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:00 smithi037 ceph-mon[31460]: daemon mds.e on smithi059 is in error state 2024-03-28T18:30:01.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:00 smithi037 ceph-mon[31460]: daemon mds.b on smithi059 is in error state 2024-03-28T18:30:01.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:00 smithi037 ceph-mon[31460]: daemon mds.l on smithi170 is in unknown state 2024-03-28T18:30:01.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:00 smithi037 ceph-mon[31460]: daemon mds.f on smithi170 is in error state 2024-03-28T18:30:01.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:00 smithi037 ceph-mon[31460]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-03-28T18:30:01.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:00 smithi037 ceph-mon[31460]: mds.a(mds.0): 1 slow requests are blocked > 180 secs 2024-03-28T18:30:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:01 smithi059 ceph-mon[36190]: 2 slow requests, 1 included below; oldest blocked for > 196.097106 secs 2024-03-28T18:30:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:01 smithi059 ceph-mon[36190]: slow request 180.516781 seconds old, received at 2024-03-28T18:27:00.096181+0000: client_request(client.25553:180 mksnap #0x1//scheduled-2024-03-28-18_27_00_UTC 2024-03-28T18:27:00.095960+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:30:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:01 smithi170 ceph-mon[36280]: 2 slow requests, 1 included below; oldest blocked for > 196.097106 secs 2024-03-28T18:30:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:01 smithi170 ceph-mon[36280]: slow request 180.516781 seconds old, received at 2024-03-28T18:27:00.096181+0000: client_request(client.25553:180 mksnap #0x1//scheduled-2024-03-28-18_27_00_UTC 2024-03-28T18:27:00.095960+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:30:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:01 smithi037 ceph-mon[31460]: 2 slow requests, 1 included below; oldest blocked for > 196.097106 secs 2024-03-28T18:30:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:01 smithi037 ceph-mon[31460]: slow request 180.516781 seconds old, received at 2024-03-28T18:27:00.096181+0000: client_request(client.25553:180 mksnap #0x1//scheduled-2024-03-28-18_27_00_UTC 2024-03-28T18:27:00.095960+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:30:03.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:02 smithi059 ceph-mon[36190]: pgmap v643: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:30:03.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:02 smithi170 ceph-mon[36280]: pgmap v643: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:30:03.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:02 smithi037 ceph-mon[31460]: pgmap v643: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:30:05.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:04 smithi059 ceph-mon[36190]: pgmap v644: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:05.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:04 smithi170 ceph-mon[36280]: pgmap v644: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:05.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:04 smithi037 ceph-mon[31460]: pgmap v644: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:06.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:05 smithi059 ceph-mon[36190]: mds.? [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] up:active 2024-03-28T18:30:06.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:05 smithi059 ceph-mon[36190]: fsmap cephfs:3 {0=a=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-03-28T18:30:06.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:05 smithi170 ceph-mon[36280]: mds.? [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] up:active 2024-03-28T18:30:06.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:05 smithi170 ceph-mon[36280]: fsmap cephfs:3 {0=a=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-03-28T18:30:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:05 smithi037 ceph-mon[31460]: mds.? [v2:172.21.15.37:6840/1923263622,v1:172.21.15.37:6841/1923263622] up:active 2024-03-28T18:30:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:05 smithi037 ceph-mon[31460]: fsmap cephfs:3 {0=a=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-03-28T18:30:07.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:06 smithi059 ceph-mon[36190]: pgmap v645: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:07.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 201.097184 secs 2024-03-28T18:30:07.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:06 smithi170 ceph-mon[36280]: pgmap v645: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:07.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 201.097184 secs 2024-03-28T18:30:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:06 smithi037 ceph-mon[31460]: pgmap v645: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 201.097184 secs 2024-03-28T18:30:09.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:08 smithi059 ceph-mon[36190]: pgmap v646: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:09.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:08 smithi170 ceph-mon[36280]: pgmap v646: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:09.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:08 smithi037 ceph-mon[31460]: pgmap v646: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:11.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:10 smithi059 ceph-mon[36190]: pgmap v647: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:11.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:10 smithi170 ceph-mon[36280]: pgmap v647: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:11.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:10 smithi037 ceph-mon[31460]: pgmap v647: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 206.097263 secs 2024-03-28T18:30:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 206.097263 secs 2024-03-28T18:30:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 206.097263 secs 2024-03-28T18:30:13.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:12 smithi059 ceph-mon[36190]: pgmap v648: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:13.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:12 smithi170 ceph-mon[36280]: pgmap v648: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:13.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:12 smithi037 ceph-mon[31460]: pgmap v648: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:14 smithi059 ceph-mon[36190]: pgmap v649: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:15.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:14 smithi170 ceph-mon[36280]: pgmap v649: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:15.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:14 smithi037 ceph-mon[31460]: pgmap v649: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:17.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:16 smithi059 ceph-mon[36190]: pgmap v650: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:17.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 211.097381 secs 2024-03-28T18:30:17.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:16 smithi170 ceph-mon[36280]: pgmap v650: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:17.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 211.097381 secs 2024-03-28T18:30:17.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:16 smithi037 ceph-mon[31460]: pgmap v650: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:17.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 211.097381 secs 2024-03-28T18:30:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:18 smithi059 ceph-mon[36190]: pgmap v651: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:19.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:18 smithi170 ceph-mon[36280]: pgmap v651: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:19.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:18 smithi037 ceph-mon[31460]: pgmap v651: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:21.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:20 smithi059 ceph-mon[36190]: pgmap v652: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:21.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:20 smithi170 ceph-mon[36280]: pgmap v652: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:21.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:20 smithi037 ceph-mon[31460]: pgmap v652: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 216.097459 secs 2024-03-28T18:30:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 216.097459 secs 2024-03-28T18:30:22.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 216.097459 secs 2024-03-28T18:30:23.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:22 smithi059 ceph-mon[36190]: pgmap v653: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:23.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:22 smithi170 ceph-mon[36280]: pgmap v653: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:23.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:22 smithi037 ceph-mon[31460]: pgmap v653: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:25.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:24 smithi059 ceph-mon[36190]: pgmap v654: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:25.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:24 smithi170 ceph-mon[36280]: pgmap v654: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:25.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:24 smithi037 ceph-mon[31460]: pgmap v654: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:26.026 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:25 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 221.097559 secs 2024-03-28T18:30:26.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:25 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 221.097559 secs 2024-03-28T18:30:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:25 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 221.097559 secs 2024-03-28T18:30:27.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:26 smithi037 ceph-mon[31460]: pgmap v655: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:26 smithi059 ceph-mon[36190]: pgmap v655: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:26 smithi170 ceph-mon[36280]: pgmap v655: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:29.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:28 smithi037 ceph-mon[31460]: pgmap v656: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:29.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:28 smithi059 ceph-mon[36190]: pgmap v656: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:29.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:28 smithi170 ceph-mon[36280]: pgmap v656: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:31.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:30 smithi037 ceph-mon[31460]: pgmap v657: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:31.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:30 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 226.097684 secs 2024-03-28T18:30:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:30 smithi059 ceph-mon[36190]: pgmap v657: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:30 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 226.097684 secs 2024-03-28T18:30:31.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:30 smithi170 ceph-mon[36280]: pgmap v657: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:31.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:30 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 226.097684 secs 2024-03-28T18:30:33.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:32 smithi037 ceph-mon[31460]: pgmap v658: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:32 smithi059 ceph-mon[36190]: pgmap v658: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:33.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:32 smithi170 ceph-mon[36280]: pgmap v658: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:35.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:34 smithi037 ceph-mon[31460]: pgmap v659: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:34 smithi059 ceph-mon[36190]: pgmap v659: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:35.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:34 smithi170 ceph-mon[36280]: pgmap v659: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:36.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:35 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 231.097803 secs 2024-03-28T18:30:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:35 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 231.097803 secs 2024-03-28T18:30:36.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:35 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 231.097803 secs 2024-03-28T18:30:37.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:36 smithi037 ceph-mon[31460]: pgmap v660: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:36 smithi059 ceph-mon[36190]: pgmap v660: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:36 smithi170 ceph-mon[36280]: pgmap v660: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:39.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:38 smithi037 ceph-mon[31460]: pgmap v661: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:39.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:38 smithi059 ceph-mon[36190]: pgmap v661: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:39.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:38 smithi170 ceph-mon[36280]: pgmap v661: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:41.087 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:40 smithi059 ceph-mon[36190]: pgmap v662: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:41.087 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:40 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 236.097921 secs 2024-03-28T18:30:41.088 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:40 smithi170 ceph-mon[36280]: pgmap v662: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:41.088 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:40 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 236.097921 secs 2024-03-28T18:30:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:40 smithi037 ceph-mon[31460]: pgmap v662: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:41.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:40 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 236.097921 secs 2024-03-28T18:30:43.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:42 smithi037 ceph-mon[31460]: pgmap v663: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:43.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:42 smithi059 ceph-mon[36190]: pgmap v663: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:43.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:42 smithi170 ceph-mon[36280]: pgmap v663: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:45.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:44 smithi037 ceph-mon[31460]: pgmap v664: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:44 smithi059 ceph-mon[36190]: pgmap v664: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:45.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:44 smithi170 ceph-mon[36280]: pgmap v664: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:45.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:45 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 241.098000 secs 2024-03-28T18:30:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:45 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 241.098000 secs 2024-03-28T18:30:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:45 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 241.098000 secs 2024-03-28T18:30:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:46 smithi037 ceph-mon[31460]: pgmap v665: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:46 smithi059 ceph-mon[36190]: pgmap v665: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:46 smithi170 ceph-mon[36280]: pgmap v665: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:49.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:48 smithi037 ceph-mon[31460]: pgmap v666: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:48 smithi059 ceph-mon[36190]: pgmap v666: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:48 smithi170 ceph-mon[36280]: pgmap v666: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:51.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:50 smithi037 ceph-mon[31460]: pgmap v667: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:51.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:50 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 246.098108 secs 2024-03-28T18:30:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:50 smithi059 ceph-mon[36190]: pgmap v667: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:50 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 246.098108 secs 2024-03-28T18:30:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:50 smithi170 ceph-mon[36280]: pgmap v667: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:50 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 246.098108 secs 2024-03-28T18:30:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:51 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:51 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:51 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:53.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:52 smithi037 ceph-mon[31460]: pgmap v668: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:53.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:52 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:53.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:52 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:53.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:52 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:30:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:52 smithi059 ceph-mon[36190]: pgmap v668: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:52 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:52 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:52 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:30:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:52 smithi170 ceph-mon[36280]: pgmap v668: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:52 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:53.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:52 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:53.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:52 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:30:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:53 smithi059 ceph-mon[36190]: pgmap v669: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:30:54.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:53 smithi170 ceph-mon[36280]: pgmap v669: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:30:54.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:53 smithi037 ceph-mon[31460]: pgmap v669: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:30:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:55 smithi059 ceph-mon[36190]: pgmap v670: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:30:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:55 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 251.098202 secs 2024-03-28T18:30:56.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:55 smithi170 ceph-mon[36280]: pgmap v670: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:30:56.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:55 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 251.098202 secs 2024-03-28T18:30:56.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:55 smithi037 ceph-mon[31460]: pgmap v670: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:30:56.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:55 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 251.098202 secs 2024-03-28T18:30:58.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:57 smithi059 ceph-mon[36190]: pgmap v671: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:30:58.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:57 smithi170 ceph-mon[36280]: pgmap v671: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:30:58.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:57 smithi037 ceph-mon[31460]: pgmap v671: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:31:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:30:59 smithi059 ceph-mon[36190]: pgmap v672: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:31:00.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:30:59 smithi170 ceph-mon[36280]: pgmap v672: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:31:00.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:30:59 smithi037 ceph-mon[31460]: pgmap v672: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:31:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:00 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 256.098296 secs 2024-03-28T18:31:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:00 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 256.098296 secs 2024-03-28T18:31:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:00 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 256.098296 secs 2024-03-28T18:31:02.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:01 smithi059 ceph-mon[36190]: pgmap v673: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:31:02.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:01 smithi170 ceph-mon[36280]: pgmap v673: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:31:02.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:01 smithi037 ceph-mon[31460]: pgmap v673: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:31:04.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:03 smithi059 ceph-mon[36190]: pgmap v674: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:31:04.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:03 smithi170 ceph-mon[36280]: pgmap v674: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:31:04.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:03 smithi037 ceph-mon[31460]: pgmap v674: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:31:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:05 smithi059 ceph-mon[36190]: pgmap v675: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:05 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 261.098392 secs 2024-03-28T18:31:06.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:05 smithi170 ceph-mon[36280]: pgmap v675: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:06.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:05 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 261.098392 secs 2024-03-28T18:31:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:05 smithi037 ceph-mon[31460]: pgmap v675: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:05 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 261.098392 secs 2024-03-28T18:31:08.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:07 smithi059 ceph-mon[36190]: pgmap v676: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:08.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:07 smithi170 ceph-mon[36280]: pgmap v676: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:08.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:07 smithi037 ceph-mon[31460]: pgmap v676: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:10.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:09 smithi059 ceph-mon[36190]: pgmap v677: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:10.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:09 smithi170 ceph-mon[36280]: pgmap v677: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:10.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:09 smithi037 ceph-mon[31460]: pgmap v677: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:11.186 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:10 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 266.098483 secs 2024-03-28T18:31:11.187 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:10 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 266.098483 secs 2024-03-28T18:31:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:10 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 266.098483 secs 2024-03-28T18:31:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:11 smithi059 ceph-mon[36190]: pgmap v678: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:12.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:11 smithi170 ceph-mon[36280]: pgmap v678: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:12.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:11 smithi037 ceph-mon[31460]: pgmap v678: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:13 smithi059 ceph-mon[36190]: pgmap v679: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:14.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:13 smithi170 ceph-mon[36280]: pgmap v679: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:14.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:13 smithi037 ceph-mon[31460]: pgmap v679: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:16.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:15 smithi059 ceph-mon[36190]: pgmap v680: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:16.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:15 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 271.098579 secs 2024-03-28T18:31:16.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:15 smithi170 ceph-mon[36280]: pgmap v680: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:16.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:15 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 271.098579 secs 2024-03-28T18:31:16.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:15 smithi037 ceph-mon[31460]: pgmap v680: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:16.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:15 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 271.098579 secs 2024-03-28T18:31:18.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:17 smithi059 ceph-mon[36190]: pgmap v681: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:18.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:17 smithi170 ceph-mon[36280]: pgmap v681: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:18.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:17 smithi037 ceph-mon[31460]: pgmap v681: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:19 smithi059 ceph-mon[36190]: pgmap v682: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:31:20.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:19 smithi170 ceph-mon[36280]: pgmap v682: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:31:20.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:19 smithi037 ceph-mon[31460]: pgmap v682: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:31:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:20 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 276.098699 secs 2024-03-28T18:31:21.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:20 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 276.098699 secs 2024-03-28T18:31:21.378 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:20 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 276.098699 secs 2024-03-28T18:31:22.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:21 smithi059 ceph-mon[36190]: pgmap v683: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:31:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:21 smithi170 ceph-mon[36280]: pgmap v683: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:31:22.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:21 smithi037 ceph-mon[31460]: pgmap v683: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:31:24.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:23 smithi059 ceph-mon[36190]: pgmap v684: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:31:24.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:23 smithi170 ceph-mon[36280]: pgmap v684: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:31:24.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:23 smithi037 ceph-mon[31460]: pgmap v684: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:31:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:25 smithi059 ceph-mon[36190]: pgmap v685: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:31:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:25 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 281.098790 secs 2024-03-28T18:31:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:25 smithi170 ceph-mon[36280]: pgmap v685: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:31:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:25 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 281.098790 secs 2024-03-28T18:31:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:25 smithi037 ceph-mon[31460]: pgmap v685: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:31:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:25 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 281.098790 secs 2024-03-28T18:31:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:27 smithi059 ceph-mon[36190]: pgmap v686: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:31:28.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:27 smithi170 ceph-mon[36280]: pgmap v686: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:31:28.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:27 smithi037 ceph-mon[31460]: pgmap v686: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:31:30.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:29 smithi059 ceph-mon[36190]: pgmap v687: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:31:30.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:29 smithi170 ceph-mon[36280]: pgmap v687: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:31:30.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:29 smithi037 ceph-mon[31460]: pgmap v687: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:31:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:30 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 286.098908 secs 2024-03-28T18:31:31.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:30 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 286.098908 secs 2024-03-28T18:31:31.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:30 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 286.098908 secs 2024-03-28T18:31:32.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:31 smithi059 ceph-mon[36190]: pgmap v688: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:32.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:31 smithi170 ceph-mon[36280]: pgmap v688: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:32.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:31 smithi037 ceph-mon[31460]: pgmap v688: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:34 smithi059 ceph-mon[36190]: pgmap v689: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:34.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:34 smithi170 ceph-mon[36280]: pgmap v689: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:34.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:34 smithi037 ceph-mon[31460]: pgmap v689: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:36 smithi059 ceph-mon[36190]: pgmap v690: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 291.099063 secs 2024-03-28T18:31:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:36 smithi170 ceph-mon[36280]: pgmap v690: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 291.099063 secs 2024-03-28T18:31:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:36 smithi037 ceph-mon[31460]: pgmap v690: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 291.099063 secs 2024-03-28T18:31:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:37 smithi037 ceph-mon[31460]: pgmap v691: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:37 smithi059 ceph-mon[36190]: pgmap v691: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:37.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:37 smithi170 ceph-mon[36280]: pgmap v691: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:39.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:39 smithi059 ceph-mon[36190]: pgmap v692: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:39.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:39 smithi170 ceph-mon[36280]: pgmap v692: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:39.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:39 smithi037 ceph-mon[31460]: pgmap v692: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:41.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:41 smithi059 ceph-mon[36190]: pgmap v693: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:41.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 296.099140 secs 2024-03-28T18:31:41.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:41 smithi170 ceph-mon[36280]: pgmap v693: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:41.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 296.099140 secs 2024-03-28T18:31:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:41 smithi037 ceph-mon[31460]: pgmap v693: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 296.099140 secs 2024-03-28T18:31:43.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:43 smithi059 ceph-mon[36190]: pgmap v694: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:43.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:43 smithi170 ceph-mon[36280]: pgmap v694: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:43.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:43 smithi037 ceph-mon[31460]: pgmap v694: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:45.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:45 smithi059 ceph-mon[36190]: pgmap v695: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:45.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:45 smithi170 ceph-mon[36280]: pgmap v695: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:45.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:45 smithi037 ceph-mon[31460]: pgmap v695: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:46.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 301.099259 secs 2024-03-28T18:31:47.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 301.099259 secs 2024-03-28T18:31:47.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 301.099259 secs 2024-03-28T18:31:47.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:47 smithi037 ceph-mon[31460]: pgmap v696: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:48.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:47 smithi059 ceph-mon[36190]: pgmap v696: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:48.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:47 smithi170 ceph-mon[36280]: pgmap v696: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:49.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:49 smithi037 ceph-mon[31460]: pgmap v697: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:50.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:49 smithi059 ceph-mon[36190]: pgmap v697: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:49 smithi170 ceph-mon[36280]: pgmap v697: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:51.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:51 smithi037 ceph-mon[31460]: pgmap v698: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:51.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 306.099379 secs 2024-03-28T18:31:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:51 smithi059 ceph-mon[36190]: pgmap v698: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:52.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 306.099379 secs 2024-03-28T18:31:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:51 smithi170 ceph-mon[36280]: pgmap v698: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 306.099379 secs 2024-03-28T18:31:53.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:53 smithi037 ceph-mon[31460]: pgmap v699: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:53.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:53 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:53.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:53 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:53.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:53 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:53.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:53 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:53.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:53 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:53.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:53 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:53.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:53 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:54.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:53 smithi059 ceph-mon[36190]: pgmap v699: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:54.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:53 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:54.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:53 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:54.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:53 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:54.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:53 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:54.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:53 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:54.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:53 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:54.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:53 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:54.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:53 smithi170 ceph-mon[36280]: pgmap v699: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:54.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:53 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:54.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:53 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:54.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:53 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:54.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:53 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:54.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:53 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:54.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:53 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:54.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:53 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:55.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:55 smithi037 ceph-mon[31460]: pgmap v700: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:55.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:55 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:55.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:55 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:55 smithi059 ceph-mon[36190]: pgmap v700: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:55 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:55 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:55 smithi170 ceph-mon[36280]: pgmap v700: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:55 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:55 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:56.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 311.099467 secs 2024-03-28T18:31:56.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:56 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:56.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:56 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:56.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:56 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:56.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:56 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:56.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:56 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:56.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:56 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:56.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:56 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:56.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:56 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:56.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:56 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:56.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:56 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:56.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:56 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:56.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:56 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:56.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:56 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:56.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:56 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:56.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:56 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:56.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:56 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:56.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:56 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:56.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:56 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 311.099467 secs 2024-03-28T18:31:57.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:56 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:57.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:56 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:57.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:56 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:56 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:56 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:56 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:56 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:56 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:56 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:56 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:56 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:56 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:56 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:56 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:56 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:56 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:56 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:56 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 311.099467 secs 2024-03-28T18:31:57.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:56 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:57.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:56 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:57.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:56 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:56 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:56 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:56 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:56 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:56 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:56 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:56 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:56 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:56 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:56 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:56 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:56 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:56 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:56 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:57.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:56 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:57 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T18:31:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:57 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T18:31:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:57 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T18:31:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:57 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T18:31:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:57 smithi059 ceph-mon[36190]: pgmap v701: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:57 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:57 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:58.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:57 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T18:31:58.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:57 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T18:31:58.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:57 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T18:31:58.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:57 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T18:31:58.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:57 smithi170 ceph-mon[36280]: pgmap v701: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:58.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:57 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:58.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:57 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:58.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:57 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T18:31:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:57 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T18:31:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:57 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T18:31:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:57 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T18:31:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:57 smithi037 ceph-mon[31460]: pgmap v701: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:57 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:58.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:57 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:31:59.931 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:59 smithi037 ceph-mon[31460]: pgmap v702: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:59.931 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:59 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:59.931 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:59 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:59.931 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:59 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:59.931 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:59 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:59.931 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:59 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:59.932 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:59 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:59.932 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:59 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:59.932 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:59 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:59.932 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:59 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T18:31:59.932 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:59 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:31:59.932 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:59 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:59.932 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:59 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:31:59.932 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:59 smithi037 ceph-mon[31460]: pgmap v703: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:31:59.932 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:59 smithi037 ceph-mon[31460]: pgmap v704: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:31:59.932 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:31:59 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:32:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:59 smithi059 ceph-mon[36190]: pgmap v702: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:59 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:59 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:59 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:59 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:00.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:59 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:00.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:59 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:00.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:59 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:00.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:59 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:00.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:59 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T18:32:00.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:59 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:32:00.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:59 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:00.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:59 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:32:00.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:59 smithi059 ceph-mon[36190]: pgmap v703: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:32:00.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:59 smithi059 ceph-mon[36190]: pgmap v704: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:32:00.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:31:59 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:32:00.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:59 smithi170 ceph-mon[36280]: pgmap v702: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:00.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:59 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:00.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:59 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:00.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:59 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:00.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:59 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:00.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:59 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:00.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:59 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:00.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:59 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:00.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:59 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:00.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:59 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T18:32:00.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:59 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:32:00.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:59 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:00.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:59 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:32:00.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:59 smithi170 ceph-mon[36280]: pgmap v703: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:32:00.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:59 smithi170 ceph-mon[36280]: pgmap v704: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:32:00.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:31:59 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:32:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:00 smithi059 ceph-mon[36190]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-28T18:32:01.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:00 smithi170 ceph-mon[36280]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-28T18:32:01.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:00 smithi037 ceph-mon[31460]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-28T18:32:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 316.099588 secs 2024-03-28T18:32:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:01 smithi059 ceph-mon[36190]: pgmap v705: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:32:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 316.099588 secs 2024-03-28T18:32:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:01 smithi170 ceph-mon[36280]: pgmap v705: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:32:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 316.099588 secs 2024-03-28T18:32:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:01 smithi037 ceph-mon[31460]: pgmap v705: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:32:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:03 smithi059 ceph-mon[36190]: pgmap v706: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:32:04.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:03 smithi170 ceph-mon[36280]: pgmap v706: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:32:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:03 smithi037 ceph-mon[31460]: pgmap v706: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:32:06.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:05 smithi059 ceph-mon[36190]: pgmap v707: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:32:06.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:05 smithi170 ceph-mon[36280]: pgmap v707: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:32:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:05 smithi037 ceph-mon[31460]: pgmap v707: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:32:07.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 321.099703 secs 2024-03-28T18:32:07.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 321.099703 secs 2024-03-28T18:32:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 321.099703 secs 2024-03-28T18:32:08.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:07 smithi059 ceph-mon[36190]: pgmap v708: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:32:08.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:07 smithi170 ceph-mon[36280]: pgmap v708: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:32:08.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:07 smithi037 ceph-mon[31460]: pgmap v708: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:32:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:09 smithi059 ceph-mon[36190]: pgmap v709: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:32:10.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:09 smithi170 ceph-mon[36280]: pgmap v709: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:32:10.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:09 smithi037 ceph-mon[31460]: pgmap v709: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:32:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 326.099786 secs 2024-03-28T18:32:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:11 smithi059 ceph-mon[36190]: pgmap v710: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 326.099786 secs 2024-03-28T18:32:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:11 smithi170 ceph-mon[36280]: pgmap v710: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 326.099786 secs 2024-03-28T18:32:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:11 smithi037 ceph-mon[31460]: pgmap v710: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:14.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:13 smithi059 ceph-mon[36190]: pgmap v711: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:14.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:13 smithi170 ceph-mon[36280]: pgmap v711: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:13 smithi037 ceph-mon[31460]: pgmap v711: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:15 smithi059 ceph-mon[36190]: pgmap v712: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:16.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:15 smithi170 ceph-mon[36280]: pgmap v712: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:16.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:15 smithi037 ceph-mon[31460]: pgmap v712: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:17.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 331.099905 secs 2024-03-28T18:32:17.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 331.099905 secs 2024-03-28T18:32:17.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 331.099905 secs 2024-03-28T18:32:18.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:17 smithi170 ceph-mon[36280]: pgmap v713: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:18.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:17 smithi037 ceph-mon[31460]: pgmap v713: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:18.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:17 smithi059 ceph-mon[36190]: pgmap v713: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:20.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:19 smithi037 ceph-mon[31460]: pgmap v714: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:19 smithi059 ceph-mon[36190]: pgmap v714: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:20.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:19 smithi170 ceph-mon[36280]: pgmap v714: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:21.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:20 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 336.100046 secs 2024-03-28T18:32:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:20 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 336.100046 secs 2024-03-28T18:32:21.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:20 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 336.100046 secs 2024-03-28T18:32:22.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:21 smithi037 ceph-mon[31460]: pgmap v715: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:22.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:21 smithi059 ceph-mon[36190]: pgmap v715: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:21 smithi170 ceph-mon[36280]: pgmap v715: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:23 smithi059 ceph-mon[36190]: pgmap v716: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:24.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:23 smithi037 ceph-mon[31460]: pgmap v716: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:24.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:23 smithi170 ceph-mon[36280]: pgmap v716: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:26.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:25 smithi037 ceph-mon[31460]: pgmap v717: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:25 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 341.100141 secs 2024-03-28T18:32:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:25 smithi059 ceph-mon[36190]: pgmap v717: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:25 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 341.100141 secs 2024-03-28T18:32:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:25 smithi170 ceph-mon[36280]: pgmap v717: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:25 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 341.100141 secs 2024-03-28T18:32:28.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:27 smithi037 ceph-mon[31460]: pgmap v718: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:27 smithi059 ceph-mon[36190]: pgmap v718: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:28.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:27 smithi170 ceph-mon[36280]: pgmap v718: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:30.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:29 smithi037 ceph-mon[31460]: pgmap v719: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:30.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:29 smithi059 ceph-mon[36190]: pgmap v719: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:30.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:29 smithi170 ceph-mon[36280]: pgmap v719: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:31.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:30 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 346.100263 secs 2024-03-28T18:32:31.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:30 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 346.100263 secs 2024-03-28T18:32:31.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:30 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 346.100263 secs 2024-03-28T18:32:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:31 smithi037 ceph-mon[31460]: pgmap v720: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:32.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:31 smithi059 ceph-mon[36190]: pgmap v720: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:31 smithi170 ceph-mon[36280]: pgmap v720: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:34.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:33 smithi037 ceph-mon[31460]: pgmap v721: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:33 smithi059 ceph-mon[36190]: pgmap v721: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:34.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:33 smithi170 ceph-mon[36280]: pgmap v721: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:36.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:35 smithi037 ceph-mon[31460]: pgmap v722: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:36.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:35 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 351.100382 secs 2024-03-28T18:32:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:35 smithi059 ceph-mon[36190]: pgmap v722: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:35 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 351.100382 secs 2024-03-28T18:32:36.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:35 smithi170 ceph-mon[36280]: pgmap v722: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:36.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:35 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 351.100382 secs 2024-03-28T18:32:38.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:37 smithi037 ceph-mon[31460]: pgmap v723: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:38.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:37 smithi059 ceph-mon[36190]: pgmap v723: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:38.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:37 smithi170 ceph-mon[36280]: pgmap v723: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:40.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:39 smithi037 ceph-mon[31460]: pgmap v724: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:39 smithi059 ceph-mon[36190]: pgmap v724: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:40.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:39 smithi170 ceph-mon[36280]: pgmap v724: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:41.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:40 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 356.100504 secs 2024-03-28T18:32:41.186 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:40 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 356.100504 secs 2024-03-28T18:32:41.187 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:40 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 356.100504 secs 2024-03-28T18:32:42.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:41 smithi037 ceph-mon[31460]: pgmap v725: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:41 smithi059 ceph-mon[36190]: pgmap v725: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:42.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:41 smithi170 ceph-mon[36280]: pgmap v725: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:44.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:43 smithi037 ceph-mon[31460]: pgmap v726: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:43 smithi059 ceph-mon[36190]: pgmap v726: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:44.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:43 smithi170 ceph-mon[36280]: pgmap v726: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:46.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:45 smithi037 ceph-mon[31460]: pgmap v727: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:46.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:45 smithi037 ceph-mon[31460]: 2 slow requests, 1 included below; oldest blocked for > 361.100625 secs 2024-03-28T18:32:46.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:45 smithi037 ceph-mon[31460]: slow request 361.100624 seconds old, received at 2024-03-28T18:26:44.515856+0000: peer_request:client.25216:13687 currently acquired locks 2024-03-28T18:32:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:45 smithi059 ceph-mon[36190]: pgmap v727: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:45 smithi059 ceph-mon[36190]: 2 slow requests, 1 included below; oldest blocked for > 361.100625 secs 2024-03-28T18:32:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:45 smithi059 ceph-mon[36190]: slow request 361.100624 seconds old, received at 2024-03-28T18:26:44.515856+0000: peer_request:client.25216:13687 currently acquired locks 2024-03-28T18:32:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:45 smithi170 ceph-mon[36280]: pgmap v727: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:45 smithi170 ceph-mon[36280]: 2 slow requests, 1 included below; oldest blocked for > 361.100625 secs 2024-03-28T18:32:46.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:45 smithi170 ceph-mon[36280]: slow request 361.100624 seconds old, received at 2024-03-28T18:26:44.515856+0000: peer_request:client.25216:13687 currently acquired locks 2024-03-28T18:32:48.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:47 smithi037 ceph-mon[31460]: pgmap v728: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:48.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:47 smithi059 ceph-mon[36190]: pgmap v728: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:48.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:47 smithi170 ceph-mon[36280]: pgmap v728: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:49 smithi059 ceph-mon[36190]: pgmap v729: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:50.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:49 smithi170 ceph-mon[36280]: pgmap v729: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:50.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:49 smithi037 ceph-mon[31460]: pgmap v729: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:50 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 366.100744 secs 2024-03-28T18:32:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:50 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 366.100744 secs 2024-03-28T18:32:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:50 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 366.100744 secs 2024-03-28T18:32:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:51 smithi059 ceph-mon[36190]: pgmap v730: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:51 smithi170 ceph-mon[36280]: pgmap v730: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:52.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:51 smithi037 ceph-mon[31460]: pgmap v730: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:53 smithi059 ceph-mon[36190]: pgmap v731: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:54.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:53 smithi170 ceph-mon[36280]: pgmap v731: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:54.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:53 smithi037 ceph-mon[31460]: pgmap v731: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:55 smithi059 ceph-mon[36190]: pgmap v732: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:55 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 371.100864 secs 2024-03-28T18:32:56.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:55 smithi170 ceph-mon[36280]: pgmap v732: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:56.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:55 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 371.100864 secs 2024-03-28T18:32:56.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:55 smithi037 ceph-mon[31460]: pgmap v732: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:56.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:55 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 371.100864 secs 2024-03-28T18:32:58.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:57 smithi059 ceph-mon[36190]: pgmap v733: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:58.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:57 smithi170 ceph-mon[36280]: pgmap v733: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:58.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:57 smithi037 ceph-mon[31460]: pgmap v733: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:59 smithi059 ceph-mon[36190]: pgmap v734: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:32:59 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:00.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:59 smithi170 ceph-mon[36280]: pgmap v734: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:00.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:32:59 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:00.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:59 smithi037 ceph-mon[31460]: pgmap v734: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:00.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:32:59 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:00 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:00 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:33:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:00 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:33:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:00 smithi059 ceph-mon[36190]: 2 slow requests, 1 included below; oldest blocked for > 376.100984 secs 2024-03-28T18:33:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:00 smithi059 ceph-mon[36190]: slow request 360.520658 seconds old, received at 2024-03-28T18:27:00.096181+0000: client_request(client.25553:180 mksnap #0x1//scheduled-2024-03-28-18_27_00_UTC 2024-03-28T18:27:00.095960+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:33:01.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:00 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:01.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:00 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:33:01.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:00 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:33:01.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:00 smithi170 ceph-mon[36280]: 2 slow requests, 1 included below; oldest blocked for > 376.100984 secs 2024-03-28T18:33:01.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:00 smithi170 ceph-mon[36280]: slow request 360.520658 seconds old, received at 2024-03-28T18:27:00.096181+0000: client_request(client.25553:180 mksnap #0x1//scheduled-2024-03-28-18_27_00_UTC 2024-03-28T18:27:00.095960+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:33:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:00 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:00 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:33:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:00 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:33:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:00 smithi037 ceph-mon[31460]: 2 slow requests, 1 included below; oldest blocked for > 376.100984 secs 2024-03-28T18:33:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:00 smithi037 ceph-mon[31460]: slow request 360.520658 seconds old, received at 2024-03-28T18:27:00.096181+0000: client_request(client.25553:180 mksnap #0x1//scheduled-2024-03-28-18_27_00_UTC 2024-03-28T18:27:00.095960+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:33:02.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:01 smithi037 ceph-mon[31460]: pgmap v735: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:02.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:01 smithi037 ceph-mon[31460]: pgmap v736: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:02.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:01 smithi059 ceph-mon[36190]: pgmap v735: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:02.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:01 smithi059 ceph-mon[36190]: pgmap v736: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:02.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:01 smithi170 ceph-mon[36280]: pgmap v735: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:02.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:01 smithi170 ceph-mon[36280]: pgmap v736: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:04.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:03 smithi059 ceph-mon[36190]: pgmap v737: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:04.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:03 smithi170 ceph-mon[36280]: pgmap v737: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:04.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:03 smithi037 ceph-mon[31460]: pgmap v737: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:05 smithi059 ceph-mon[36190]: pgmap v738: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:05 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 381.101107 secs 2024-03-28T18:33:06.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:05 smithi170 ceph-mon[36280]: pgmap v738: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:06.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:05 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 381.101107 secs 2024-03-28T18:33:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:05 smithi037 ceph-mon[31460]: pgmap v738: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:05 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 381.101107 secs 2024-03-28T18:33:08.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:07 smithi059 ceph-mon[36190]: pgmap v739: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:33:08.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:07 smithi170 ceph-mon[36280]: pgmap v739: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:33:08.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:07 smithi037 ceph-mon[31460]: pgmap v739: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:33:10.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:09 smithi059 ceph-mon[36190]: pgmap v740: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:10.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:09 smithi170 ceph-mon[36280]: pgmap v740: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:10.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:09 smithi037 ceph-mon[31460]: pgmap v740: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:10 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 386.101226 secs 2024-03-28T18:33:11.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:10 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 386.101226 secs 2024-03-28T18:33:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:10 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 386.101226 secs 2024-03-28T18:33:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:11 smithi059 ceph-mon[36190]: pgmap v741: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:33:12.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:11 smithi170 ceph-mon[36280]: pgmap v741: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:33:12.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:11 smithi037 ceph-mon[31460]: pgmap v741: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:33:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:13 smithi059 ceph-mon[36190]: pgmap v742: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:14.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:13 smithi170 ceph-mon[36280]: pgmap v742: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:14.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:13 smithi037 ceph-mon[31460]: pgmap v742: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:16.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:15 smithi059 ceph-mon[36190]: pgmap v743: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:16.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:15 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 391.101346 secs 2024-03-28T18:33:16.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:15 smithi170 ceph-mon[36280]: pgmap v743: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:16.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:15 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 391.101346 secs 2024-03-28T18:33:16.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:15 smithi037 ceph-mon[31460]: pgmap v743: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:16.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:15 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 391.101346 secs 2024-03-28T18:33:18.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:17 smithi059 ceph-mon[36190]: pgmap v744: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:18.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:17 smithi170 ceph-mon[36280]: pgmap v744: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:18.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:17 smithi037 ceph-mon[31460]: pgmap v744: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:19 smithi059 ceph-mon[36190]: pgmap v745: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:20.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:19 smithi170 ceph-mon[36280]: pgmap v745: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:20.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:19 smithi037 ceph-mon[31460]: pgmap v745: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:20 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 396.101423 secs 2024-03-28T18:33:21.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:20 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 396.101423 secs 2024-03-28T18:33:21.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:20 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 396.101423 secs 2024-03-28T18:33:22.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:21 smithi059 ceph-mon[36190]: pgmap v746: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:22.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:21 smithi170 ceph-mon[36280]: pgmap v746: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:22.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:21 smithi037 ceph-mon[31460]: pgmap v746: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:24.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:23 smithi059 ceph-mon[36190]: pgmap v747: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:24.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:23 smithi170 ceph-mon[36280]: pgmap v747: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:24.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:23 smithi037 ceph-mon[31460]: pgmap v747: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:25 smithi059 ceph-mon[36190]: pgmap v748: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:25 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 401.101541 secs 2024-03-28T18:33:26.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:25 smithi170 ceph-mon[36280]: pgmap v748: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:26.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:25 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 401.101541 secs 2024-03-28T18:33:26.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:25 smithi037 ceph-mon[31460]: pgmap v748: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:26.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:25 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 401.101541 secs 2024-03-28T18:33:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:27 smithi059 ceph-mon[36190]: pgmap v749: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:28.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:27 smithi170 ceph-mon[36280]: pgmap v749: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:28.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:27 smithi037 ceph-mon[31460]: pgmap v749: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:30.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:29 smithi059 ceph-mon[36190]: pgmap v750: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:30.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:29 smithi170 ceph-mon[36280]: pgmap v750: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:30.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:29 smithi037 ceph-mon[31460]: pgmap v750: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 406.101661 secs 2024-03-28T18:33:31.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 406.101661 secs 2024-03-28T18:33:31.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 406.101661 secs 2024-03-28T18:33:32.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:32 smithi059 ceph-mon[36190]: pgmap v751: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:32 smithi170 ceph-mon[36280]: pgmap v751: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:32.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:32 smithi037 ceph-mon[31460]: pgmap v751: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:34 smithi059 ceph-mon[36190]: pgmap v752: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:34.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:34 smithi170 ceph-mon[36280]: pgmap v752: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:34.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:34 smithi037 ceph-mon[31460]: pgmap v752: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:36 smithi059 ceph-mon[36190]: pgmap v753: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 411.101746 secs 2024-03-28T18:33:36.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:36 smithi170 ceph-mon[36280]: pgmap v753: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:36.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 411.101746 secs 2024-03-28T18:33:36.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:36 smithi037 ceph-mon[31460]: pgmap v753: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:36.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 411.101746 secs 2024-03-28T18:33:38.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:38 smithi059 ceph-mon[36190]: pgmap v754: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:38.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:38 smithi170 ceph-mon[36280]: pgmap v754: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:38.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:38 smithi037 ceph-mon[31460]: pgmap v754: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:40 smithi059 ceph-mon[36190]: pgmap v755: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:40.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:40 smithi170 ceph-mon[36280]: pgmap v755: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:40.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:40 smithi037 ceph-mon[31460]: pgmap v755: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 416.101863 secs 2024-03-28T18:33:41.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 416.101863 secs 2024-03-28T18:33:41.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 416.101863 secs 2024-03-28T18:33:42.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:42 smithi037 ceph-mon[31460]: pgmap v756: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:42.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:42 smithi059 ceph-mon[36190]: pgmap v756: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:42.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:42 smithi170 ceph-mon[36280]: pgmap v756: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:43.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:43 smithi037 ceph-mon[31460]: pgmap v757: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:43.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:43 smithi059 ceph-mon[36190]: pgmap v757: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:43.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:43 smithi170 ceph-mon[36280]: pgmap v757: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:45.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:45 smithi059 ceph-mon[36190]: pgmap v758: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:45.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:45 smithi170 ceph-mon[36280]: pgmap v758: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:45.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:45 smithi037 ceph-mon[31460]: pgmap v758: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:46.526 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 421.101985 secs 2024-03-28T18:33:46.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 421.101985 secs 2024-03-28T18:33:46.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 421.101985 secs 2024-03-28T18:33:47.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:47 smithi059 ceph-mon[36190]: pgmap v759: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:47.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:47 smithi170 ceph-mon[36280]: pgmap v759: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:47.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:47 smithi037 ceph-mon[31460]: pgmap v759: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:49.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:49 smithi059 ceph-mon[36190]: pgmap v760: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:49.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:49 smithi170 ceph-mon[36280]: pgmap v760: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:49.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:49 smithi037 ceph-mon[31460]: pgmap v760: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:51.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:51 smithi059 ceph-mon[36190]: pgmap v761: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:51.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 426.102103 secs 2024-03-28T18:33:51.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:51 smithi170 ceph-mon[36280]: pgmap v761: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:51.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 426.102103 secs 2024-03-28T18:33:51.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:51 smithi037 ceph-mon[31460]: pgmap v761: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:51.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 426.102103 secs 2024-03-28T18:33:53.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:53 smithi059 ceph-mon[36190]: pgmap v762: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:53.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:53 smithi170 ceph-mon[36280]: pgmap v762: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:53.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:53 smithi037 ceph-mon[31460]: pgmap v762: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:55.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:55 smithi059 ceph-mon[36190]: pgmap v763: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:55.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:55 smithi170 ceph-mon[36280]: pgmap v763: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:55.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:55 smithi037 ceph-mon[31460]: pgmap v763: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 431.102221 secs 2024-03-28T18:33:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 431.102221 secs 2024-03-28T18:33:56.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 431.102221 secs 2024-03-28T18:33:57.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:57 smithi037 ceph-mon[31460]: pgmap v764: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:57 smithi059 ceph-mon[36190]: pgmap v764: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:57.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:57 smithi170 ceph-mon[36280]: pgmap v764: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:59.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:33:59 smithi037 ceph-mon[31460]: pgmap v765: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:33:59 smithi059 ceph-mon[36190]: pgmap v765: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:59.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:33:59 smithi170 ceph-mon[36280]: pgmap v765: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:01.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:01 smithi037 ceph-mon[31460]: pgmap v766: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:01.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:01 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:01.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 436.102291 secs 2024-03-28T18:34:01.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:01 smithi059 ceph-mon[36190]: pgmap v766: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:01.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:01 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:01.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 436.102291 secs 2024-03-28T18:34:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:01 smithi170 ceph-mon[36280]: pgmap v766: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:01 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 436.102291 secs 2024-03-28T18:34:02.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:02 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:02.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:02 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:02.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:02 smithi037 ceph-mon[31460]: pgmap v767: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:34:02.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:02 smithi037 ceph-mon[31460]: pgmap v768: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:02.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:02 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:34:02.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:02 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:02.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:02 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:02.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:02 smithi059 ceph-mon[36190]: pgmap v767: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:34:02.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:02 smithi059 ceph-mon[36190]: pgmap v768: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:02.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:02 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:34:02.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:02 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:02.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:02 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:02.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:02 smithi170 ceph-mon[36280]: pgmap v767: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:34:02.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:02 smithi170 ceph-mon[36280]: pgmap v768: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:02.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:02 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:34:05.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:04 smithi059 ceph-mon[36190]: pgmap v769: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:05.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:04 smithi170 ceph-mon[36280]: pgmap v769: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:05.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:04 smithi037 ceph-mon[31460]: pgmap v769: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:07.037 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 441.102411 secs 2024-03-28T18:34:07.037 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:06 smithi059 ceph-mon[36190]: pgmap v770: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:07.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 441.102411 secs 2024-03-28T18:34:07.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:06 smithi170 ceph-mon[36280]: pgmap v770: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 441.102411 secs 2024-03-28T18:34:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:06 smithi037 ceph-mon[31460]: pgmap v770: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:09.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:08 smithi059 ceph-mon[36190]: pgmap v771: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:09.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:08 smithi170 ceph-mon[36280]: pgmap v771: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:09.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:08 smithi037 ceph-mon[31460]: pgmap v771: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:10.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:10 smithi059 ceph-mon[36190]: pgmap v772: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:10.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:10 smithi170 ceph-mon[36280]: pgmap v772: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:10.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:10 smithi037 ceph-mon[31460]: pgmap v772: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:11.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 446.102530 secs 2024-03-28T18:34:11.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 446.102530 secs 2024-03-28T18:34:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 446.102530 secs 2024-03-28T18:34:12.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:12 smithi059 ceph-mon[36190]: pgmap v773: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:34:12.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:12 smithi170 ceph-mon[36280]: pgmap v773: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:34:12.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:12 smithi037 ceph-mon[31460]: pgmap v773: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:34:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:14 smithi059 ceph-mon[36190]: pgmap v774: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:15.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:14 smithi170 ceph-mon[36280]: pgmap v774: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:15.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:14 smithi037 ceph-mon[31460]: pgmap v774: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:17.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 451.102650 secs 2024-03-28T18:34:17.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:16 smithi059 ceph-mon[36190]: pgmap v775: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:17.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 451.102650 secs 2024-03-28T18:34:17.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:16 smithi170 ceph-mon[36280]: pgmap v775: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:17.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 451.102650 secs 2024-03-28T18:34:17.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:16 smithi037 ceph-mon[31460]: pgmap v775: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:18 smithi059 ceph-mon[36190]: pgmap v776: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:19.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:18 smithi170 ceph-mon[36280]: pgmap v776: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:19.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:18 smithi037 ceph-mon[31460]: pgmap v776: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:21.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:20 smithi059 ceph-mon[36190]: pgmap v777: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:21.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:20 smithi170 ceph-mon[36280]: pgmap v777: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:21.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:20 smithi037 ceph-mon[31460]: pgmap v777: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 456.102770 secs 2024-03-28T18:34:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 456.102770 secs 2024-03-28T18:34:22.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 456.102770 secs 2024-03-28T18:34:23.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:22 smithi059 ceph-mon[36190]: pgmap v778: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:23.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:22 smithi170 ceph-mon[36280]: pgmap v778: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:23.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:22 smithi037 ceph-mon[31460]: pgmap v778: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:25.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:24 smithi059 ceph-mon[36190]: pgmap v779: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:25.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:24 smithi170 ceph-mon[36280]: pgmap v779: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:25.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:24 smithi037 ceph-mon[31460]: pgmap v779: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 461.102891 secs 2024-03-28T18:34:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:26 smithi059 ceph-mon[36190]: pgmap v780: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:27.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 461.102891 secs 2024-03-28T18:34:27.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:26 smithi170 ceph-mon[36280]: pgmap v780: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:27.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 461.102891 secs 2024-03-28T18:34:27.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:26 smithi037 ceph-mon[31460]: pgmap v780: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:29.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:28 smithi059 ceph-mon[36190]: pgmap v781: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:29.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:28 smithi170 ceph-mon[36280]: pgmap v781: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:29.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:28 smithi037 ceph-mon[31460]: pgmap v781: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:31.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:30 smithi059 ceph-mon[36190]: pgmap v782: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:31.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:30 smithi170 ceph-mon[36280]: pgmap v782: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:31.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:30 smithi037 ceph-mon[31460]: pgmap v782: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 466.102985 secs 2024-03-28T18:34:32.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 466.102985 secs 2024-03-28T18:34:32.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 466.102985 secs 2024-03-28T18:34:33.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:32 smithi059 ceph-mon[36190]: pgmap v783: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:33.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:32 smithi170 ceph-mon[36280]: pgmap v783: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:33.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:32 smithi037 ceph-mon[31460]: pgmap v783: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:34 smithi059 ceph-mon[36190]: pgmap v784: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:35.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:34 smithi170 ceph-mon[36280]: pgmap v784: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:35.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:34 smithi037 ceph-mon[31460]: pgmap v784: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:37.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 471.103108 secs 2024-03-28T18:34:37.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:36 smithi059 ceph-mon[36190]: pgmap v785: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:37.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 471.103108 secs 2024-03-28T18:34:37.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:36 smithi170 ceph-mon[36280]: pgmap v785: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:37.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 471.103108 secs 2024-03-28T18:34:37.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:36 smithi037 ceph-mon[31460]: pgmap v785: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:39.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:38 smithi059 ceph-mon[36190]: pgmap v786: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:39.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:38 smithi170 ceph-mon[36280]: pgmap v786: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:39.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:38 smithi037 ceph-mon[31460]: pgmap v786: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:40 smithi059 ceph-mon[36190]: pgmap v787: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:41.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:40 smithi170 ceph-mon[36280]: pgmap v787: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:41.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:40 smithi037 ceph-mon[31460]: pgmap v787: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:42.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 476.103228 secs 2024-03-28T18:34:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 476.103228 secs 2024-03-28T18:34:42.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 476.103228 secs 2024-03-28T18:34:43.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:42 smithi037 ceph-mon[31460]: pgmap v788: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:43.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:42 smithi059 ceph-mon[36190]: pgmap v788: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:43.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:42 smithi170 ceph-mon[36280]: pgmap v788: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:45.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:44 smithi037 ceph-mon[31460]: pgmap v789: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:44 smithi059 ceph-mon[36190]: pgmap v789: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:45.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:44 smithi170 ceph-mon[36280]: pgmap v789: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 481.103349 secs 2024-03-28T18:34:47.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:46 smithi037 ceph-mon[31460]: pgmap v790: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 481.103349 secs 2024-03-28T18:34:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:46 smithi059 ceph-mon[36190]: pgmap v790: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 481.103349 secs 2024-03-28T18:34:47.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:46 smithi170 ceph-mon[36280]: pgmap v790: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:49.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:48 smithi037 ceph-mon[31460]: pgmap v791: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:48 smithi059 ceph-mon[36190]: pgmap v791: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:48 smithi170 ceph-mon[36280]: pgmap v791: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:51.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:50 smithi037 ceph-mon[31460]: pgmap v792: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:50 smithi059 ceph-mon[36190]: pgmap v792: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:50 smithi170 ceph-mon[36280]: pgmap v792: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 486.103469 secs 2024-03-28T18:34:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 486.103469 secs 2024-03-28T18:34:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 486.103469 secs 2024-03-28T18:34:53.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:52 smithi037 ceph-mon[31460]: pgmap v793: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:52 smithi059 ceph-mon[36190]: pgmap v793: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:52 smithi170 ceph-mon[36280]: pgmap v793: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:55.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:54 smithi037 ceph-mon[31460]: pgmap v794: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:54 smithi059 ceph-mon[36190]: pgmap v794: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:55.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:54 smithi170 ceph-mon[36280]: pgmap v794: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:56.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:55 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 491.103587 secs 2024-03-28T18:34:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:55 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 491.103587 secs 2024-03-28T18:34:56.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:55 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 491.103587 secs 2024-03-28T18:34:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:56 smithi037 ceph-mon[31460]: pgmap v795: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:56 smithi059 ceph-mon[36190]: pgmap v795: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:56 smithi170 ceph-mon[36280]: pgmap v795: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:59.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:34:58 smithi037 ceph-mon[31460]: pgmap v796: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:59.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:34:58 smithi059 ceph-mon[36190]: pgmap v796: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:59.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:34:58 smithi170 ceph-mon[36280]: pgmap v796: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:01.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:00 smithi037 ceph-mon[31460]: pgmap v797: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:01.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:00 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 496.103706 secs 2024-03-28T18:35:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:00 smithi059 ceph-mon[36190]: pgmap v797: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:00 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 496.103706 secs 2024-03-28T18:35:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:00 smithi170 ceph-mon[36280]: pgmap v797: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:00 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 496.103706 secs 2024-03-28T18:35:03.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:02 smithi037 ceph-mon[31460]: pgmap v798: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:03.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:02 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:02 smithi059 ceph-mon[36190]: pgmap v798: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:02 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:03.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:02 smithi170 ceph-mon[36280]: pgmap v798: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:03.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:02 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:04.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:03 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:04.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:03 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:04.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:03 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:35:04.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:03 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:04.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:03 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:04.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:03 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:35:04.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:03 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:04.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:03 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:04.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:03 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:35:05.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:04 smithi037 ceph-mon[31460]: pgmap v799: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:05.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:04 smithi037 ceph-mon[31460]: pgmap v800: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:05.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:04 smithi059 ceph-mon[36190]: pgmap v799: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:05.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:04 smithi059 ceph-mon[36190]: pgmap v800: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:05.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:04 smithi170 ceph-mon[36280]: pgmap v799: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:05.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:04 smithi170 ceph-mon[36280]: pgmap v800: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:05 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 501.103830 secs 2024-03-28T18:35:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:05 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 501.103830 secs 2024-03-28T18:35:06.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:05 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 501.103830 secs 2024-03-28T18:35:07.026 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:06 smithi037 ceph-mon[31460]: pgmap v801: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:07.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:06 smithi059 ceph-mon[36190]: pgmap v801: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:07.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:06 smithi170 ceph-mon[36280]: pgmap v801: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:08 smithi059 ceph-mon[36190]: pgmap v802: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:09.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:08 smithi170 ceph-mon[36280]: pgmap v802: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:09.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:08 smithi037 ceph-mon[31460]: pgmap v802: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:10 smithi059 ceph-mon[36190]: pgmap v803: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:10 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 506.103949 secs 2024-03-28T18:35:11.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:10 smithi170 ceph-mon[36280]: pgmap v803: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:11.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:10 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 506.103949 secs 2024-03-28T18:35:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:10 smithi037 ceph-mon[31460]: pgmap v803: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:10 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 506.103949 secs 2024-03-28T18:35:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:12 smithi059 ceph-mon[36190]: pgmap v804: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:35:13.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:12 smithi170 ceph-mon[36280]: pgmap v804: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:35:13.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:12 smithi037 ceph-mon[31460]: pgmap v804: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:35:15.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:14 smithi059 ceph-mon[36190]: pgmap v805: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:35:15.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:14 smithi170 ceph-mon[36280]: pgmap v805: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:35:15.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:14 smithi037 ceph-mon[31460]: pgmap v805: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:35:16.276 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:15 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 511.104070 secs 2024-03-28T18:35:16.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:15 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 511.104070 secs 2024-03-28T18:35:16.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:15 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 511.104070 secs 2024-03-28T18:35:17.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:16 smithi059 ceph-mon[36190]: pgmap v806: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:16 smithi170 ceph-mon[36280]: pgmap v806: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:17.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:16 smithi037 ceph-mon[31460]: pgmap v806: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:19.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:18 smithi059 ceph-mon[36190]: pgmap v807: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:19.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:18 smithi170 ceph-mon[36280]: pgmap v807: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:19.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:18 smithi037 ceph-mon[31460]: pgmap v807: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:20 smithi059 ceph-mon[36190]: pgmap v808: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:20 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 516.104190 secs 2024-03-28T18:35:21.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:20 smithi170 ceph-mon[36280]: pgmap v808: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:21.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:20 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 516.104190 secs 2024-03-28T18:35:21.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:20 smithi037 ceph-mon[31460]: pgmap v808: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:21.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:20 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 516.104190 secs 2024-03-28T18:35:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:22 smithi059 ceph-mon[36190]: pgmap v809: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:23.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:22 smithi170 ceph-mon[36280]: pgmap v809: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:23.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:22 smithi037 ceph-mon[31460]: pgmap v809: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:25.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:24 smithi059 ceph-mon[36190]: pgmap v810: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:25.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:24 smithi170 ceph-mon[36280]: pgmap v810: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:25.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:24 smithi037 ceph-mon[31460]: pgmap v810: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:25 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 521.104311 secs 2024-03-28T18:35:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:25 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 521.104311 secs 2024-03-28T18:35:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:25 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 521.104311 secs 2024-03-28T18:35:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:26 smithi059 ceph-mon[36190]: pgmap v811: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:26 smithi170 ceph-mon[36280]: pgmap v811: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:27.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:26 smithi037 ceph-mon[31460]: pgmap v811: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:29.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:28 smithi059 ceph-mon[36190]: pgmap v812: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:29.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:28 smithi170 ceph-mon[36280]: pgmap v812: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:29.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:28 smithi037 ceph-mon[31460]: pgmap v812: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:30 smithi059 ceph-mon[36190]: pgmap v813: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:30 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 526.104433 secs 2024-03-28T18:35:31.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:30 smithi170 ceph-mon[36280]: pgmap v813: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:31.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:30 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 526.104433 secs 2024-03-28T18:35:31.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:30 smithi037 ceph-mon[31460]: pgmap v813: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:31.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:30 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 526.104433 secs 2024-03-28T18:35:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:32 smithi059 ceph-mon[36190]: pgmap v814: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:33.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:32 smithi170 ceph-mon[36280]: pgmap v814: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:33.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:32 smithi037 ceph-mon[31460]: pgmap v814: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:34 smithi059 ceph-mon[36190]: pgmap v815: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:35.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:34 smithi170 ceph-mon[36280]: pgmap v815: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:35.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:34 smithi037 ceph-mon[31460]: pgmap v815: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:35 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 531.104554 secs 2024-03-28T18:35:36.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:35 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 531.104554 secs 2024-03-28T18:35:36.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:35 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 531.104554 secs 2024-03-28T18:35:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:36 smithi059 ceph-mon[36190]: pgmap v816: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:36 smithi170 ceph-mon[36280]: pgmap v816: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:36 smithi037 ceph-mon[31460]: pgmap v816: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:39.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:38 smithi059 ceph-mon[36190]: pgmap v817: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:39.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:38 smithi170 ceph-mon[36280]: pgmap v817: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:39.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:38 smithi037 ceph-mon[31460]: pgmap v817: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:40 smithi059 ceph-mon[36190]: pgmap v818: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:40 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 536.104672 secs 2024-03-28T18:35:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:40 smithi170 ceph-mon[36280]: pgmap v818: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:40 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 536.104672 secs 2024-03-28T18:35:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:40 smithi037 ceph-mon[31460]: pgmap v818: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:40 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 536.104672 secs 2024-03-28T18:35:43.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:42 smithi059 ceph-mon[36190]: pgmap v819: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:43.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:42 smithi170 ceph-mon[36280]: pgmap v819: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:43.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:42 smithi037 ceph-mon[31460]: pgmap v819: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:44 smithi059 ceph-mon[36190]: pgmap v820: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:45.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:44 smithi170 ceph-mon[36280]: pgmap v820: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:45.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:44 smithi037 ceph-mon[31460]: pgmap v820: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:46.276 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:45 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 541.104792 secs 2024-03-28T18:35:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:45 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 541.104792 secs 2024-03-28T18:35:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:45 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 541.104792 secs 2024-03-28T18:35:47.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:46 smithi059 ceph-mon[36190]: pgmap v821: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:47.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:46 smithi170 ceph-mon[36280]: pgmap v821: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:47.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:46 smithi037 ceph-mon[31460]: pgmap v821: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:49 smithi059 ceph-mon[36190]: pgmap v822: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:49 smithi170 ceph-mon[36280]: pgmap v822: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:49.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:49 smithi037 ceph-mon[31460]: pgmap v822: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:51 smithi059 ceph-mon[36190]: pgmap v823: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:51.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 546.104913 secs 2024-03-28T18:35:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:51 smithi170 ceph-mon[36280]: pgmap v823: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 546.104913 secs 2024-03-28T18:35:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:51 smithi037 ceph-mon[31460]: pgmap v823: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:51.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 546.104913 secs 2024-03-28T18:35:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:53 smithi037 ceph-mon[31460]: pgmap v824: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:53 smithi059 ceph-mon[36190]: pgmap v824: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:53 smithi170 ceph-mon[36280]: pgmap v824: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:55 smithi059 ceph-mon[36190]: pgmap v825: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:55.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:55 smithi170 ceph-mon[36280]: pgmap v825: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:55.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:55 smithi037 ceph-mon[31460]: pgmap v825: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 551.105061 secs 2024-03-28T18:35:56.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 551.105061 secs 2024-03-28T18:35:56.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 551.105061 secs 2024-03-28T18:35:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:57 smithi059 ceph-mon[36190]: pgmap v826: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:57 smithi170 ceph-mon[36280]: pgmap v826: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:57.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:57 smithi037 ceph-mon[31460]: pgmap v826: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:59.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:35:59 smithi059 ceph-mon[36190]: pgmap v827: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:59.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:35:59 smithi170 ceph-mon[36280]: pgmap v827: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:59.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:35:59 smithi037 ceph-mon[31460]: pgmap v827: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:01 smithi059 ceph-mon[36190]: pgmap v828: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 556.105178 secs 2024-03-28T18:36:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:01 smithi170 ceph-mon[36280]: pgmap v828: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:01.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 556.105178 secs 2024-03-28T18:36:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:01 smithi037 ceph-mon[31460]: pgmap v828: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 556.105178 secs 2024-03-28T18:36:03.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:03 smithi037 ceph-mon[31460]: pgmap v829: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:03 smithi059 ceph-mon[36190]: pgmap v829: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:03.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:03 smithi170 ceph-mon[36280]: pgmap v829: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:04.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:04 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:04.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:04 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:04.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:04 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:05.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:05 smithi037 ceph-mon[31460]: pgmap v830: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:05.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:05 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:05.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:05 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:36:05.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:05 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:36:05.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:05 smithi059 ceph-mon[36190]: pgmap v830: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:05.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:05 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:05.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:05 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:36:05.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:05 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:36:05.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:05 smithi170 ceph-mon[36280]: pgmap v830: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:05.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:05 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:05.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:05 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:36:05.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:05 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:36:06.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:06 smithi037 ceph-mon[31460]: pgmap v831: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:06.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:06 smithi037 ceph-mon[31460]: pgmap v832: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:06.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 561.105297 secs 2024-03-28T18:36:06.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:06 smithi059 ceph-mon[36190]: pgmap v831: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:06.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:06 smithi059 ceph-mon[36190]: pgmap v832: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:06.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 561.105297 secs 2024-03-28T18:36:06.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:06 smithi170 ceph-mon[36280]: pgmap v831: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:06.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:06 smithi170 ceph-mon[36280]: pgmap v832: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:06.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 561.105297 secs 2024-03-28T18:36:07.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:07 smithi037 ceph-mon[31460]: pgmap v833: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:07.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:07 smithi059 ceph-mon[36190]: pgmap v833: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:07.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:07 smithi170 ceph-mon[36280]: pgmap v833: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:10.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:09 smithi059 ceph-mon[36190]: pgmap v834: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:10.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:09 smithi170 ceph-mon[36280]: pgmap v834: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:10.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:09 smithi037 ceph-mon[31460]: pgmap v834: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:11 smithi059 ceph-mon[36190]: pgmap v835: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 566.105377 secs 2024-03-28T18:36:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:11 smithi170 ceph-mon[36280]: pgmap v835: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 566.105377 secs 2024-03-28T18:36:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:11 smithi037 ceph-mon[31460]: pgmap v835: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 566.105377 secs 2024-03-28T18:36:14.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:13 smithi059 ceph-mon[36190]: pgmap v836: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:14.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:13 smithi170 ceph-mon[36280]: pgmap v836: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:13 smithi037 ceph-mon[31460]: pgmap v836: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:16.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:15 smithi059 ceph-mon[36190]: pgmap v837: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:36:16.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:15 smithi170 ceph-mon[36280]: pgmap v837: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:36:16.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:15 smithi037 ceph-mon[31460]: pgmap v837: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:36:17.025 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 571.105471 secs 2024-03-28T18:36:17.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 571.105471 secs 2024-03-28T18:36:17.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 571.105471 secs 2024-03-28T18:36:18.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:17 smithi059 ceph-mon[36190]: pgmap v838: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:18.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:17 smithi170 ceph-mon[36280]: pgmap v838: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:18.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:17 smithi037 ceph-mon[31460]: pgmap v838: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:19 smithi059 ceph-mon[36190]: pgmap v839: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:20.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:19 smithi170 ceph-mon[36280]: pgmap v839: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:20.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:19 smithi037 ceph-mon[31460]: pgmap v839: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:21 smithi059 ceph-mon[36190]: pgmap v840: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 576.105564 secs 2024-03-28T18:36:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:21 smithi170 ceph-mon[36280]: pgmap v840: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 576.105564 secs 2024-03-28T18:36:22.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:21 smithi037 ceph-mon[31460]: pgmap v840: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:22.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 576.105564 secs 2024-03-28T18:36:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:23 smithi059 ceph-mon[36190]: pgmap v841: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:23 smithi170 ceph-mon[36280]: pgmap v841: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:24.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:23 smithi037 ceph-mon[31460]: pgmap v841: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:25 smithi059 ceph-mon[36190]: pgmap v842: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:26.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:25 smithi170 ceph-mon[36280]: pgmap v842: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:26.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:25 smithi037 ceph-mon[31460]: pgmap v842: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 581.105684 secs 2024-03-28T18:36:27.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 581.105684 secs 2024-03-28T18:36:27.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 581.105684 secs 2024-03-28T18:36:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:27 smithi059 ceph-mon[36190]: pgmap v843: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:28.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:27 smithi170 ceph-mon[36280]: pgmap v843: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:28.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:27 smithi037 ceph-mon[31460]: pgmap v843: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:29 smithi059 ceph-mon[36190]: pgmap v844: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:30.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:29 smithi170 ceph-mon[36280]: pgmap v844: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:30.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:29 smithi037 ceph-mon[31460]: pgmap v844: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 586.105805 secs 2024-03-28T18:36:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:31 smithi059 ceph-mon[36190]: pgmap v845: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:32.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 586.105805 secs 2024-03-28T18:36:32.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:31 smithi170 ceph-mon[36280]: pgmap v845: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:32.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 586.105805 secs 2024-03-28T18:36:32.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:31 smithi037 ceph-mon[31460]: pgmap v845: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:34.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:33 smithi059 ceph-mon[36190]: pgmap v846: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:34.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:33 smithi170 ceph-mon[36280]: pgmap v846: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:34.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:33 smithi037 ceph-mon[31460]: pgmap v846: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:35 smithi059 ceph-mon[36190]: pgmap v847: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:36.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:35 smithi170 ceph-mon[36280]: pgmap v847: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:36.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:35 smithi037 ceph-mon[31460]: pgmap v847: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:37.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 591.105926 secs 2024-03-28T18:36:37.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 591.105926 secs 2024-03-28T18:36:37.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 591.105926 secs 2024-03-28T18:36:38.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:37 smithi059 ceph-mon[36190]: pgmap v848: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:38.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:37 smithi170 ceph-mon[36280]: pgmap v848: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:38.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:37 smithi037 ceph-mon[31460]: pgmap v848: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:39 smithi059 ceph-mon[36190]: pgmap v849: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:40.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:39 smithi170 ceph-mon[36280]: pgmap v849: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:40.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:39 smithi037 ceph-mon[31460]: pgmap v849: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:42.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 596.106068 secs 2024-03-28T18:36:42.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:41 smithi059 ceph-mon[36190]: pgmap v850: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:42.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 596.106068 secs 2024-03-28T18:36:42.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:41 smithi170 ceph-mon[36280]: pgmap v850: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:42.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 596.106068 secs 2024-03-28T18:36:42.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:41 smithi037 ceph-mon[31460]: pgmap v850: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:44.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:43 smithi059 ceph-mon[36190]: pgmap v851: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:44.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:43 smithi170 ceph-mon[36280]: pgmap v851: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:44.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:43 smithi037 ceph-mon[31460]: pgmap v851: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:46.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:45 smithi059 ceph-mon[36190]: pgmap v852: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:46.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:45 smithi170 ceph-mon[36280]: pgmap v852: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:46.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:45 smithi037 ceph-mon[31460]: pgmap v852: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 601.106200 secs 2024-03-28T18:36:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 601.106200 secs 2024-03-28T18:36:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 601.106200 secs 2024-03-28T18:36:48.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:47 smithi037 ceph-mon[31460]: pgmap v853: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:48.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:47 smithi059 ceph-mon[36190]: pgmap v853: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:48.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:47 smithi170 ceph-mon[36280]: pgmap v853: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:49 smithi037 ceph-mon[31460]: pgmap v854: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:49 smithi059 ceph-mon[36190]: pgmap v854: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:50.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:49 smithi170 ceph-mon[36280]: pgmap v854: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 606.106318 secs 2024-03-28T18:36:52.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:51 smithi037 ceph-mon[31460]: pgmap v855: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 606.106318 secs 2024-03-28T18:36:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:51 smithi059 ceph-mon[36190]: pgmap v855: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 606.106318 secs 2024-03-28T18:36:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:51 smithi170 ceph-mon[36280]: pgmap v855: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:54.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:53 smithi037 ceph-mon[31460]: pgmap v856: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:53 smithi059 ceph-mon[36190]: pgmap v856: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:54.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:53 smithi170 ceph-mon[36280]: pgmap v856: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:56.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:55 smithi037 ceph-mon[31460]: pgmap v857: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:55 smithi059 ceph-mon[36190]: pgmap v857: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:56.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:55 smithi170 ceph-mon[36280]: pgmap v857: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 611.106407 secs 2024-03-28T18:36:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 611.106407 secs 2024-03-28T18:36:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 611.106407 secs 2024-03-28T18:36:58.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:57 smithi037 ceph-mon[31460]: pgmap v858: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:58.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:57 smithi059 ceph-mon[36190]: pgmap v858: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:58.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:57 smithi170 ceph-mon[36280]: pgmap v858: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:00.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:36:59 smithi037 ceph-mon[31460]: pgmap v859: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:36:59 smithi059 ceph-mon[36190]: pgmap v859: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:00.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:36:59 smithi170 ceph-mon[36280]: pgmap v859: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 616.106507 secs 2024-03-28T18:37:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:01 smithi037 ceph-mon[31460]: pgmap v860: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:02.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 616.106507 secs 2024-03-28T18:37:02.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:01 smithi059 ceph-mon[36190]: pgmap v860: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:02.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 616.106507 secs 2024-03-28T18:37:02.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:01 smithi170 ceph-mon[36280]: pgmap v860: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:03 smithi037 ceph-mon[31460]: pgmap v861: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:04.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:03 smithi059 ceph-mon[36190]: pgmap v861: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:04.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:03 smithi170 ceph-mon[36280]: pgmap v861: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:05 smithi037 ceph-mon[31460]: pgmap v862: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:05 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:37:06.224 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:05 smithi059 ceph-mon[36190]: pgmap v862: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:06.225 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:05 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:37:06.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:05 smithi170 ceph-mon[36280]: pgmap v862: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:06.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:05 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:37:07.122 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 621.106586 secs 2024-03-28T18:37:07.248 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 621.106586 secs 2024-03-28T18:37:07.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 621.106586 secs 2024-03-28T18:37:08.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:07 smithi037 ceph-mon[31460]: pgmap v863: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:08.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:07 smithi170 ceph-mon[36280]: pgmap v863: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:08.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:07 smithi059 ceph-mon[36190]: pgmap v863: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:10.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:09 smithi037 ceph-mon[31460]: pgmap v864: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:09 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:37:10.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:09 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:37:10.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:09 smithi059 ceph-mon[36190]: pgmap v864: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:10.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:09 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:37:10.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:09 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:37:10.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:09 smithi170 ceph-mon[36280]: pgmap v864: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:10.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:09 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:37:10.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:09 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:37:11.025 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:10 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 626.106669 secs 2024-03-28T18:37:11.186 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:10 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 626.106669 secs 2024-03-28T18:37:11.187 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:10 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 626.106669 secs 2024-03-28T18:37:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:11 smithi037 ceph-mon[31460]: pgmap v865: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:11 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:37:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:11 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:37:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:11 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:11 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:12.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:11 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:37:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:11 smithi059 ceph-mon[36190]: pgmap v865: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:11 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:37:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:11 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:37:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:11 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:11 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:11 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:37:12.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:11 smithi170 ceph-mon[36280]: pgmap v865: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:12.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:11 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:37:12.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:11 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:37:12.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:11 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:12.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:11 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:12.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:11 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:37:13.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:12 smithi037 ceph-mon[31460]: pgmap v866: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T18:37:13.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:12 smithi037 ceph-mon[31460]: pgmap v867: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:37:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:12 smithi059 ceph-mon[36190]: pgmap v866: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T18:37:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:12 smithi059 ceph-mon[36190]: pgmap v867: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:37:13.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:12 smithi170 ceph-mon[36280]: pgmap v866: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T18:37:13.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:12 smithi170 ceph-mon[36280]: pgmap v867: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:37:15.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:14 smithi037 ceph-mon[31460]: pgmap v868: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:37:15.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:14 smithi059 ceph-mon[36190]: pgmap v868: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:37:15.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:14 smithi170 ceph-mon[36280]: pgmap v868: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:37:16.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:15 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 631.106791 secs 2024-03-28T18:37:16.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:15 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 631.106791 secs 2024-03-28T18:37:16.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:15 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 631.106791 secs 2024-03-28T18:37:17.276 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:16 smithi037 ceph-mon[31460]: pgmap v869: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:37:17.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:16 smithi059 ceph-mon[36190]: pgmap v869: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:37:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:16 smithi170 ceph-mon[36280]: pgmap v869: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:37:19.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:18 smithi059 ceph-mon[36190]: pgmap v870: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:37:19.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:18 smithi170 ceph-mon[36280]: pgmap v870: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:37:19.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:18 smithi037 ceph-mon[31460]: pgmap v870: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:37:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:20 smithi059 ceph-mon[36190]: pgmap v871: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:37:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:20 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 636.106908 secs 2024-03-28T18:37:21.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:20 smithi170 ceph-mon[36280]: pgmap v871: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:37:21.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:20 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 636.106908 secs 2024-03-28T18:37:21.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:20 smithi037 ceph-mon[31460]: pgmap v871: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:37:21.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:20 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 636.106908 secs 2024-03-28T18:37:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:22 smithi059 ceph-mon[36190]: pgmap v872: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:37:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:22 smithi170 ceph-mon[36280]: pgmap v872: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:37:23.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:22 smithi037 ceph-mon[31460]: pgmap v872: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:37:25.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:24 smithi059 ceph-mon[36190]: pgmap v873: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:25.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:24 smithi170 ceph-mon[36280]: pgmap v873: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:25.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:24 smithi037 ceph-mon[31460]: pgmap v873: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:25 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 641.107047 secs 2024-03-28T18:37:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:25 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 641.107047 secs 2024-03-28T18:37:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:25 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 641.107047 secs 2024-03-28T18:37:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:26 smithi059 ceph-mon[36190]: pgmap v874: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:26 smithi170 ceph-mon[36280]: pgmap v874: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:26 smithi037 ceph-mon[31460]: pgmap v874: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:29.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:28 smithi059 ceph-mon[36190]: pgmap v875: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:29.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:28 smithi170 ceph-mon[36280]: pgmap v875: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:29.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:28 smithi037 ceph-mon[31460]: pgmap v875: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:30 smithi059 ceph-mon[36190]: pgmap v876: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:30 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 646.107164 secs 2024-03-28T18:37:31.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:30 smithi170 ceph-mon[36280]: pgmap v876: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:31.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:30 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 646.107164 secs 2024-03-28T18:37:31.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:30 smithi037 ceph-mon[31460]: pgmap v876: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:31.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:30 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 646.107164 secs 2024-03-28T18:37:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:32 smithi059 ceph-mon[36190]: pgmap v877: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:33.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:32 smithi170 ceph-mon[36280]: pgmap v877: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:33.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:32 smithi037 ceph-mon[31460]: pgmap v877: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:34 smithi059 ceph-mon[36190]: pgmap v878: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:35.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:34 smithi170 ceph-mon[36280]: pgmap v878: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:35.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:34 smithi037 ceph-mon[31460]: pgmap v878: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:35 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 651.107284 secs 2024-03-28T18:37:36.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:35 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 651.107284 secs 2024-03-28T18:37:36.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:35 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 651.107284 secs 2024-03-28T18:37:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:36 smithi059 ceph-mon[36190]: pgmap v879: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:36 smithi170 ceph-mon[36280]: pgmap v879: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:36 smithi037 ceph-mon[31460]: pgmap v879: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:39.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:38 smithi059 ceph-mon[36190]: pgmap v880: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:39.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:38 smithi170 ceph-mon[36280]: pgmap v880: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:39.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:38 smithi037 ceph-mon[31460]: pgmap v880: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:40 smithi059 ceph-mon[36190]: pgmap v881: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:40 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 656.107363 secs 2024-03-28T18:37:41.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:40 smithi170 ceph-mon[36280]: pgmap v881: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:40 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 656.107363 secs 2024-03-28T18:37:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:40 smithi037 ceph-mon[31460]: pgmap v881: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:40 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 656.107363 secs 2024-03-28T18:37:43.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:42 smithi059 ceph-mon[36190]: pgmap v882: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:43.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:42 smithi170 ceph-mon[36280]: pgmap v882: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:43.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:42 smithi037 ceph-mon[31460]: pgmap v882: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:44 smithi059 ceph-mon[36190]: pgmap v883: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:45.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:44 smithi170 ceph-mon[36280]: pgmap v883: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:45.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:44 smithi037 ceph-mon[31460]: pgmap v883: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:45 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 661.107483 secs 2024-03-28T18:37:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:45 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 661.107483 secs 2024-03-28T18:37:46.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:45 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 661.107483 secs 2024-03-28T18:37:47.276 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:46 smithi037 ceph-mon[31460]: pgmap v884: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:46 smithi059 ceph-mon[36190]: pgmap v884: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:46 smithi170 ceph-mon[36280]: pgmap v884: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:48 smithi059 ceph-mon[36190]: pgmap v885: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:48 smithi170 ceph-mon[36280]: pgmap v885: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:49.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:48 smithi037 ceph-mon[31460]: pgmap v885: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:50 smithi059 ceph-mon[36190]: pgmap v886: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:50 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 666.107599 secs 2024-03-28T18:37:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:50 smithi170 ceph-mon[36280]: pgmap v886: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:50 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 666.107599 secs 2024-03-28T18:37:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:50 smithi037 ceph-mon[31460]: pgmap v886: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:50 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 666.107599 secs 2024-03-28T18:37:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:52 smithi059 ceph-mon[36190]: pgmap v887: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:52 smithi170 ceph-mon[36280]: pgmap v887: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:53.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:52 smithi037 ceph-mon[31460]: pgmap v887: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:54 smithi059 ceph-mon[36190]: pgmap v888: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:55.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:54 smithi170 ceph-mon[36280]: pgmap v888: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:55.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:54 smithi037 ceph-mon[31460]: pgmap v888: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:55 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 671.107720 secs 2024-03-28T18:37:56.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:55 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 671.107720 secs 2024-03-28T18:37:56.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:55 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 671.107720 secs 2024-03-28T18:37:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:57 smithi059 ceph-mon[36190]: pgmap v889: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:57 smithi170 ceph-mon[36280]: pgmap v889: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:57.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:57 smithi037 ceph-mon[31460]: pgmap v889: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:59.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:37:59 smithi059 ceph-mon[36190]: pgmap v890: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:59.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:37:59 smithi170 ceph-mon[36280]: pgmap v890: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:59.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:37:59 smithi037 ceph-mon[31460]: pgmap v890: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:01 smithi059 ceph-mon[36190]: pgmap v891: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 676.107835 secs 2024-03-28T18:38:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:01 smithi170 ceph-mon[36280]: pgmap v891: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:01.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 676.107835 secs 2024-03-28T18:38:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:01 smithi037 ceph-mon[31460]: pgmap v891: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 676.107835 secs 2024-03-28T18:38:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:03 smithi059 ceph-mon[36190]: pgmap v892: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:03.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:03 smithi170 ceph-mon[36280]: pgmap v892: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:03.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:03 smithi037 ceph-mon[31460]: pgmap v892: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:05.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:05 smithi059 ceph-mon[36190]: pgmap v893: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:05.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:05 smithi170 ceph-mon[36280]: pgmap v893: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:05.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:05 smithi037 ceph-mon[31460]: pgmap v893: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 681.107955 secs 2024-03-28T18:38:06.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 681.107955 secs 2024-03-28T18:38:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 681.107955 secs 2024-03-28T18:38:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:07 smithi059 ceph-mon[36190]: pgmap v894: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:07.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:07 smithi170 ceph-mon[36280]: pgmap v894: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:07.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:07 smithi037 ceph-mon[31460]: pgmap v894: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:09 smithi059 ceph-mon[36190]: pgmap v895: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:09.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:09 smithi170 ceph-mon[36280]: pgmap v895: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:09.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:09 smithi037 ceph-mon[31460]: pgmap v895: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:11.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:11 smithi037 ceph-mon[31460]: pgmap v896: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:11.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 686.108066 secs 2024-03-28T18:38:11.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:11 smithi059 ceph-mon[36190]: pgmap v896: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:11.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 686.108066 secs 2024-03-28T18:38:11.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:11 smithi170 ceph-mon[36280]: pgmap v896: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:11.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 686.108066 secs 2024-03-28T18:38:12.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:12 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:12.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:12 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:12.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:12 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:13.296 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:13 smithi170 ceph-mon[36280]: pgmap v897: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:13.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:13 smithi037 ceph-mon[31460]: pgmap v897: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:13.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:13 smithi059 ceph-mon[36190]: pgmap v897: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:15.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:15 smithi170 ceph-mon[36280]: pgmap v898: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:15.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:15 smithi037 ceph-mon[31460]: pgmap v898: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:15.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:15 smithi059 ceph-mon[36190]: pgmap v898: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:16.274 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 691.108185 secs 2024-03-28T18:38:16.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 691.108185 secs 2024-03-28T18:38:16.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 691.108185 secs 2024-03-28T18:38:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:17 smithi170 ceph-mon[36280]: pgmap v899: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:17 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:38:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:17 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:38:17.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:17 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:17.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:17 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:17.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:17 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:38:17.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:17 smithi037 ceph-mon[31460]: pgmap v899: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:17.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:17 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:38:17.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:17 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:38:17.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:17 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:17.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:17 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:17.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:17 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:38:17.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:17 smithi059 ceph-mon[36190]: pgmap v899: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:17.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:17 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:38:17.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:17 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:38:17.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:17 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:17.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:17 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:17.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:17 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:38:18.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:18 smithi037 ceph-mon[31460]: pgmap v900: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:38:18.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:18 smithi037 ceph-mon[31460]: pgmap v901: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:38:18.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:18 smithi059 ceph-mon[36190]: pgmap v900: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:38:18.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:18 smithi059 ceph-mon[36190]: pgmap v901: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:38:18.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:18 smithi170 ceph-mon[36280]: pgmap v900: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:38:18.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:18 smithi170 ceph-mon[36280]: pgmap v901: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:38:20.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:20 smithi037 ceph-mon[31460]: pgmap v902: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:38:20.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:20 smithi059 ceph-mon[36190]: pgmap v902: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:38:20.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:20 smithi170 ceph-mon[36280]: pgmap v902: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:38:21.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 696.108304 secs 2024-03-28T18:38:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 696.108304 secs 2024-03-28T18:38:21.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 696.108304 secs 2024-03-28T18:38:22.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:22 smithi037 ceph-mon[31460]: pgmap v903: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:38:22.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:22 smithi059 ceph-mon[36190]: pgmap v903: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:38:22.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:22 smithi170 ceph-mon[36280]: pgmap v903: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:38:24.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:24 smithi037 ceph-mon[31460]: pgmap v904: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:38:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:24 smithi059 ceph-mon[36190]: pgmap v904: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:38:24.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:24 smithi170 ceph-mon[36280]: pgmap v904: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:38:25.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:25 smithi059 ceph-mon[36190]: pgmap v905: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:38:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:25 smithi170 ceph-mon[36280]: pgmap v905: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:38:25.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:25 smithi037 ceph-mon[31460]: pgmap v905: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:38:26.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 701.108426 secs 2024-03-28T18:38:26.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 701.108426 secs 2024-03-28T18:38:26.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 701.108426 secs 2024-03-28T18:38:27.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:27 smithi059 ceph-mon[36190]: pgmap v906: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:38:27.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:27 smithi170 ceph-mon[36280]: pgmap v906: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:38:27.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:27 smithi037 ceph-mon[31460]: pgmap v906: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:38:29.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:29 smithi059 ceph-mon[36190]: pgmap v907: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:29.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:29 smithi170 ceph-mon[36280]: pgmap v907: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:29.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:29 smithi037 ceph-mon[31460]: pgmap v907: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:31.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:31 smithi059 ceph-mon[36190]: pgmap v908: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:31.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 706.108546 secs 2024-03-28T18:38:31.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:31 smithi170 ceph-mon[36280]: pgmap v908: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:31.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 706.108546 secs 2024-03-28T18:38:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:31 smithi037 ceph-mon[31460]: pgmap v908: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 706.108546 secs 2024-03-28T18:38:33.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:33 smithi059 ceph-mon[36190]: pgmap v909: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:33.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:33 smithi170 ceph-mon[36280]: pgmap v909: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:33.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:33 smithi037 ceph-mon[31460]: pgmap v909: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:35.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:35 smithi059 ceph-mon[36190]: pgmap v910: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:35.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:35 smithi170 ceph-mon[36280]: pgmap v910: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:35.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:35 smithi037 ceph-mon[31460]: pgmap v910: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:36.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 711.108647 secs 2024-03-28T18:38:36.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 711.108647 secs 2024-03-28T18:38:36.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 711.108647 secs 2024-03-28T18:38:37.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:37 smithi059 ceph-mon[36190]: pgmap v911: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:37.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:37 smithi170 ceph-mon[36280]: pgmap v911: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:37.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:37 smithi037 ceph-mon[31460]: pgmap v911: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:39.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:39 smithi059 ceph-mon[36190]: pgmap v912: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:39.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:39 smithi170 ceph-mon[36280]: pgmap v912: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:39.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:39 smithi037 ceph-mon[31460]: pgmap v912: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:41.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:41 smithi059 ceph-mon[36190]: pgmap v913: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:41.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 716.108766 secs 2024-03-28T18:38:41.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:41 smithi170 ceph-mon[36280]: pgmap v913: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:41.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 716.108766 secs 2024-03-28T18:38:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:41 smithi037 ceph-mon[31460]: pgmap v913: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:41.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 716.108766 secs 2024-03-28T18:38:43.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:43 smithi059 ceph-mon[36190]: pgmap v914: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:38:43.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:43 smithi170 ceph-mon[36280]: pgmap v914: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:38:43.839 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:43 smithi037 ceph-mon[31460]: pgmap v914: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:38:45.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:45 smithi059 ceph-mon[36190]: pgmap v915: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:38:45.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:45 smithi170 ceph-mon[36280]: pgmap v915: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:38:45.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:45 smithi037 ceph-mon[31460]: pgmap v915: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:38:46.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:46 smithi059 ceph-mon[36190]: 2 slow requests, 1 included below; oldest blocked for > 721.108887 secs 2024-03-28T18:38:46.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:46 smithi059 ceph-mon[36190]: slow request 721.108886 seconds old, received at 2024-03-28T18:26:44.515856+0000: peer_request:client.25216:13687 currently acquired locks 2024-03-28T18:38:46.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:46 smithi170 ceph-mon[36280]: 2 slow requests, 1 included below; oldest blocked for > 721.108887 secs 2024-03-28T18:38:46.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:46 smithi170 ceph-mon[36280]: slow request 721.108886 seconds old, received at 2024-03-28T18:26:44.515856+0000: peer_request:client.25216:13687 currently acquired locks 2024-03-28T18:38:46.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:46 smithi037 ceph-mon[31460]: 2 slow requests, 1 included below; oldest blocked for > 721.108887 secs 2024-03-28T18:38:46.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:46 smithi037 ceph-mon[31460]: slow request 721.108886 seconds old, received at 2024-03-28T18:26:44.515856+0000: peer_request:client.25216:13687 currently acquired locks 2024-03-28T18:38:47.776 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:47 smithi037 ceph-mon[31460]: pgmap v916: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:38:47.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:47 smithi059 ceph-mon[36190]: pgmap v916: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:38:47.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:47 smithi170 ceph-mon[36280]: pgmap v916: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:38:49.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:49 smithi059 ceph-mon[36190]: pgmap v917: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:38:49.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:49 smithi170 ceph-mon[36280]: pgmap v917: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:38:49.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:49 smithi037 ceph-mon[31460]: pgmap v917: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:38:51.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:51 smithi059 ceph-mon[36190]: pgmap v918: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:38:51.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 726.109008 secs 2024-03-28T18:38:51.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:51 smithi170 ceph-mon[36280]: pgmap v918: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:38:51.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 726.109008 secs 2024-03-28T18:38:51.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:51 smithi037 ceph-mon[31460]: pgmap v918: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:38:51.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 726.109008 secs 2024-03-28T18:38:53.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:53 smithi059 ceph-mon[36190]: pgmap v919: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:38:53.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:53 smithi170 ceph-mon[36280]: pgmap v919: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:38:53.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:53 smithi037 ceph-mon[31460]: pgmap v919: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:38:55.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:55 smithi059 ceph-mon[36190]: pgmap v920: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:55.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:55 smithi170 ceph-mon[36280]: pgmap v920: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:55.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:55 smithi037 ceph-mon[31460]: pgmap v920: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 731.109128 secs 2024-03-28T18:38:56.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 731.109128 secs 2024-03-28T18:38:56.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 731.109128 secs 2024-03-28T18:38:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:57 smithi059 ceph-mon[36190]: pgmap v921: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:57.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:57 smithi170 ceph-mon[36280]: pgmap v921: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:57.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:57 smithi037 ceph-mon[31460]: pgmap v921: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:38:59 smithi059 ceph-mon[36190]: pgmap v922: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:59.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:38:59 smithi170 ceph-mon[36280]: pgmap v922: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:59.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:38:59 smithi037 ceph-mon[31460]: pgmap v922: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:01.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:01 smithi059 ceph-mon[36190]: pgmap v923: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:01.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:01 smithi059 ceph-mon[36190]: 2 slow requests, 1 included below; oldest blocked for > 736.109246 secs 2024-03-28T18:39:01.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:01 smithi059 ceph-mon[36190]: slow request 720.528921 seconds old, received at 2024-03-28T18:27:00.096181+0000: client_request(client.25553:180 mksnap #0x1//scheduled-2024-03-28-18_27_00_UTC 2024-03-28T18:27:00.095960+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:39:01.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:01 smithi170 ceph-mon[36280]: pgmap v923: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:01.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:01 smithi170 ceph-mon[36280]: 2 slow requests, 1 included below; oldest blocked for > 736.109246 secs 2024-03-28T18:39:01.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:01 smithi170 ceph-mon[36280]: slow request 720.528921 seconds old, received at 2024-03-28T18:27:00.096181+0000: client_request(client.25553:180 mksnap #0x1//scheduled-2024-03-28-18_27_00_UTC 2024-03-28T18:27:00.095960+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:39:01.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:01 smithi037 ceph-mon[31460]: pgmap v923: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:01.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:01 smithi037 ceph-mon[31460]: 2 slow requests, 1 included below; oldest blocked for > 736.109246 secs 2024-03-28T18:39:01.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:01 smithi037 ceph-mon[31460]: slow request 720.528921 seconds old, received at 2024-03-28T18:27:00.096181+0000: client_request(client.25553:180 mksnap #0x1//scheduled-2024-03-28-18_27_00_UTC 2024-03-28T18:27:00.095960+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:39:03.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:03 smithi059 ceph-mon[36190]: pgmap v924: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:03.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:03 smithi170 ceph-mon[36280]: pgmap v924: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:03.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:03 smithi037 ceph-mon[31460]: pgmap v924: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:05.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:05 smithi059 ceph-mon[36190]: pgmap v925: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:05.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:05 smithi170 ceph-mon[36280]: pgmap v925: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:05.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:05 smithi037 ceph-mon[31460]: pgmap v925: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:06.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 741.109367 secs 2024-03-28T18:39:06.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 741.109367 secs 2024-03-28T18:39:06.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 741.109367 secs 2024-03-28T18:39:07.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:07 smithi059 ceph-mon[36190]: pgmap v926: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:07.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:07 smithi170 ceph-mon[36280]: pgmap v926: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:07.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:07 smithi037 ceph-mon[31460]: pgmap v926: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:09.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:09 smithi059 ceph-mon[36190]: pgmap v927: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:09.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:09 smithi170 ceph-mon[36280]: pgmap v927: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:09.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:09 smithi037 ceph-mon[31460]: pgmap v927: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:11.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:11 smithi059 ceph-mon[36190]: pgmap v928: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:11.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 746.109459 secs 2024-03-28T18:39:11.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:11 smithi170 ceph-mon[36280]: pgmap v928: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:11.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 746.109459 secs 2024-03-28T18:39:11.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:11 smithi037 ceph-mon[31460]: pgmap v928: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:11.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 746.109459 secs 2024-03-28T18:39:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:13 smithi059 ceph-mon[36190]: pgmap v929: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:13.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:13 smithi170 ceph-mon[36280]: pgmap v929: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:13.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:13 smithi037 ceph-mon[31460]: pgmap v929: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:15.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:15 smithi059 ceph-mon[36190]: pgmap v930: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:15.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:15 smithi170 ceph-mon[36280]: pgmap v930: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:15.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:15 smithi037 ceph-mon[31460]: pgmap v930: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:16.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 751.109566 secs 2024-03-28T18:39:17.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 751.109566 secs 2024-03-28T18:39:17.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 751.109566 secs 2024-03-28T18:39:17.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:17 smithi037 ceph-mon[31460]: pgmap v931: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:17.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:17 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:39:18.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:17 smithi059 ceph-mon[36190]: pgmap v931: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:18.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:17 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:39:18.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:17 smithi170 ceph-mon[36280]: pgmap v931: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:18.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:17 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:39:18.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:18 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:18.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:18 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:39:18.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:18 smithi037 ceph-mon[31460]: pgmap v932: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:39:18.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:18 smithi037 ceph-mon[31460]: pgmap v933: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:18.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:18 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:39:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:18 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:18 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:39:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:18 smithi059 ceph-mon[36190]: pgmap v932: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:39:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:18 smithi059 ceph-mon[36190]: pgmap v933: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:18 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:39:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:18 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:18 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:39:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:18 smithi170 ceph-mon[36280]: pgmap v932: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:39:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:18 smithi170 ceph-mon[36280]: pgmap v933: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:19.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:18 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:39:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:20 smithi059 ceph-mon[36190]: pgmap v934: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:20 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 756.109687 secs 2024-03-28T18:39:21.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:20 smithi170 ceph-mon[36280]: pgmap v934: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:21.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:20 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 756.109687 secs 2024-03-28T18:39:21.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:20 smithi037 ceph-mon[31460]: pgmap v934: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:21.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:20 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 756.109687 secs 2024-03-28T18:39:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:22 smithi059 ceph-mon[36190]: pgmap v935: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:23.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:22 smithi170 ceph-mon[36280]: pgmap v935: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:23.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:22 smithi037 ceph-mon[31460]: pgmap v935: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:25.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:24 smithi059 ceph-mon[36190]: pgmap v936: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:25.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:24 smithi170 ceph-mon[36280]: pgmap v936: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:25.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:24 smithi037 ceph-mon[31460]: pgmap v936: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:25 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 761.109803 secs 2024-03-28T18:39:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:25 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 761.109803 secs 2024-03-28T18:39:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:25 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 761.109803 secs 2024-03-28T18:39:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:26 smithi059 ceph-mon[36190]: pgmap v937: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:26 smithi170 ceph-mon[36280]: pgmap v937: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:26 smithi037 ceph-mon[31460]: pgmap v937: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:29.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:28 smithi059 ceph-mon[36190]: pgmap v938: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:39:29.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:28 smithi170 ceph-mon[36280]: pgmap v938: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:39:29.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:28 smithi037 ceph-mon[31460]: pgmap v938: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:39:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:30 smithi059 ceph-mon[36190]: pgmap v939: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:39:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:30 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 766.109918 secs 2024-03-28T18:39:31.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:30 smithi170 ceph-mon[36280]: pgmap v939: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:39:31.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:30 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 766.109918 secs 2024-03-28T18:39:31.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:30 smithi037 ceph-mon[31460]: pgmap v939: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:39:31.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:30 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 766.109918 secs 2024-03-28T18:39:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:32 smithi059 ceph-mon[36190]: pgmap v940: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:39:33.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:32 smithi170 ceph-mon[36280]: pgmap v940: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:39:33.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:32 smithi037 ceph-mon[31460]: pgmap v940: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:39:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:34 smithi059 ceph-mon[36190]: pgmap v941: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:39:35.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:34 smithi170 ceph-mon[36280]: pgmap v941: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:39:35.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:34 smithi037 ceph-mon[31460]: pgmap v941: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:39:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:35 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 771.110053 secs 2024-03-28T18:39:36.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:35 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 771.110053 secs 2024-03-28T18:39:36.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:35 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 771.110053 secs 2024-03-28T18:39:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:36 smithi059 ceph-mon[36190]: pgmap v942: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:39:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:36 smithi170 ceph-mon[36280]: pgmap v942: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:39:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:36 smithi037 ceph-mon[31460]: pgmap v942: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:39:39.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:38 smithi059 ceph-mon[36190]: pgmap v943: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:39:39.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:38 smithi170 ceph-mon[36280]: pgmap v943: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:39:39.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:38 smithi037 ceph-mon[31460]: pgmap v943: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:39:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:40 smithi059 ceph-mon[36190]: pgmap v944: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:40 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 776.110173 secs 2024-03-28T18:39:41.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:40 smithi170 ceph-mon[36280]: pgmap v944: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:41.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:40 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 776.110173 secs 2024-03-28T18:39:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:40 smithi037 ceph-mon[31460]: pgmap v944: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:41.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:40 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 776.110173 secs 2024-03-28T18:39:43.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:42 smithi059 ceph-mon[36190]: pgmap v945: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:43.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:42 smithi170 ceph-mon[36280]: pgmap v945: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:43.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:42 smithi037 ceph-mon[31460]: pgmap v945: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:44 smithi059 ceph-mon[36190]: pgmap v946: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:45.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:44 smithi170 ceph-mon[36280]: pgmap v946: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:45.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:44 smithi037 ceph-mon[31460]: pgmap v946: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:46.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:45 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 781.110241 secs 2024-03-28T18:39:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:45 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 781.110241 secs 2024-03-28T18:39:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:45 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 781.110241 secs 2024-03-28T18:39:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:46 smithi059 ceph-mon[36190]: pgmap v947: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:46 smithi170 ceph-mon[36280]: pgmap v947: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:47.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:46 smithi037 ceph-mon[31460]: pgmap v947: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:48 smithi059 ceph-mon[36190]: pgmap v948: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:48 smithi170 ceph-mon[36280]: pgmap v948: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:49.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:48 smithi037 ceph-mon[31460]: pgmap v948: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:50 smithi059 ceph-mon[36190]: pgmap v949: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:50 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 786.110309 secs 2024-03-28T18:39:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:50 smithi170 ceph-mon[36280]: pgmap v949: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:50 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 786.110309 secs 2024-03-28T18:39:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:50 smithi037 ceph-mon[31460]: pgmap v949: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:50 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 786.110309 secs 2024-03-28T18:39:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:52 smithi059 ceph-mon[36190]: pgmap v950: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:52 smithi170 ceph-mon[36280]: pgmap v950: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:53.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:52 smithi037 ceph-mon[31460]: pgmap v950: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:55 smithi059 ceph-mon[36190]: pgmap v951: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:55.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:55 smithi170 ceph-mon[36280]: pgmap v951: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:55.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:55 smithi037 ceph-mon[31460]: pgmap v951: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 791.110425 secs 2024-03-28T18:39:56.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 791.110425 secs 2024-03-28T18:39:56.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 791.110425 secs 2024-03-28T18:39:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:57 smithi059 ceph-mon[36190]: pgmap v952: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:57 smithi170 ceph-mon[36280]: pgmap v952: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:57.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:57 smithi037 ceph-mon[31460]: pgmap v952: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:59.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:39:59 smithi059 ceph-mon[36190]: pgmap v953: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:59.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:39:59 smithi170 ceph-mon[36280]: pgmap v953: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:59.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:39:59 smithi037 ceph-mon[31460]: pgmap v953: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:00 smithi059 ceph-mon[36190]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T18:40:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:00 smithi059 ceph-mon[36190]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-03-28T18:40:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:00 smithi059 ceph-mon[36190]: daemon mds.j on smithi037 is in error state 2024-03-28T18:40:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:00 smithi059 ceph-mon[36190]: daemon mds.d on smithi037 is in error state 2024-03-28T18:40:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:00 smithi059 ceph-mon[36190]: daemon mds.g on smithi037 is in error state 2024-03-28T18:40:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:00 smithi059 ceph-mon[36190]: daemon mds.k on smithi059 is in error state 2024-03-28T18:40:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:00 smithi059 ceph-mon[36190]: daemon mds.e on smithi059 is in error state 2024-03-28T18:40:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:00 smithi059 ceph-mon[36190]: daemon mds.b on smithi059 is in error state 2024-03-28T18:40:00.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:00 smithi059 ceph-mon[36190]: daemon mds.l on smithi170 is in error state 2024-03-28T18:40:00.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:00 smithi059 ceph-mon[36190]: daemon mds.f on smithi170 is in error state 2024-03-28T18:40:00.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:00 smithi059 ceph-mon[36190]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-03-28T18:40:00.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:00 smithi059 ceph-mon[36190]: mds.a(mds.0): 2 slow requests are blocked > 180 secs 2024-03-28T18:40:00.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:00 smithi170 ceph-mon[36280]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T18:40:00.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:00 smithi170 ceph-mon[36280]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-03-28T18:40:00.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:00 smithi170 ceph-mon[36280]: daemon mds.j on smithi037 is in error state 2024-03-28T18:40:00.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:00 smithi170 ceph-mon[36280]: daemon mds.d on smithi037 is in error state 2024-03-28T18:40:00.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:00 smithi170 ceph-mon[36280]: daemon mds.g on smithi037 is in error state 2024-03-28T18:40:00.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:00 smithi170 ceph-mon[36280]: daemon mds.k on smithi059 is in error state 2024-03-28T18:40:00.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:00 smithi170 ceph-mon[36280]: daemon mds.e on smithi059 is in error state 2024-03-28T18:40:00.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:00 smithi170 ceph-mon[36280]: daemon mds.b on smithi059 is in error state 2024-03-28T18:40:00.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:00 smithi170 ceph-mon[36280]: daemon mds.l on smithi170 is in error state 2024-03-28T18:40:00.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:00 smithi170 ceph-mon[36280]: daemon mds.f on smithi170 is in error state 2024-03-28T18:40:00.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:00 smithi170 ceph-mon[36280]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-03-28T18:40:00.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:00 smithi170 ceph-mon[36280]: mds.a(mds.0): 2 slow requests are blocked > 180 secs 2024-03-28T18:40:00.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:00 smithi037 ceph-mon[31460]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T18:40:00.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:00 smithi037 ceph-mon[31460]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-03-28T18:40:00.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:00 smithi037 ceph-mon[31460]: daemon mds.j on smithi037 is in error state 2024-03-28T18:40:00.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:00 smithi037 ceph-mon[31460]: daemon mds.d on smithi037 is in error state 2024-03-28T18:40:00.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:00 smithi037 ceph-mon[31460]: daemon mds.g on smithi037 is in error state 2024-03-28T18:40:00.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:00 smithi037 ceph-mon[31460]: daemon mds.k on smithi059 is in error state 2024-03-28T18:40:00.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:00 smithi037 ceph-mon[31460]: daemon mds.e on smithi059 is in error state 2024-03-28T18:40:00.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:00 smithi037 ceph-mon[31460]: daemon mds.b on smithi059 is in error state 2024-03-28T18:40:00.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:00 smithi037 ceph-mon[31460]: daemon mds.l on smithi170 is in error state 2024-03-28T18:40:00.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:00 smithi037 ceph-mon[31460]: daemon mds.f on smithi170 is in error state 2024-03-28T18:40:00.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:00 smithi037 ceph-mon[31460]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-03-28T18:40:00.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:00 smithi037 ceph-mon[31460]: mds.a(mds.0): 2 slow requests are blocked > 180 secs 2024-03-28T18:40:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:01 smithi059 ceph-mon[36190]: pgmap v954: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 796.110548 secs 2024-03-28T18:40:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:01 smithi170 ceph-mon[36280]: pgmap v954: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 796.110548 secs 2024-03-28T18:40:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:01 smithi037 ceph-mon[31460]: pgmap v954: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 796.110548 secs 2024-03-28T18:40:03.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:03 smithi037 ceph-mon[31460]: pgmap v955: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:03 smithi059 ceph-mon[36190]: pgmap v955: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:03.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:03 smithi170 ceph-mon[36280]: pgmap v955: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:05.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:05 smithi037 ceph-mon[31460]: pgmap v956: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:05.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:05 smithi059 ceph-mon[36190]: pgmap v956: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:05.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:05 smithi170 ceph-mon[36280]: pgmap v956: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 801.110668 secs 2024-03-28T18:40:06.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 801.110668 secs 2024-03-28T18:40:06.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 801.110668 secs 2024-03-28T18:40:07.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:07 smithi037 ceph-mon[31460]: pgmap v957: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:07.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:07 smithi059 ceph-mon[36190]: pgmap v957: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:07.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:07 smithi170 ceph-mon[36280]: pgmap v957: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:09.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:09 smithi037 ceph-mon[31460]: pgmap v958: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:09.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:09 smithi059 ceph-mon[36190]: pgmap v958: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:09.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:09 smithi170 ceph-mon[36280]: pgmap v958: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:11.337 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:11 smithi170 ceph-mon[36280]: pgmap v959: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:11.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 806.110787 secs 2024-03-28T18:40:11.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:11 smithi037 ceph-mon[31460]: pgmap v959: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:11.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 806.110787 secs 2024-03-28T18:40:11.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:11 smithi059 ceph-mon[36190]: pgmap v959: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:11.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 806.110787 secs 2024-03-28T18:40:13.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:13 smithi037 ceph-mon[31460]: pgmap v960: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:13.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:13 smithi059 ceph-mon[36190]: pgmap v960: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:13.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:13 smithi170 ceph-mon[36280]: pgmap v960: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:15.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:15 smithi037 ceph-mon[31460]: pgmap v961: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:15.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:15 smithi059 ceph-mon[36190]: pgmap v961: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:15.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:15 smithi170 ceph-mon[36280]: pgmap v961: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:16.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 811.110886 secs 2024-03-28T18:40:16.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 811.110886 secs 2024-03-28T18:40:16.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 811.110886 secs 2024-03-28T18:40:17.377 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:17 smithi037 ceph-mon[31460]: pgmap v962: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:17.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:17 smithi059 ceph-mon[36190]: pgmap v962: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:17.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:17 smithi170 ceph-mon[36280]: pgmap v962: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:19.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:19 smithi037 ceph-mon[31460]: pgmap v963: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:19.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:19 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:40:19.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:19 smithi059 ceph-mon[36190]: pgmap v963: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:19.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:19 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:40:19.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:19 smithi170 ceph-mon[36280]: pgmap v963: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:19.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:19 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:40:20.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:20 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:20.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:20 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:40:20.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:20 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:40:20.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:20 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:20.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:20 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:40:20.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:20 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:40:20.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:20 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:20.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:20 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:40:20.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:20 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:40:21.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:21 smithi037 ceph-mon[31460]: pgmap v964: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:21.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:21 smithi037 ceph-mon[31460]: pgmap v965: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:21.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 816.111010 secs 2024-03-28T18:40:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:21 smithi059 ceph-mon[36190]: pgmap v964: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:21 smithi059 ceph-mon[36190]: pgmap v965: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 816.111010 secs 2024-03-28T18:40:21.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:21 smithi170 ceph-mon[36280]: pgmap v964: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:21.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:21 smithi170 ceph-mon[36280]: pgmap v965: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:21.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 816.111010 secs 2024-03-28T18:40:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:23 smithi037 ceph-mon[31460]: pgmap v966: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:40:23.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:23 smithi059 ceph-mon[36190]: pgmap v966: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:40:23.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:23 smithi170 ceph-mon[36280]: pgmap v966: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:40:25.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:25 smithi037 ceph-mon[31460]: pgmap v967: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:25.537 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:25 smithi059 ceph-mon[36190]: pgmap v967: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:25.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:25 smithi170 ceph-mon[36280]: pgmap v967: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:26 smithi037 ceph-mon[31460]: pgmap v968: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:26.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 821.111131 secs 2024-03-28T18:40:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:26 smithi059 ceph-mon[36190]: pgmap v968: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 821.111131 secs 2024-03-28T18:40:26.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:26 smithi170 ceph-mon[36280]: pgmap v968: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:26.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 821.111131 secs 2024-03-28T18:40:28.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:28 smithi059 ceph-mon[36190]: pgmap v969: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:28.790 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:28 smithi170 ceph-mon[36280]: pgmap v969: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:28.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:28 smithi037 ceph-mon[31460]: pgmap v969: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:30.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:30 smithi059 ceph-mon[36190]: pgmap v970: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:40:30.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:30 smithi170 ceph-mon[36280]: pgmap v970: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:40:30.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:30 smithi037 ceph-mon[31460]: pgmap v970: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:40:31.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 826.111210 secs 2024-03-28T18:40:31.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 826.111210 secs 2024-03-28T18:40:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 826.111210 secs 2024-03-28T18:40:32.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:32 smithi059 ceph-mon[36190]: pgmap v971: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:32.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:32 smithi170 ceph-mon[36280]: pgmap v971: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:32.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:32 smithi037 ceph-mon[31460]: pgmap v971: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:34.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:34 smithi059 ceph-mon[36190]: pgmap v972: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:34.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:34 smithi170 ceph-mon[36280]: pgmap v972: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:34.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:34 smithi037 ceph-mon[31460]: pgmap v972: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:36.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:36 smithi059 ceph-mon[36190]: pgmap v973: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:36.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 831.111329 secs 2024-03-28T18:40:36.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:36 smithi170 ceph-mon[36280]: pgmap v973: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:36.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 831.111329 secs 2024-03-28T18:40:36.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:36 smithi037 ceph-mon[31460]: pgmap v973: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:36.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 831.111329 secs 2024-03-28T18:40:38.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:38 smithi059 ceph-mon[36190]: pgmap v974: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:38.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:38 smithi170 ceph-mon[36280]: pgmap v974: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:38.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:38 smithi037 ceph-mon[31460]: pgmap v974: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:40.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:40 smithi059 ceph-mon[36190]: pgmap v975: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:40.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:40 smithi170 ceph-mon[36280]: pgmap v975: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:40.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:40 smithi037 ceph-mon[31460]: pgmap v975: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:41.686 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 836.111450 secs 2024-03-28T18:40:41.687 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 836.111450 secs 2024-03-28T18:40:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 836.111450 secs 2024-03-28T18:40:42.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:42 smithi059 ceph-mon[36190]: pgmap v976: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:42.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:42 smithi170 ceph-mon[36280]: pgmap v976: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:42.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:42 smithi037 ceph-mon[31460]: pgmap v976: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:44.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:44 smithi059 ceph-mon[36190]: pgmap v977: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:44.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:44 smithi170 ceph-mon[36280]: pgmap v977: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:44.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:44 smithi037 ceph-mon[31460]: pgmap v977: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:46.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:46 smithi059 ceph-mon[36190]: pgmap v978: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:46.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 841.111568 secs 2024-03-28T18:40:46.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:46 smithi170 ceph-mon[36280]: pgmap v978: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:46.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 841.111568 secs 2024-03-28T18:40:46.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:46 smithi037 ceph-mon[31460]: pgmap v978: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:46.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 841.111568 secs 2024-03-28T18:40:48.764 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:48 smithi037 ceph-mon[31460]: pgmap v979: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:48.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:48 smithi059 ceph-mon[36190]: pgmap v979: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:48.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:48 smithi170 ceph-mon[36280]: pgmap v979: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:50.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:50 smithi059 ceph-mon[36190]: pgmap v980: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:50.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:50 smithi170 ceph-mon[36280]: pgmap v980: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:50.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:50 smithi037 ceph-mon[31460]: pgmap v980: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:51.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 846.111688 secs 2024-03-28T18:40:51.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 846.111688 secs 2024-03-28T18:40:51.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 846.111688 secs 2024-03-28T18:40:52.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:52 smithi059 ceph-mon[36190]: pgmap v981: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:52.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:52 smithi170 ceph-mon[36280]: pgmap v981: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:52.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:52 smithi037 ceph-mon[31460]: pgmap v981: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:54.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:54 smithi059 ceph-mon[36190]: pgmap v982: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:54.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:54 smithi170 ceph-mon[36280]: pgmap v982: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:54.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:54 smithi037 ceph-mon[31460]: pgmap v982: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:56 smithi059 ceph-mon[36190]: pgmap v983: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 851.111807 secs 2024-03-28T18:40:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:56 smithi170 ceph-mon[36280]: pgmap v983: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 851.111807 secs 2024-03-28T18:40:56.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:56 smithi037 ceph-mon[31460]: pgmap v983: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:56.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 851.111807 secs 2024-03-28T18:40:58.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:40:58 smithi059 ceph-mon[36190]: pgmap v984: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:58.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:40:58 smithi170 ceph-mon[36280]: pgmap v984: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:58.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:40:58 smithi037 ceph-mon[31460]: pgmap v984: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:00.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:00 smithi059 ceph-mon[36190]: pgmap v985: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:00.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:00 smithi170 ceph-mon[36280]: pgmap v985: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:00.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:00 smithi037 ceph-mon[31460]: pgmap v985: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:01.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 856.111926 secs 2024-03-28T18:41:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 856.111926 secs 2024-03-28T18:41:01.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 856.111926 secs 2024-03-28T18:41:02.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:02 smithi059 ceph-mon[36190]: pgmap v986: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:02.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:02 smithi170 ceph-mon[36280]: pgmap v986: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:02.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:02 smithi037 ceph-mon[31460]: pgmap v986: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:04.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:04 smithi059 ceph-mon[36190]: pgmap v987: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:04.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:04 smithi170 ceph-mon[36280]: pgmap v987: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:04.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:04 smithi037 ceph-mon[31460]: pgmap v987: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:06.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:06 smithi059 ceph-mon[36190]: pgmap v988: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:06.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 861.112064 secs 2024-03-28T18:41:06.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:06 smithi170 ceph-mon[36280]: pgmap v988: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:06.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 861.112064 secs 2024-03-28T18:41:06.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:06 smithi037 ceph-mon[31460]: pgmap v988: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:06.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 861.112064 secs 2024-03-28T18:41:08.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:08 smithi059 ceph-mon[36190]: pgmap v989: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:08.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:08 smithi170 ceph-mon[36280]: pgmap v989: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:08.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:08 smithi037 ceph-mon[31460]: pgmap v989: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:10.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:10 smithi059 ceph-mon[36190]: pgmap v990: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:10.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:10 smithi170 ceph-mon[36280]: pgmap v990: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:10.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:10 smithi037 ceph-mon[31460]: pgmap v990: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:11.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 866.112184 secs 2024-03-28T18:41:11.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 866.112184 secs 2024-03-28T18:41:11.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 866.112184 secs 2024-03-28T18:41:12.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:12 smithi059 ceph-mon[36190]: pgmap v991: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:12.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:12 smithi170 ceph-mon[36280]: pgmap v991: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:12.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:12 smithi037 ceph-mon[31460]: pgmap v991: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:14.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:14 smithi059 ceph-mon[36190]: pgmap v992: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:14.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:14 smithi170 ceph-mon[36280]: pgmap v992: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:14.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:14 smithi037 ceph-mon[31460]: pgmap v992: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:16.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:16 smithi059 ceph-mon[36190]: pgmap v993: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:16.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 871.112281 secs 2024-03-28T18:41:16.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:16 smithi170 ceph-mon[36280]: pgmap v993: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:16.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 871.112281 secs 2024-03-28T18:41:16.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:16 smithi037 ceph-mon[31460]: pgmap v993: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:16.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 871.112281 secs 2024-03-28T18:41:18.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:18 smithi059 ceph-mon[36190]: pgmap v994: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:18.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:18 smithi170 ceph-mon[36280]: pgmap v994: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:18.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:18 smithi037 ceph-mon[31460]: pgmap v994: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:20.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:20 smithi059 ceph-mon[36190]: pgmap v995: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:20.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:20 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:41:20.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:20 smithi170 ceph-mon[36280]: pgmap v995: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:20.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:20 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:41:20.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:20 smithi037 ceph-mon[31460]: pgmap v995: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:20.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:20 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:41:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 876.112394 secs 2024-03-28T18:41:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:21 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:21 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:21 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:21 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:21 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:22.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:21 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:22.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:21 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:22.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:21 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:41:22.126 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:21 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:22.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 876.112394 secs 2024-03-28T18:41:22.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:21 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:22.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:21 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:22.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:21 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:22.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:21 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:22.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:21 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:22.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:21 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:22.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:21 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:22.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:21 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:41:22.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:21 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:22.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 876.112394 secs 2024-03-28T18:41:22.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:21 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:22.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:21 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:22.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:21 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:22.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:21 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:22.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:21 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:22.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:21 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:22.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:21 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:22.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:21 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:41:22.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:21 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:41:23.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:22 smithi037 ceph-mon[31460]: pgmap v996: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:23.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:22 smithi037 ceph-mon[31460]: pgmap v997: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:41:23.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:22 smithi037 ceph-mon[31460]: pgmap v998: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:41:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:22 smithi059 ceph-mon[36190]: pgmap v996: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:22 smithi059 ceph-mon[36190]: pgmap v997: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:41:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:22 smithi059 ceph-mon[36190]: pgmap v998: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:41:23.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:22 smithi170 ceph-mon[36280]: pgmap v996: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:23.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:22 smithi170 ceph-mon[36280]: pgmap v997: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T18:41:23.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:22 smithi170 ceph-mon[36280]: pgmap v998: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:41:25.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:24 smithi059 ceph-mon[36190]: pgmap v999: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:41:25.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:24 smithi170 ceph-mon[36280]: pgmap v999: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:41:25.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:24 smithi037 ceph-mon[31460]: pgmap v999: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:41:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:25 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 881.112513 secs 2024-03-28T18:41:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:25 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 881.112513 secs 2024-03-28T18:41:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:25 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 881.112513 secs 2024-03-28T18:41:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:26 smithi059 ceph-mon[36190]: pgmap v1000: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:41:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:26 smithi170 ceph-mon[36280]: pgmap v1000: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:41:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:26 smithi037 ceph-mon[31460]: pgmap v1000: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:41:29.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:28 smithi059 ceph-mon[36190]: pgmap v1001: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:41:29.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:28 smithi170 ceph-mon[36280]: pgmap v1001: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:41:29.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:28 smithi037 ceph-mon[31460]: pgmap v1001: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:41:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:30 smithi059 ceph-mon[36190]: pgmap v1002: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:41:31.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:30 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 886.112634 secs 2024-03-28T18:41:31.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:30 smithi170 ceph-mon[36280]: pgmap v1002: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:41:31.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:30 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 886.112634 secs 2024-03-28T18:41:31.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:30 smithi037 ceph-mon[31460]: pgmap v1002: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:41:31.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:30 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 886.112634 secs 2024-03-28T18:41:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:32 smithi059 ceph-mon[36190]: pgmap v1003: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:41:33.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:32 smithi170 ceph-mon[36280]: pgmap v1003: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:41:33.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:32 smithi037 ceph-mon[31460]: pgmap v1003: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:41:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:34 smithi059 ceph-mon[36190]: pgmap v1004: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:35.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:34 smithi170 ceph-mon[36280]: pgmap v1004: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:35.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:34 smithi037 ceph-mon[31460]: pgmap v1004: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:36.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:35 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 891.112717 secs 2024-03-28T18:41:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:35 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 891.112717 secs 2024-03-28T18:41:36.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:35 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 891.112717 secs 2024-03-28T18:41:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:36 smithi059 ceph-mon[36190]: pgmap v1005: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:36 smithi170 ceph-mon[36280]: pgmap v1005: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:36 smithi037 ceph-mon[31460]: pgmap v1005: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:39.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:38 smithi059 ceph-mon[36190]: pgmap v1006: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:39.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:38 smithi170 ceph-mon[36280]: pgmap v1006: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:39.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:38 smithi037 ceph-mon[31460]: pgmap v1006: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:40 smithi059 ceph-mon[36190]: pgmap v1007: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:40 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 896.112786 secs 2024-03-28T18:41:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:40 smithi170 ceph-mon[36280]: pgmap v1007: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:40 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 896.112786 secs 2024-03-28T18:41:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:40 smithi037 ceph-mon[31460]: pgmap v1007: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:40 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 896.112786 secs 2024-03-28T18:41:43.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:42 smithi059 ceph-mon[36190]: pgmap v1008: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:43.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:42 smithi170 ceph-mon[36280]: pgmap v1008: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:43.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:42 smithi037 ceph-mon[31460]: pgmap v1008: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:44 smithi059 ceph-mon[36190]: pgmap v1009: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:45.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:44 smithi170 ceph-mon[36280]: pgmap v1009: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:45.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:44 smithi037 ceph-mon[31460]: pgmap v1009: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:45 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 901.112854 secs 2024-03-28T18:41:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:45 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 901.112854 secs 2024-03-28T18:41:46.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:45 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 901.112854 secs 2024-03-28T18:41:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:46 smithi059 ceph-mon[36190]: pgmap v1010: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:46 smithi170 ceph-mon[36280]: pgmap v1010: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:47.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:46 smithi037 ceph-mon[31460]: pgmap v1010: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:48 smithi059 ceph-mon[36190]: pgmap v1011: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:49.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:48 smithi170 ceph-mon[36280]: pgmap v1011: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:49.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:48 smithi037 ceph-mon[31460]: pgmap v1011: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:50 smithi059 ceph-mon[36190]: pgmap v1012: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:50 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 906.112966 secs 2024-03-28T18:41:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:50 smithi170 ceph-mon[36280]: pgmap v1012: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:50 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 906.112966 secs 2024-03-28T18:41:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:50 smithi037 ceph-mon[31460]: pgmap v1012: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:51.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:50 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 906.112966 secs 2024-03-28T18:41:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:52 smithi059 ceph-mon[36190]: pgmap v1013: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:53.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:52 smithi170 ceph-mon[36280]: pgmap v1013: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:53.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:52 smithi037 ceph-mon[31460]: pgmap v1013: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:54 smithi059 ceph-mon[36190]: pgmap v1014: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:55.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:54 smithi170 ceph-mon[36280]: pgmap v1014: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:55.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:54 smithi037 ceph-mon[31460]: pgmap v1014: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:55 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 911.113102 secs 2024-03-28T18:41:56.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:55 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 911.113102 secs 2024-03-28T18:41:56.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:55 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 911.113102 secs 2024-03-28T18:41:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:56 smithi059 ceph-mon[36190]: pgmap v1015: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:56 smithi170 ceph-mon[36280]: pgmap v1015: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:57.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:56 smithi037 ceph-mon[31460]: pgmap v1015: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:59.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:41:58 smithi059 ceph-mon[36190]: pgmap v1016: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:59.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:41:58 smithi170 ceph-mon[36280]: pgmap v1016: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:59.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:41:58 smithi037 ceph-mon[31460]: pgmap v1016: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:00 smithi059 ceph-mon[36190]: pgmap v1017: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:00 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 916.113221 secs 2024-03-28T18:42:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:00 smithi170 ceph-mon[36280]: pgmap v1017: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:00 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 916.113221 secs 2024-03-28T18:42:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:00 smithi037 ceph-mon[31460]: pgmap v1017: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:00 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 916.113221 secs 2024-03-28T18:42:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:03 smithi059 ceph-mon[36190]: pgmap v1018: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:03.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:03 smithi170 ceph-mon[36280]: pgmap v1018: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:03.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:03 smithi037 ceph-mon[31460]: pgmap v1018: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:05.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:05 smithi059 ceph-mon[36190]: pgmap v1019: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:05.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:05 smithi170 ceph-mon[36280]: pgmap v1019: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:05.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:05 smithi037 ceph-mon[31460]: pgmap v1019: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 921.113301 secs 2024-03-28T18:42:06.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 921.113301 secs 2024-03-28T18:42:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 921.113301 secs 2024-03-28T18:42:07.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:07 smithi170 ceph-mon[36280]: pgmap v1020: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:07.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:07 smithi059 ceph-mon[36190]: pgmap v1020: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:07.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:07 smithi037 ceph-mon[31460]: pgmap v1020: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:09 smithi059 ceph-mon[36190]: pgmap v1021: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:09.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:09 smithi170 ceph-mon[36280]: pgmap v1021: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:09.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:09 smithi037 ceph-mon[31460]: pgmap v1021: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:11 smithi059 ceph-mon[36190]: pgmap v1022: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 926.113420 secs 2024-03-28T18:42:11.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:11 smithi170 ceph-mon[36280]: pgmap v1022: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:11.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 926.113420 secs 2024-03-28T18:42:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:11 smithi037 ceph-mon[31460]: pgmap v1022: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:11.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 926.113420 secs 2024-03-28T18:42:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:13 smithi059 ceph-mon[36190]: pgmap v1023: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:13.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:13 smithi170 ceph-mon[36280]: pgmap v1023: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:13.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:13 smithi037 ceph-mon[31460]: pgmap v1023: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:15.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:15 smithi170 ceph-mon[36280]: pgmap v1024: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:15.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:15 smithi037 ceph-mon[31460]: pgmap v1024: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:15.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:15 smithi059 ceph-mon[36190]: pgmap v1024: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:16.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 931.113522 secs 2024-03-28T18:42:16.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 931.113522 secs 2024-03-28T18:42:16.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 931.113522 secs 2024-03-28T18:42:17.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:17 smithi037 ceph-mon[31460]: pgmap v1025: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:17.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:17 smithi059 ceph-mon[36190]: pgmap v1025: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:17.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:17 smithi170 ceph-mon[36280]: pgmap v1025: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:19.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:19 smithi037 ceph-mon[31460]: pgmap v1026: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:19.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:19 smithi059 ceph-mon[36190]: pgmap v1026: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:19.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:19 smithi170 ceph-mon[36280]: pgmap v1026: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:21.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:21 smithi037 ceph-mon[31460]: pgmap v1027: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:21.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 936.113642 secs 2024-03-28T18:42:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:21 smithi059 ceph-mon[36190]: pgmap v1027: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 936.113642 secs 2024-03-28T18:42:21.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:21 smithi170 ceph-mon[36280]: pgmap v1027: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:21.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 936.113642 secs 2024-03-28T18:42:22.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:22 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:42:22.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:22 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:42:22.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:22 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:42:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:23 smithi037 ceph-mon[31460]: pgmap v1028: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:23 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:23 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:23 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:23 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:23.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:23 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:23.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:23 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:23.376 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:23 smithi170 ceph-mon[36280]: pgmap v1028: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:23.376 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:23 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:23.376 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:23 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:23.376 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:23 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:23.376 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:23 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:23.376 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:23 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:23.376 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:23 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:23.401 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:23 smithi059 ceph-mon[36190]: pgmap v1028: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:23.402 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:23 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:23.402 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:23 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:23.402 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:23 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:23.402 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:23 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:23.402 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:23 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:23.402 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:23 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:25.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:25 smithi037 ceph-mon[31460]: pgmap v1029: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:25.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:25 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:25.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:25 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:25.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:25 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:25.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:25 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:25.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:25 smithi059 ceph-mon[36190]: pgmap v1029: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:25.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:25 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:25.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:25 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:25.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:25 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:25.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:25 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:25.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:25 smithi170 ceph-mon[36280]: pgmap v1029: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:25.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:25 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:25.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:25 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:25.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:25 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:25.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:25 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:26.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:26 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:26 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:26 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:26 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:26 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:26 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:26 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:26 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:26 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:26 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:26 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:26 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:26 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:26 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:26 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:26 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 941.113758 secs 2024-03-28T18:42:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:26 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:26 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:26 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:26 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:26 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:26 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:26 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:26 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:26 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:26 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:26 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:26 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:26 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:26 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:26 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:26 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 941.113758 secs 2024-03-28T18:42:26.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:26 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:26 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:26 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:26 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:26 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:26 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:26 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:26 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:26 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:26 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:26 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:26 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:26 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:26 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:26 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:26 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:26.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 941.113758 secs 2024-03-28T18:42:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:27 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T18:42:27.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:27 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T18:42:27.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:27 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T18:42:27.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:27 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T18:42:27.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:27 smithi037 ceph-mon[31460]: pgmap v1030: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:27.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:27 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:27.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:27 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:27.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:27 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T18:42:27.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:27 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T18:42:27.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:27 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T18:42:27.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:27 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T18:42:27.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:27 smithi059 ceph-mon[36190]: pgmap v1030: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:27.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:27 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:27.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:27 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:27.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:27 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T18:42:27.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:27 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T18:42:27.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:27 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T18:42:27.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:27 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T18:42:27.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:27 smithi170 ceph-mon[36280]: pgmap v1030: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:27.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:27 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:27.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:27 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:28.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:28 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:28 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:28 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:28 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:28 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:28 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:28 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:28 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:28 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:42:28.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:28 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:42:28.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:28 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:28.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:28 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:28 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:28 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:28 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:28 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:28 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:28 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:28 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:28 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:42:28.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:28 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:42:28.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:28 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:28.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:28 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:28 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:28 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:28 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:28 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:28 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:28 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:28 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:42:28.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:28 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:42:28.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:28 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:42:28.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:28 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:42:29.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:29 smithi037 ceph-mon[31460]: pgmap v1031: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:29.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:29 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T18:42:29.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:29 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:42:29.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:29 smithi037 ceph-mon[31460]: pgmap v1032: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:42:29.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:29 smithi037 ceph-mon[31460]: pgmap v1033: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:42:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:29 smithi059 ceph-mon[36190]: pgmap v1031: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:29 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T18:42:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:29 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:42:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:29 smithi059 ceph-mon[36190]: pgmap v1032: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:42:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:29 smithi059 ceph-mon[36190]: pgmap v1033: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:42:29.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:29 smithi170 ceph-mon[36280]: pgmap v1031: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:29.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:29 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T18:42:29.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:29 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:42:29.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:29 smithi170 ceph-mon[36280]: pgmap v1032: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:42:29.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:29 smithi170 ceph-mon[36280]: pgmap v1033: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:42:31.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:31 smithi037 ceph-mon[31460]: pgmap v1034: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:42:31.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 946.113877 secs 2024-03-28T18:42:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:31 smithi059 ceph-mon[36190]: pgmap v1034: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:42:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 946.113877 secs 2024-03-28T18:42:31.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:31 smithi170 ceph-mon[36280]: pgmap v1034: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:42:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 946.113877 secs 2024-03-28T18:42:33.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:33 smithi037 ceph-mon[31460]: pgmap v1035: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:42:33.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:33 smithi059 ceph-mon[36190]: pgmap v1035: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:42:33.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:33 smithi170 ceph-mon[36280]: pgmap v1035: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:42:35.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:35 smithi037 ceph-mon[31460]: pgmap v1036: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:42:35.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:35 smithi059 ceph-mon[36190]: pgmap v1036: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:42:35.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:35 smithi170 ceph-mon[36280]: pgmap v1036: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:42:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 951.113996 secs 2024-03-28T18:42:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 951.113996 secs 2024-03-28T18:42:36.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 951.113996 secs 2024-03-28T18:42:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:37 smithi037 ceph-mon[31460]: pgmap v1037: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:42:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:37 smithi059 ceph-mon[36190]: pgmap v1037: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:42:37.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:37 smithi170 ceph-mon[36280]: pgmap v1037: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:42:39.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:39 smithi037 ceph-mon[31460]: pgmap v1038: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:42:39.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:39 smithi059 ceph-mon[36190]: pgmap v1038: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:42:39.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:39 smithi170 ceph-mon[36280]: pgmap v1038: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:42:40.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:40 smithi059 ceph-mon[36190]: pgmap v1039: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:40.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:40 smithi170 ceph-mon[36280]: pgmap v1039: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:40.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:40 smithi037 ceph-mon[31460]: pgmap v1039: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:41.436 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 956.114115 secs 2024-03-28T18:42:41.437 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 956.114115 secs 2024-03-28T18:42:41.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 956.114115 secs 2024-03-28T18:42:42.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:42 smithi059 ceph-mon[36190]: pgmap v1040: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:42.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:42 smithi170 ceph-mon[36280]: pgmap v1040: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:42.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:42 smithi037 ceph-mon[31460]: pgmap v1040: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:44 smithi059 ceph-mon[36190]: pgmap v1041: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:45.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:44 smithi170 ceph-mon[36280]: pgmap v1041: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:45.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:44 smithi037 ceph-mon[31460]: pgmap v1041: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:45 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 961.114235 secs 2024-03-28T18:42:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:45 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 961.114235 secs 2024-03-28T18:42:46.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:45 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 961.114235 secs 2024-03-28T18:42:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:46 smithi059 ceph-mon[36190]: pgmap v1042: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:46 smithi170 ceph-mon[36280]: pgmap v1042: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:47.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:46 smithi037 ceph-mon[31460]: pgmap v1042: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:49.177 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:48 smithi037 ceph-mon[31460]: pgmap v1043: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:48 smithi059 ceph-mon[36190]: pgmap v1043: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:48 smithi170 ceph-mon[36280]: pgmap v1043: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:50 smithi059 ceph-mon[36190]: pgmap v1044: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:50 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 966.114354 secs 2024-03-28T18:42:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:50 smithi170 ceph-mon[36280]: pgmap v1044: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:50 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 966.114354 secs 2024-03-28T18:42:51.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:50 smithi037 ceph-mon[31460]: pgmap v1044: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:51.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:50 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 966.114354 secs 2024-03-28T18:42:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:52 smithi059 ceph-mon[36190]: pgmap v1045: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:53.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:52 smithi170 ceph-mon[36280]: pgmap v1045: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:53.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:52 smithi037 ceph-mon[31460]: pgmap v1045: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:54 smithi059 ceph-mon[36190]: pgmap v1046: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:55.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:54 smithi170 ceph-mon[36280]: pgmap v1046: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:55.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:54 smithi037 ceph-mon[31460]: pgmap v1046: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:55 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 971.114475 secs 2024-03-28T18:42:56.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:55 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 971.114475 secs 2024-03-28T18:42:56.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:55 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 971.114475 secs 2024-03-28T18:42:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:56 smithi059 ceph-mon[36190]: pgmap v1047: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:56 smithi170 ceph-mon[36280]: pgmap v1047: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:57.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:56 smithi037 ceph-mon[31460]: pgmap v1047: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:59.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:42:58 smithi059 ceph-mon[36190]: pgmap v1048: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:59.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:42:58 smithi170 ceph-mon[36280]: pgmap v1048: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:59.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:42:58 smithi037 ceph-mon[31460]: pgmap v1048: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:00 smithi059 ceph-mon[36190]: pgmap v1049: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:00 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 976.114594 secs 2024-03-28T18:43:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:00 smithi170 ceph-mon[36280]: pgmap v1049: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:01.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:00 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 976.114594 secs 2024-03-28T18:43:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:00 smithi037 ceph-mon[31460]: pgmap v1049: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:00 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 976.114594 secs 2024-03-28T18:43:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:02 smithi059 ceph-mon[36190]: pgmap v1050: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:03.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:02 smithi170 ceph-mon[36280]: pgmap v1050: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:03.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:02 smithi037 ceph-mon[31460]: pgmap v1050: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:05.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:04 smithi059 ceph-mon[36190]: pgmap v1051: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:05.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:04 smithi170 ceph-mon[36280]: pgmap v1051: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:05.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:04 smithi037 ceph-mon[31460]: pgmap v1051: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:05 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 981.114715 secs 2024-03-28T18:43:06.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:05 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 981.114715 secs 2024-03-28T18:43:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:05 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 981.114715 secs 2024-03-28T18:43:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:06 smithi059 ceph-mon[36190]: pgmap v1052: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:07.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:06 smithi170 ceph-mon[36280]: pgmap v1052: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:07.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:06 smithi037 ceph-mon[31460]: pgmap v1052: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:08 smithi059 ceph-mon[36190]: pgmap v1053: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:09.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:08 smithi170 ceph-mon[36280]: pgmap v1053: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:09.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:08 smithi037 ceph-mon[31460]: pgmap v1053: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:10.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:10 smithi037 ceph-mon[31460]: pgmap v1054: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:11.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:10 smithi059 ceph-mon[36190]: pgmap v1054: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:11.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:10 smithi170 ceph-mon[36280]: pgmap v1054: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:11.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 986.114837 secs 2024-03-28T18:43:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 986.114837 secs 2024-03-28T18:43:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 986.114837 secs 2024-03-28T18:43:12.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:12 smithi037 ceph-mon[31460]: pgmap v1055: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:13.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:12 smithi059 ceph-mon[36190]: pgmap v1055: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:13.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:12 smithi170 ceph-mon[36280]: pgmap v1055: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:15.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:14 smithi059 ceph-mon[36190]: pgmap v1056: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:15.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:14 smithi170 ceph-mon[36280]: pgmap v1056: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:15.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:14 smithi037 ceph-mon[31460]: pgmap v1056: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:16.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:15 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 991.114956 secs 2024-03-28T18:43:16.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:15 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 991.114956 secs 2024-03-28T18:43:16.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:15 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 991.114956 secs 2024-03-28T18:43:17.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:16 smithi059 ceph-mon[36190]: pgmap v1057: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:16 smithi170 ceph-mon[36280]: pgmap v1057: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:17.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:16 smithi037 ceph-mon[31460]: pgmap v1057: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:19.177 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:18 smithi037 ceph-mon[31460]: pgmap v1058: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:19.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:18 smithi059 ceph-mon[36190]: pgmap v1058: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:19.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:18 smithi170 ceph-mon[36280]: pgmap v1058: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:20 smithi059 ceph-mon[36190]: pgmap v1059: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:20 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 996.115076 secs 2024-03-28T18:43:21.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:20 smithi170 ceph-mon[36280]: pgmap v1059: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:21.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:20 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 996.115076 secs 2024-03-28T18:43:21.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:20 smithi037 ceph-mon[31460]: pgmap v1059: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:21.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:20 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 996.115076 secs 2024-03-28T18:43:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:22 smithi059 ceph-mon[36190]: pgmap v1060: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:23.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:22 smithi170 ceph-mon[36280]: pgmap v1060: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:23.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:22 smithi037 ceph-mon[31460]: pgmap v1060: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:25.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:24 smithi059 ceph-mon[36190]: pgmap v1061: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:25.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:24 smithi170 ceph-mon[36280]: pgmap v1061: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:25.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:24 smithi037 ceph-mon[31460]: pgmap v1061: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:25 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1001.115197 secs 2024-03-28T18:43:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:25 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1001.115197 secs 2024-03-28T18:43:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:25 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1001.115197 secs 2024-03-28T18:43:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:26 smithi059 ceph-mon[36190]: pgmap v1062: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:26 smithi170 ceph-mon[36280]: pgmap v1062: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:26 smithi037 ceph-mon[31460]: pgmap v1062: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:29.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:28 smithi059 ceph-mon[36190]: pgmap v1063: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:29.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:28 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:43:29.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:28 smithi170 ceph-mon[36280]: pgmap v1063: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:29.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:28 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:43:29.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:28 smithi037 ceph-mon[31460]: pgmap v1063: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:29.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:28 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:43:30.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:29 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:43:30.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:29 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:43:30.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:29 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:43:30.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:29 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:43:30.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:29 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:43:30.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:29 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:43:30.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:29 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:43:30.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:29 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:43:30.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:29 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:43:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:30 smithi059 ceph-mon[36190]: pgmap v1064: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:30 smithi059 ceph-mon[36190]: pgmap v1065: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:30 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1006.115315 secs 2024-03-28T18:43:31.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:30 smithi170 ceph-mon[36280]: pgmap v1064: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:31.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:30 smithi170 ceph-mon[36280]: pgmap v1065: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:31.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:30 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1006.115315 secs 2024-03-28T18:43:31.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:30 smithi037 ceph-mon[31460]: pgmap v1064: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:31.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:30 smithi037 ceph-mon[31460]: pgmap v1065: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:31.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:30 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1006.115315 secs 2024-03-28T18:43:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:32 smithi059 ceph-mon[36190]: pgmap v1066: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:43:33.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:32 smithi170 ceph-mon[36280]: pgmap v1066: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:43:33.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:32 smithi037 ceph-mon[31460]: pgmap v1066: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:43:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:34 smithi059 ceph-mon[36190]: pgmap v1067: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:35.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:34 smithi170 ceph-mon[36280]: pgmap v1067: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:35.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:34 smithi037 ceph-mon[31460]: pgmap v1067: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:35 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1011.115440 secs 2024-03-28T18:43:36.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:35 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1011.115440 secs 2024-03-28T18:43:36.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:35 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1011.115440 secs 2024-03-28T18:43:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:36 smithi059 ceph-mon[36190]: pgmap v1068: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:36 smithi170 ceph-mon[36280]: pgmap v1068: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:36 smithi037 ceph-mon[31460]: pgmap v1068: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:39.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:38 smithi059 ceph-mon[36190]: pgmap v1069: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:39.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:38 smithi170 ceph-mon[36280]: pgmap v1069: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:39.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:38 smithi037 ceph-mon[31460]: pgmap v1069: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:40 smithi059 ceph-mon[36190]: pgmap v1070: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:43:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:40 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1016.115560 secs 2024-03-28T18:43:41.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:40 smithi170 ceph-mon[36280]: pgmap v1070: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:43:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:40 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1016.115560 secs 2024-03-28T18:43:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:40 smithi037 ceph-mon[31460]: pgmap v1070: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:43:41.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:40 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1016.115560 secs 2024-03-28T18:43:43.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:42 smithi059 ceph-mon[36190]: pgmap v1071: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:43.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:42 smithi170 ceph-mon[36280]: pgmap v1071: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:43.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:42 smithi037 ceph-mon[31460]: pgmap v1071: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:45 smithi059 ceph-mon[36190]: pgmap v1072: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:45.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:45 smithi170 ceph-mon[36280]: pgmap v1072: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:45.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:45 smithi037 ceph-mon[31460]: pgmap v1072: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1021.115678 secs 2024-03-28T18:43:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1021.115678 secs 2024-03-28T18:43:46.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1021.115678 secs 2024-03-28T18:43:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:47 smithi059 ceph-mon[36190]: pgmap v1073: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:47.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:47 smithi170 ceph-mon[36280]: pgmap v1073: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:47.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:47 smithi037 ceph-mon[31460]: pgmap v1073: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:49 smithi059 ceph-mon[36190]: pgmap v1074: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:49.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:49 smithi170 ceph-mon[36280]: pgmap v1074: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:49.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:49 smithi037 ceph-mon[31460]: pgmap v1074: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:51 smithi059 ceph-mon[36190]: pgmap v1075: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1026.115796 secs 2024-03-28T18:43:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:51 smithi170 ceph-mon[36280]: pgmap v1075: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1026.115796 secs 2024-03-28T18:43:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:51 smithi037 ceph-mon[31460]: pgmap v1075: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:51.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1026.115796 secs 2024-03-28T18:43:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:53 smithi059 ceph-mon[36190]: pgmap v1076: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:53 smithi170 ceph-mon[36280]: pgmap v1076: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:53.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:53 smithi037 ceph-mon[31460]: pgmap v1076: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:55 smithi059 ceph-mon[36190]: pgmap v1077: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:55.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:55 smithi170 ceph-mon[36280]: pgmap v1077: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:55.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:55 smithi037 ceph-mon[31460]: pgmap v1077: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:56.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1031.115867 secs 2024-03-28T18:43:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1031.115867 secs 2024-03-28T18:43:56.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1031.115867 secs 2024-03-28T18:43:57.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:57 smithi037 ceph-mon[31460]: pgmap v1078: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:57.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:57 smithi059 ceph-mon[36190]: pgmap v1078: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:57.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:57 smithi170 ceph-mon[36280]: pgmap v1078: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:59.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:43:59 smithi037 ceph-mon[31460]: pgmap v1079: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:59.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:43:59 smithi059 ceph-mon[36190]: pgmap v1079: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:59.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:43:59 smithi170 ceph-mon[36280]: pgmap v1079: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:01 smithi037 ceph-mon[31460]: pgmap v1080: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1036.115984 secs 2024-03-28T18:44:01.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:01 smithi059 ceph-mon[36190]: pgmap v1080: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:01.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1036.115984 secs 2024-03-28T18:44:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:01 smithi170 ceph-mon[36280]: pgmap v1080: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1036.115984 secs 2024-03-28T18:44:03.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:03 smithi037 ceph-mon[31460]: pgmap v1081: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:03 smithi059 ceph-mon[36190]: pgmap v1081: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:03.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:03 smithi170 ceph-mon[36280]: pgmap v1081: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:05.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:05 smithi037 ceph-mon[31460]: pgmap v1082: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:05.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:05 smithi059 ceph-mon[36190]: pgmap v1082: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:05.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:05 smithi170 ceph-mon[36280]: pgmap v1082: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1041.116106 secs 2024-03-28T18:44:06.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1041.116106 secs 2024-03-28T18:44:06.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1041.116106 secs 2024-03-28T18:44:07.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:07 smithi037 ceph-mon[31460]: pgmap v1083: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:07.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:07 smithi059 ceph-mon[36190]: pgmap v1083: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:07.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:07 smithi170 ceph-mon[36280]: pgmap v1083: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:09.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:09 smithi037 ceph-mon[31460]: pgmap v1084: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:09.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:09 smithi059 ceph-mon[36190]: pgmap v1084: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:09.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:09 smithi170 ceph-mon[36280]: pgmap v1084: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:11.338 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:11 smithi059 ceph-mon[36190]: pgmap v1085: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:11.338 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1046.116200 secs 2024-03-28T18:44:11.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:11 smithi170 ceph-mon[36280]: pgmap v1085: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:11.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1046.116200 secs 2024-03-28T18:44:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:11 smithi037 ceph-mon[31460]: pgmap v1085: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:11.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1046.116200 secs 2024-03-28T18:44:13.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:13 smithi037 ceph-mon[31460]: pgmap v1086: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:13.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:13 smithi059 ceph-mon[36190]: pgmap v1086: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:13.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:13 smithi170 ceph-mon[36280]: pgmap v1086: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:15.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:15 smithi037 ceph-mon[31460]: pgmap v1087: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:15.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:15 smithi059 ceph-mon[36190]: pgmap v1087: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:15.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:15 smithi170 ceph-mon[36280]: pgmap v1087: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:16.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1051.116294 secs 2024-03-28T18:44:16.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1051.116294 secs 2024-03-28T18:44:16.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1051.116294 secs 2024-03-28T18:44:17.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:17 smithi037 ceph-mon[31460]: pgmap v1088: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:17.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:17 smithi059 ceph-mon[36190]: pgmap v1088: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:17.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:17 smithi170 ceph-mon[36280]: pgmap v1088: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:19.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:19 smithi037 ceph-mon[31460]: pgmap v1089: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:19.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:19 smithi059 ceph-mon[36190]: pgmap v1089: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:19.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:19 smithi170 ceph-mon[36280]: pgmap v1089: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:21.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:21 smithi037 ceph-mon[31460]: pgmap v1090: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:21.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1056.116390 secs 2024-03-28T18:44:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:21 smithi059 ceph-mon[36190]: pgmap v1090: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1056.116390 secs 2024-03-28T18:44:21.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:21 smithi170 ceph-mon[36280]: pgmap v1090: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:21.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1056.116390 secs 2024-03-28T18:44:23.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:23 smithi037 ceph-mon[31460]: pgmap v1091: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:23.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:23 smithi059 ceph-mon[36190]: pgmap v1091: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:23.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:23 smithi170 ceph-mon[36280]: pgmap v1091: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:25.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:25 smithi037 ceph-mon[31460]: pgmap v1092: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:25.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:25 smithi059 ceph-mon[36190]: pgmap v1092: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:25.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:25 smithi170 ceph-mon[36280]: pgmap v1092: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1061.116509 secs 2024-03-28T18:44:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1061.116509 secs 2024-03-28T18:44:26.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1061.116509 secs 2024-03-28T18:44:27.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:27 smithi059 ceph-mon[36190]: pgmap v1093: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:27.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:27 smithi170 ceph-mon[36280]: pgmap v1093: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:27.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:27 smithi037 ceph-mon[31460]: pgmap v1093: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:29 smithi059 ceph-mon[36190]: pgmap v1094: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:29.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:29 smithi170 ceph-mon[36280]: pgmap v1094: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:29.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:29 smithi037 ceph-mon[31460]: pgmap v1094: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:30.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:30 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:44:30.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:30 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:44:30.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:30 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:44:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:31 smithi059 ceph-mon[36190]: pgmap v1095: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1066.116606 secs 2024-03-28T18:44:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:31 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:44:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:31 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:44:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:31 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:44:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:31 smithi170 ceph-mon[36280]: pgmap v1095: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1066.116606 secs 2024-03-28T18:44:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:31 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:44:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:31 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:44:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:31 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:44:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:31 smithi037 ceph-mon[31460]: pgmap v1095: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:31.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1066.116606 secs 2024-03-28T18:44:31.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:31 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:44:31.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:31 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:44:31.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:31 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:44:32.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:32 smithi059 ceph-mon[36190]: pgmap v1096: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:32.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:32 smithi059 ceph-mon[36190]: pgmap v1097: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:32.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:32 smithi170 ceph-mon[36280]: pgmap v1096: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:32.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:32 smithi170 ceph-mon[36280]: pgmap v1097: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:32.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:32 smithi037 ceph-mon[31460]: pgmap v1096: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:32.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:32 smithi037 ceph-mon[31460]: pgmap v1097: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:33.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:33 smithi059 ceph-mon[36190]: pgmap v1098: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:33.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:33 smithi170 ceph-mon[36280]: pgmap v1098: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:33.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:33 smithi037 ceph-mon[31460]: pgmap v1098: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:35 smithi059 ceph-mon[36190]: pgmap v1099: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:36.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:35 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1071.116695 secs 2024-03-28T18:44:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:35 smithi170 ceph-mon[36280]: pgmap v1099: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:35 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1071.116695 secs 2024-03-28T18:44:36.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:35 smithi037 ceph-mon[31460]: pgmap v1099: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:36.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:35 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1071.116695 secs 2024-03-28T18:44:38.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:37 smithi059 ceph-mon[36190]: pgmap v1100: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:38.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:37 smithi170 ceph-mon[36280]: pgmap v1100: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:38.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:37 smithi037 ceph-mon[31460]: pgmap v1100: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:39 smithi059 ceph-mon[36190]: pgmap v1101: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:40.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:39 smithi170 ceph-mon[36280]: pgmap v1101: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:40.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:39 smithi037 ceph-mon[31460]: pgmap v1101: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:40 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1076.116814 secs 2024-03-28T18:44:41.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:40 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1076.116814 secs 2024-03-28T18:44:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:40 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1076.116814 secs 2024-03-28T18:44:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:42 smithi059 ceph-mon[36190]: pgmap v1102: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:44:42.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:42 smithi170 ceph-mon[36280]: pgmap v1102: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:44:42.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:42 smithi037 ceph-mon[31460]: pgmap v1102: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:44:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:44 smithi059 ceph-mon[36190]: pgmap v1103: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:44.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:44 smithi170 ceph-mon[36280]: pgmap v1103: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:44.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:44 smithi037 ceph-mon[31460]: pgmap v1103: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:46 smithi059 ceph-mon[36190]: pgmap v1104: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1081.116932 secs 2024-03-28T18:44:46.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:46 smithi170 ceph-mon[36280]: pgmap v1104: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:46.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1081.116932 secs 2024-03-28T18:44:46.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:46 smithi037 ceph-mon[31460]: pgmap v1104: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:46.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1081.116932 secs 2024-03-28T18:44:48.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:48 smithi059 ceph-mon[36190]: pgmap v1105: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:48.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:48 smithi170 ceph-mon[36280]: pgmap v1105: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:48.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:48 smithi037 ceph-mon[31460]: pgmap v1105: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:50 smithi059 ceph-mon[36190]: pgmap v1106: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:50.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:50 smithi170 ceph-mon[36280]: pgmap v1106: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:50.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:50 smithi037 ceph-mon[31460]: pgmap v1106: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1086.117014 secs 2024-03-28T18:44:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1086.117014 secs 2024-03-28T18:44:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1086.117014 secs 2024-03-28T18:44:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:52 smithi059 ceph-mon[36190]: pgmap v1107: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:52.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:52 smithi170 ceph-mon[36280]: pgmap v1107: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:52.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:52 smithi037 ceph-mon[31460]: pgmap v1107: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:54.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:54 smithi037 ceph-mon[31460]: pgmap v1108: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:54.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:54 smithi059 ceph-mon[36190]: pgmap v1108: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:54.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:54 smithi170 ceph-mon[36280]: pgmap v1108: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:56.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:56 smithi037 ceph-mon[31460]: pgmap v1109: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:56.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1091.117133 secs 2024-03-28T18:44:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:56 smithi059 ceph-mon[36190]: pgmap v1109: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1091.117133 secs 2024-03-28T18:44:56.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:56 smithi170 ceph-mon[36280]: pgmap v1109: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:56.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1091.117133 secs 2024-03-28T18:44:58.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:44:58 smithi037 ceph-mon[31460]: pgmap v1110: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:58.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:44:58 smithi059 ceph-mon[36190]: pgmap v1110: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:58.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:44:58 smithi170 ceph-mon[36280]: pgmap v1110: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:00.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:00 smithi037 ceph-mon[31460]: pgmap v1111: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:00.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:00 smithi059 ceph-mon[36190]: pgmap v1111: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:00.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:00 smithi170 ceph-mon[36280]: pgmap v1111: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1096.117252 secs 2024-03-28T18:45:01.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1096.117252 secs 2024-03-28T18:45:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1096.117252 secs 2024-03-28T18:45:02.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:02 smithi037 ceph-mon[31460]: pgmap v1112: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:02.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:02 smithi059 ceph-mon[36190]: pgmap v1112: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:02.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:02 smithi170 ceph-mon[36280]: pgmap v1112: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:04.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:04 smithi037 ceph-mon[31460]: pgmap v1113: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:04.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:04 smithi059 ceph-mon[36190]: pgmap v1113: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:04.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:04 smithi170 ceph-mon[36280]: pgmap v1113: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:06 smithi037 ceph-mon[31460]: pgmap v1114: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:06.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1101.117338 secs 2024-03-28T18:45:06.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:06 smithi059 ceph-mon[36190]: pgmap v1114: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:06.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1101.117338 secs 2024-03-28T18:45:06.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:06 smithi170 ceph-mon[36280]: pgmap v1114: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:06.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1101.117338 secs 2024-03-28T18:45:08.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:08 smithi037 ceph-mon[31460]: pgmap v1115: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:08.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:08 smithi059 ceph-mon[36190]: pgmap v1115: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:08.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:08 smithi170 ceph-mon[36280]: pgmap v1115: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:10.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:10 smithi037 ceph-mon[31460]: pgmap v1116: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:10.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:10 smithi059 ceph-mon[36190]: pgmap v1116: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:10.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:10 smithi170 ceph-mon[36280]: pgmap v1116: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1106.117419 secs 2024-03-28T18:45:11.436 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1106.117419 secs 2024-03-28T18:45:11.437 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1106.117419 secs 2024-03-28T18:45:12.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:12 smithi037 ceph-mon[31460]: pgmap v1117: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:12.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:12 smithi059 ceph-mon[36190]: pgmap v1117: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:12.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:12 smithi170 ceph-mon[36280]: pgmap v1117: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:14.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:14 smithi037 ceph-mon[31460]: pgmap v1118: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:14.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:14 smithi059 ceph-mon[36190]: pgmap v1118: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:14.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:14 smithi170 ceph-mon[36280]: pgmap v1118: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:16.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:16 smithi037 ceph-mon[31460]: pgmap v1119: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:16.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1111.117513 secs 2024-03-28T18:45:16.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:16 smithi059 ceph-mon[36190]: pgmap v1119: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:16.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1111.117513 secs 2024-03-28T18:45:16.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:16 smithi170 ceph-mon[36280]: pgmap v1119: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:16.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1111.117513 secs 2024-03-28T18:45:18.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:18 smithi037 ceph-mon[31460]: pgmap v1120: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:18.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:18 smithi059 ceph-mon[36190]: pgmap v1120: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:18.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:18 smithi170 ceph-mon[36280]: pgmap v1120: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:20.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:20 smithi037 ceph-mon[31460]: pgmap v1121: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:20.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:20 smithi059 ceph-mon[36190]: pgmap v1121: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:20.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:20 smithi170 ceph-mon[36280]: pgmap v1121: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:21.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1116.117632 secs 2024-03-28T18:45:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1116.117632 secs 2024-03-28T18:45:21.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1116.117632 secs 2024-03-28T18:45:22.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:22 smithi037 ceph-mon[31460]: pgmap v1122: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:22.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:22 smithi059 ceph-mon[36190]: pgmap v1122: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:22.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:22 smithi170 ceph-mon[36280]: pgmap v1122: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:24 smithi059 ceph-mon[36190]: pgmap v1123: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:24.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:24 smithi170 ceph-mon[36280]: pgmap v1123: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:24.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:24 smithi037 ceph-mon[31460]: pgmap v1123: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:26 smithi059 ceph-mon[36190]: pgmap v1124: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1121.117752 secs 2024-03-28T18:45:26.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:26 smithi170 ceph-mon[36280]: pgmap v1124: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:26.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1121.117752 secs 2024-03-28T18:45:26.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:26 smithi037 ceph-mon[31460]: pgmap v1124: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:26.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1121.117752 secs 2024-03-28T18:45:28.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:28 smithi059 ceph-mon[36190]: pgmap v1125: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:28.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:28 smithi170 ceph-mon[36280]: pgmap v1125: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:28.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:28 smithi037 ceph-mon[31460]: pgmap v1125: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:30.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:30 smithi059 ceph-mon[36190]: pgmap v1126: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:30.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:30 smithi170 ceph-mon[36280]: pgmap v1126: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:30.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:30 smithi037 ceph-mon[31460]: pgmap v1126: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1126.117873 secs 2024-03-28T18:45:31.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1126.117873 secs 2024-03-28T18:45:31.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1126.117873 secs 2024-03-28T18:45:32.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:32 smithi059 ceph-mon[36190]: pgmap v1127: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:32.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:32 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:45:32.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:32 smithi170 ceph-mon[36280]: pgmap v1127: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:32.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:32 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:45:32.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:32 smithi037 ceph-mon[31460]: pgmap v1127: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:32.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:32 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:45:33.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:33 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:45:33.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:33 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:45:33.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:33 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:45:33.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:33 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:45:33.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:33 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:45:33.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:33 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:45:33.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:33 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:45:33.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:33 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:45:33.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:33 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:45:34.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:34 smithi059 ceph-mon[36190]: pgmap v1128: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T18:45:34.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:34 smithi059 ceph-mon[36190]: pgmap v1129: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:45:34.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:34 smithi170 ceph-mon[36280]: pgmap v1128: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T18:45:34.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:34 smithi170 ceph-mon[36280]: pgmap v1129: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:45:34.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:34 smithi037 ceph-mon[31460]: pgmap v1128: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T18:45:34.652 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:34 smithi037 ceph-mon[31460]: pgmap v1129: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:45:35.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:35 smithi059 ceph-mon[36190]: pgmap v1130: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:45:35.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:35 smithi170 ceph-mon[36280]: pgmap v1130: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:45:35.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:35 smithi037 ceph-mon[31460]: pgmap v1130: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:45:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1131.117994 secs 2024-03-28T18:45:36.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1131.117994 secs 2024-03-28T18:45:36.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1131.117994 secs 2024-03-28T18:45:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:37 smithi059 ceph-mon[36190]: pgmap v1131: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 977 B/s rd, 2 op/s 2024-03-28T18:45:37.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:37 smithi170 ceph-mon[36280]: pgmap v1131: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 977 B/s rd, 2 op/s 2024-03-28T18:45:37.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:37 smithi037 ceph-mon[31460]: pgmap v1131: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 977 B/s rd, 2 op/s 2024-03-28T18:45:39.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:39 smithi059 ceph-mon[36190]: pgmap v1132: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:45:39.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:39 smithi170 ceph-mon[36280]: pgmap v1132: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:45:39.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:39 smithi037 ceph-mon[31460]: pgmap v1132: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:45:41.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:41 smithi059 ceph-mon[36190]: pgmap v1133: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:45:41.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1136.118100 secs 2024-03-28T18:45:41.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:41 smithi170 ceph-mon[36280]: pgmap v1133: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:45:41.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1136.118100 secs 2024-03-28T18:45:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:41 smithi037 ceph-mon[31460]: pgmap v1133: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:45:41.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1136.118100 secs 2024-03-28T18:45:43.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:43 smithi059 ceph-mon[36190]: pgmap v1134: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:45:43.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:43 smithi170 ceph-mon[36280]: pgmap v1134: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:45:43.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:43 smithi037 ceph-mon[31460]: pgmap v1134: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:45:45.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:45 smithi059 ceph-mon[36190]: pgmap v1135: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:45.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:45 smithi170 ceph-mon[36280]: pgmap v1135: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:45.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:45 smithi037 ceph-mon[31460]: pgmap v1135: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:46.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1141.118195 secs 2024-03-28T18:45:46.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1141.118195 secs 2024-03-28T18:45:46.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1141.118195 secs 2024-03-28T18:45:47.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:47 smithi059 ceph-mon[36190]: pgmap v1136: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:47.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:47 smithi170 ceph-mon[36280]: pgmap v1136: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:47.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:47 smithi037 ceph-mon[31460]: pgmap v1136: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:49.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:49 smithi059 ceph-mon[36190]: pgmap v1137: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:49.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:49 smithi170 ceph-mon[36280]: pgmap v1137: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:49.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:49 smithi037 ceph-mon[31460]: pgmap v1137: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:51.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:51 smithi059 ceph-mon[36190]: pgmap v1138: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:51.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1146.118291 secs 2024-03-28T18:45:51.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:51 smithi170 ceph-mon[36280]: pgmap v1138: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:51.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1146.118291 secs 2024-03-28T18:45:51.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:51 smithi037 ceph-mon[31460]: pgmap v1138: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:51.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1146.118291 secs 2024-03-28T18:45:53.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:53 smithi059 ceph-mon[36190]: pgmap v1139: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:53.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:53 smithi170 ceph-mon[36280]: pgmap v1139: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:53.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:53 smithi037 ceph-mon[31460]: pgmap v1139: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:55.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:55 smithi170 ceph-mon[36280]: pgmap v1140: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:55.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:55 smithi059 ceph-mon[36190]: pgmap v1140: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:55.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:55 smithi037 ceph-mon[31460]: pgmap v1140: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1151.118381 secs 2024-03-28T18:45:56.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1151.118381 secs 2024-03-28T18:45:56.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1151.118381 secs 2024-03-28T18:45:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:57 smithi059 ceph-mon[36190]: pgmap v1141: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:57.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:57 smithi170 ceph-mon[36280]: pgmap v1141: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:57.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:57 smithi037 ceph-mon[31460]: pgmap v1141: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:45:59 smithi059 ceph-mon[36190]: pgmap v1142: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:59.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:45:59 smithi170 ceph-mon[36280]: pgmap v1142: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:59.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:45:59 smithi037 ceph-mon[31460]: pgmap v1142: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:01.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:01 smithi059 ceph-mon[36190]: pgmap v1143: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:01.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1156.118506 secs 2024-03-28T18:46:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:01 smithi170 ceph-mon[36280]: pgmap v1143: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:01.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1156.118506 secs 2024-03-28T18:46:01.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:01 smithi037 ceph-mon[31460]: pgmap v1143: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:01.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1156.118506 secs 2024-03-28T18:46:03.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:03 smithi059 ceph-mon[36190]: pgmap v1144: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:03.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:03 smithi170 ceph-mon[36280]: pgmap v1144: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:03.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:03 smithi037 ceph-mon[31460]: pgmap v1144: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:05.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:05 smithi059 ceph-mon[36190]: pgmap v1145: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:05.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:05 smithi170 ceph-mon[36280]: pgmap v1145: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:05.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:05 smithi037 ceph-mon[31460]: pgmap v1145: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:06.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1161.118606 secs 2024-03-28T18:46:06.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1161.118606 secs 2024-03-28T18:46:06.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1161.118606 secs 2024-03-28T18:46:07.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:07 smithi059 ceph-mon[36190]: pgmap v1146: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:07.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:07 smithi170 ceph-mon[36280]: pgmap v1146: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:07.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:07 smithi037 ceph-mon[31460]: pgmap v1146: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:09.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:09 smithi059 ceph-mon[36190]: pgmap v1147: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:09.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:09 smithi170 ceph-mon[36280]: pgmap v1147: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:09.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:09 smithi037 ceph-mon[31460]: pgmap v1147: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:11.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:11 smithi059 ceph-mon[36190]: pgmap v1148: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:11.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1166.118717 secs 2024-03-28T18:46:11.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:11 smithi170 ceph-mon[36280]: pgmap v1148: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:11.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1166.118717 secs 2024-03-28T18:46:11.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:11 smithi037 ceph-mon[31460]: pgmap v1148: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:11.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1166.118717 secs 2024-03-28T18:46:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:13 smithi059 ceph-mon[36190]: pgmap v1149: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:13.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:13 smithi170 ceph-mon[36280]: pgmap v1149: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:13.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:13 smithi037 ceph-mon[31460]: pgmap v1149: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:15.677 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:15 smithi037 ceph-mon[31460]: pgmap v1150: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:15.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:15 smithi059 ceph-mon[36190]: pgmap v1150: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:15.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:15 smithi170 ceph-mon[36280]: pgmap v1150: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:16.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1171.118853 secs 2024-03-28T18:46:17.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1171.118853 secs 2024-03-28T18:46:17.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1171.118853 secs 2024-03-28T18:46:17.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:17 smithi037 ceph-mon[31460]: pgmap v1151: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:18.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:17 smithi059 ceph-mon[36190]: pgmap v1151: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:18.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:17 smithi170 ceph-mon[36280]: pgmap v1151: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:19.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:19 smithi037 ceph-mon[31460]: pgmap v1152: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:19 smithi059 ceph-mon[36190]: pgmap v1152: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:20.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:19 smithi170 ceph-mon[36280]: pgmap v1152: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:21.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:21 smithi037 ceph-mon[31460]: pgmap v1153: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1176.118970 secs 2024-03-28T18:46:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:21 smithi170 ceph-mon[36280]: pgmap v1153: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1176.118970 secs 2024-03-28T18:46:22.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:21 smithi059 ceph-mon[36190]: pgmap v1153: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:22.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1176.118970 secs 2024-03-28T18:46:23.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:23 smithi037 ceph-mon[31460]: pgmap v1154: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:23 smithi059 ceph-mon[36190]: pgmap v1154: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:23 smithi170 ceph-mon[36280]: pgmap v1154: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:25.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:25 smithi037 ceph-mon[31460]: pgmap v1155: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:25 smithi059 ceph-mon[36190]: pgmap v1155: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:26.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:25 smithi170 ceph-mon[36280]: pgmap v1155: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:26.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1181.119075 secs 2024-03-28T18:46:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1181.119075 secs 2024-03-28T18:46:27.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1181.119075 secs 2024-03-28T18:46:27.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:27 smithi037 ceph-mon[31460]: pgmap v1156: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:27 smithi059 ceph-mon[36190]: pgmap v1156: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:28.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:27 smithi170 ceph-mon[36280]: pgmap v1156: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:29.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:29 smithi037 ceph-mon[31460]: pgmap v1157: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:29 smithi059 ceph-mon[36190]: pgmap v1157: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:30.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:29 smithi170 ceph-mon[36280]: pgmap v1157: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:31.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:31 smithi037 ceph-mon[31460]: pgmap v1158: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:31.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1186.119196 secs 2024-03-28T18:46:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:31 smithi059 ceph-mon[36190]: pgmap v1158: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1186.119196 secs 2024-03-28T18:46:32.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:31 smithi170 ceph-mon[36280]: pgmap v1158: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:32.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1186.119196 secs 2024-03-28T18:46:33.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:33 smithi037 ceph-mon[31460]: pgmap v1159: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:33.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:33 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:46:34.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:33 smithi059 ceph-mon[36190]: pgmap v1159: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:34.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:33 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:46:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:33 smithi170 ceph-mon[36280]: pgmap v1159: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:33 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:46:34.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:34 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:46:34.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:34 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:46:34.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:34 smithi037 ceph-mon[31460]: pgmap v1160: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:34.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:34 smithi037 ceph-mon[31460]: pgmap v1161: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:46:34.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:34 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:46:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:34 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:46:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:34 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:46:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:34 smithi059 ceph-mon[36190]: pgmap v1160: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:34 smithi059 ceph-mon[36190]: pgmap v1161: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:46:35.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:34 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:46:35.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:34 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:46:35.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:34 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:46:35.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:34 smithi170 ceph-mon[36280]: pgmap v1160: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:35.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:34 smithi170 ceph-mon[36280]: pgmap v1161: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:46:35.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:34 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:46:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1191.119262 secs 2024-03-28T18:46:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:36 smithi059 ceph-mon[36190]: pgmap v1162: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:46:37.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1191.119262 secs 2024-03-28T18:46:37.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:36 smithi170 ceph-mon[36280]: pgmap v1162: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:46:37.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1191.119262 secs 2024-03-28T18:46:37.392 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:36 smithi037 ceph-mon[31460]: pgmap v1162: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:46:39.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:38 smithi059 ceph-mon[36190]: pgmap v1163: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:46:39.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:38 smithi170 ceph-mon[36280]: pgmap v1163: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:46:39.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:38 smithi037 ceph-mon[31460]: pgmap v1163: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:46:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:40 smithi059 ceph-mon[36190]: pgmap v1164: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:46:41.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:40 smithi170 ceph-mon[36280]: pgmap v1164: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:46:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:40 smithi037 ceph-mon[31460]: pgmap v1164: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:46:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1196.119353 secs 2024-03-28T18:46:42.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1196.119353 secs 2024-03-28T18:46:42.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1196.119353 secs 2024-03-28T18:46:43.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:42 smithi059 ceph-mon[36190]: pgmap v1165: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:46:43.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:42 smithi170 ceph-mon[36280]: pgmap v1165: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:46:43.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:42 smithi037 ceph-mon[31460]: pgmap v1165: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:46:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:44 smithi059 ceph-mon[36190]: pgmap v1166: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:46:45.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:44 smithi170 ceph-mon[36280]: pgmap v1166: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:46:45.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:44 smithi037 ceph-mon[31460]: pgmap v1166: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:46:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1201.119472 secs 2024-03-28T18:46:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:46 smithi059 ceph-mon[36190]: pgmap v1167: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1201.119472 secs 2024-03-28T18:46:47.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:46 smithi170 ceph-mon[36280]: pgmap v1167: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:47.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1201.119472 secs 2024-03-28T18:46:47.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:46 smithi037 ceph-mon[31460]: pgmap v1167: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:48 smithi059 ceph-mon[36190]: pgmap v1168: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:49.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:48 smithi170 ceph-mon[36280]: pgmap v1168: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:49.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:48 smithi037 ceph-mon[31460]: pgmap v1168: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:51.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:50 smithi059 ceph-mon[36190]: pgmap v1169: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:51.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:50 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1206.119562 secs 2024-03-28T18:46:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:50 smithi170 ceph-mon[36280]: pgmap v1169: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:50 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1206.119562 secs 2024-03-28T18:46:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:50 smithi037 ceph-mon[31460]: pgmap v1169: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:51.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:50 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1206.119562 secs 2024-03-28T18:46:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:52 smithi059 ceph-mon[36190]: pgmap v1170: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:52 smithi170 ceph-mon[36280]: pgmap v1170: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:53.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:52 smithi037 ceph-mon[31460]: pgmap v1170: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:54 smithi059 ceph-mon[36190]: pgmap v1171: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:55.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:54 smithi170 ceph-mon[36280]: pgmap v1171: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:55.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:54 smithi037 ceph-mon[31460]: pgmap v1171: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:55 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1211.119638 secs 2024-03-28T18:46:56.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:55 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1211.119638 secs 2024-03-28T18:46:56.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:55 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1211.119638 secs 2024-03-28T18:46:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:56 smithi059 ceph-mon[36190]: pgmap v1172: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:56 smithi170 ceph-mon[36280]: pgmap v1172: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:57.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:56 smithi037 ceph-mon[31460]: pgmap v1172: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:59.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:46:58 smithi059 ceph-mon[36190]: pgmap v1173: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:59.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:46:58 smithi170 ceph-mon[36280]: pgmap v1173: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:59.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:46:58 smithi037 ceph-mon[31460]: pgmap v1173: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:00 smithi059 ceph-mon[36190]: pgmap v1174: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:00 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1216.119750 secs 2024-03-28T18:47:01.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:00 smithi170 ceph-mon[36280]: pgmap v1174: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:01.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:00 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1216.119750 secs 2024-03-28T18:47:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:00 smithi037 ceph-mon[31460]: pgmap v1174: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:00 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1216.119750 secs 2024-03-28T18:47:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:02 smithi059 ceph-mon[36190]: pgmap v1175: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:03.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:02 smithi170 ceph-mon[36280]: pgmap v1175: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:03.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:02 smithi037 ceph-mon[31460]: pgmap v1175: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:05.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:04 smithi059 ceph-mon[36190]: pgmap v1176: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:05.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:04 smithi170 ceph-mon[36280]: pgmap v1176: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:05.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:04 smithi037 ceph-mon[31460]: pgmap v1176: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:05 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1221.119867 secs 2024-03-28T18:47:06.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:05 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1221.119867 secs 2024-03-28T18:47:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:05 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1221.119867 secs 2024-03-28T18:47:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:06 smithi059 ceph-mon[36190]: pgmap v1177: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:07.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:06 smithi170 ceph-mon[36280]: pgmap v1177: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:07.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:06 smithi037 ceph-mon[31460]: pgmap v1177: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:08 smithi059 ceph-mon[36190]: pgmap v1178: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:09.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:08 smithi170 ceph-mon[36280]: pgmap v1178: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:09.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:08 smithi037 ceph-mon[31460]: pgmap v1178: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:11.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:10 smithi059 ceph-mon[36190]: pgmap v1179: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:11.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:10 smithi170 ceph-mon[36280]: pgmap v1179: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:11.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:10 smithi037 ceph-mon[31460]: pgmap v1179: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1226.119982 secs 2024-03-28T18:47:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1226.119982 secs 2024-03-28T18:47:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1226.119982 secs 2024-03-28T18:47:13.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:12 smithi059 ceph-mon[36190]: pgmap v1180: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:13.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:12 smithi170 ceph-mon[36280]: pgmap v1180: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:13.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:12 smithi037 ceph-mon[31460]: pgmap v1180: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:15.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:14 smithi059 ceph-mon[36190]: pgmap v1181: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:15.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:14 smithi170 ceph-mon[36280]: pgmap v1181: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:15.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:14 smithi037 ceph-mon[31460]: pgmap v1181: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:17.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1231.120105 secs 2024-03-28T18:47:17.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:16 smithi059 ceph-mon[36190]: pgmap v1182: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:17.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1231.120105 secs 2024-03-28T18:47:17.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:16 smithi170 ceph-mon[36280]: pgmap v1182: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:17.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1231.120105 secs 2024-03-28T18:47:17.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:16 smithi037 ceph-mon[31460]: pgmap v1182: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:19.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:18 smithi059 ceph-mon[36190]: pgmap v1183: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:19.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:18 smithi170 ceph-mon[36280]: pgmap v1183: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:19.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:18 smithi037 ceph-mon[31460]: pgmap v1183: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:21.178 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:20 smithi037 ceph-mon[31460]: pgmap v1184: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:20 smithi059 ceph-mon[36190]: pgmap v1184: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:21.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:20 smithi170 ceph-mon[36280]: pgmap v1184: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:22.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1236.120221 secs 2024-03-28T18:47:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1236.120221 secs 2024-03-28T18:47:22.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1236.120221 secs 2024-03-28T18:47:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:22 smithi059 ceph-mon[36190]: pgmap v1185: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:47:23.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:22 smithi170 ceph-mon[36280]: pgmap v1185: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:47:23.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:22 smithi037 ceph-mon[31460]: pgmap v1185: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:47:25.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:24 smithi059 ceph-mon[36190]: pgmap v1186: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:47:25.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:24 smithi170 ceph-mon[36280]: pgmap v1186: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:47:25.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:24 smithi037 ceph-mon[31460]: pgmap v1186: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:47:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:25 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1241.120302 secs 2024-03-28T18:47:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:25 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1241.120302 secs 2024-03-28T18:47:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:25 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1241.120302 secs 2024-03-28T18:47:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:26 smithi059 ceph-mon[36190]: pgmap v1187: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:47:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:26 smithi170 ceph-mon[36280]: pgmap v1187: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:47:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:26 smithi037 ceph-mon[31460]: pgmap v1187: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:47:29.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:28 smithi059 ceph-mon[36190]: pgmap v1188: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:47:29.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:28 smithi170 ceph-mon[36280]: pgmap v1188: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:47:29.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:28 smithi037 ceph-mon[31460]: pgmap v1188: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:47:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:30 smithi059 ceph-mon[36190]: pgmap v1189: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:47:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:30 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1246.120422 secs 2024-03-28T18:47:31.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:30 smithi170 ceph-mon[36280]: pgmap v1189: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:47:31.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:30 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1246.120422 secs 2024-03-28T18:47:31.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:30 smithi037 ceph-mon[31460]: pgmap v1189: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:47:31.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:30 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1246.120422 secs 2024-03-28T18:47:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:32 smithi059 ceph-mon[36190]: pgmap v1190: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:47:33.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:32 smithi170 ceph-mon[36280]: pgmap v1190: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:47:33.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:32 smithi037 ceph-mon[31460]: pgmap v1190: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:47:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:34 smithi059 ceph-mon[36190]: pgmap v1191: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:34 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:47:35.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:34 smithi170 ceph-mon[36280]: pgmap v1191: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:35.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:34 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:47:35.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:34 smithi037 ceph-mon[31460]: pgmap v1191: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:35.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:34 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:47:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:35 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:47:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:35 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:47:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:35 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:47:36.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:35 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1251.120541 secs 2024-03-28T18:47:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:35 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:47:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:35 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:47:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:35 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:47:36.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:35 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1251.120541 secs 2024-03-28T18:47:36.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:35 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:47:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:35 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:47:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:35 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:47:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:35 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1251.120541 secs 2024-03-28T18:47:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:36 smithi059 ceph-mon[36190]: pgmap v1192: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:36 smithi059 ceph-mon[36190]: pgmap v1193: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:36 smithi170 ceph-mon[36280]: pgmap v1192: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:37.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:36 smithi170 ceph-mon[36280]: pgmap v1193: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:36 smithi037 ceph-mon[31460]: pgmap v1192: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:36 smithi037 ceph-mon[31460]: pgmap v1193: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:39.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:38 smithi059 ceph-mon[36190]: pgmap v1194: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:39.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:38 smithi170 ceph-mon[36280]: pgmap v1194: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:39.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:38 smithi037 ceph-mon[31460]: pgmap v1194: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:40 smithi059 ceph-mon[36190]: pgmap v1195: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:40 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1256.120673 secs 2024-03-28T18:47:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:40 smithi170 ceph-mon[36280]: pgmap v1195: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:40 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1256.120673 secs 2024-03-28T18:47:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:40 smithi037 ceph-mon[31460]: pgmap v1195: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:41.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:40 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1256.120673 secs 2024-03-28T18:47:43.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:42 smithi059 ceph-mon[36190]: pgmap v1196: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:47:43.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:42 smithi170 ceph-mon[36280]: pgmap v1196: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:47:43.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:42 smithi037 ceph-mon[31460]: pgmap v1196: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:47:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:44 smithi059 ceph-mon[36190]: pgmap v1197: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:45.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:44 smithi170 ceph-mon[36280]: pgmap v1197: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:45.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:44 smithi037 ceph-mon[31460]: pgmap v1197: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1261.120790 secs 2024-03-28T18:47:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1261.120790 secs 2024-03-28T18:47:46.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1261.120790 secs 2024-03-28T18:47:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:47 smithi059 ceph-mon[36190]: pgmap v1198: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:47:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:47 smithi170 ceph-mon[36280]: pgmap v1198: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:47:47.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:47 smithi037 ceph-mon[31460]: pgmap v1198: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:47:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:49 smithi059 ceph-mon[36190]: pgmap v1199: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:49 smithi170 ceph-mon[36280]: pgmap v1199: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:49.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:49 smithi037 ceph-mon[31460]: pgmap v1199: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:51 smithi059 ceph-mon[36190]: pgmap v1200: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1266.120909 secs 2024-03-28T18:47:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:51 smithi170 ceph-mon[36280]: pgmap v1200: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1266.120909 secs 2024-03-28T18:47:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:51 smithi037 ceph-mon[31460]: pgmap v1200: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:51.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1266.120909 secs 2024-03-28T18:47:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:53 smithi059 ceph-mon[36190]: pgmap v1201: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:53 smithi170 ceph-mon[36280]: pgmap v1201: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:53.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:53 smithi037 ceph-mon[31460]: pgmap v1201: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:55 smithi059 ceph-mon[36190]: pgmap v1202: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:55.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:55 smithi170 ceph-mon[36280]: pgmap v1202: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:55.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:55 smithi037 ceph-mon[31460]: pgmap v1202: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:56.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1271.121054 secs 2024-03-28T18:47:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1271.121054 secs 2024-03-28T18:47:56.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1271.121054 secs 2024-03-28T18:47:57.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:57 smithi037 ceph-mon[31460]: pgmap v1203: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:57.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:57 smithi059 ceph-mon[36190]: pgmap v1203: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:57.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:57 smithi170 ceph-mon[36280]: pgmap v1203: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:59.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:47:59 smithi037 ceph-mon[31460]: pgmap v1204: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:59.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:47:59 smithi059 ceph-mon[36190]: pgmap v1204: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:59.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:47:59 smithi170 ceph-mon[36280]: pgmap v1204: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:01 smithi037 ceph-mon[31460]: pgmap v1205: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1276.121174 secs 2024-03-28T18:48:01.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:01 smithi059 ceph-mon[36190]: pgmap v1205: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:01.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1276.121174 secs 2024-03-28T18:48:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:01 smithi170 ceph-mon[36280]: pgmap v1205: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1276.121174 secs 2024-03-28T18:48:03.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:03 smithi037 ceph-mon[31460]: pgmap v1206: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:03 smithi059 ceph-mon[36190]: pgmap v1206: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:03.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:03 smithi170 ceph-mon[36280]: pgmap v1206: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:05.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:05 smithi037 ceph-mon[31460]: pgmap v1207: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:05.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:05 smithi059 ceph-mon[36190]: pgmap v1207: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:05.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:05 smithi170 ceph-mon[36280]: pgmap v1207: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1281.121294 secs 2024-03-28T18:48:06.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1281.121294 secs 2024-03-28T18:48:06.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1281.121294 secs 2024-03-28T18:48:07.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:07 smithi037 ceph-mon[31460]: pgmap v1208: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:07.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:07 smithi059 ceph-mon[36190]: pgmap v1208: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:07.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:07 smithi170 ceph-mon[36280]: pgmap v1208: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:09.378 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:09 smithi037 ceph-mon[31460]: pgmap v1209: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:09.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:09 smithi059 ceph-mon[36190]: pgmap v1209: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:09.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:09 smithi170 ceph-mon[36280]: pgmap v1209: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:11.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:11 smithi170 ceph-mon[36280]: pgmap v1210: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:11.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1286.121416 secs 2024-03-28T18:48:11.338 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:11 smithi059 ceph-mon[36190]: pgmap v1210: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:11.338 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1286.121416 secs 2024-03-28T18:48:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:11 smithi037 ceph-mon[31460]: pgmap v1210: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1286.121416 secs 2024-03-28T18:48:13.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:13 smithi037 ceph-mon[31460]: pgmap v1211: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:13.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:13 smithi059 ceph-mon[36190]: pgmap v1211: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:13.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:13 smithi170 ceph-mon[36280]: pgmap v1211: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:15.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:15 smithi037 ceph-mon[31460]: pgmap v1212: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:15.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:15 smithi059 ceph-mon[36190]: pgmap v1212: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:15.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:15 smithi170 ceph-mon[36280]: pgmap v1212: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:16.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1291.121535 secs 2024-03-28T18:48:16.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1291.121535 secs 2024-03-28T18:48:16.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1291.121535 secs 2024-03-28T18:48:17.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:17 smithi037 ceph-mon[31460]: pgmap v1213: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:17.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:17 smithi059 ceph-mon[36190]: pgmap v1213: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:17.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:17 smithi170 ceph-mon[36280]: pgmap v1213: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:19.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:19 smithi037 ceph-mon[31460]: pgmap v1214: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:19.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:19 smithi059 ceph-mon[36190]: pgmap v1214: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:19.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:19 smithi170 ceph-mon[36280]: pgmap v1214: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:21.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:21 smithi037 ceph-mon[31460]: pgmap v1215: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:21.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1296.121624 secs 2024-03-28T18:48:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:21 smithi059 ceph-mon[36190]: pgmap v1215: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1296.121624 secs 2024-03-28T18:48:21.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:21 smithi170 ceph-mon[36280]: pgmap v1215: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:21.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1296.121624 secs 2024-03-28T18:48:23.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:23 smithi037 ceph-mon[31460]: pgmap v1216: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:23.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:23 smithi059 ceph-mon[36190]: pgmap v1216: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:23.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:23 smithi170 ceph-mon[36280]: pgmap v1216: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:25.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:25 smithi037 ceph-mon[31460]: pgmap v1217: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:25.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:25 smithi059 ceph-mon[36190]: pgmap v1217: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:25.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:25 smithi170 ceph-mon[36280]: pgmap v1217: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1301.121746 secs 2024-03-28T18:48:26.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1301.121746 secs 2024-03-28T18:48:26.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1301.121746 secs 2024-03-28T18:48:27.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:27 smithi059 ceph-mon[36190]: pgmap v1218: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:27.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:27 smithi170 ceph-mon[36280]: pgmap v1218: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:27.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:27 smithi037 ceph-mon[31460]: pgmap v1218: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:29.538 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:29 smithi059 ceph-mon[36190]: pgmap v1219: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:29.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:29 smithi170 ceph-mon[36280]: pgmap v1219: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:29.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:29 smithi037 ceph-mon[31460]: pgmap v1219: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:31 smithi059 ceph-mon[36190]: pgmap v1220: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1306.122017 secs 2024-03-28T18:48:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:31 smithi170 ceph-mon[36280]: pgmap v1220: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1306.122017 secs 2024-03-28T18:48:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:31 smithi037 ceph-mon[31460]: pgmap v1220: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1306.122017 secs 2024-03-28T18:48:33.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:33 smithi059 ceph-mon[36190]: pgmap v1221: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:33.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:33 smithi170 ceph-mon[36280]: pgmap v1221: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:33.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:33 smithi037 ceph-mon[31460]: pgmap v1221: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:35.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:35 smithi059 ceph-mon[36190]: pgmap v1222: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:35.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:35 smithi170 ceph-mon[36280]: pgmap v1222: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:35.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:35 smithi037 ceph-mon[31460]: pgmap v1222: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1311.122105 secs 2024-03-28T18:48:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:36 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:48:36.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1311.122105 secs 2024-03-28T18:48:36.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:36 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:48:36.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1311.122105 secs 2024-03-28T18:48:36.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:36 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:48:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:37 smithi059 ceph-mon[36190]: pgmap v1223: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:37 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:48:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:37 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:48:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:37 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:48:37.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:37 smithi170 ceph-mon[36280]: pgmap v1223: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:37.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:37 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:48:37.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:37 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:48:37.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:37 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:48:37.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:37 smithi037 ceph-mon[31460]: pgmap v1223: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:37.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:37 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:48:37.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:37 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:48:37.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:37 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:48:38.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:38 smithi059 ceph-mon[36190]: pgmap v1224: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:48:38.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:38 smithi059 ceph-mon[36190]: pgmap v1225: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:38.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:38 smithi170 ceph-mon[36280]: pgmap v1224: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:48:38.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:38 smithi170 ceph-mon[36280]: pgmap v1225: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:38.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:38 smithi037 ceph-mon[31460]: pgmap v1224: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:48:38.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:38 smithi037 ceph-mon[31460]: pgmap v1225: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:40.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:40 smithi059 ceph-mon[36190]: pgmap v1226: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:40.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:40 smithi170 ceph-mon[36280]: pgmap v1226: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:40.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:40 smithi037 ceph-mon[31460]: pgmap v1226: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:41.436 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1316.122192 secs 2024-03-28T18:48:41.437 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1316.122192 secs 2024-03-28T18:48:41.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1316.122192 secs 2024-03-28T18:48:42.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:42 smithi059 ceph-mon[36190]: pgmap v1227: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:42.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:42 smithi170 ceph-mon[36280]: pgmap v1227: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:42.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:42 smithi037 ceph-mon[31460]: pgmap v1227: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:44.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:44 smithi059 ceph-mon[36190]: pgmap v1228: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:44.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:44 smithi170 ceph-mon[36280]: pgmap v1228: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:44.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:44 smithi037 ceph-mon[31460]: pgmap v1228: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:45.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:45 smithi059 ceph-mon[36190]: pgmap v1229: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:45.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:45 smithi170 ceph-mon[36280]: pgmap v1229: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:45.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:45 smithi037 ceph-mon[31460]: pgmap v1229: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:46.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1321.122312 secs 2024-03-28T18:48:46.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1321.122312 secs 2024-03-28T18:48:46.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1321.122312 secs 2024-03-28T18:48:47.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:47 smithi059 ceph-mon[36190]: pgmap v1230: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:48:47.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:47 smithi170 ceph-mon[36280]: pgmap v1230: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:48:47.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:47 smithi037 ceph-mon[31460]: pgmap v1230: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:48:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:49 smithi059 ceph-mon[36190]: pgmap v1231: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:50.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:49 smithi170 ceph-mon[36280]: pgmap v1231: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:50.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:49 smithi037 ceph-mon[31460]: pgmap v1231: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:51.276 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:50 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1326.122433 secs 2024-03-28T18:48:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:50 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1326.122433 secs 2024-03-28T18:48:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:50 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1326.122433 secs 2024-03-28T18:48:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:51 smithi059 ceph-mon[36190]: pgmap v1232: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:51 smithi170 ceph-mon[36280]: pgmap v1232: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:52.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:51 smithi037 ceph-mon[31460]: pgmap v1232: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:54.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:53 smithi059 ceph-mon[36190]: pgmap v1233: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:54.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:53 smithi170 ceph-mon[36280]: pgmap v1233: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:54.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:53 smithi037 ceph-mon[31460]: pgmap v1233: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:55 smithi059 ceph-mon[36190]: pgmap v1234: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:56.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:55 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1331.122552 secs 2024-03-28T18:48:56.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:55 smithi170 ceph-mon[36280]: pgmap v1234: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:56.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:55 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1331.122552 secs 2024-03-28T18:48:56.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:55 smithi037 ceph-mon[31460]: pgmap v1234: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:56.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:55 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1331.122552 secs 2024-03-28T18:48:58.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:57 smithi059 ceph-mon[36190]: pgmap v1235: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:58.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:57 smithi170 ceph-mon[36280]: pgmap v1235: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:58.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:57 smithi037 ceph-mon[31460]: pgmap v1235: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:00.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:48:59 smithi059 ceph-mon[36190]: pgmap v1236: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:00.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:48:59 smithi170 ceph-mon[36280]: pgmap v1236: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:00.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:48:59 smithi037 ceph-mon[31460]: pgmap v1236: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:01.274 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:00 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1336.122631 secs 2024-03-28T18:49:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:00 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1336.122631 secs 2024-03-28T18:49:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:00 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1336.122631 secs 2024-03-28T18:49:02.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:01 smithi059 ceph-mon[36190]: pgmap v1237: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:02.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:01 smithi170 ceph-mon[36280]: pgmap v1237: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:02.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:01 smithi037 ceph-mon[31460]: pgmap v1237: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:04.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:03 smithi059 ceph-mon[36190]: pgmap v1238: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:04.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:03 smithi170 ceph-mon[36280]: pgmap v1238: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:04.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:03 smithi037 ceph-mon[31460]: pgmap v1238: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:06 smithi059 ceph-mon[36190]: pgmap v1239: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:06.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1341.122728 secs 2024-03-28T18:49:06.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:06 smithi170 ceph-mon[36280]: pgmap v1239: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:06.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1341.122728 secs 2024-03-28T18:49:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:06 smithi037 ceph-mon[31460]: pgmap v1239: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:06.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1341.122728 secs 2024-03-28T18:49:08.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:08 smithi059 ceph-mon[36190]: pgmap v1240: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:08.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:08 smithi170 ceph-mon[36280]: pgmap v1240: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:08.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:08 smithi037 ceph-mon[31460]: pgmap v1240: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:10.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:10 smithi059 ceph-mon[36190]: pgmap v1241: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:10.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:10 smithi170 ceph-mon[36280]: pgmap v1241: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:10.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:10 smithi037 ceph-mon[31460]: pgmap v1241: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1346.122847 secs 2024-03-28T18:49:11.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1346.122847 secs 2024-03-28T18:49:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1346.122847 secs 2024-03-28T18:49:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:12 smithi059 ceph-mon[36190]: pgmap v1242: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:12.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:12 smithi170 ceph-mon[36280]: pgmap v1242: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:12.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:12 smithi037 ceph-mon[31460]: pgmap v1242: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:14 smithi059 ceph-mon[36190]: pgmap v1243: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:14.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:14 smithi170 ceph-mon[36280]: pgmap v1243: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:14.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:14 smithi037 ceph-mon[31460]: pgmap v1243: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:16.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:16 smithi170 ceph-mon[36280]: pgmap v1244: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:16.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1351.122961 secs 2024-03-28T18:49:16.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:16 smithi037 ceph-mon[31460]: pgmap v1244: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:16.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1351.122961 secs 2024-03-28T18:49:16.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:16 smithi059 ceph-mon[36190]: pgmap v1244: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:16.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1351.122961 secs 2024-03-28T18:49:18.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:18 smithi037 ceph-mon[31460]: pgmap v1245: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:18.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:18 smithi059 ceph-mon[36190]: pgmap v1245: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:18.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:18 smithi170 ceph-mon[36280]: pgmap v1245: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:20.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:20 smithi037 ceph-mon[31460]: pgmap v1246: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:20.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:20 smithi059 ceph-mon[36190]: pgmap v1246: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:20.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:20 smithi170 ceph-mon[36280]: pgmap v1246: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:21.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1356.123081 secs 2024-03-28T18:49:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1356.123081 secs 2024-03-28T18:49:21.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1356.123081 secs 2024-03-28T18:49:22.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:22 smithi037 ceph-mon[31460]: pgmap v1247: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:22.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:22 smithi059 ceph-mon[36190]: pgmap v1247: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:22.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:22 smithi170 ceph-mon[36280]: pgmap v1247: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:24.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:24 smithi037 ceph-mon[31460]: pgmap v1248: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:24 smithi059 ceph-mon[36190]: pgmap v1248: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:24.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:24 smithi170 ceph-mon[36280]: pgmap v1248: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:26.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:26 smithi037 ceph-mon[31460]: pgmap v1249: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:26.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1361.123202 secs 2024-03-28T18:49:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:26 smithi059 ceph-mon[36190]: pgmap v1249: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1361.123202 secs 2024-03-28T18:49:26.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:26 smithi170 ceph-mon[36280]: pgmap v1249: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:26.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1361.123202 secs 2024-03-28T18:49:28.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:28 smithi037 ceph-mon[31460]: pgmap v1250: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:28.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:28 smithi059 ceph-mon[36190]: pgmap v1250: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:28.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:28 smithi170 ceph-mon[36280]: pgmap v1250: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:30.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:30 smithi037 ceph-mon[31460]: pgmap v1251: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:30.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:30 smithi059 ceph-mon[36190]: pgmap v1251: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:30.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:30 smithi170 ceph-mon[36280]: pgmap v1251: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:31.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1366.123323 secs 2024-03-28T18:49:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1366.123323 secs 2024-03-28T18:49:31.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1366.123323 secs 2024-03-28T18:49:32.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:32 smithi037 ceph-mon[31460]: pgmap v1252: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:32.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:32 smithi059 ceph-mon[36190]: pgmap v1252: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:32.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:32 smithi170 ceph-mon[36280]: pgmap v1252: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:34.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:34 smithi037 ceph-mon[31460]: pgmap v1253: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:34.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:34 smithi059 ceph-mon[36190]: pgmap v1253: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:34.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:34 smithi170 ceph-mon[36280]: pgmap v1253: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:36.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:36 smithi037 ceph-mon[31460]: pgmap v1254: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:36.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1371.123442 secs 2024-03-28T18:49:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:36 smithi059 ceph-mon[36190]: pgmap v1254: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1371.123442 secs 2024-03-28T18:49:36.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:36 smithi170 ceph-mon[36280]: pgmap v1254: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:36.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1371.123442 secs 2024-03-28T18:49:38.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:38 smithi037 ceph-mon[31460]: pgmap v1255: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:38.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:38 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:49:38.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:38 smithi059 ceph-mon[36190]: pgmap v1255: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:38.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:38 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:49:38.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:38 smithi170 ceph-mon[36280]: pgmap v1255: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:38.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:38 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:49:39.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:39 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:49:39.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:39 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:49:39.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:39 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:49:39.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:39 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:49:39.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:39 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:49:39.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:39 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:49:39.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:39 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:49:39.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:39 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:49:39.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:39 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:49:40.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:40 smithi037 ceph-mon[31460]: pgmap v1256: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:40.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:40 smithi037 ceph-mon[31460]: pgmap v1257: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:49:40.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:40 smithi059 ceph-mon[36190]: pgmap v1256: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:40.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:40 smithi059 ceph-mon[36190]: pgmap v1257: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:49:40.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:40 smithi170 ceph-mon[36280]: pgmap v1256: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:40.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:40 smithi170 ceph-mon[36280]: pgmap v1257: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:49:41.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1376.123562 secs 2024-03-28T18:49:41.437 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1376.123562 secs 2024-03-28T18:49:41.437 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1376.123562 secs 2024-03-28T18:49:42.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:42 smithi037 ceph-mon[31460]: pgmap v1258: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:49:42.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:42 smithi059 ceph-mon[36190]: pgmap v1258: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:49:42.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:42 smithi170 ceph-mon[36280]: pgmap v1258: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:49:44.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:44 smithi059 ceph-mon[36190]: pgmap v1259: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:49:44.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:44 smithi170 ceph-mon[36280]: pgmap v1259: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:49:44.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:44 smithi037 ceph-mon[31460]: pgmap v1259: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:49:46.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:46 smithi059 ceph-mon[36190]: pgmap v1260: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:49:46.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1381.123683 secs 2024-03-28T18:49:46.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:46 smithi170 ceph-mon[36280]: pgmap v1260: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:49:46.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1381.123683 secs 2024-03-28T18:49:46.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:46 smithi037 ceph-mon[31460]: pgmap v1260: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:49:46.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1381.123683 secs 2024-03-28T18:49:48.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:48 smithi059 ceph-mon[36190]: pgmap v1261: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:49:48.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:48 smithi170 ceph-mon[36280]: pgmap v1261: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:49:48.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:48 smithi037 ceph-mon[31460]: pgmap v1261: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:49:50.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:50 smithi059 ceph-mon[36190]: pgmap v1262: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:49:50.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:50 smithi170 ceph-mon[36280]: pgmap v1262: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:49:50.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:50 smithi037 ceph-mon[31460]: pgmap v1262: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:49:51.526 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1386.123805 secs 2024-03-28T18:49:51.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1386.123805 secs 2024-03-28T18:49:51.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1386.123805 secs 2024-03-28T18:49:52.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:52 smithi059 ceph-mon[36190]: pgmap v1263: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:52.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:52 smithi170 ceph-mon[36280]: pgmap v1263: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:52.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:52 smithi037 ceph-mon[31460]: pgmap v1263: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:54.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:54 smithi059 ceph-mon[36190]: pgmap v1264: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:54.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:54 smithi170 ceph-mon[36280]: pgmap v1264: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:54.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:54 smithi037 ceph-mon[31460]: pgmap v1264: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:56 smithi059 ceph-mon[36190]: pgmap v1265: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:56.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1391.123886 secs 2024-03-28T18:49:56.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:56 smithi170 ceph-mon[36280]: pgmap v1265: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:56.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1391.123886 secs 2024-03-28T18:49:56.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:56 smithi037 ceph-mon[31460]: pgmap v1265: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:56.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1391.123886 secs 2024-03-28T18:49:58.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:49:58 smithi059 ceph-mon[36190]: pgmap v1266: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:49:58.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:49:58 smithi170 ceph-mon[36280]: pgmap v1266: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:49:58.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:49:58 smithi037 ceph-mon[31460]: pgmap v1266: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:00.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:00 smithi059 ceph-mon[36190]: pgmap v1267: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:50:00.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:00 smithi059 ceph-mon[36190]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T18:50:00.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:00 smithi170 ceph-mon[36280]: pgmap v1267: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:50:00.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:00 smithi170 ceph-mon[36280]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T18:50:00.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:00 smithi037 ceph-mon[31460]: pgmap v1267: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:50:00.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:00 smithi037 ceph-mon[31460]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T18:50:01.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1396.124000 secs 2024-03-28T18:50:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1396.124000 secs 2024-03-28T18:50:01.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1396.124000 secs 2024-03-28T18:50:02.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:02 smithi059 ceph-mon[36190]: pgmap v1268: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:02.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:02 smithi170 ceph-mon[36280]: pgmap v1268: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:02.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:02 smithi037 ceph-mon[31460]: pgmap v1268: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:04.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:04 smithi059 ceph-mon[36190]: pgmap v1269: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:04.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:04 smithi170 ceph-mon[36280]: pgmap v1269: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:04.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:04 smithi037 ceph-mon[31460]: pgmap v1269: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:06.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:06 smithi059 ceph-mon[36190]: pgmap v1270: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:06.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1401.124119 secs 2024-03-28T18:50:06.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:06 smithi170 ceph-mon[36280]: pgmap v1270: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:06.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1401.124119 secs 2024-03-28T18:50:06.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:06 smithi037 ceph-mon[31460]: pgmap v1270: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:06.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1401.124119 secs 2024-03-28T18:50:08.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:08 smithi059 ceph-mon[36190]: pgmap v1271: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:08.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:08 smithi170 ceph-mon[36280]: pgmap v1271: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:08.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:08 smithi037 ceph-mon[31460]: pgmap v1271: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:10.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:10 smithi059 ceph-mon[36190]: pgmap v1272: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:50:10.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:10 smithi170 ceph-mon[36280]: pgmap v1272: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:50:10.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:10 smithi037 ceph-mon[31460]: pgmap v1272: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:50:11.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1406.124204 secs 2024-03-28T18:50:11.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1406.124204 secs 2024-03-28T18:50:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1406.124204 secs 2024-03-28T18:50:12.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:12 smithi059 ceph-mon[36190]: pgmap v1273: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:12.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:12 smithi170 ceph-mon[36280]: pgmap v1273: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:12.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:12 smithi037 ceph-mon[31460]: pgmap v1273: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:13.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:13 smithi170 ceph-mon[36280]: pgmap v1274: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:13.539 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:13 smithi059 ceph-mon[36190]: pgmap v1274: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:13.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:13 smithi037 ceph-mon[31460]: pgmap v1274: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:15.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:15 smithi059 ceph-mon[36190]: pgmap v1275: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:15.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:15 smithi170 ceph-mon[36280]: pgmap v1275: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:15.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:15 smithi037 ceph-mon[31460]: pgmap v1275: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:16.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1411.124326 secs 2024-03-28T18:50:16.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1411.124326 secs 2024-03-28T18:50:16.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1411.124326 secs 2024-03-28T18:50:17.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:17 smithi059 ceph-mon[36190]: pgmap v1276: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:17.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:17 smithi170 ceph-mon[36280]: pgmap v1276: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:17.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:17 smithi037 ceph-mon[31460]: pgmap v1276: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:19.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:19 smithi059 ceph-mon[36190]: pgmap v1277: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:19.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:19 smithi170 ceph-mon[36280]: pgmap v1277: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:19.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:19 smithi037 ceph-mon[31460]: pgmap v1277: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:21.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:21 smithi037 ceph-mon[31460]: pgmap v1278: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:21.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1416.124443 secs 2024-03-28T18:50:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:21 smithi059 ceph-mon[36190]: pgmap v1278: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:22.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1416.124443 secs 2024-03-28T18:50:22.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:21 smithi170 ceph-mon[36280]: pgmap v1278: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:22.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1416.124443 secs 2024-03-28T18:50:23.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:23 smithi037 ceph-mon[31460]: pgmap v1279: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:24.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:23 smithi059 ceph-mon[36190]: pgmap v1279: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:24.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:23 smithi170 ceph-mon[36280]: pgmap v1279: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:25.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:25 smithi037 ceph-mon[31460]: pgmap v1280: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:26.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:25 smithi059 ceph-mon[36190]: pgmap v1280: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:26.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:25 smithi170 ceph-mon[36280]: pgmap v1280: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:26.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1421.124563 secs 2024-03-28T18:50:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1421.124563 secs 2024-03-28T18:50:27.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1421.124563 secs 2024-03-28T18:50:27.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:27 smithi037 ceph-mon[31460]: pgmap v1281: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:28.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:27 smithi059 ceph-mon[36190]: pgmap v1281: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:28.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:27 smithi170 ceph-mon[36280]: pgmap v1281: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:29.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:29 smithi037 ceph-mon[31460]: pgmap v1282: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:30.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:29 smithi059 ceph-mon[36190]: pgmap v1282: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:30.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:29 smithi170 ceph-mon[36280]: pgmap v1282: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:31.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:31 smithi037 ceph-mon[31460]: pgmap v1283: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:31.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1426.124684 secs 2024-03-28T18:50:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:31 smithi059 ceph-mon[36190]: pgmap v1283: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:32.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1426.124684 secs 2024-03-28T18:50:32.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:31 smithi170 ceph-mon[36280]: pgmap v1283: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:32.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1426.124684 secs 2024-03-28T18:50:33.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:33 smithi037 ceph-mon[31460]: pgmap v1284: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:34.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:33 smithi059 ceph-mon[36190]: pgmap v1284: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:33 smithi170 ceph-mon[36280]: pgmap v1284: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:35.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:35 smithi037 ceph-mon[31460]: pgmap v1285: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:36.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:35 smithi059 ceph-mon[36190]: pgmap v1285: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:36.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:35 smithi170 ceph-mon[36280]: pgmap v1285: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:36.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1431.124807 secs 2024-03-28T18:50:37.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1431.124807 secs 2024-03-28T18:50:37.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1431.124807 secs 2024-03-28T18:50:37.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:37 smithi037 ceph-mon[31460]: pgmap v1286: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:38.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:37 smithi059 ceph-mon[36190]: pgmap v1286: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:38.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:37 smithi170 ceph-mon[36280]: pgmap v1286: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:39.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:39 smithi037 ceph-mon[31460]: pgmap v1287: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:39.878 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:39 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:50:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:39 smithi059 ceph-mon[36190]: pgmap v1287: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:40.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:39 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:50:40.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:39 smithi170 ceph-mon[36280]: pgmap v1287: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:40.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:39 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:50:40.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:40 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:50:40.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:40 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:50:40.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:40 smithi037 ceph-mon[31460]: pgmap v1288: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:40.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:40 smithi037 ceph-mon[31460]: pgmap v1289: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:40.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:40 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:50:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:40 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:50:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:40 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:50:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:40 smithi059 ceph-mon[36190]: pgmap v1288: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:40 smithi059 ceph-mon[36190]: pgmap v1289: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:41.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:40 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:50:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:40 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:50:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:40 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:50:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:40 smithi170 ceph-mon[36280]: pgmap v1288: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:40 smithi170 ceph-mon[36280]: pgmap v1289: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:41.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:40 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:50:41.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1436.124923 secs 2024-03-28T18:50:41.936 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1436.124923 secs 2024-03-28T18:50:41.937 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1436.124923 secs 2024-03-28T18:50:42.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:42 smithi037 ceph-mon[31460]: pgmap v1290: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:42.950 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:42 smithi059 ceph-mon[36190]: pgmap v1290: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:42.950 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:42 smithi170 ceph-mon[36280]: pgmap v1290: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:45.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:44 smithi059 ceph-mon[36190]: pgmap v1291: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:45.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:44 smithi170 ceph-mon[36280]: pgmap v1291: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:45.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:44 smithi037 ceph-mon[31460]: pgmap v1291: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:46 smithi059 ceph-mon[36190]: 2 slow requests, 1 included below; oldest blocked for > 1441.125067 secs 2024-03-28T18:50:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:46 smithi059 ceph-mon[36190]: slow request 1441.125066 seconds old, received at 2024-03-28T18:26:44.515856+0000: peer_request:client.25216:13687 currently acquired locks 2024-03-28T18:50:46.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:46 smithi170 ceph-mon[36280]: 2 slow requests, 1 included below; oldest blocked for > 1441.125067 secs 2024-03-28T18:50:46.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:46 smithi170 ceph-mon[36280]: slow request 1441.125066 seconds old, received at 2024-03-28T18:26:44.515856+0000: peer_request:client.25216:13687 currently acquired locks 2024-03-28T18:50:46.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:46 smithi037 ceph-mon[31460]: 2 slow requests, 1 included below; oldest blocked for > 1441.125067 secs 2024-03-28T18:50:46.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:46 smithi037 ceph-mon[31460]: slow request 1441.125066 seconds old, received at 2024-03-28T18:26:44.515856+0000: peer_request:client.25216:13687 currently acquired locks 2024-03-28T18:50:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:47 smithi059 ceph-mon[36190]: pgmap v1292: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:50:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:47 smithi170 ceph-mon[36280]: pgmap v1292: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:50:47.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:47 smithi037 ceph-mon[31460]: pgmap v1292: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:50:49.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:49 smithi059 ceph-mon[36190]: pgmap v1293: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:49.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:49 smithi170 ceph-mon[36280]: pgmap v1293: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:49.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:49 smithi037 ceph-mon[31460]: pgmap v1293: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:51 smithi059 ceph-mon[36190]: pgmap v1294: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:50:51.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1446.125187 secs 2024-03-28T18:50:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:51 smithi170 ceph-mon[36280]: pgmap v1294: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:50:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1446.125187 secs 2024-03-28T18:50:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:51 smithi037 ceph-mon[31460]: pgmap v1294: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:50:51.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1446.125187 secs 2024-03-28T18:50:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:53 smithi037 ceph-mon[31460]: pgmap v1295: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:53 smithi059 ceph-mon[36190]: pgmap v1295: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:53 smithi170 ceph-mon[36280]: pgmap v1295: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:55 smithi059 ceph-mon[36190]: pgmap v1296: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:55.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:55 smithi170 ceph-mon[36280]: pgmap v1296: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:55.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:55 smithi037 ceph-mon[31460]: pgmap v1296: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:56.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1451.125306 secs 2024-03-28T18:50:56.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1451.125306 secs 2024-03-28T18:50:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1451.125306 secs 2024-03-28T18:50:57.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:57 smithi037 ceph-mon[31460]: pgmap v1297: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:57.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:57 smithi059 ceph-mon[36190]: pgmap v1297: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:57.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:57 smithi170 ceph-mon[36280]: pgmap v1297: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:59.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:50:59 smithi037 ceph-mon[31460]: pgmap v1298: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:59.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:50:59 smithi059 ceph-mon[36190]: pgmap v1298: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:59.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:50:59 smithi170 ceph-mon[36280]: pgmap v1298: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:01 smithi037 ceph-mon[31460]: pgmap v1299: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:01 smithi037 ceph-mon[31460]: 2 slow requests, 1 included below; oldest blocked for > 1456.125411 secs 2024-03-28T18:51:01.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:01 smithi037 ceph-mon[31460]: slow request 1440.545085 seconds old, received at 2024-03-28T18:27:00.096181+0000: client_request(client.25553:180 mksnap #0x1//scheduled-2024-03-28-18_27_00_UTC 2024-03-28T18:27:00.095960+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:51:01.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:01 smithi059 ceph-mon[36190]: pgmap v1299: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:01.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:01 smithi059 ceph-mon[36190]: 2 slow requests, 1 included below; oldest blocked for > 1456.125411 secs 2024-03-28T18:51:01.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:01 smithi059 ceph-mon[36190]: slow request 1440.545085 seconds old, received at 2024-03-28T18:27:00.096181+0000: client_request(client.25553:180 mksnap #0x1//scheduled-2024-03-28-18_27_00_UTC 2024-03-28T18:27:00.095960+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:51:01.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:01 smithi170 ceph-mon[36280]: pgmap v1299: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:01.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:01 smithi170 ceph-mon[36280]: 2 slow requests, 1 included below; oldest blocked for > 1456.125411 secs 2024-03-28T18:51:01.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:01 smithi170 ceph-mon[36280]: slow request 1440.545085 seconds old, received at 2024-03-28T18:27:00.096181+0000: client_request(client.25553:180 mksnap #0x1//scheduled-2024-03-28-18_27_00_UTC 2024-03-28T18:27:00.095960+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:51:03.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:03 smithi037 ceph-mon[31460]: pgmap v1300: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:03 smithi059 ceph-mon[36190]: pgmap v1300: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:03.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:03 smithi170 ceph-mon[36280]: pgmap v1300: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:05.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:05 smithi037 ceph-mon[31460]: pgmap v1301: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:05.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:05 smithi059 ceph-mon[36190]: pgmap v1301: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:05.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:05 smithi170 ceph-mon[36280]: pgmap v1301: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1461.125512 secs 2024-03-28T18:51:06.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1461.125512 secs 2024-03-28T18:51:06.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1461.125512 secs 2024-03-28T18:51:07.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:07 smithi037 ceph-mon[31460]: pgmap v1302: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:07.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:07 smithi059 ceph-mon[36190]: pgmap v1302: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:07.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:07 smithi170 ceph-mon[36280]: pgmap v1302: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:09.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:09 smithi037 ceph-mon[31460]: pgmap v1303: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:09.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:09 smithi059 ceph-mon[36190]: pgmap v1303: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:09.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:09 smithi170 ceph-mon[36280]: pgmap v1303: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:11.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:10 smithi037 ceph-mon[31460]: pgmap v1304: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:10 smithi059 ceph-mon[36190]: pgmap v1304: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:11.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:10 smithi170 ceph-mon[36280]: pgmap v1304: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1466.125606 secs 2024-03-28T18:51:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1466.125606 secs 2024-03-28T18:51:12.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1466.125606 secs 2024-03-28T18:51:13.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:12 smithi037 ceph-mon[31460]: pgmap v1305: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:12 smithi059 ceph-mon[36190]: pgmap v1305: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:13.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:12 smithi170 ceph-mon[36280]: pgmap v1305: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:15.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:15 smithi059 ceph-mon[36190]: pgmap v1306: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:15.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:15 smithi170 ceph-mon[36280]: pgmap v1306: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:15.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:15 smithi037 ceph-mon[31460]: pgmap v1306: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:16.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1471.125727 secs 2024-03-28T18:51:16.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1471.125727 secs 2024-03-28T18:51:16.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1471.125727 secs 2024-03-28T18:51:17.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:17 smithi059 ceph-mon[36190]: pgmap v1307: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:17.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:17 smithi170 ceph-mon[36280]: pgmap v1307: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:17.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:17 smithi037 ceph-mon[31460]: pgmap v1307: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:19.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:19 smithi059 ceph-mon[36190]: pgmap v1308: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:19.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:19 smithi170 ceph-mon[36280]: pgmap v1308: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:19.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:19 smithi037 ceph-mon[31460]: pgmap v1308: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:21 smithi059 ceph-mon[36190]: pgmap v1309: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1476.125821 secs 2024-03-28T18:51:21.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:21 smithi170 ceph-mon[36280]: pgmap v1309: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:21.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1476.125821 secs 2024-03-28T18:51:21.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:21 smithi037 ceph-mon[31460]: pgmap v1309: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:21.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1476.125821 secs 2024-03-28T18:51:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:23 smithi059 ceph-mon[36190]: pgmap v1310: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:23 smithi170 ceph-mon[36280]: pgmap v1310: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:23.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:23 smithi037 ceph-mon[31460]: pgmap v1310: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:25.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:25 smithi037 ceph-mon[31460]: pgmap v1311: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:25.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:25 smithi059 ceph-mon[36190]: pgmap v1311: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:25.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:25 smithi170 ceph-mon[36280]: pgmap v1311: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1481.125938 secs 2024-03-28T18:51:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1481.125938 secs 2024-03-28T18:51:26.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1481.125938 secs 2024-03-28T18:51:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:27 smithi037 ceph-mon[31460]: pgmap v1312: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:27.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:27 smithi059 ceph-mon[36190]: pgmap v1312: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:27.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:27 smithi170 ceph-mon[36280]: pgmap v1312: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:29.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:29 smithi037 ceph-mon[31460]: pgmap v1313: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:29 smithi059 ceph-mon[36190]: pgmap v1313: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:29.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:29 smithi170 ceph-mon[36280]: pgmap v1313: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:31.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:31 smithi037 ceph-mon[31460]: pgmap v1314: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:31.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1486.126017 secs 2024-03-28T18:51:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:31 smithi059 ceph-mon[36190]: pgmap v1314: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1486.126017 secs 2024-03-28T18:51:31.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:31 smithi170 ceph-mon[36280]: pgmap v1314: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:31.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1486.126017 secs 2024-03-28T18:51:33.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:33 smithi037 ceph-mon[31460]: pgmap v1315: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:33.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:33 smithi059 ceph-mon[36190]: pgmap v1315: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:33.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:33 smithi170 ceph-mon[36280]: pgmap v1315: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:35.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:35 smithi037 ceph-mon[31460]: pgmap v1316: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:35.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:35 smithi059 ceph-mon[36190]: pgmap v1316: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:35.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:35 smithi170 ceph-mon[36280]: pgmap v1316: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:36.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1491.126136 secs 2024-03-28T18:51:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1491.126136 secs 2024-03-28T18:51:36.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1491.126136 secs 2024-03-28T18:51:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:37 smithi037 ceph-mon[31460]: pgmap v1317: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:37 smithi059 ceph-mon[36190]: pgmap v1317: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:37.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:37 smithi170 ceph-mon[36280]: pgmap v1317: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:39.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:39 smithi037 ceph-mon[31460]: pgmap v1318: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:39.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:39 smithi059 ceph-mon[36190]: pgmap v1318: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:39.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:39 smithi170 ceph-mon[36280]: pgmap v1318: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:41.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:41 smithi170 ceph-mon[36280]: pgmap v1319: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:41.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:41 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:51:41.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1496.126256 secs 2024-03-28T18:51:41.338 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:41 smithi059 ceph-mon[36190]: pgmap v1319: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:41.338 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:41 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:51:41.338 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1496.126256 secs 2024-03-28T18:51:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:41 smithi037 ceph-mon[31460]: pgmap v1319: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:41 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:51:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1496.126256 secs 2024-03-28T18:51:42.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:42 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:51:42.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:42 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:51:42.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:42 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:51:42.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:42 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:51:42.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:42 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:51:42.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:42 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:51:42.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:42 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:51:42.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:42 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:51:42.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:42 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:51:43.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:43 smithi037 ceph-mon[31460]: pgmap v1320: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:43.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:43 smithi037 ceph-mon[31460]: pgmap v1321: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:51:43.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:43 smithi059 ceph-mon[36190]: pgmap v1320: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:43.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:43 smithi059 ceph-mon[36190]: pgmap v1321: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:51:43.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:43 smithi170 ceph-mon[36280]: pgmap v1320: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:43.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:43 smithi170 ceph-mon[36280]: pgmap v1321: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:51:45.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:45 smithi037 ceph-mon[31460]: pgmap v1322: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:51:45.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:45 smithi059 ceph-mon[36190]: pgmap v1322: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:51:45.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:45 smithi170 ceph-mon[36280]: pgmap v1322: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:51:46.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1501.126325 secs 2024-03-28T18:51:46.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1501.126325 secs 2024-03-28T18:51:46.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1501.126325 secs 2024-03-28T18:51:47.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:47 smithi037 ceph-mon[31460]: pgmap v1323: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:51:47.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:47 smithi059 ceph-mon[36190]: pgmap v1323: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:51:47.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:47 smithi170 ceph-mon[36280]: pgmap v1323: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:51:49.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:49 smithi037 ceph-mon[31460]: pgmap v1324: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:51:49.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:49 smithi059 ceph-mon[36190]: pgmap v1324: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:51:49.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:49 smithi170 ceph-mon[36280]: pgmap v1324: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:51:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:51 smithi037 ceph-mon[31460]: pgmap v1325: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:51:51.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1506.126395 secs 2024-03-28T18:51:51.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:51 smithi059 ceph-mon[36190]: pgmap v1325: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:51:51.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1506.126395 secs 2024-03-28T18:51:51.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:51 smithi170 ceph-mon[36280]: pgmap v1325: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:51:51.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1506.126395 secs 2024-03-28T18:51:53.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:53 smithi059 ceph-mon[36190]: pgmap v1326: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:51:53.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:53 smithi170 ceph-mon[36280]: pgmap v1326: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:51:53.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:53 smithi037 ceph-mon[31460]: pgmap v1326: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:51:55.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:55 smithi059 ceph-mon[36190]: pgmap v1327: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:55.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:55 smithi170 ceph-mon[36280]: pgmap v1327: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:55.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:55 smithi037 ceph-mon[31460]: pgmap v1327: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1511.126465 secs 2024-03-28T18:51:56.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1511.126465 secs 2024-03-28T18:51:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1511.126465 secs 2024-03-28T18:51:57.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:57 smithi059 ceph-mon[36190]: pgmap v1328: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:57.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:57 smithi170 ceph-mon[36280]: pgmap v1328: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:57.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:57 smithi037 ceph-mon[31460]: pgmap v1328: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:59.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:51:59 smithi059 ceph-mon[36190]: pgmap v1329: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:59.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:51:59 smithi170 ceph-mon[36280]: pgmap v1329: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:59.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:51:59 smithi037 ceph-mon[31460]: pgmap v1329: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:01.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:01 smithi059 ceph-mon[36190]: pgmap v1330: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:01.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1516.126581 secs 2024-03-28T18:52:01.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:01 smithi170 ceph-mon[36280]: pgmap v1330: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:01.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1516.126581 secs 2024-03-28T18:52:01.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:01 smithi037 ceph-mon[31460]: pgmap v1330: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:01.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1516.126581 secs 2024-03-28T18:52:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:03 smithi059 ceph-mon[36190]: pgmap v1331: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:03.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:03 smithi170 ceph-mon[36280]: pgmap v1331: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:03.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:03 smithi037 ceph-mon[31460]: pgmap v1331: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:05.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:05 smithi059 ceph-mon[36190]: pgmap v1332: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:05.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:05 smithi170 ceph-mon[36280]: pgmap v1332: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:05.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:05 smithi037 ceph-mon[31460]: pgmap v1332: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:06.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1521.126702 secs 2024-03-28T18:52:06.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1521.126702 secs 2024-03-28T18:52:06.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1521.126702 secs 2024-03-28T18:52:07.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:07 smithi059 ceph-mon[36190]: pgmap v1333: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:07.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:07 smithi170 ceph-mon[36280]: pgmap v1333: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:07.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:07 smithi037 ceph-mon[31460]: pgmap v1333: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:09.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:09 smithi059 ceph-mon[36190]: pgmap v1334: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:09.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:09 smithi170 ceph-mon[36280]: pgmap v1334: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:09.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:09 smithi037 ceph-mon[31460]: pgmap v1334: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:11.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:11 smithi059 ceph-mon[36190]: pgmap v1335: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:11.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1526.126811 secs 2024-03-28T18:52:11.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:11 smithi170 ceph-mon[36280]: pgmap v1335: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:11.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1526.126811 secs 2024-03-28T18:52:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:11 smithi037 ceph-mon[31460]: pgmap v1335: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1526.126811 secs 2024-03-28T18:52:13.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:13 smithi059 ceph-mon[36190]: pgmap v1336: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:13.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:13 smithi170 ceph-mon[36280]: pgmap v1336: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:13.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:13 smithi037 ceph-mon[31460]: pgmap v1336: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:15.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:15 smithi059 ceph-mon[36190]: pgmap v1337: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:15.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:15 smithi170 ceph-mon[36280]: pgmap v1337: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:15.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:15 smithi037 ceph-mon[31460]: pgmap v1337: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:16.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1531.126929 secs 2024-03-28T18:52:16.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1531.126929 secs 2024-03-28T18:52:16.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1531.126929 secs 2024-03-28T18:52:17.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:17 smithi059 ceph-mon[36190]: pgmap v1338: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:17.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:17 smithi170 ceph-mon[36280]: pgmap v1338: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:17.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:17 smithi037 ceph-mon[31460]: pgmap v1338: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:19.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:19 smithi059 ceph-mon[36190]: pgmap v1339: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:19.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:19 smithi170 ceph-mon[36280]: pgmap v1339: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:19.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:19 smithi037 ceph-mon[31460]: pgmap v1339: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:21 smithi059 ceph-mon[36190]: pgmap v1340: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1536.127048 secs 2024-03-28T18:52:21.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:21 smithi170 ceph-mon[36280]: pgmap v1340: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:21.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1536.127048 secs 2024-03-28T18:52:21.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:21 smithi037 ceph-mon[31460]: pgmap v1340: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:21.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1536.127048 secs 2024-03-28T18:52:23.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:23 smithi059 ceph-mon[36190]: pgmap v1341: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:23.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:23 smithi170 ceph-mon[36280]: pgmap v1341: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:23.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:23 smithi037 ceph-mon[31460]: pgmap v1341: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:25.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:25 smithi059 ceph-mon[36190]: pgmap v1342: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:25.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:25 smithi170 ceph-mon[36280]: pgmap v1342: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:25.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:25 smithi037 ceph-mon[31460]: pgmap v1342: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1541.127170 secs 2024-03-28T18:52:26.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1541.127170 secs 2024-03-28T18:52:26.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1541.127170 secs 2024-03-28T18:52:27.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:27 smithi059 ceph-mon[36190]: pgmap v1343: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:27.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:27 smithi170 ceph-mon[36280]: pgmap v1343: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:27.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:27 smithi037 ceph-mon[31460]: pgmap v1343: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:29.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:29 smithi059 ceph-mon[36190]: pgmap v1344: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:29.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:29 smithi170 ceph-mon[36280]: pgmap v1344: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:29.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:29 smithi037 ceph-mon[31460]: pgmap v1344: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:31 smithi059 ceph-mon[36190]: pgmap v1345: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:31.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1546.127258 secs 2024-03-28T18:52:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:31 smithi170 ceph-mon[36280]: pgmap v1345: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1546.127258 secs 2024-03-28T18:52:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:31 smithi037 ceph-mon[31460]: pgmap v1345: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:31.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1546.127258 secs 2024-03-28T18:52:32.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:32 smithi037 ceph-mon[31460]: pgmap v1346: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:32.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:32 smithi059 ceph-mon[36190]: pgmap v1346: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:32.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:32 smithi170 ceph-mon[36280]: pgmap v1346: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:34.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:34 smithi059 ceph-mon[36190]: pgmap v1347: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:34.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:34 smithi170 ceph-mon[36280]: pgmap v1347: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:34.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:34 smithi037 ceph-mon[31460]: pgmap v1347: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:36.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:36 smithi059 ceph-mon[36190]: pgmap v1348: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:36.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1551.127343 secs 2024-03-28T18:52:36.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:36 smithi170 ceph-mon[36280]: pgmap v1348: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:36.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1551.127343 secs 2024-03-28T18:52:36.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:36 smithi037 ceph-mon[31460]: pgmap v1348: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:36.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1551.127343 secs 2024-03-28T18:52:38.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:38 smithi059 ceph-mon[36190]: pgmap v1349: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:38.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:38 smithi170 ceph-mon[36280]: pgmap v1349: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:38.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:38 smithi037 ceph-mon[31460]: pgmap v1349: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:40.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:40 smithi059 ceph-mon[36190]: pgmap v1350: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:40.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:40 smithi170 ceph-mon[36280]: pgmap v1350: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:40.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:40 smithi037 ceph-mon[31460]: pgmap v1350: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:41.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1556.127460 secs 2024-03-28T18:52:41.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1556.127460 secs 2024-03-28T18:52:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1556.127460 secs 2024-03-28T18:52:42.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:42 smithi059 ceph-mon[36190]: pgmap v1351: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:42.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:42 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:52:42.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:42 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:42.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:42 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:42.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:42 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:42.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:42 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:42.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:42 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:42.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:42 smithi170 ceph-mon[36280]: pgmap v1351: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:42.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:42 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:52:42.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:42 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:42.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:42 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:42.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:42 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:42.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:42 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:42.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:42 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:42.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:42 smithi037 ceph-mon[31460]: pgmap v1351: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:42.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:42 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:52:42.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:42 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:42.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:42 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:42.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:42 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:42.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:42 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:42.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:42 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:43.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:43 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:43.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:43 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:43.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:43 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:44.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:44 smithi059 ceph-mon[36190]: pgmap v1352: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:44.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:44 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:44.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:44 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:44.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:44 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:44.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:44 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:44.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:44 smithi170 ceph-mon[36280]: pgmap v1352: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:44.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:44 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:44.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:44 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:44.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:44 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:44.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:44 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:44.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:44 smithi037 ceph-mon[31460]: pgmap v1352: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:44.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:44 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:44.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:44 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:44.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:44 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:44.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:44 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:45.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:45 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:45 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:45 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:45 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:45 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:45 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:45 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:45 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:45 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:45 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:45 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T18:52:45.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:45 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:45 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:45 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T18:52:45.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:45 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:45 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:45 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:45 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:45 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T18:52:45.788 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:45 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T18:52:45.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:45 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:45 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:45 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:45 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:45 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:45 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:45 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:45 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:45 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:45 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:45 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T18:52:45.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:45 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:45 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:45 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T18:52:45.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:45 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.790 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:45 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.790 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:45 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.790 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:45 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.790 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:45 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T18:52:45.790 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:45 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T18:52:45.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:45 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:45 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:45 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:45 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:45 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:45 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:45 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:45 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:45 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:45 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:45 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T18:52:45.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:45 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:45 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:45 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T18:52:45.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:45 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:45 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:45 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:45 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:45.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:45 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T18:52:45.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:45 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T18:52:46.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:46 smithi037 ceph-mon[31460]: pgmap v1353: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:46.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1561.127558 secs 2024-03-28T18:52:46.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:46 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:46.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:46 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:47.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:46 smithi059 ceph-mon[36190]: pgmap v1353: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:47.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1561.127558 secs 2024-03-28T18:52:47.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:46 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:47.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:46 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:47.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:46 smithi170 ceph-mon[36280]: pgmap v1353: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:47.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1561.127558 secs 2024-03-28T18:52:47.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:46 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:47.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:46 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:48.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:48 smithi037 ceph-mon[31460]: pgmap v1354: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:48.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:48 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:48.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:48 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:48.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:48 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:48.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:48 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:48.875 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:48 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:48.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:48 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:48.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:48 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:48.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:48 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:48.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:48 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T18:52:48.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:48 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:52:48.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:48 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:52:48.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:48 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:52:48.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:48 smithi037 ceph-mon[31460]: pgmap v1355: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:52:48.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:48 smithi037 ceph-mon[31460]: pgmap v1356: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:52:48.876 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:48 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:49.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:48 smithi059 ceph-mon[36190]: pgmap v1354: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:49.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:48 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:49.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:48 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:49.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:48 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:49.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:48 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:49.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:48 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:49.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:48 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:49.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:48 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:49.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:48 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:49.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:48 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T18:52:49.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:48 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:52:49.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:48 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:52:49.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:48 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:52:49.036 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:48 smithi059 ceph-mon[36190]: pgmap v1355: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:52:49.037 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:48 smithi059 ceph-mon[36190]: pgmap v1356: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:52:49.037 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:48 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:49.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:48 smithi170 ceph-mon[36280]: pgmap v1354: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:49.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:48 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:49.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:48 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:49.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:48 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:49.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:48 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:49.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:48 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:49.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:48 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:49.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:48 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:49.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:48 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:52:49.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:48 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T18:52:49.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:48 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:52:49.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:48 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:52:49.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:48 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:52:49.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:48 smithi170 ceph-mon[36280]: pgmap v1355: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:52:49.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:48 smithi170 ceph-mon[36280]: pgmap v1356: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:52:49.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:48 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:52:51.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:50 smithi059 ceph-mon[36190]: pgmap v1357: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:52:51.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:50 smithi170 ceph-mon[36280]: pgmap v1357: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:52:51.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:50 smithi037 ceph-mon[31460]: pgmap v1357: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:52:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1566.127676 secs 2024-03-28T18:52:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1566.127676 secs 2024-03-28T18:52:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1566.127676 secs 2024-03-28T18:52:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:52 smithi037 ceph-mon[31460]: pgmap v1358: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:52:53.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:52 smithi059 ceph-mon[36190]: pgmap v1358: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:52:53.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:52 smithi170 ceph-mon[36280]: pgmap v1358: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:52:55.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:54 smithi037 ceph-mon[31460]: pgmap v1359: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:52:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:54 smithi059 ceph-mon[36190]: pgmap v1359: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:52:55.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:54 smithi170 ceph-mon[36280]: pgmap v1359: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:52:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1571.127796 secs 2024-03-28T18:52:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:56 smithi037 ceph-mon[31460]: pgmap v1360: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:52:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1571.127796 secs 2024-03-28T18:52:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:56 smithi059 ceph-mon[36190]: pgmap v1360: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:52:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1571.127796 secs 2024-03-28T18:52:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:56 smithi170 ceph-mon[36280]: pgmap v1360: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:52:59.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:52:58 smithi037 ceph-mon[31460]: pgmap v1361: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:52:59.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:52:58 smithi059 ceph-mon[36190]: pgmap v1361: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:52:59.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:52:58 smithi170 ceph-mon[36280]: pgmap v1361: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:53:01.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:00 smithi037 ceph-mon[31460]: pgmap v1362: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:01.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:00 smithi059 ceph-mon[36190]: pgmap v1362: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:00 smithi170 ceph-mon[36280]: pgmap v1362: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:02.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1576.127915 secs 2024-03-28T18:53:02.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1576.127915 secs 2024-03-28T18:53:02.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1576.127915 secs 2024-03-28T18:53:03.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:02 smithi037 ceph-mon[31460]: pgmap v1363: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:03.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:02 smithi059 ceph-mon[36190]: pgmap v1363: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:03.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:02 smithi170 ceph-mon[36280]: pgmap v1363: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:05.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:04 smithi037 ceph-mon[31460]: pgmap v1364: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:05.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:04 smithi059 ceph-mon[36190]: pgmap v1364: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:05.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:04 smithi170 ceph-mon[36280]: pgmap v1364: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1581.128061 secs 2024-03-28T18:53:07.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:06 smithi037 ceph-mon[31460]: pgmap v1365: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1581.128061 secs 2024-03-28T18:53:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:06 smithi059 ceph-mon[36190]: pgmap v1365: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:07.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1581.128061 secs 2024-03-28T18:53:07.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:06 smithi170 ceph-mon[36280]: pgmap v1365: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:09.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:08 smithi037 ceph-mon[31460]: pgmap v1366: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:09.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:08 smithi059 ceph-mon[36190]: pgmap v1366: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:09.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:08 smithi170 ceph-mon[36280]: pgmap v1366: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:11.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:10 smithi037 ceph-mon[31460]: pgmap v1367: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:10 smithi059 ceph-mon[36190]: pgmap v1367: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:11.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:10 smithi170 ceph-mon[36280]: pgmap v1367: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1586.128182 secs 2024-03-28T18:53:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1586.128182 secs 2024-03-28T18:53:12.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1586.128182 secs 2024-03-28T18:53:13.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:12 smithi037 ceph-mon[31460]: pgmap v1368: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:13.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:12 smithi059 ceph-mon[36190]: pgmap v1368: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:13.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:12 smithi170 ceph-mon[36280]: pgmap v1368: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:15.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:14 smithi037 ceph-mon[31460]: pgmap v1369: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:15.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:14 smithi059 ceph-mon[36190]: pgmap v1369: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:15.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:14 smithi170 ceph-mon[36280]: pgmap v1369: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:17.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1591.128304 secs 2024-03-28T18:53:17.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:16 smithi037 ceph-mon[31460]: pgmap v1370: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:17.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1591.128304 secs 2024-03-28T18:53:17.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:16 smithi059 ceph-mon[36190]: pgmap v1370: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1591.128304 secs 2024-03-28T18:53:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:16 smithi170 ceph-mon[36280]: pgmap v1370: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:19.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:18 smithi037 ceph-mon[31460]: pgmap v1371: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:19.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:18 smithi059 ceph-mon[36190]: pgmap v1371: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:19.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:18 smithi170 ceph-mon[36280]: pgmap v1371: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:21.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:20 smithi037 ceph-mon[31460]: pgmap v1372: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:21.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:20 smithi059 ceph-mon[36190]: pgmap v1372: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:21.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:20 smithi170 ceph-mon[36280]: pgmap v1372: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:22.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1596.128425 secs 2024-03-28T18:53:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1596.128425 secs 2024-03-28T18:53:22.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1596.128425 secs 2024-03-28T18:53:23.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:22 smithi059 ceph-mon[36190]: pgmap v1373: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:23.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:22 smithi170 ceph-mon[36280]: pgmap v1373: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:23.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:22 smithi037 ceph-mon[31460]: pgmap v1373: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:25.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:24 smithi059 ceph-mon[36190]: pgmap v1374: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:25.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:24 smithi170 ceph-mon[36280]: pgmap v1374: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:25.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:24 smithi037 ceph-mon[31460]: pgmap v1374: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1601.128546 secs 2024-03-28T18:53:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:26 smithi059 ceph-mon[36190]: pgmap v1375: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:27.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1601.128546 secs 2024-03-28T18:53:27.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:26 smithi170 ceph-mon[36280]: pgmap v1375: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1601.128546 secs 2024-03-28T18:53:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:26 smithi037 ceph-mon[31460]: pgmap v1375: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:29.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:28 smithi059 ceph-mon[36190]: pgmap v1376: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:29.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:28 smithi170 ceph-mon[36280]: pgmap v1376: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:29.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:28 smithi037 ceph-mon[31460]: pgmap v1376: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:31.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:30 smithi059 ceph-mon[36190]: pgmap v1377: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:31.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:30 smithi170 ceph-mon[36280]: pgmap v1377: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:31.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:30 smithi037 ceph-mon[31460]: pgmap v1377: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:32.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1606.128665 secs 2024-03-28T18:53:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1606.128665 secs 2024-03-28T18:53:32.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1606.128665 secs 2024-03-28T18:53:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:32 smithi059 ceph-mon[36190]: pgmap v1378: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:33.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:32 smithi170 ceph-mon[36280]: pgmap v1378: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:33.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:32 smithi037 ceph-mon[31460]: pgmap v1378: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:34 smithi059 ceph-mon[36190]: pgmap v1379: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:35.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:34 smithi170 ceph-mon[36280]: pgmap v1379: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:35.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:34 smithi037 ceph-mon[31460]: pgmap v1379: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1611.128786 secs 2024-03-28T18:53:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:36 smithi059 ceph-mon[36190]: pgmap v1380: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:37.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1611.128786 secs 2024-03-28T18:53:37.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:36 smithi170 ceph-mon[36280]: pgmap v1380: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1611.128786 secs 2024-03-28T18:53:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:36 smithi037 ceph-mon[31460]: pgmap v1380: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:39.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:38 smithi059 ceph-mon[36190]: pgmap v1381: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:39.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:38 smithi170 ceph-mon[36280]: pgmap v1381: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:39.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:38 smithi037 ceph-mon[31460]: pgmap v1381: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:40 smithi059 ceph-mon[36190]: pgmap v1382: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:41.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:40 smithi170 ceph-mon[36280]: pgmap v1382: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:40 smithi037 ceph-mon[31460]: pgmap v1382: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1616.128906 secs 2024-03-28T18:53:42.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1616.128906 secs 2024-03-28T18:53:42.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1616.128906 secs 2024-03-28T18:53:43.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:42 smithi059 ceph-mon[36190]: pgmap v1383: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:43.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:42 smithi170 ceph-mon[36280]: pgmap v1383: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:43.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:42 smithi037 ceph-mon[31460]: pgmap v1383: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:44 smithi059 ceph-mon[36190]: pgmap v1384: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:45.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:44 smithi170 ceph-mon[36280]: pgmap v1384: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:45.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:44 smithi037 ceph-mon[31460]: pgmap v1384: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1621.129053 secs 2024-03-28T18:53:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:46 smithi059 ceph-mon[36190]: pgmap v1385: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:47.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1621.129053 secs 2024-03-28T18:53:47.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:46 smithi170 ceph-mon[36280]: pgmap v1385: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:47.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1621.129053 secs 2024-03-28T18:53:47.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:46 smithi037 ceph-mon[31460]: pgmap v1385: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:49.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:48 smithi059 ceph-mon[36190]: pgmap v1386: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:49.287 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:48 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:53:49.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:48 smithi170 ceph-mon[36280]: pgmap v1386: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:49.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:48 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:53:49.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:48 smithi037 ceph-mon[31460]: pgmap v1386: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:49.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:48 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:53:50.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:49 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:53:50.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:49 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:53:50.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:49 smithi170 ceph-mon[36280]: pgmap v1387: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:50.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:49 smithi170 ceph-mon[36280]: pgmap v1388: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:53:50.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:49 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:53:50.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:49 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:53:50.287 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:49 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:53:50.287 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:49 smithi059 ceph-mon[36190]: pgmap v1387: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:50.287 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:49 smithi059 ceph-mon[36190]: pgmap v1388: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:53:50.287 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:49 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:53:50.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:49 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:53:50.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:49 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:53:50.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:49 smithi037 ceph-mon[31460]: pgmap v1387: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:50.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:49 smithi037 ceph-mon[31460]: pgmap v1388: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:53:50.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:49 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:53:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1626.129171 secs 2024-03-28T18:53:51.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1626.129171 secs 2024-03-28T18:53:51.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1626.129171 secs 2024-03-28T18:53:52.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:52 smithi037 ceph-mon[31460]: pgmap v1389: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:53:52.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:52 smithi059 ceph-mon[36190]: pgmap v1389: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:53:52.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:52 smithi170 ceph-mon[36280]: pgmap v1389: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:53:53.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:53 smithi037 ceph-mon[31460]: pgmap v1390: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:53:53.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:53 smithi059 ceph-mon[36190]: pgmap v1390: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:53:53.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:53 smithi170 ceph-mon[36280]: pgmap v1390: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:53:55.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:55 smithi037 ceph-mon[31460]: pgmap v1391: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:53:55.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:55 smithi059 ceph-mon[36190]: pgmap v1391: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:53:55.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:55 smithi170 ceph-mon[36280]: pgmap v1391: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:53:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1631.129293 secs 2024-03-28T18:53:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1631.129293 secs 2024-03-28T18:53:56.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1631.129293 secs 2024-03-28T18:53:57.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:57 smithi037 ceph-mon[31460]: pgmap v1392: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:53:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:57 smithi059 ceph-mon[36190]: pgmap v1392: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:53:57.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:57 smithi170 ceph-mon[36280]: pgmap v1392: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:53:59.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:53:59 smithi037 ceph-mon[31460]: pgmap v1393: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:53:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:53:59 smithi059 ceph-mon[36190]: pgmap v1393: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:53:59.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:53:59 smithi170 ceph-mon[36280]: pgmap v1393: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:54:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1636.129414 secs 2024-03-28T18:54:01.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1636.129414 secs 2024-03-28T18:54:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1636.129414 secs 2024-03-28T18:54:02.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:02 smithi037 ceph-mon[31460]: pgmap v1394: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:54:02.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:02 smithi059 ceph-mon[36190]: pgmap v1394: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:54:02.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:02 smithi170 ceph-mon[36280]: pgmap v1394: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:54:03.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:03 smithi037 ceph-mon[31460]: pgmap v1395: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:54:03.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:03 smithi059 ceph-mon[36190]: pgmap v1395: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:54:03.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:03 smithi170 ceph-mon[36280]: pgmap v1395: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:54:05.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:05 smithi037 ceph-mon[31460]: pgmap v1396: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:05.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:05 smithi059 ceph-mon[36190]: pgmap v1396: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:05.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:05 smithi170 ceph-mon[36280]: pgmap v1396: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:06.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1641.129546 secs 2024-03-28T18:54:06.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1641.129546 secs 2024-03-28T18:54:06.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1641.129546 secs 2024-03-28T18:54:07.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:07 smithi037 ceph-mon[31460]: pgmap v1397: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:07.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:07 smithi059 ceph-mon[36190]: pgmap v1397: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:07.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:07 smithi170 ceph-mon[36280]: pgmap v1397: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:09.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:09 smithi037 ceph-mon[31460]: pgmap v1398: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:09.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:09 smithi059 ceph-mon[36190]: pgmap v1398: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:09.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:09 smithi170 ceph-mon[36280]: pgmap v1398: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1646.129660 secs 2024-03-28T18:54:11.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1646.129660 secs 2024-03-28T18:54:11.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1646.129660 secs 2024-03-28T18:54:12.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:12 smithi170 ceph-mon[36280]: pgmap v1399: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:12.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:12 smithi037 ceph-mon[31460]: pgmap v1399: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:12.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:12 smithi059 ceph-mon[36190]: pgmap v1399: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:13.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:13 smithi037 ceph-mon[31460]: pgmap v1400: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:13.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:13 smithi059 ceph-mon[36190]: pgmap v1400: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:13.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:13 smithi170 ceph-mon[36280]: pgmap v1400: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:15.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:15 smithi037 ceph-mon[31460]: pgmap v1401: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:15.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:15 smithi059 ceph-mon[36190]: pgmap v1401: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:15.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:15 smithi170 ceph-mon[36280]: pgmap v1401: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:16.626 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1651.129781 secs 2024-03-28T18:54:16.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1651.129781 secs 2024-03-28T18:54:16.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1651.129781 secs 2024-03-28T18:54:17.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:17 smithi037 ceph-mon[31460]: pgmap v1402: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:17.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:17 smithi059 ceph-mon[36190]: pgmap v1402: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:17.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:17 smithi170 ceph-mon[36280]: pgmap v1402: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:19.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:19 smithi037 ceph-mon[31460]: pgmap v1403: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:19.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:19 smithi059 ceph-mon[36190]: pgmap v1403: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:19.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:19 smithi170 ceph-mon[36280]: pgmap v1403: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:21.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1656.129899 secs 2024-03-28T18:54:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1656.129899 secs 2024-03-28T18:54:21.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1656.129899 secs 2024-03-28T18:54:22.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:22 smithi037 ceph-mon[31460]: pgmap v1404: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:22.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:22 smithi059 ceph-mon[36190]: pgmap v1404: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:22.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:22 smithi170 ceph-mon[36280]: pgmap v1404: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:23.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:23 smithi037 ceph-mon[31460]: pgmap v1405: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:23.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:23 smithi059 ceph-mon[36190]: pgmap v1405: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:23.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:23 smithi170 ceph-mon[36280]: pgmap v1405: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:25.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:25 smithi037 ceph-mon[31460]: pgmap v1406: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:25.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:25 smithi059 ceph-mon[36190]: pgmap v1406: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:25.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:25 smithi170 ceph-mon[36280]: pgmap v1406: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:26.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1661.130020 secs 2024-03-28T18:54:26.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1661.130020 secs 2024-03-28T18:54:26.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1661.130020 secs 2024-03-28T18:54:27.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:27 smithi037 ceph-mon[31460]: pgmap v1407: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:27.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:27 smithi059 ceph-mon[36190]: pgmap v1407: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:27.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:27 smithi170 ceph-mon[36280]: pgmap v1407: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:29.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:29 smithi037 ceph-mon[31460]: pgmap v1408: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:29.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:29 smithi059 ceph-mon[36190]: pgmap v1408: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:29.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:29 smithi170 ceph-mon[36280]: pgmap v1408: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:31.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1666.130138 secs 2024-03-28T18:54:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1666.130138 secs 2024-03-28T18:54:31.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1666.130138 secs 2024-03-28T18:54:32.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:32 smithi037 ceph-mon[31460]: pgmap v1409: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:32.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:32 smithi059 ceph-mon[36190]: pgmap v1409: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:32.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:32 smithi170 ceph-mon[36280]: pgmap v1409: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:33.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:33 smithi037 ceph-mon[31460]: pgmap v1410: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:33.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:33 smithi059 ceph-mon[36190]: pgmap v1410: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:33.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:33 smithi170 ceph-mon[36280]: pgmap v1410: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:35.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:35 smithi037 ceph-mon[31460]: pgmap v1411: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:35.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:35 smithi059 ceph-mon[36190]: pgmap v1411: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:35.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:35 smithi170 ceph-mon[36280]: pgmap v1411: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:36.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1671.130258 secs 2024-03-28T18:54:36.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1671.130258 secs 2024-03-28T18:54:36.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1671.130258 secs 2024-03-28T18:54:37.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:37 smithi037 ceph-mon[31460]: pgmap v1412: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:37.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:37 smithi059 ceph-mon[36190]: pgmap v1412: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:37.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:37 smithi170 ceph-mon[36280]: pgmap v1412: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:39.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:39 smithi037 ceph-mon[31460]: pgmap v1413: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:39.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:39 smithi059 ceph-mon[36190]: pgmap v1413: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:39.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:39 smithi170 ceph-mon[36280]: pgmap v1413: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1676.130336 secs 2024-03-28T18:54:41.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1676.130336 secs 2024-03-28T18:54:41.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1676.130336 secs 2024-03-28T18:54:42.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:42 smithi037 ceph-mon[31460]: pgmap v1414: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:42.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:42 smithi059 ceph-mon[36190]: pgmap v1414: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:42.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:42 smithi170 ceph-mon[36280]: pgmap v1414: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:43.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:43 smithi037 ceph-mon[31460]: pgmap v1415: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:43.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:43 smithi059 ceph-mon[36190]: pgmap v1415: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:43.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:43 smithi170 ceph-mon[36280]: pgmap v1415: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:45.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:45 smithi037 ceph-mon[31460]: pgmap v1416: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:45.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:45 smithi059 ceph-mon[36190]: pgmap v1416: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:45.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:45 smithi170 ceph-mon[36280]: pgmap v1416: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:46.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1681.130456 secs 2024-03-28T18:54:46.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1681.130456 secs 2024-03-28T18:54:46.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1681.130456 secs 2024-03-28T18:54:47.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:47 smithi037 ceph-mon[31460]: pgmap v1417: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:47.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:47 smithi059 ceph-mon[36190]: pgmap v1417: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:47.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:47 smithi170 ceph-mon[36280]: pgmap v1417: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:49.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:49 smithi037 ceph-mon[31460]: pgmap v1418: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:49.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:49 smithi059 ceph-mon[36190]: pgmap v1418: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:49.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:49 smithi170 ceph-mon[36280]: pgmap v1418: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:50.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:50 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:54:50.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:50 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:54:50.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:50 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:54:51.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1686.130541 secs 2024-03-28T18:54:51.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:51 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:54:51.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:51 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:54:51.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:51 smithi037 ceph-mon[31460]: pgmap v1419: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:51.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:51 smithi037 ceph-mon[31460]: pgmap v1420: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:51.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:51 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:54:51.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1686.130541 secs 2024-03-28T18:54:51.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:51 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:54:51.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:51 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:54:51.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:51 smithi059 ceph-mon[36190]: pgmap v1419: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:51.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:51 smithi059 ceph-mon[36190]: pgmap v1420: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:51.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:51 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:54:51.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1686.130541 secs 2024-03-28T18:54:51.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:51 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:54:51.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:51 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:54:51.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:51 smithi170 ceph-mon[36280]: pgmap v1419: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:51.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:51 smithi170 ceph-mon[36280]: pgmap v1420: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:51.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:51 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:54:54.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:53 smithi059 ceph-mon[36190]: pgmap v1421: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:54.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:53 smithi170 ceph-mon[36280]: pgmap v1421: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:54.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:53 smithi037 ceph-mon[31460]: pgmap v1421: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:56.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:55 smithi059 ceph-mon[36190]: pgmap v1422: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:56.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:55 smithi170 ceph-mon[36280]: pgmap v1422: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:56.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:55 smithi037 ceph-mon[31460]: pgmap v1422: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:57.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1691.130636 secs 2024-03-28T18:54:57.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1691.130636 secs 2024-03-28T18:54:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1691.130636 secs 2024-03-28T18:54:58.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:57 smithi059 ceph-mon[36190]: pgmap v1423: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:58.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:57 smithi170 ceph-mon[36280]: pgmap v1423: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:58.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:57 smithi037 ceph-mon[31460]: pgmap v1423: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:00.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:54:59 smithi059 ceph-mon[36190]: pgmap v1424: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:00.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:54:59 smithi170 ceph-mon[36280]: pgmap v1424: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:00.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:54:59 smithi037 ceph-mon[31460]: pgmap v1424: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1696.130716 secs 2024-03-28T18:55:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:01 smithi059 ceph-mon[36190]: pgmap v1425: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:55:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1696.130716 secs 2024-03-28T18:55:02.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:01 smithi170 ceph-mon[36280]: pgmap v1425: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:55:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1696.130716 secs 2024-03-28T18:55:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:01 smithi037 ceph-mon[31460]: pgmap v1425: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:55:04.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:03 smithi059 ceph-mon[36190]: pgmap v1426: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:04.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:03 smithi170 ceph-mon[36280]: pgmap v1426: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:03 smithi037 ceph-mon[31460]: pgmap v1426: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:06.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:05 smithi059 ceph-mon[36190]: pgmap v1427: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:06.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:05 smithi170 ceph-mon[36280]: pgmap v1427: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:05 smithi037 ceph-mon[31460]: pgmap v1427: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:07.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1701.130836 secs 2024-03-28T18:55:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1701.130836 secs 2024-03-28T18:55:07.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1701.130836 secs 2024-03-28T18:55:08.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:07 smithi037 ceph-mon[31460]: pgmap v1428: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:08.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:07 smithi059 ceph-mon[36190]: pgmap v1428: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:08.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:07 smithi170 ceph-mon[36280]: pgmap v1428: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:10.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:09 smithi037 ceph-mon[31460]: pgmap v1429: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:10.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:09 smithi059 ceph-mon[36190]: pgmap v1429: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:10.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:09 smithi170 ceph-mon[36280]: pgmap v1429: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1706.130930 secs 2024-03-28T18:55:12.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:11 smithi059 ceph-mon[36190]: pgmap v1430: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:12.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1706.130930 secs 2024-03-28T18:55:12.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:11 smithi170 ceph-mon[36280]: pgmap v1430: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:12.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1706.130930 secs 2024-03-28T18:55:12.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:11 smithi037 ceph-mon[31460]: pgmap v1430: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:14.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:13 smithi059 ceph-mon[36190]: pgmap v1431: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:14.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:13 smithi170 ceph-mon[36280]: pgmap v1431: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:14.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:13 smithi037 ceph-mon[31460]: pgmap v1431: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:16.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:15 smithi059 ceph-mon[36190]: pgmap v1432: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:16.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:15 smithi170 ceph-mon[36280]: pgmap v1432: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:16.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:15 smithi037 ceph-mon[31460]: pgmap v1432: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:17.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1711.131049 secs 2024-03-28T18:55:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1711.131049 secs 2024-03-28T18:55:17.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1711.131049 secs 2024-03-28T18:55:18.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:17 smithi059 ceph-mon[36190]: pgmap v1433: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:18.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:17 smithi170 ceph-mon[36280]: pgmap v1433: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:18.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:17 smithi037 ceph-mon[31460]: pgmap v1433: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:20.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:19 smithi059 ceph-mon[36190]: pgmap v1434: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:20.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:19 smithi170 ceph-mon[36280]: pgmap v1434: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:20.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:19 smithi037 ceph-mon[31460]: pgmap v1434: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:22.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1716.131143 secs 2024-03-28T18:55:22.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:21 smithi059 ceph-mon[36190]: pgmap v1435: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1716.131143 secs 2024-03-28T18:55:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:21 smithi170 ceph-mon[36280]: pgmap v1435: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:22.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1716.131143 secs 2024-03-28T18:55:22.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:21 smithi037 ceph-mon[31460]: pgmap v1435: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:24.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:23 smithi059 ceph-mon[36190]: pgmap v1436: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:24.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:23 smithi170 ceph-mon[36280]: pgmap v1436: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:24.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:23 smithi037 ceph-mon[31460]: pgmap v1436: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:26.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:25 smithi059 ceph-mon[36190]: pgmap v1437: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:25 smithi170 ceph-mon[36280]: pgmap v1437: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:25 smithi037 ceph-mon[31460]: pgmap v1437: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1721.131264 secs 2024-03-28T18:55:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1721.131264 secs 2024-03-28T18:55:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1721.131264 secs 2024-03-28T18:55:28.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:27 smithi059 ceph-mon[36190]: pgmap v1438: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:28.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:27 smithi170 ceph-mon[36280]: pgmap v1438: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:28.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:27 smithi037 ceph-mon[31460]: pgmap v1438: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:30.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:29 smithi059 ceph-mon[36190]: pgmap v1439: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:30.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:29 smithi170 ceph-mon[36280]: pgmap v1439: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:30.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:29 smithi037 ceph-mon[31460]: pgmap v1439: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:32.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1726.131387 secs 2024-03-28T18:55:32.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:31 smithi059 ceph-mon[36190]: pgmap v1440: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1726.131387 secs 2024-03-28T18:55:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:31 smithi170 ceph-mon[36280]: pgmap v1440: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:32.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1726.131387 secs 2024-03-28T18:55:32.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:31 smithi037 ceph-mon[31460]: pgmap v1440: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:34.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:33 smithi059 ceph-mon[36190]: pgmap v1441: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:34.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:33 smithi170 ceph-mon[36280]: pgmap v1441: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:34.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:33 smithi037 ceph-mon[31460]: pgmap v1441: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:36.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:35 smithi059 ceph-mon[36190]: pgmap v1442: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:36.296 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:35 smithi170 ceph-mon[36280]: pgmap v1442: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:36.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:35 smithi037 ceph-mon[31460]: pgmap v1442: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1731.131507 secs 2024-03-28T18:55:37.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1731.131507 secs 2024-03-28T18:55:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1731.131507 secs 2024-03-28T18:55:38.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:37 smithi059 ceph-mon[36190]: pgmap v1443: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:38.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:37 smithi170 ceph-mon[36280]: pgmap v1443: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:38.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:37 smithi037 ceph-mon[31460]: pgmap v1443: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:40 smithi059 ceph-mon[36190]: pgmap v1444: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:40.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:40 smithi170 ceph-mon[36280]: pgmap v1444: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:40.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:40 smithi037 ceph-mon[31460]: pgmap v1444: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:42 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1736.131626 secs 2024-03-28T18:55:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:42 smithi059 ceph-mon[36190]: pgmap v1445: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:42.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:42 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1736.131626 secs 2024-03-28T18:55:42.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:42 smithi170 ceph-mon[36280]: pgmap v1445: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:42.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:42 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1736.131626 secs 2024-03-28T18:55:42.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:42 smithi037 ceph-mon[31460]: pgmap v1445: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:44 smithi059 ceph-mon[36190]: pgmap v1446: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:44.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:44 smithi170 ceph-mon[36280]: pgmap v1446: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:44.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:44 smithi037 ceph-mon[31460]: pgmap v1446: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:46 smithi059 ceph-mon[36190]: pgmap v1447: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1741.131746 secs 2024-03-28T18:55:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:46 smithi170 ceph-mon[36280]: pgmap v1447: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:46.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1741.131746 secs 2024-03-28T18:55:46.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:46 smithi037 ceph-mon[31460]: pgmap v1447: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:46.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1741.131746 secs 2024-03-28T18:55:48.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:48 smithi059 ceph-mon[36190]: pgmap v1448: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:48.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:48 smithi170 ceph-mon[36280]: pgmap v1448: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:48.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:48 smithi037 ceph-mon[31460]: pgmap v1448: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:50.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:50 smithi059 ceph-mon[36190]: pgmap v1449: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:50.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:50 smithi170 ceph-mon[36280]: pgmap v1449: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:50.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:50 smithi037 ceph-mon[31460]: pgmap v1449: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1746.131900 secs 2024-03-28T18:55:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:51 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:55:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1746.131900 secs 2024-03-28T18:55:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:51 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:55:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1746.131900 secs 2024-03-28T18:55:51.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:51 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:55:52.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:52 smithi059 ceph-mon[36190]: pgmap v1450: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:52 smithi170 ceph-mon[36280]: pgmap v1450: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:52.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:52 smithi037 ceph-mon[31460]: pgmap v1450: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:53.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:53 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:55:53.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:53 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:55:53.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:53 smithi037 ceph-mon[31460]: pgmap v1451: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:55:53.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:53 smithi037 ceph-mon[31460]: pgmap v1452: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:53.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:53 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:55:53.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:53 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:55:53.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:53 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:55:53.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:53 smithi059 ceph-mon[36190]: pgmap v1451: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:55:53.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:53 smithi059 ceph-mon[36190]: pgmap v1452: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:53.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:53 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:55:53.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:53 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:55:53.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:53 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:55:53.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:53 smithi170 ceph-mon[36280]: pgmap v1451: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:55:53.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:53 smithi170 ceph-mon[36280]: pgmap v1452: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:53.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:53 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:55:55.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:55 smithi059 ceph-mon[36190]: pgmap v1453: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:55.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:55 smithi170 ceph-mon[36280]: pgmap v1453: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:55.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:55 smithi037 ceph-mon[31460]: pgmap v1453: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1751.132020 secs 2024-03-28T18:55:56.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1751.132020 secs 2024-03-28T18:55:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1751.132020 secs 2024-03-28T18:55:57.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:57 smithi059 ceph-mon[36190]: pgmap v1454: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:55:57.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:57 smithi170 ceph-mon[36280]: pgmap v1454: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:55:57.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:57 smithi037 ceph-mon[31460]: pgmap v1454: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:55:59.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:55:59 smithi059 ceph-mon[36190]: pgmap v1455: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:59.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:55:59 smithi170 ceph-mon[36280]: pgmap v1455: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:59.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:55:59 smithi037 ceph-mon[31460]: pgmap v1455: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:01.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:01 smithi059 ceph-mon[36190]: pgmap v1456: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:01.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1756.132138 secs 2024-03-28T18:56:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:01 smithi170 ceph-mon[36280]: pgmap v1456: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1756.132138 secs 2024-03-28T18:56:01.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:01 smithi037 ceph-mon[31460]: pgmap v1456: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:01.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1756.132138 secs 2024-03-28T18:56:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:03 smithi059 ceph-mon[36190]: pgmap v1457: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:56:03.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:03 smithi170 ceph-mon[36280]: pgmap v1457: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:56:03.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:03 smithi037 ceph-mon[31460]: pgmap v1457: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:56:05.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:05 smithi059 ceph-mon[36190]: pgmap v1458: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:05.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:05 smithi170 ceph-mon[36280]: pgmap v1458: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:05.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:05 smithi037 ceph-mon[31460]: pgmap v1458: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:06.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1761.132260 secs 2024-03-28T18:56:06.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1761.132260 secs 2024-03-28T18:56:06.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1761.132260 secs 2024-03-28T18:56:07.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:07 smithi059 ceph-mon[36190]: pgmap v1459: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:07.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:07 smithi170 ceph-mon[36280]: pgmap v1459: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:07.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:07 smithi037 ceph-mon[31460]: pgmap v1459: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:08.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:08 smithi059 ceph-mon[36190]: pgmap v1460: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:08.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:08 smithi170 ceph-mon[36280]: pgmap v1460: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:08.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:08 smithi037 ceph-mon[31460]: pgmap v1460: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:11.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:11 smithi059 ceph-mon[36190]: pgmap v1461: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:11.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1766.132381 secs 2024-03-28T18:56:11.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:11 smithi170 ceph-mon[36280]: pgmap v1461: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:11.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1766.132381 secs 2024-03-28T18:56:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:11 smithi037 ceph-mon[31460]: pgmap v1461: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1766.132381 secs 2024-03-28T18:56:13.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:13 smithi059 ceph-mon[36190]: pgmap v1462: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:13.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:13 smithi170 ceph-mon[36280]: pgmap v1462: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:13.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:13 smithi037 ceph-mon[31460]: pgmap v1462: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:15.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:15 smithi059 ceph-mon[36190]: pgmap v1463: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:15.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:15 smithi170 ceph-mon[36280]: pgmap v1463: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:15.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:15 smithi037 ceph-mon[31460]: pgmap v1463: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:16.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1771.132500 secs 2024-03-28T18:56:16.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1771.132500 secs 2024-03-28T18:56:16.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1771.132500 secs 2024-03-28T18:56:17.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:17 smithi059 ceph-mon[36190]: pgmap v1464: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:17.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:17 smithi170 ceph-mon[36280]: pgmap v1464: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:17.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:17 smithi037 ceph-mon[31460]: pgmap v1464: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:19.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:19 smithi059 ceph-mon[36190]: pgmap v1465: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:19.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:19 smithi170 ceph-mon[36280]: pgmap v1465: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:19.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:19 smithi037 ceph-mon[31460]: pgmap v1465: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:21 smithi059 ceph-mon[36190]: pgmap v1466: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1776.132618 secs 2024-03-28T18:56:21.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:21 smithi170 ceph-mon[36280]: pgmap v1466: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:21.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1776.132618 secs 2024-03-28T18:56:21.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:21 smithi037 ceph-mon[31460]: pgmap v1466: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:21.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1776.132618 secs 2024-03-28T18:56:23.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:23 smithi059 ceph-mon[36190]: pgmap v1467: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:23.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:23 smithi170 ceph-mon[36280]: pgmap v1467: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:23.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:23 smithi037 ceph-mon[31460]: pgmap v1467: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:24.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:24 smithi037 ceph-mon[31460]: pgmap v1468: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:24.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:24 smithi059 ceph-mon[36190]: pgmap v1468: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:24.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:24 smithi170 ceph-mon[36280]: pgmap v1468: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1781.132737 secs 2024-03-28T18:56:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1781.132737 secs 2024-03-28T18:56:26.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1781.132737 secs 2024-03-28T18:56:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:27 smithi037 ceph-mon[31460]: pgmap v1469: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:27.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:27 smithi059 ceph-mon[36190]: pgmap v1469: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:27.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:27 smithi170 ceph-mon[36280]: pgmap v1469: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:29.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:29 smithi059 ceph-mon[36190]: pgmap v1470: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:29.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:29 smithi170 ceph-mon[36280]: pgmap v1470: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:29.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:29 smithi037 ceph-mon[31460]: pgmap v1470: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:31 smithi059 ceph-mon[36190]: pgmap v1471: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1786.132857 secs 2024-03-28T18:56:31.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:31 smithi170 ceph-mon[36280]: pgmap v1471: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:31.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1786.132857 secs 2024-03-28T18:56:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:31 smithi037 ceph-mon[31460]: pgmap v1471: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1786.132857 secs 2024-03-28T18:56:33.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:33 smithi059 ceph-mon[36190]: pgmap v1472: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:33.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:33 smithi170 ceph-mon[36280]: pgmap v1472: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:33.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:33 smithi037 ceph-mon[31460]: pgmap v1472: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:35.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:35 smithi059 ceph-mon[36190]: pgmap v1473: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:35.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:35 smithi170 ceph-mon[36280]: pgmap v1473: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:35.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:35 smithi037 ceph-mon[31460]: pgmap v1473: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1791.132975 secs 2024-03-28T18:56:36.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1791.132975 secs 2024-03-28T18:56:36.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1791.132975 secs 2024-03-28T18:56:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:37 smithi059 ceph-mon[36190]: pgmap v1474: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:37.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:37 smithi170 ceph-mon[36280]: pgmap v1474: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:37.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:37 smithi037 ceph-mon[31460]: pgmap v1474: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:38.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:38 smithi037 ceph-mon[31460]: pgmap v1475: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:38.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:38 smithi059 ceph-mon[36190]: pgmap v1475: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:38.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:38 smithi170 ceph-mon[36280]: pgmap v1475: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:41.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:41 smithi059 ceph-mon[36190]: pgmap v1476: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:41.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1796.133092 secs 2024-03-28T18:56:41.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:41 smithi170 ceph-mon[36280]: pgmap v1476: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:41.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1796.133092 secs 2024-03-28T18:56:41.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:41 smithi037 ceph-mon[31460]: pgmap v1476: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:41.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1796.133092 secs 2024-03-28T18:56:43.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:43 smithi059 ceph-mon[36190]: pgmap v1477: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:43.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:43 smithi170 ceph-mon[36280]: pgmap v1477: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:43.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:43 smithi037 ceph-mon[31460]: pgmap v1477: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:45.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:45 smithi059 ceph-mon[36190]: pgmap v1478: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:45.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:45 smithi170 ceph-mon[36280]: pgmap v1478: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:45.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:45 smithi037 ceph-mon[31460]: pgmap v1478: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:46.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1801.133214 secs 2024-03-28T18:56:46.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1801.133214 secs 2024-03-28T18:56:46.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1801.133214 secs 2024-03-28T18:56:47.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:47 smithi059 ceph-mon[36190]: pgmap v1479: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:47.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:47 smithi170 ceph-mon[36280]: pgmap v1479: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:47.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:47 smithi037 ceph-mon[31460]: pgmap v1479: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:49.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:49 smithi059 ceph-mon[36190]: pgmap v1480: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:49.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:49 smithi170 ceph-mon[36280]: pgmap v1480: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:49.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:49 smithi037 ceph-mon[31460]: pgmap v1480: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:50.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:50 smithi037 ceph-mon[31460]: pgmap v1481: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:50.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:50 smithi059 ceph-mon[36190]: pgmap v1481: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:50.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:50 smithi170 ceph-mon[36280]: pgmap v1481: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:51.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1806.133300 secs 2024-03-28T18:56:51.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1806.133300 secs 2024-03-28T18:56:51.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1806.133300 secs 2024-03-28T18:56:52.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:52 smithi037 ceph-mon[31460]: pgmap v1482: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:52.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:52 smithi059 ceph-mon[36190]: pgmap v1482: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:52.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:52 smithi170 ceph-mon[36280]: pgmap v1482: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:53.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:53 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:56:53.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:53 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:56:53.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:53 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:56:54.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:54 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:56:54.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:54 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:56:54.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:54 smithi037 ceph-mon[31460]: pgmap v1483: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:54.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:54 smithi037 ceph-mon[31460]: pgmap v1484: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:54.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:54 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:56:54.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:54 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:56:54.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:54 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:56:54.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:54 smithi059 ceph-mon[36190]: pgmap v1483: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:54.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:54 smithi059 ceph-mon[36190]: pgmap v1484: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:54.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:54 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:56:54.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:54 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:56:54.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:54 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:56:54.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:54 smithi170 ceph-mon[36280]: pgmap v1483: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:54.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:54 smithi170 ceph-mon[36280]: pgmap v1484: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:54.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:54 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:56:57.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:56 smithi059 ceph-mon[36190]: pgmap v1485: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:57.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1811.133420 secs 2024-03-28T18:56:57.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:56 smithi170 ceph-mon[36280]: pgmap v1485: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:57.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1811.133420 secs 2024-03-28T18:56:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:56 smithi037 ceph-mon[31460]: pgmap v1485: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1811.133420 secs 2024-03-28T18:56:59.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:56:58 smithi059 ceph-mon[36190]: pgmap v1486: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:59.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:56:58 smithi170 ceph-mon[36280]: pgmap v1486: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:59.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:56:58 smithi037 ceph-mon[31460]: pgmap v1486: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:57:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:00 smithi059 ceph-mon[36190]: pgmap v1487: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:57:01.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:00 smithi170 ceph-mon[36280]: pgmap v1487: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:57:01.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:00 smithi037 ceph-mon[31460]: pgmap v1487: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:57:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1816.133540 secs 2024-03-28T18:57:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1816.133540 secs 2024-03-28T18:57:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1816.133540 secs 2024-03-28T18:57:03.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:02 smithi059 ceph-mon[36190]: pgmap v1488: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:57:03.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:02 smithi170 ceph-mon[36280]: pgmap v1488: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:57:03.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:02 smithi037 ceph-mon[31460]: pgmap v1488: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:57:05.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:04 smithi059 ceph-mon[36190]: pgmap v1489: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:57:05.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:04 smithi170 ceph-mon[36280]: pgmap v1489: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:57:05.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:04 smithi037 ceph-mon[31460]: pgmap v1489: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:57:07.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:06 smithi059 ceph-mon[36190]: pgmap v1490: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:07.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1821.133659 secs 2024-03-28T18:57:07.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:06 smithi170 ceph-mon[36280]: pgmap v1490: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:07.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1821.133659 secs 2024-03-28T18:57:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:06 smithi037 ceph-mon[31460]: pgmap v1490: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1821.133659 secs 2024-03-28T18:57:09.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:08 smithi059 ceph-mon[36190]: pgmap v1491: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:09.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:08 smithi170 ceph-mon[36280]: pgmap v1491: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:09.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:08 smithi037 ceph-mon[31460]: pgmap v1491: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:11.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:10 smithi059 ceph-mon[36190]: pgmap v1492: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:11.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:10 smithi170 ceph-mon[36280]: pgmap v1492: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:11.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:10 smithi037 ceph-mon[31460]: pgmap v1492: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:12.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1826.133780 secs 2024-03-28T18:57:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1826.133780 secs 2024-03-28T18:57:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1826.133780 secs 2024-03-28T18:57:13.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:12 smithi059 ceph-mon[36190]: pgmap v1493: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:13.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:12 smithi170 ceph-mon[36280]: pgmap v1493: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:13.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:12 smithi037 ceph-mon[31460]: pgmap v1493: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:15.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:14 smithi059 ceph-mon[36190]: pgmap v1494: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:15.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:14 smithi170 ceph-mon[36280]: pgmap v1494: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:15.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:14 smithi037 ceph-mon[31460]: pgmap v1494: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:17.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:16 smithi059 ceph-mon[36190]: pgmap v1495: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:17.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1831.133902 secs 2024-03-28T18:57:17.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:16 smithi170 ceph-mon[36280]: pgmap v1495: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:17.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1831.133902 secs 2024-03-28T18:57:17.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:16 smithi037 ceph-mon[31460]: pgmap v1495: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:17.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1831.133902 secs 2024-03-28T18:57:19.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:18 smithi059 ceph-mon[36190]: pgmap v1496: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:19.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:18 smithi170 ceph-mon[36280]: pgmap v1496: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:19.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:18 smithi037 ceph-mon[31460]: pgmap v1496: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:21.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:20 smithi059 ceph-mon[36190]: pgmap v1497: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:21.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:20 smithi170 ceph-mon[36280]: pgmap v1497: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:21.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:20 smithi037 ceph-mon[31460]: pgmap v1497: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:22.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1836.134020 secs 2024-03-28T18:57:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1836.134020 secs 2024-03-28T18:57:22.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1836.134020 secs 2024-03-28T18:57:23.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:22 smithi059 ceph-mon[36190]: pgmap v1498: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:23.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:22 smithi170 ceph-mon[36280]: pgmap v1498: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:23.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:22 smithi037 ceph-mon[31460]: pgmap v1498: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:25.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:24 smithi059 ceph-mon[36190]: pgmap v1499: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:25.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:24 smithi170 ceph-mon[36280]: pgmap v1499: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:25.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:24 smithi037 ceph-mon[31460]: pgmap v1499: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:27.026 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:26 smithi037 ceph-mon[31460]: pgmap v1500: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:27.026 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1841.134099 secs 2024-03-28T18:57:27.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:26 smithi059 ceph-mon[36190]: pgmap v1500: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:27.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1841.134099 secs 2024-03-28T18:57:27.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:26 smithi170 ceph-mon[36280]: pgmap v1500: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:27.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1841.134099 secs 2024-03-28T18:57:29.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:28 smithi059 ceph-mon[36190]: pgmap v1501: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:29.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:28 smithi170 ceph-mon[36280]: pgmap v1501: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:29.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:28 smithi037 ceph-mon[31460]: pgmap v1501: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:31.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:30 smithi059 ceph-mon[36190]: pgmap v1502: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:31.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:30 smithi170 ceph-mon[36280]: pgmap v1502: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:31.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:30 smithi037 ceph-mon[31460]: pgmap v1502: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:32.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1846.134181 secs 2024-03-28T18:57:32.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1846.134181 secs 2024-03-28T18:57:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1846.134181 secs 2024-03-28T18:57:33.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:32 smithi037 ceph-mon[31460]: pgmap v1503: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:33.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:32 smithi059 ceph-mon[36190]: pgmap v1503: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:33.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:32 smithi170 ceph-mon[36280]: pgmap v1503: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:35.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:34 smithi037 ceph-mon[31460]: pgmap v1504: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:35.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:34 smithi059 ceph-mon[36190]: pgmap v1504: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:35.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:34 smithi170 ceph-mon[36280]: pgmap v1504: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:37.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:36 smithi037 ceph-mon[31460]: pgmap v1505: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:37.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1851.134299 secs 2024-03-28T18:57:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:36 smithi059 ceph-mon[36190]: pgmap v1505: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:37.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1851.134299 secs 2024-03-28T18:57:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:36 smithi170 ceph-mon[36280]: pgmap v1505: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1851.134299 secs 2024-03-28T18:57:39.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:38 smithi037 ceph-mon[31460]: pgmap v1506: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:39.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:38 smithi059 ceph-mon[36190]: pgmap v1506: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:39.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:38 smithi170 ceph-mon[36280]: pgmap v1506: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:41.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:40 smithi037 ceph-mon[31460]: pgmap v1507: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:41.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:40 smithi059 ceph-mon[36190]: pgmap v1507: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:41.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:40 smithi170 ceph-mon[36280]: pgmap v1507: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:42.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1856.134417 secs 2024-03-28T18:57:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1856.134417 secs 2024-03-28T18:57:42.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1856.134417 secs 2024-03-28T18:57:43.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:42 smithi037 ceph-mon[31460]: pgmap v1508: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:43.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:42 smithi059 ceph-mon[36190]: pgmap v1508: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:43.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:42 smithi170 ceph-mon[36280]: pgmap v1508: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:45.087 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:44 smithi059 ceph-mon[36190]: pgmap v1509: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:45.087 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:44 smithi170 ceph-mon[36280]: pgmap v1509: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:45.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:44 smithi037 ceph-mon[31460]: pgmap v1509: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:46 smithi037 ceph-mon[31460]: pgmap v1510: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1861.134506 secs 2024-03-28T18:57:47.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:46 smithi059 ceph-mon[36190]: pgmap v1510: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:47.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1861.134506 secs 2024-03-28T18:57:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:46 smithi170 ceph-mon[36280]: pgmap v1510: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1861.134506 secs 2024-03-28T18:57:49.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:48 smithi037 ceph-mon[31460]: pgmap v1511: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:49.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:48 smithi059 ceph-mon[36190]: pgmap v1511: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:48 smithi170 ceph-mon[36280]: pgmap v1511: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:51.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:50 smithi037 ceph-mon[31460]: pgmap v1512: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:51.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:50 smithi059 ceph-mon[36190]: pgmap v1512: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:50 smithi170 ceph-mon[36280]: pgmap v1512: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1866.134627 secs 2024-03-28T18:57:52.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1866.134627 secs 2024-03-28T18:57:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1866.134627 secs 2024-03-28T18:57:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:52 smithi037 ceph-mon[31460]: pgmap v1513: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:57:53.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:52 smithi059 ceph-mon[36190]: pgmap v1513: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:57:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:52 smithi170 ceph-mon[36280]: pgmap v1513: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:57:55.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:54 smithi037 ceph-mon[31460]: pgmap v1514: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:57:55.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:54 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:57:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:54 smithi059 ceph-mon[36190]: pgmap v1514: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:57:55.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:54 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:57:55.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:54 smithi170 ceph-mon[36280]: pgmap v1514: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:57:55.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:54 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:57:56.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:55 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:57:56.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:55 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:57:56.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:55 smithi037 ceph-mon[31460]: pgmap v1515: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:56.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:55 smithi037 ceph-mon[31460]: pgmap v1516: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:57:56.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:55 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:57:56.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:55 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:57:56.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:55 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:57:56.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:55 smithi059 ceph-mon[36190]: pgmap v1515: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:56.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:55 smithi059 ceph-mon[36190]: pgmap v1516: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:57:56.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:55 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:57:56.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:55 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:57:56.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:55 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:57:56.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:55 smithi170 ceph-mon[36280]: pgmap v1515: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:56.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:55 smithi170 ceph-mon[36280]: pgmap v1516: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:57:56.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:55 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:57:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1871.134747 secs 2024-03-28T18:57:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1871.134747 secs 2024-03-28T18:57:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1871.134747 secs 2024-03-28T18:57:58.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:57:57 smithi059 ceph-mon[36190]: pgmap v1517: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:57:58.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:57:57 smithi170 ceph-mon[36280]: pgmap v1517: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:57:58.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:57:57 smithi037 ceph-mon[31460]: pgmap v1517: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:58:00.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:00 smithi059 ceph-mon[36190]: pgmap v1518: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:58:00.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:00 smithi170 ceph-mon[36280]: pgmap v1518: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:58:00.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:00 smithi037 ceph-mon[31460]: pgmap v1518: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:58:01.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1876.134863 secs 2024-03-28T18:58:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1876.134863 secs 2024-03-28T18:58:01.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1876.134863 secs 2024-03-28T18:58:02.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:02 smithi059 ceph-mon[36190]: pgmap v1519: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:58:02.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:02 smithi170 ceph-mon[36280]: pgmap v1519: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:58:02.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:02 smithi037 ceph-mon[31460]: pgmap v1519: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:58:04.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:04 smithi059 ceph-mon[36190]: pgmap v1520: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:04.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:04 smithi170 ceph-mon[36280]: pgmap v1520: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:04.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:04 smithi037 ceph-mon[31460]: pgmap v1520: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:06.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:06 smithi059 ceph-mon[36190]: pgmap v1521: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:58:06.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1881.134983 secs 2024-03-28T18:58:06.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:06 smithi170 ceph-mon[36280]: pgmap v1521: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:58:06.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1881.134983 secs 2024-03-28T18:58:06.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:06 smithi037 ceph-mon[31460]: pgmap v1521: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:58:06.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1881.134983 secs 2024-03-28T18:58:08.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:08 smithi059 ceph-mon[36190]: pgmap v1522: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:08.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:08 smithi170 ceph-mon[36280]: pgmap v1522: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:08.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:08 smithi037 ceph-mon[31460]: pgmap v1522: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:10.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:10 smithi059 ceph-mon[36190]: pgmap v1523: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:10.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:10 smithi170 ceph-mon[36280]: pgmap v1523: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:10.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:10 smithi037 ceph-mon[31460]: pgmap v1523: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:11.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1886.135104 secs 2024-03-28T18:58:11.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1886.135104 secs 2024-03-28T18:58:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1886.135104 secs 2024-03-28T18:58:12.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:12 smithi059 ceph-mon[36190]: pgmap v1524: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:12.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:12 smithi170 ceph-mon[36280]: pgmap v1524: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:12.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:12 smithi037 ceph-mon[31460]: pgmap v1524: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:14.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:14 smithi059 ceph-mon[36190]: pgmap v1525: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:14.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:14 smithi170 ceph-mon[36280]: pgmap v1525: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:14.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:14 smithi037 ceph-mon[31460]: pgmap v1525: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:16.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:16 smithi059 ceph-mon[36190]: pgmap v1526: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:16.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1891.135183 secs 2024-03-28T18:58:16.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:16 smithi170 ceph-mon[36280]: pgmap v1526: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:16.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1891.135183 secs 2024-03-28T18:58:16.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:16 smithi037 ceph-mon[31460]: pgmap v1526: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:16.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1891.135183 secs 2024-03-28T18:58:18.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:18 smithi059 ceph-mon[36190]: pgmap v1527: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:18.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:18 smithi170 ceph-mon[36280]: pgmap v1527: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:18.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:18 smithi037 ceph-mon[31460]: pgmap v1527: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:20.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:20 smithi059 ceph-mon[36190]: pgmap v1528: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:20.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:20 smithi170 ceph-mon[36280]: pgmap v1528: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:20.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:20 smithi037 ceph-mon[31460]: pgmap v1528: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1896.135302 secs 2024-03-28T18:58:21.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1896.135302 secs 2024-03-28T18:58:21.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1896.135302 secs 2024-03-28T18:58:22.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:22 smithi059 ceph-mon[36190]: pgmap v1529: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:22.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:22 smithi170 ceph-mon[36280]: pgmap v1529: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:22.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:22 smithi037 ceph-mon[31460]: pgmap v1529: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:24.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:24 smithi059 ceph-mon[36190]: pgmap v1530: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:24.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:24 smithi170 ceph-mon[36280]: pgmap v1530: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:24.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:24 smithi037 ceph-mon[31460]: pgmap v1530: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:26.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:26 smithi059 ceph-mon[36190]: pgmap v1531: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1901.135423 secs 2024-03-28T18:58:26.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:26 smithi170 ceph-mon[36280]: pgmap v1531: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:26.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1901.135423 secs 2024-03-28T18:58:26.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:26 smithi037 ceph-mon[31460]: pgmap v1531: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:26.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1901.135423 secs 2024-03-28T18:58:28.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:28 smithi059 ceph-mon[36190]: pgmap v1532: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:28.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:28 smithi170 ceph-mon[36280]: pgmap v1532: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:28.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:28 smithi037 ceph-mon[31460]: pgmap v1532: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:30.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:30 smithi059 ceph-mon[36190]: pgmap v1533: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:30.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:30 smithi170 ceph-mon[36280]: pgmap v1533: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:30.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:30 smithi037 ceph-mon[31460]: pgmap v1533: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:31.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1906.135527 secs 2024-03-28T18:58:31.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1906.135527 secs 2024-03-28T18:58:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1906.135527 secs 2024-03-28T18:58:32.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:32 smithi059 ceph-mon[36190]: pgmap v1534: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:32.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:32 smithi170 ceph-mon[36280]: pgmap v1534: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:32.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:32 smithi037 ceph-mon[31460]: pgmap v1534: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:34.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:34 smithi059 ceph-mon[36190]: pgmap v1535: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:34.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:34 smithi170 ceph-mon[36280]: pgmap v1535: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:34.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:34 smithi037 ceph-mon[31460]: pgmap v1535: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:36 smithi059 ceph-mon[36190]: pgmap v1536: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1911.135647 secs 2024-03-28T18:58:36.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:36 smithi170 ceph-mon[36280]: pgmap v1536: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:36.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1911.135647 secs 2024-03-28T18:58:36.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:36 smithi037 ceph-mon[31460]: pgmap v1536: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:36.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1911.135647 secs 2024-03-28T18:58:38.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:38 smithi059 ceph-mon[36190]: pgmap v1537: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:38.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:38 smithi170 ceph-mon[36280]: pgmap v1537: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:38.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:38 smithi037 ceph-mon[31460]: pgmap v1537: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:40.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:40 smithi059 ceph-mon[36190]: pgmap v1538: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:40.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:40 smithi170 ceph-mon[36280]: pgmap v1538: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:40.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:40 smithi037 ceph-mon[31460]: pgmap v1538: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:41.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1916.135778 secs 2024-03-28T18:58:41.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1916.135778 secs 2024-03-28T18:58:41.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1916.135778 secs 2024-03-28T18:58:42.527 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:42 smithi037 ceph-mon[31460]: pgmap v1539: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:42.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:42 smithi059 ceph-mon[36190]: pgmap v1539: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:42.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:42 smithi170 ceph-mon[36280]: pgmap v1539: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:44.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:44 smithi059 ceph-mon[36190]: pgmap v1540: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:44.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:44 smithi170 ceph-mon[36280]: pgmap v1540: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:44.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:44 smithi037 ceph-mon[31460]: pgmap v1540: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:46.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:46 smithi059 ceph-mon[36190]: pgmap v1541: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:46.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1921.135892 secs 2024-03-28T18:58:46.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:46 smithi170 ceph-mon[36280]: pgmap v1541: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:46.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1921.135892 secs 2024-03-28T18:58:46.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:46 smithi037 ceph-mon[31460]: pgmap v1541: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:46.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1921.135892 secs 2024-03-28T18:58:48.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:48 smithi059 ceph-mon[36190]: pgmap v1542: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:48.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:48 smithi170 ceph-mon[36280]: pgmap v1542: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:48.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:48 smithi037 ceph-mon[31460]: pgmap v1542: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:50.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:50 smithi059 ceph-mon[36190]: pgmap v1543: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:50.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:50 smithi170 ceph-mon[36280]: pgmap v1543: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:50.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:50 smithi037 ceph-mon[31460]: pgmap v1543: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:51.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1926.136012 secs 2024-03-28T18:58:51.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1926.136012 secs 2024-03-28T18:58:51.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1926.136012 secs 2024-03-28T18:58:52.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:52 smithi037 ceph-mon[31460]: pgmap v1544: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:52.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:52 smithi059 ceph-mon[36190]: pgmap v1544: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:52.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:52 smithi170 ceph-mon[36280]: pgmap v1544: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:54.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:54 smithi037 ceph-mon[31460]: pgmap v1545: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:54.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:54 smithi059 ceph-mon[36190]: pgmap v1545: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:54.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:54 smithi170 ceph-mon[36280]: pgmap v1545: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:56 smithi037 ceph-mon[31460]: pgmap v1546: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:56 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:58:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1931.136090 secs 2024-03-28T18:58:56.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:56 smithi059 ceph-mon[36190]: pgmap v1546: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:56 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:58:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1931.136090 secs 2024-03-28T18:58:56.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:56 smithi170 ceph-mon[36280]: pgmap v1546: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:56.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:56 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:58:56.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1931.136090 secs 2024-03-28T18:58:57.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:57 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:58:57.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:57 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:58:57.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:57 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:58:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:57 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:58:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:57 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:58:57.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:57 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:58:57.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:57 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:58:57.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:57 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:58:57.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:57 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:58:58.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:58 smithi037 ceph-mon[31460]: pgmap v1547: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:58:58.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:58 smithi037 ceph-mon[31460]: pgmap v1548: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:58.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:58 smithi059 ceph-mon[36190]: pgmap v1547: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:58:58.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:58 smithi059 ceph-mon[36190]: pgmap v1548: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:58.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:58 smithi170 ceph-mon[36280]: pgmap v1547: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:58:58.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:58 smithi170 ceph-mon[36280]: pgmap v1548: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:59.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:58:59 smithi037 ceph-mon[31460]: pgmap v1549: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:58:59 smithi059 ceph-mon[36190]: pgmap v1549: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:59.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:58:59 smithi170 ceph-mon[36280]: pgmap v1549: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:02.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:02 smithi037 ceph-mon[31460]: pgmap v1550: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:02.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:02 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1936.136210 secs 2024-03-28T18:59:02.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:02 smithi059 ceph-mon[36190]: pgmap v1550: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:02.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:02 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1936.136210 secs 2024-03-28T18:59:02.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:02 smithi170 ceph-mon[36280]: pgmap v1550: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:02.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:02 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1936.136210 secs 2024-03-28T18:59:04.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:04 smithi037 ceph-mon[31460]: pgmap v1551: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:04.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:04 smithi059 ceph-mon[36190]: pgmap v1551: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:04.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:04 smithi170 ceph-mon[36280]: pgmap v1551: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:06 smithi037 ceph-mon[31460]: pgmap v1552: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:06.375 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1941.136329 secs 2024-03-28T18:59:06.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:06 smithi059 ceph-mon[36190]: pgmap v1552: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:06.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1941.136329 secs 2024-03-28T18:59:06.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:06 smithi170 ceph-mon[36280]: pgmap v1552: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:06.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1941.136329 secs 2024-03-28T18:59:08.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:08 smithi037 ceph-mon[31460]: pgmap v1553: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:59:08.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:08 smithi059 ceph-mon[36190]: pgmap v1553: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:59:08.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:08 smithi170 ceph-mon[36280]: pgmap v1553: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:59:10.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:10 smithi037 ceph-mon[31460]: pgmap v1554: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:10.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:10 smithi059 ceph-mon[36190]: pgmap v1554: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:10.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:10 smithi170 ceph-mon[36280]: pgmap v1554: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1946.136448 secs 2024-03-28T18:59:11.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1946.136448 secs 2024-03-28T18:59:11.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1946.136448 secs 2024-03-28T18:59:12.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:12 smithi037 ceph-mon[31460]: pgmap v1555: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:12.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:12 smithi059 ceph-mon[36190]: pgmap v1555: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:12.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:12 smithi170 ceph-mon[36280]: pgmap v1555: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:14.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:14 smithi037 ceph-mon[31460]: pgmap v1556: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:14.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:14 smithi059 ceph-mon[36190]: pgmap v1556: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:14.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:14 smithi170 ceph-mon[36280]: pgmap v1556: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:16.337 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:16 smithi170 ceph-mon[36280]: pgmap v1557: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:16.337 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1951.136546 secs 2024-03-28T18:59:16.338 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:16 smithi059 ceph-mon[36190]: pgmap v1557: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:16.338 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1951.136546 secs 2024-03-28T18:59:16.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:16 smithi037 ceph-mon[31460]: pgmap v1557: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:16.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1951.136546 secs 2024-03-28T18:59:18.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:18 smithi037 ceph-mon[31460]: pgmap v1558: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:18.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:18 smithi059 ceph-mon[36190]: pgmap v1558: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:18.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:18 smithi170 ceph-mon[36280]: pgmap v1558: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:20.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:20 smithi037 ceph-mon[31460]: pgmap v1559: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:20.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:20 smithi059 ceph-mon[36190]: pgmap v1559: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:20.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:20 smithi170 ceph-mon[36280]: pgmap v1559: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:21.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1956.136665 secs 2024-03-28T18:59:21.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1956.136665 secs 2024-03-28T18:59:21.558 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1956.136665 secs 2024-03-28T18:59:22.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:22 smithi037 ceph-mon[31460]: pgmap v1560: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:22.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:22 smithi059 ceph-mon[36190]: pgmap v1560: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:22.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:22 smithi170 ceph-mon[36280]: pgmap v1560: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:24.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:24 smithi037 ceph-mon[31460]: pgmap v1561: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:24.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:24 smithi059 ceph-mon[36190]: pgmap v1561: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:24.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:24 smithi170 ceph-mon[36280]: pgmap v1561: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:26 smithi037 ceph-mon[31460]: pgmap v1562: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1961.136784 secs 2024-03-28T18:59:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:26 smithi059 ceph-mon[36190]: pgmap v1562: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1961.136784 secs 2024-03-28T18:59:26.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:26 smithi170 ceph-mon[36280]: pgmap v1562: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:26.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1961.136784 secs 2024-03-28T18:59:28.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:28 smithi037 ceph-mon[31460]: pgmap v1563: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:28.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:28 smithi059 ceph-mon[36190]: pgmap v1563: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:28.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:28 smithi170 ceph-mon[36280]: pgmap v1563: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:30.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:30 smithi037 ceph-mon[31460]: pgmap v1564: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:30.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:30 smithi059 ceph-mon[36190]: pgmap v1564: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:30.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:30 smithi170 ceph-mon[36280]: pgmap v1564: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:31.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1966.136906 secs 2024-03-28T18:59:31.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1966.136906 secs 2024-03-28T18:59:31.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1966.136906 secs 2024-03-28T18:59:32.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:32 smithi059 ceph-mon[36190]: pgmap v1565: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:32.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:32 smithi170 ceph-mon[36280]: pgmap v1565: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:32.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:32 smithi037 ceph-mon[31460]: pgmap v1565: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:34.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:34 smithi059 ceph-mon[36190]: pgmap v1566: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:34.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:34 smithi170 ceph-mon[36280]: pgmap v1566: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:34.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:34 smithi037 ceph-mon[31460]: pgmap v1566: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:36 smithi059 ceph-mon[36190]: pgmap v1567: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1971.137053 secs 2024-03-28T18:59:36.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:36 smithi170 ceph-mon[36280]: pgmap v1567: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:36.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1971.137053 secs 2024-03-28T18:59:36.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:36 smithi037 ceph-mon[31460]: pgmap v1567: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:36.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1971.137053 secs 2024-03-28T18:59:38.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:38 smithi059 ceph-mon[36190]: pgmap v1568: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:38.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:38 smithi170 ceph-mon[36280]: pgmap v1568: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:38.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:38 smithi037 ceph-mon[31460]: pgmap v1568: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:40.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:40 smithi059 ceph-mon[36190]: pgmap v1569: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:40.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:40 smithi170 ceph-mon[36280]: pgmap v1569: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:40.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:40 smithi037 ceph-mon[31460]: pgmap v1569: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:41.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1976.137170 secs 2024-03-28T18:59:41.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1976.137170 secs 2024-03-28T18:59:41.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1976.137170 secs 2024-03-28T18:59:42.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:42 smithi059 ceph-mon[36190]: pgmap v1570: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:42.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:42 smithi170 ceph-mon[36280]: pgmap v1570: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:42.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:42 smithi037 ceph-mon[31460]: pgmap v1570: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:44.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:44 smithi059 ceph-mon[36190]: pgmap v1571: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:44.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:44 smithi170 ceph-mon[36280]: pgmap v1571: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:44.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:44 smithi037 ceph-mon[31460]: pgmap v1571: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:46.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:46 smithi059 ceph-mon[36190]: pgmap v1572: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:46.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1981.137262 secs 2024-03-28T18:59:46.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:46 smithi170 ceph-mon[36280]: pgmap v1572: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:46.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1981.137262 secs 2024-03-28T18:59:46.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:46 smithi037 ceph-mon[31460]: pgmap v1572: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:46.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1981.137262 secs 2024-03-28T18:59:48.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:48 smithi059 ceph-mon[36190]: pgmap v1573: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:48.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:48 smithi170 ceph-mon[36280]: pgmap v1573: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:48.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:48 smithi037 ceph-mon[31460]: pgmap v1573: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:50.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:50 smithi059 ceph-mon[36190]: pgmap v1574: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:50.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:50 smithi170 ceph-mon[36280]: pgmap v1574: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:50.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:50 smithi037 ceph-mon[31460]: pgmap v1574: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:51.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1986.137382 secs 2024-03-28T18:59:51.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1986.137382 secs 2024-03-28T18:59:51.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1986.137382 secs 2024-03-28T18:59:52.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:52 smithi059 ceph-mon[36190]: pgmap v1575: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:52.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:52 smithi170 ceph-mon[36280]: pgmap v1575: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:52.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:52 smithi037 ceph-mon[31460]: pgmap v1575: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:54.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:54 smithi059 ceph-mon[36190]: pgmap v1576: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:54.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:54 smithi170 ceph-mon[36280]: pgmap v1576: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:54.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:54 smithi037 ceph-mon[31460]: pgmap v1576: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:56 smithi059 ceph-mon[36190]: pgmap v1577: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1991.137503 secs 2024-03-28T18:59:56.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:56 smithi170 ceph-mon[36280]: pgmap v1577: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:56.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1991.137503 secs 2024-03-28T18:59:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:56 smithi037 ceph-mon[31460]: pgmap v1577: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1991.137503 secs 2024-03-28T18:59:57.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:57 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:59:57.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:57 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:59:57.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:57 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:59:58.526 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:58 smithi037 ceph-mon[31460]: pgmap v1578: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:58.526 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:58 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:59:58.526 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:58 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:59:58.526 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:58 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:59:58.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:58 smithi059 ceph-mon[36190]: pgmap v1578: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:58.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:58 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:59:58.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:58 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:59:58.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:58 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:59:58.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:58 smithi170 ceph-mon[36280]: pgmap v1578: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:58.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:58 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:59:58.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:58 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:59:58.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:58 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T18:59:59.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:59 smithi059 ceph-mon[36190]: pgmap v1579: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:59.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 18:59:59 smithi059 ceph-mon[36190]: pgmap v1580: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:59.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:59 smithi170 ceph-mon[36280]: pgmap v1579: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:59.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 18:59:59 smithi170 ceph-mon[36280]: pgmap v1580: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:59.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:59 smithi037 ceph-mon[31460]: pgmap v1579: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:59.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 18:59:59 smithi037 ceph-mon[31460]: pgmap v1580: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:00:00.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:00 smithi059 ceph-mon[36190]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T19:00:00.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:00 smithi170 ceph-mon[36280]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T19:00:00.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:00 smithi037 ceph-mon[31460]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T19:00:01.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:01 smithi059 ceph-mon[36190]: pgmap v1581: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:00:01.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 1996.137622 secs 2024-03-28T19:00:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:01 smithi170 ceph-mon[36280]: pgmap v1581: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:00:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 1996.137622 secs 2024-03-28T19:00:01.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:01 smithi037 ceph-mon[31460]: pgmap v1581: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:00:01.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 1996.137622 secs 2024-03-28T19:00:03.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:03 smithi059 ceph-mon[36190]: pgmap v1582: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:00:03.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:03 smithi170 ceph-mon[36280]: pgmap v1582: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:00:03.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:03 smithi037 ceph-mon[31460]: pgmap v1582: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:00:05.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:05 smithi059 ceph-mon[36190]: pgmap v1583: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:00:05.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:05 smithi170 ceph-mon[36280]: pgmap v1583: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:00:05.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:05 smithi037 ceph-mon[31460]: pgmap v1583: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:00:06.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2001.137741 secs 2024-03-28T19:00:06.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2001.137741 secs 2024-03-28T19:00:06.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2001.137741 secs 2024-03-28T19:00:07.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:07 smithi059 ceph-mon[36190]: pgmap v1584: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:00:07.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:07 smithi170 ceph-mon[36280]: pgmap v1584: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:00:07.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:07 smithi037 ceph-mon[31460]: pgmap v1584: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:00:09.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:09 smithi059 ceph-mon[36190]: pgmap v1585: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:00:09.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:09 smithi170 ceph-mon[36280]: pgmap v1585: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:00:09.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:09 smithi037 ceph-mon[31460]: pgmap v1585: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:00:11.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:11 smithi059 ceph-mon[36190]: pgmap v1586: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:11.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2006.137859 secs 2024-03-28T19:00:11.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:11 smithi170 ceph-mon[36280]: pgmap v1586: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:11.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2006.137859 secs 2024-03-28T19:00:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:11 smithi037 ceph-mon[31460]: pgmap v1586: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2006.137859 secs 2024-03-28T19:00:13.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:13 smithi059 ceph-mon[36190]: pgmap v1587: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:13.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:13 smithi170 ceph-mon[36280]: pgmap v1587: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:13.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:13 smithi037 ceph-mon[31460]: pgmap v1587: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:15.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:15 smithi059 ceph-mon[36190]: pgmap v1588: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:15.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:15 smithi170 ceph-mon[36280]: pgmap v1588: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:15.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:15 smithi037 ceph-mon[31460]: pgmap v1588: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:16.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2011.137979 secs 2024-03-28T19:00:16.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2011.137979 secs 2024-03-28T19:00:16.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2011.137979 secs 2024-03-28T19:00:17.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:17 smithi059 ceph-mon[36190]: pgmap v1589: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:17.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:17 smithi170 ceph-mon[36280]: pgmap v1589: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:17.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:17 smithi037 ceph-mon[31460]: pgmap v1589: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:19.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:19 smithi037 ceph-mon[31460]: pgmap v1590: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:19.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:19 smithi059 ceph-mon[36190]: pgmap v1590: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:19.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:19 smithi170 ceph-mon[36280]: pgmap v1590: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:21.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:21 smithi037 ceph-mon[31460]: pgmap v1591: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:21.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2016.138098 secs 2024-03-28T19:00:21.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:21 smithi059 ceph-mon[36190]: pgmap v1591: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:21.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2016.138098 secs 2024-03-28T19:00:21.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:21 smithi170 ceph-mon[36280]: pgmap v1591: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:21.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2016.138098 secs 2024-03-28T19:00:23.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:23 smithi037 ceph-mon[31460]: pgmap v1592: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:23.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:23 smithi059 ceph-mon[36190]: pgmap v1592: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:23.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:23 smithi170 ceph-mon[36280]: pgmap v1592: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:25.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:25 smithi037 ceph-mon[31460]: pgmap v1593: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:25.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:25 smithi059 ceph-mon[36190]: pgmap v1593: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:25 smithi170 ceph-mon[36280]: pgmap v1593: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:26.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2021.138177 secs 2024-03-28T19:00:26.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2021.138177 secs 2024-03-28T19:00:26.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2021.138177 secs 2024-03-28T19:00:27.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:27 smithi037 ceph-mon[31460]: pgmap v1594: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:27.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:27 smithi059 ceph-mon[36190]: pgmap v1594: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:27.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:27 smithi170 ceph-mon[36280]: pgmap v1594: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:29.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:29 smithi037 ceph-mon[31460]: pgmap v1595: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:29.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:29 smithi059 ceph-mon[36190]: pgmap v1595: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:29.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:29 smithi170 ceph-mon[36280]: pgmap v1595: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:31 smithi037 ceph-mon[31460]: pgmap v1596: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2026.138296 secs 2024-03-28T19:00:31.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:31 smithi059 ceph-mon[36190]: pgmap v1596: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:31.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2026.138296 secs 2024-03-28T19:00:31.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:31 smithi170 ceph-mon[36280]: pgmap v1596: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:31.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2026.138296 secs 2024-03-28T19:00:32.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:32 smithi037 ceph-mon[31460]: pgmap v1597: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:32.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:32 smithi059 ceph-mon[36190]: pgmap v1597: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:32.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:32 smithi170 ceph-mon[36280]: pgmap v1597: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:35.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:35 smithi059 ceph-mon[36190]: pgmap v1598: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:35.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:35 smithi170 ceph-mon[36280]: pgmap v1598: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:35.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:35 smithi037 ceph-mon[31460]: pgmap v1598: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:36.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2031.138386 secs 2024-03-28T19:00:36.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2031.138386 secs 2024-03-28T19:00:36.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2031.138386 secs 2024-03-28T19:00:37.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:37 smithi059 ceph-mon[36190]: pgmap v1599: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:37.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:37 smithi170 ceph-mon[36280]: pgmap v1599: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:37.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:37 smithi037 ceph-mon[31460]: pgmap v1599: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:39.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:39 smithi059 ceph-mon[36190]: pgmap v1600: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:39.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:39 smithi170 ceph-mon[36280]: pgmap v1600: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:39.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:39 smithi037 ceph-mon[31460]: pgmap v1600: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:41.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:41 smithi059 ceph-mon[36190]: pgmap v1601: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:41.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2036.138508 secs 2024-03-28T19:00:41.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:41 smithi170 ceph-mon[36280]: pgmap v1601: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:41.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2036.138508 secs 2024-03-28T19:00:41.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:41 smithi037 ceph-mon[31460]: pgmap v1601: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:41.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2036.138508 secs 2024-03-28T19:00:43.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:43 smithi170 ceph-mon[36280]: pgmap v1602: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:43.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:43 smithi059 ceph-mon[36190]: pgmap v1602: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:43.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:43 smithi037 ceph-mon[31460]: pgmap v1602: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:45.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:45 smithi059 ceph-mon[36190]: pgmap v1603: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:45.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:45 smithi170 ceph-mon[36280]: pgmap v1603: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:45.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:45 smithi037 ceph-mon[31460]: pgmap v1603: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:46.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2041.138599 secs 2024-03-28T19:00:46.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2041.138599 secs 2024-03-28T19:00:46.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2041.138599 secs 2024-03-28T19:00:47.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:47 smithi059 ceph-mon[36190]: pgmap v1604: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:47.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:47 smithi170 ceph-mon[36280]: pgmap v1604: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:47.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:47 smithi037 ceph-mon[31460]: pgmap v1604: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:49.489 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:49 smithi037 ceph-mon[31460]: pgmap v1605: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:49.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:49 smithi059 ceph-mon[36190]: pgmap v1605: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:49.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:49 smithi170 ceph-mon[36280]: pgmap v1605: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:51.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:51 smithi059 ceph-mon[36190]: pgmap v1606: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:51.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2046.138717 secs 2024-03-28T19:00:51.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:51 smithi170 ceph-mon[36280]: pgmap v1606: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:51.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2046.138717 secs 2024-03-28T19:00:51.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:51 smithi037 ceph-mon[31460]: pgmap v1606: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:51.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2046.138717 secs 2024-03-28T19:00:53.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:53 smithi059 ceph-mon[36190]: pgmap v1607: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:53.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:53 smithi170 ceph-mon[36280]: pgmap v1607: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:53.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:53 smithi037 ceph-mon[31460]: pgmap v1607: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:55.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:55 smithi059 ceph-mon[36190]: pgmap v1608: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:55.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:55 smithi170 ceph-mon[36280]: pgmap v1608: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:55.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:55 smithi037 ceph-mon[31460]: pgmap v1608: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:56.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2051.138837 secs 2024-03-28T19:00:56.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2051.138837 secs 2024-03-28T19:00:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2051.138837 secs 2024-03-28T19:00:57.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:57 smithi059 ceph-mon[36190]: pgmap v1609: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:57.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:57 smithi170 ceph-mon[36280]: pgmap v1609: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:57.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:57 smithi037 ceph-mon[31460]: pgmap v1609: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:59.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:59 smithi059 ceph-mon[36190]: pgmap v1610: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:59.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:00:59 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:00:59.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:59 smithi170 ceph-mon[36280]: pgmap v1610: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:59.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:00:59 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:00:59.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:59 smithi037 ceph-mon[31460]: pgmap v1610: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:59.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:00:59 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:01:00.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:00 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:01:00.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:00 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:01:00.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:00 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:01:00.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:00 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:01:00.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:00 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:01:00.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:00 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:01:00.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:00 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:01:00.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:00 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:01:00.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:00 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:01:01.522 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:01 smithi059 ceph-mon[36190]: pgmap v1611: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T19:01:01.523 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:01 smithi059 ceph-mon[36190]: pgmap v1612: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:01.523 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2056.138959 secs 2024-03-28T19:01:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:01 smithi170 ceph-mon[36280]: pgmap v1611: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T19:01:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:01 smithi170 ceph-mon[36280]: pgmap v1612: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2056.138959 secs 2024-03-28T19:01:01.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:01 smithi037 ceph-mon[31460]: pgmap v1611: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T19:01:01.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:01 smithi037 ceph-mon[31460]: pgmap v1612: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:01.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2056.138959 secs 2024-03-28T19:01:03.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:03 smithi059 ceph-mon[36190]: pgmap v1613: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:01:03.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:03 smithi170 ceph-mon[36280]: pgmap v1613: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:01:03.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:03 smithi037 ceph-mon[31460]: pgmap v1613: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:01:05.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:05 smithi059 ceph-mon[36190]: pgmap v1614: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:05.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:05 smithi170 ceph-mon[36280]: pgmap v1614: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:05.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:05 smithi037 ceph-mon[31460]: pgmap v1614: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:06.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2061.139079 secs 2024-03-28T19:01:06.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2061.139079 secs 2024-03-28T19:01:06.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2061.139079 secs 2024-03-28T19:01:07.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:07 smithi059 ceph-mon[36190]: pgmap v1615: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:07.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:07 smithi170 ceph-mon[36280]: pgmap v1615: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:07.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:07 smithi037 ceph-mon[31460]: pgmap v1615: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:09.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:09 smithi059 ceph-mon[36190]: pgmap v1616: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:09.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:09 smithi170 ceph-mon[36280]: pgmap v1616: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:09.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:09 smithi037 ceph-mon[31460]: pgmap v1616: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:11.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:11 smithi059 ceph-mon[36190]: pgmap v1617: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:01:11.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2066.139200 secs 2024-03-28T19:01:11.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:11 smithi170 ceph-mon[36280]: pgmap v1617: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:01:11.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2066.139200 secs 2024-03-28T19:01:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:11 smithi037 ceph-mon[31460]: pgmap v1617: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:01:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2066.139200 secs 2024-03-28T19:01:13.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:13 smithi059 ceph-mon[36190]: pgmap v1618: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:13.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:13 smithi170 ceph-mon[36280]: pgmap v1618: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:13.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:13 smithi037 ceph-mon[31460]: pgmap v1618: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:15.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:15 smithi059 ceph-mon[36190]: pgmap v1619: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T19:01:15.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:15 smithi170 ceph-mon[36280]: pgmap v1619: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T19:01:15.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:15 smithi037 ceph-mon[31460]: pgmap v1619: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T19:01:16.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2071.139319 secs 2024-03-28T19:01:16.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2071.139319 secs 2024-03-28T19:01:16.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2071.139319 secs 2024-03-28T19:01:17.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:17 smithi037 ceph-mon[31460]: pgmap v1620: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:17.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:17 smithi059 ceph-mon[36190]: pgmap v1620: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:17.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:17 smithi170 ceph-mon[36280]: pgmap v1620: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:19.611 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:19 smithi037 ceph-mon[31460]: pgmap v1621: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:19.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:19 smithi059 ceph-mon[36190]: pgmap v1621: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:19.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:19 smithi170 ceph-mon[36280]: pgmap v1621: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:21.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:21 smithi037 ceph-mon[31460]: pgmap v1622: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:21.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2076.139438 secs 2024-03-28T19:01:21.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:21 smithi059 ceph-mon[36190]: pgmap v1622: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:21.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2076.139438 secs 2024-03-28T19:01:21.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:21 smithi170 ceph-mon[36280]: pgmap v1622: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:21.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2076.139438 secs 2024-03-28T19:01:23.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:23 smithi037 ceph-mon[31460]: pgmap v1623: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:23.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:23 smithi059 ceph-mon[36190]: pgmap v1623: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:23.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:23 smithi170 ceph-mon[36280]: pgmap v1623: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:25.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:25 smithi037 ceph-mon[31460]: pgmap v1624: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T19:01:25.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:25 smithi059 ceph-mon[36190]: pgmap v1624: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T19:01:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:25 smithi170 ceph-mon[36280]: pgmap v1624: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T19:01:26.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2081.139557 secs 2024-03-28T19:01:26.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2081.139557 secs 2024-03-28T19:01:26.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2081.139557 secs 2024-03-28T19:01:27.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:27 smithi037 ceph-mon[31460]: pgmap v1625: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:27.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:27 smithi059 ceph-mon[36190]: pgmap v1625: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:27.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:27 smithi170 ceph-mon[36280]: pgmap v1625: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:29.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:29 smithi037 ceph-mon[31460]: pgmap v1626: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:29.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:29 smithi059 ceph-mon[36190]: pgmap v1626: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:29.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:29 smithi170 ceph-mon[36280]: pgmap v1626: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:30.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:30 smithi037 ceph-mon[31460]: pgmap v1627: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:30.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:30 smithi059 ceph-mon[36190]: pgmap v1627: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:30.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:30 smithi170 ceph-mon[36280]: pgmap v1627: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2086.139676 secs 2024-03-28T19:01:31.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2086.139676 secs 2024-03-28T19:01:31.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2086.139676 secs 2024-03-28T19:01:32.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:32 smithi037 ceph-mon[31460]: pgmap v1628: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:32.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:32 smithi059 ceph-mon[36190]: pgmap v1628: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:32.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:32 smithi170 ceph-mon[36280]: pgmap v1628: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:34.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:34 smithi037 ceph-mon[31460]: pgmap v1629: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:35.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:34 smithi059 ceph-mon[36190]: pgmap v1629: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:35.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:34 smithi170 ceph-mon[36280]: pgmap v1629: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:36.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:36 smithi037 ceph-mon[31460]: pgmap v1630: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:36.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2091.139795 secs 2024-03-28T19:01:37.037 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:36 smithi059 ceph-mon[36190]: pgmap v1630: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:37.037 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2091.139795 secs 2024-03-28T19:01:37.037 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:36 smithi170 ceph-mon[36280]: pgmap v1630: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:37.038 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2091.139795 secs 2024-03-28T19:01:38.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:38 smithi037 ceph-mon[31460]: pgmap v1631: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:39.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:38 smithi059 ceph-mon[36190]: pgmap v1631: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:39.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:38 smithi170 ceph-mon[36280]: pgmap v1631: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:41.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:40 smithi059 ceph-mon[36190]: pgmap v1632: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:41.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:40 smithi170 ceph-mon[36280]: pgmap v1632: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:41.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:40 smithi037 ceph-mon[31460]: pgmap v1632: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:42.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2096.139914 secs 2024-03-28T19:01:42.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2096.139914 secs 2024-03-28T19:01:42.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2096.139914 secs 2024-03-28T19:01:43.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:42 smithi059 ceph-mon[36190]: pgmap v1633: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:43.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:42 smithi170 ceph-mon[36280]: pgmap v1633: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:43.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:42 smithi037 ceph-mon[31460]: pgmap v1633: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:45.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:44 smithi059 ceph-mon[36190]: pgmap v1634: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:45.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:44 smithi170 ceph-mon[36280]: pgmap v1634: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:45.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:44 smithi037 ceph-mon[31460]: pgmap v1634: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:47.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:46 smithi059 ceph-mon[36190]: pgmap v1635: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:47.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2101.140060 secs 2024-03-28T19:01:47.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:46 smithi170 ceph-mon[36280]: pgmap v1635: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:47.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2101.140060 secs 2024-03-28T19:01:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:46 smithi037 ceph-mon[31460]: pgmap v1635: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2101.140060 secs 2024-03-28T19:01:49.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:48 smithi059 ceph-mon[36190]: pgmap v1636: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:49.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:48 smithi170 ceph-mon[36280]: pgmap v1636: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:49.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:48 smithi037 ceph-mon[31460]: pgmap v1636: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:51.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:50 smithi059 ceph-mon[36190]: pgmap v1637: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:51.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:50 smithi170 ceph-mon[36280]: pgmap v1637: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:51.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:50 smithi037 ceph-mon[31460]: pgmap v1637: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:52.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2106.140145 secs 2024-03-28T19:01:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2106.140145 secs 2024-03-28T19:01:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2106.140145 secs 2024-03-28T19:01:53.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:52 smithi059 ceph-mon[36190]: pgmap v1638: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:53.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:52 smithi170 ceph-mon[36280]: pgmap v1638: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:52 smithi037 ceph-mon[31460]: pgmap v1638: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:55.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:54 smithi059 ceph-mon[36190]: pgmap v1639: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:55.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:54 smithi170 ceph-mon[36280]: pgmap v1639: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:55.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:54 smithi037 ceph-mon[31460]: pgmap v1639: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:57.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:56 smithi059 ceph-mon[36190]: pgmap v1640: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:57.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2111.140230 secs 2024-03-28T19:01:57.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:56 smithi170 ceph-mon[36280]: pgmap v1640: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:57.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2111.140230 secs 2024-03-28T19:01:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:56 smithi037 ceph-mon[31460]: pgmap v1640: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2111.140230 secs 2024-03-28T19:01:59.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:01:58 smithi059 ceph-mon[36190]: pgmap v1641: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:59.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:01:58 smithi170 ceph-mon[36280]: pgmap v1641: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:59.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:01:58 smithi037 ceph-mon[31460]: pgmap v1641: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:00 smithi059 ceph-mon[36190]: pgmap v1642: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:01.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:00 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:02:01.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:00 smithi170 ceph-mon[36280]: pgmap v1642: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:01.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:00 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:02:01.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:00 smithi037 ceph-mon[31460]: pgmap v1642: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:01.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:00 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:02:02.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2116.140297 secs 2024-03-28T19:02:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:01 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:02:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:01 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:02:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:01 smithi059 ceph-mon[36190]: pgmap v1643: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:01 smithi059 ceph-mon[36190]: pgmap v1644: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:02:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:01 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:02:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2116.140297 secs 2024-03-28T19:02:02.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:01 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:02:02.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:01 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:02:02.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:01 smithi170 ceph-mon[36280]: pgmap v1643: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:02.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:01 smithi170 ceph-mon[36280]: pgmap v1644: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:02:02.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:01 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:02:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2116.140297 secs 2024-03-28T19:02:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:01 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:02:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:01 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:02:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:01 smithi037 ceph-mon[31460]: pgmap v1643: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:01 smithi037 ceph-mon[31460]: pgmap v1644: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:02:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:01 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:02:04.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:04 smithi037 ceph-mon[31460]: pgmap v1645: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:02:04.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:04 smithi059 ceph-mon[36190]: pgmap v1645: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:02:04.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:04 smithi170 ceph-mon[36280]: pgmap v1645: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:02:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:06 smithi037 ceph-mon[31460]: pgmap v1646: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:02:06.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2121.140418 secs 2024-03-28T19:02:06.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:06 smithi059 ceph-mon[36190]: pgmap v1646: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:02:06.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2121.140418 secs 2024-03-28T19:02:06.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:06 smithi170 ceph-mon[36280]: pgmap v1646: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:02:06.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2121.140418 secs 2024-03-28T19:02:08.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:08 smithi037 ceph-mon[31460]: pgmap v1647: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T19:02:08.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:08 smithi059 ceph-mon[36190]: pgmap v1647: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T19:02:08.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:08 smithi170 ceph-mon[36280]: pgmap v1647: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T19:02:10.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:10 smithi037 ceph-mon[31460]: pgmap v1648: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T19:02:10.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:10 smithi059 ceph-mon[36190]: pgmap v1648: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T19:02:10.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:10 smithi170 ceph-mon[36280]: pgmap v1648: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T19:02:11.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2126.140517 secs 2024-03-28T19:02:11.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2126.140517 secs 2024-03-28T19:02:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2126.140517 secs 2024-03-28T19:02:12.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:12 smithi059 ceph-mon[36190]: pgmap v1649: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:02:12.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:12 smithi170 ceph-mon[36280]: pgmap v1649: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:02:12.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:12 smithi037 ceph-mon[31460]: pgmap v1649: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:02:14.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:14 smithi059 ceph-mon[36190]: pgmap v1650: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:02:14.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:14 smithi170 ceph-mon[36280]: pgmap v1650: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:02:14.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:14 smithi037 ceph-mon[31460]: pgmap v1650: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:02:16.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:16 smithi059 ceph-mon[36190]: pgmap v1651: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:02:16.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2131.140618 secs 2024-03-28T19:02:16.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:16 smithi170 ceph-mon[36280]: pgmap v1651: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:02:16.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2131.140618 secs 2024-03-28T19:02:16.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:16 smithi037 ceph-mon[31460]: pgmap v1651: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:02:16.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2131.140618 secs 2024-03-28T19:02:18.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:18 smithi059 ceph-mon[36190]: pgmap v1652: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:02:18.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:18 smithi170 ceph-mon[36280]: pgmap v1652: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:02:18.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:18 smithi037 ceph-mon[31460]: pgmap v1652: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:02:20.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:20 smithi059 ceph-mon[36190]: pgmap v1653: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:20.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:20 smithi170 ceph-mon[36280]: pgmap v1653: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:20.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:20 smithi037 ceph-mon[31460]: pgmap v1653: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:21.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2136.140733 secs 2024-03-28T19:02:21.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2136.140733 secs 2024-03-28T19:02:21.564 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2136.140733 secs 2024-03-28T19:02:22.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:22 smithi059 ceph-mon[36190]: pgmap v1654: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:22.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:22 smithi170 ceph-mon[36280]: pgmap v1654: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:22.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:22 smithi037 ceph-mon[31460]: pgmap v1654: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:24.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:24 smithi059 ceph-mon[36190]: pgmap v1655: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:24.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:24 smithi170 ceph-mon[36280]: pgmap v1655: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:24.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:24 smithi037 ceph-mon[31460]: pgmap v1655: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:26 smithi059 ceph-mon[36190]: pgmap v1656: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:26.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2141.140847 secs 2024-03-28T19:02:26.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:26 smithi170 ceph-mon[36280]: pgmap v1656: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:26.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2141.140847 secs 2024-03-28T19:02:26.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:26 smithi037 ceph-mon[31460]: pgmap v1656: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:26.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2141.140847 secs 2024-03-28T19:02:28.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:28 smithi059 ceph-mon[36190]: pgmap v1657: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:28.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:28 smithi170 ceph-mon[36280]: pgmap v1657: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:28.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:28 smithi037 ceph-mon[31460]: pgmap v1657: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:30.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:30 smithi059 ceph-mon[36190]: pgmap v1658: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:30.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:30 smithi170 ceph-mon[36280]: pgmap v1658: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:30.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:30 smithi037 ceph-mon[31460]: pgmap v1658: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2146.141208 secs 2024-03-28T19:02:31.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2146.141208 secs 2024-03-28T19:02:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2146.141208 secs 2024-03-28T19:02:32.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:32 smithi059 ceph-mon[36190]: pgmap v1659: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:32.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:32 smithi170 ceph-mon[36280]: pgmap v1659: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:32.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:32 smithi037 ceph-mon[31460]: pgmap v1659: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:34.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:34 smithi059 ceph-mon[36190]: pgmap v1660: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:34.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:34 smithi170 ceph-mon[36280]: pgmap v1660: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:34.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:34 smithi037 ceph-mon[31460]: pgmap v1660: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:36.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:36 smithi059 ceph-mon[36190]: pgmap v1661: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2151.141298 secs 2024-03-28T19:02:36.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:36 smithi170 ceph-mon[36280]: pgmap v1661: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:36.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2151.141298 secs 2024-03-28T19:02:36.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:36 smithi037 ceph-mon[31460]: pgmap v1661: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:36.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2151.141298 secs 2024-03-28T19:02:38.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:38 smithi059 ceph-mon[36190]: pgmap v1662: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:38.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:38 smithi170 ceph-mon[36280]: pgmap v1662: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:38.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:38 smithi037 ceph-mon[31460]: pgmap v1662: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:40.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:40 smithi059 ceph-mon[36190]: pgmap v1663: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:40.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:40 smithi170 ceph-mon[36280]: pgmap v1663: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:40.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:40 smithi037 ceph-mon[31460]: pgmap v1663: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:41.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2156.141414 secs 2024-03-28T19:02:41.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2156.141414 secs 2024-03-28T19:02:41.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2156.141414 secs 2024-03-28T19:02:42.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:42 smithi059 ceph-mon[36190]: pgmap v1664: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:42.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:42 smithi170 ceph-mon[36280]: pgmap v1664: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:42.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:42 smithi037 ceph-mon[31460]: pgmap v1664: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:44.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:44 smithi059 ceph-mon[36190]: pgmap v1665: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:44.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:44 smithi170 ceph-mon[36280]: pgmap v1665: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:44.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:44 smithi037 ceph-mon[31460]: pgmap v1665: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:46.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:46 smithi059 ceph-mon[36190]: pgmap v1666: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:46.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2161.141516 secs 2024-03-28T19:02:46.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:46 smithi170 ceph-mon[36280]: pgmap v1666: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:46.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2161.141516 secs 2024-03-28T19:02:46.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:46 smithi037 ceph-mon[31460]: pgmap v1666: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:46.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2161.141516 secs 2024-03-28T19:02:48.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:48 smithi059 ceph-mon[36190]: pgmap v1667: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:48.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:48 smithi170 ceph-mon[36280]: pgmap v1667: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:48.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:48 smithi037 ceph-mon[31460]: pgmap v1667: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:50.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:50 smithi059 ceph-mon[36190]: pgmap v1668: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:50.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:50 smithi170 ceph-mon[36280]: pgmap v1668: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:50.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:50 smithi037 ceph-mon[31460]: pgmap v1668: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:51.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2166.141601 secs 2024-03-28T19:02:51.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2166.141601 secs 2024-03-28T19:02:51.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2166.141601 secs 2024-03-28T19:02:52.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:52 smithi059 ceph-mon[36190]: pgmap v1669: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:52.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:52 smithi170 ceph-mon[36280]: pgmap v1669: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:52.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:52 smithi037 ceph-mon[31460]: pgmap v1669: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:54.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:54 smithi059 ceph-mon[36190]: pgmap v1670: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:54.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:54 smithi170 ceph-mon[36280]: pgmap v1670: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:54.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:54 smithi037 ceph-mon[31460]: pgmap v1670: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:56.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:56 smithi059 ceph-mon[36190]: pgmap v1671: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:56.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2171.141705 secs 2024-03-28T19:02:56.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:56 smithi170 ceph-mon[36280]: pgmap v1671: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:56.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2171.141705 secs 2024-03-28T19:02:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:56 smithi037 ceph-mon[31460]: pgmap v1671: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2171.141705 secs 2024-03-28T19:02:58.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:02:58 smithi059 ceph-mon[36190]: pgmap v1672: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:58.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:02:58 smithi170 ceph-mon[36280]: pgmap v1672: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:58.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:02:58 smithi037 ceph-mon[31460]: pgmap v1672: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:00.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:00 smithi059 ceph-mon[36190]: pgmap v1673: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:00.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:00 smithi170 ceph-mon[36280]: pgmap v1673: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:00.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:00 smithi037 ceph-mon[31460]: pgmap v1673: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:01.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2176.141817 secs 2024-03-28T19:03:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2176.141817 secs 2024-03-28T19:03:01.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2176.141817 secs 2024-03-28T19:03:02.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:02 smithi059 ceph-mon[36190]: pgmap v1674: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:02.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:02 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:03:02.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:02 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:02.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:02 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:02.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:02 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:02.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:02 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:02.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:02 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:02.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:02 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:02.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:02 smithi170 ceph-mon[36280]: pgmap v1674: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:02.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:02 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:03:02.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:02 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:02.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:02 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:02.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:02 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:02.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:02 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:02.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:02 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:02.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:02 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:02.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:02 smithi037 ceph-mon[31460]: pgmap v1674: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:02.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:02 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:03:02.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:02 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:02.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:02 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:02.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:02 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:02.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:02 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:02.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:02 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:02.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:02 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:04.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:04 smithi059 ceph-mon[36190]: pgmap v1675: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:04.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:04 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:04.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:04 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:04.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:04 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:04.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:04 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:04.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:04 smithi170 ceph-mon[36280]: pgmap v1675: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:04.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:04 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:04.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:04 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:04.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:04 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:04.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:04 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:04.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:04 smithi037 ceph-mon[31460]: pgmap v1675: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:04.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:04 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:04.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:04 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:04.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:04 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:04.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:04 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:05.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:05 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:05 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:05 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:05 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:05 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:05 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:05 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:05 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:05 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:05 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:05 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:05 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:05 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:05 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:05 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:05 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:05 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:05 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:05 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:05 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:05 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:05 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:05 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:05 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:05 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:05 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:05 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:05 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:05 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:05 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:05 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:05 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:05 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:05 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:05 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:05 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:05 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:05 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:05 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:05 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:05 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:05 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:05 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:05 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:05 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:05 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:05 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:05.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:05 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:06.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:06 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T19:03:06.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:06 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T19:03:06.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:06 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T19:03:06.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:06 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T19:03:06.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:06 smithi037 ceph-mon[31460]: pgmap v1676: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:06.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2181.141906 secs 2024-03-28T19:03:06.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:06 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:06.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:06 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:06 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T19:03:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:06 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T19:03:06.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:06 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T19:03:06.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:06 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T19:03:06.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:06 smithi059 ceph-mon[36190]: pgmap v1676: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:06.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2181.141906 secs 2024-03-28T19:03:06.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:06 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:06.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:06 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:06.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:06 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T19:03:06.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:06 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T19:03:06.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:06 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T19:03:06.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:06 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T19:03:06.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:06 smithi170 ceph-mon[36280]: pgmap v1676: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:06.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2181.141906 secs 2024-03-28T19:03:06.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:06 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:06.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:06 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:07.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:07 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:07 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:07 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:07 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:07 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:07 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:07 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:07 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:07 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:03:07.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:07 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:03:07.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:07 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:07.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:07 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:07 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:07 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:07 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:07 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:07 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:07 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:07 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:07 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:03:07.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:07 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:03:07.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:07 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:07.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:07 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:07 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:07 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:07 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:07 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:07 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:07 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:07 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:03:07.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:07 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:03:07.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:07 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:03:07.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:07 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:03:08.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:08 smithi037 ceph-mon[31460]: pgmap v1677: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:08.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:08 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T19:03:08.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:08 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T19:03:08.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:08 smithi037 ceph-mon[31460]: pgmap v1678: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:03:08.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:08 smithi037 ceph-mon[31460]: pgmap v1679: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:03:08.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:08 smithi059 ceph-mon[36190]: pgmap v1677: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:08.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:08 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T19:03:08.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:08 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T19:03:08.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:08 smithi059 ceph-mon[36190]: pgmap v1678: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:03:08.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:08 smithi059 ceph-mon[36190]: pgmap v1679: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:03:08.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:08 smithi170 ceph-mon[36280]: pgmap v1677: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:08.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:08 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T19:03:08.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:08 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T19:03:08.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:08 smithi170 ceph-mon[36280]: pgmap v1678: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:03:08.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:08 smithi170 ceph-mon[36280]: pgmap v1679: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:03:10.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:10 smithi037 ceph-mon[31460]: pgmap v1680: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:03:10.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:10 smithi059 ceph-mon[36190]: pgmap v1680: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:03:10.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:10 smithi170 ceph-mon[36280]: pgmap v1680: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:03:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2186.142055 secs 2024-03-28T19:03:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2186.142055 secs 2024-03-28T19:03:11.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2186.142055 secs 2024-03-28T19:03:12.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:12 smithi037 ceph-mon[31460]: pgmap v1681: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:03:12.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:12 smithi059 ceph-mon[36190]: pgmap v1681: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:03:12.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:12 smithi170 ceph-mon[36280]: pgmap v1681: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:03:14.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:14 smithi037 ceph-mon[31460]: pgmap v1682: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:03:14.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:14 smithi059 ceph-mon[36190]: pgmap v1682: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:03:14.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:14 smithi170 ceph-mon[36280]: pgmap v1682: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:03:16.587 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:16 smithi059 ceph-mon[36190]: pgmap v1683: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:03:16.587 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2191.142173 secs 2024-03-28T19:03:16.587 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:16 smithi170 ceph-mon[36280]: pgmap v1683: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:03:16.587 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2191.142173 secs 2024-03-28T19:03:16.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:16 smithi037 ceph-mon[31460]: pgmap v1683: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:03:16.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2191.142173 secs 2024-03-28T19:03:18.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:18 smithi037 ceph-mon[31460]: pgmap v1684: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:03:18.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:18 smithi059 ceph-mon[36190]: pgmap v1684: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:03:18.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:18 smithi170 ceph-mon[36280]: pgmap v1684: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:03:20.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:20 smithi037 ceph-mon[31460]: pgmap v1685: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:20.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:20 smithi059 ceph-mon[36190]: pgmap v1685: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:20.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:20 smithi170 ceph-mon[36280]: pgmap v1685: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:21.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2196.142290 secs 2024-03-28T19:03:21.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2196.142290 secs 2024-03-28T19:03:21.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2196.142290 secs 2024-03-28T19:03:22.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:22 smithi037 ceph-mon[31460]: pgmap v1686: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:22.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:22 smithi059 ceph-mon[36190]: pgmap v1686: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:22.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:22 smithi170 ceph-mon[36280]: pgmap v1686: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:24.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:24 smithi037 ceph-mon[31460]: pgmap v1687: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:24.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:24 smithi059 ceph-mon[36190]: pgmap v1687: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:24.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:24 smithi170 ceph-mon[36280]: pgmap v1687: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:25.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:25 smithi037 ceph-mon[31460]: pgmap v1688: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:25.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:25 smithi059 ceph-mon[36190]: pgmap v1688: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:25.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:25 smithi170 ceph-mon[36280]: pgmap v1688: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:26.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2201.142375 secs 2024-03-28T19:03:26.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2201.142375 secs 2024-03-28T19:03:26.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2201.142375 secs 2024-03-28T19:03:27.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:27 smithi037 ceph-mon[31460]: pgmap v1689: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:27.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:27 smithi059 ceph-mon[36190]: pgmap v1689: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:27.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:27 smithi170 ceph-mon[36280]: pgmap v1689: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:30.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:30 smithi037 ceph-mon[31460]: pgmap v1690: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:30.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:30 smithi059 ceph-mon[36190]: pgmap v1690: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:30.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:30 smithi170 ceph-mon[36280]: pgmap v1690: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2206.142479 secs 2024-03-28T19:03:31.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2206.142479 secs 2024-03-28T19:03:31.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2206.142479 secs 2024-03-28T19:03:32.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:32 smithi037 ceph-mon[31460]: pgmap v1691: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:32.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:32 smithi059 ceph-mon[36190]: pgmap v1691: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:32.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:32 smithi170 ceph-mon[36280]: pgmap v1691: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:34.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:34 smithi037 ceph-mon[31460]: pgmap v1692: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:34.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:34 smithi059 ceph-mon[36190]: pgmap v1692: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:34.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:34 smithi170 ceph-mon[36280]: pgmap v1692: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:36.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:36 smithi037 ceph-mon[31460]: pgmap v1693: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:36.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2211.142579 secs 2024-03-28T19:03:36.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:36 smithi059 ceph-mon[36190]: pgmap v1693: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:36.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2211.142579 secs 2024-03-28T19:03:36.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:36 smithi170 ceph-mon[36280]: pgmap v1693: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:36.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2211.142579 secs 2024-03-28T19:03:37.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:37 smithi059 ceph-mon[36190]: pgmap v1694: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:37.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:37 smithi170 ceph-mon[36280]: pgmap v1694: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:37.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:37 smithi037 ceph-mon[31460]: pgmap v1694: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:40.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:40 smithi037 ceph-mon[31460]: pgmap v1695: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:40.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:40 smithi059 ceph-mon[36190]: pgmap v1695: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:40.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:40 smithi170 ceph-mon[36280]: pgmap v1695: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:41.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2216.142694 secs 2024-03-28T19:03:41.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2216.142694 secs 2024-03-28T19:03:41.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2216.142694 secs 2024-03-28T19:03:42.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:42 smithi037 ceph-mon[31460]: pgmap v1696: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:42.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:42 smithi059 ceph-mon[36190]: pgmap v1696: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:42.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:42 smithi170 ceph-mon[36280]: pgmap v1696: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:44.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:44 smithi037 ceph-mon[31460]: pgmap v1697: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:44.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:44 smithi059 ceph-mon[36190]: pgmap v1697: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:44.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:44 smithi170 ceph-mon[36280]: pgmap v1697: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:46.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:46 smithi037 ceph-mon[31460]: pgmap v1698: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:46.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2221.142808 secs 2024-03-28T19:03:46.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:46 smithi059 ceph-mon[36190]: pgmap v1698: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:46.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2221.142808 secs 2024-03-28T19:03:46.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:46 smithi170 ceph-mon[36280]: pgmap v1698: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:46.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2221.142808 secs 2024-03-28T19:03:47.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:47 smithi037 ceph-mon[31460]: pgmap v1699: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:47.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:47 smithi059 ceph-mon[36190]: pgmap v1699: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:47.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:47 smithi170 ceph-mon[36280]: pgmap v1699: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:50.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:50 smithi037 ceph-mon[31460]: pgmap v1700: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:50.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:50 smithi059 ceph-mon[36190]: pgmap v1700: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:50.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:50 smithi170 ceph-mon[36280]: pgmap v1700: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:51.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2226.142926 secs 2024-03-28T19:03:51.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2226.142926 secs 2024-03-28T19:03:51.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2226.142926 secs 2024-03-28T19:03:52.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:52 smithi037 ceph-mon[31460]: pgmap v1701: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:52.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:52 smithi059 ceph-mon[36190]: pgmap v1701: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:52.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:52 smithi170 ceph-mon[36280]: pgmap v1701: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:54.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:54 smithi037 ceph-mon[31460]: pgmap v1702: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:54.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:54 smithi059 ceph-mon[36190]: pgmap v1702: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:54.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:54 smithi170 ceph-mon[36280]: pgmap v1702: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:55.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:55 smithi037 ceph-mon[31460]: pgmap v1703: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:55.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:55 smithi059 ceph-mon[36190]: pgmap v1703: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:55.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:55 smithi170 ceph-mon[36280]: pgmap v1703: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2231.143161 secs 2024-03-28T19:03:56.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2231.143161 secs 2024-03-28T19:03:56.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2231.143161 secs 2024-03-28T19:03:57.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:03:57 smithi037 ceph-mon[31460]: pgmap v1704: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:57.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:03:57 smithi059 ceph-mon[36190]: pgmap v1704: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:57.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:03:57 smithi170 ceph-mon[36280]: pgmap v1704: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:00.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:00 smithi037 ceph-mon[31460]: pgmap v1705: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:00.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:00 smithi059 ceph-mon[36190]: pgmap v1705: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:00.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:00 smithi170 ceph-mon[36280]: pgmap v1705: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:01.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2236.143246 secs 2024-03-28T19:04:01.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2236.143246 secs 2024-03-28T19:04:01.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2236.143246 secs 2024-03-28T19:04:02.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:02 smithi037 ceph-mon[31460]: pgmap v1706: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:02.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:02 smithi059 ceph-mon[36190]: pgmap v1706: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:02.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:02 smithi170 ceph-mon[36280]: pgmap v1706: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:03.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:03 smithi037 ceph-mon[31460]: pgmap v1707: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:03.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:03 smithi059 ceph-mon[36190]: pgmap v1707: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:03.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:03 smithi170 ceph-mon[36280]: pgmap v1707: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:06.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:06 smithi037 ceph-mon[31460]: pgmap v1708: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:06.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2241.143327 secs 2024-03-28T19:04:06.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:06 smithi059 ceph-mon[36190]: pgmap v1708: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:06.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2241.143327 secs 2024-03-28T19:04:06.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:06 smithi170 ceph-mon[36280]: pgmap v1708: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:06.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2241.143327 secs 2024-03-28T19:04:08.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:08 smithi037 ceph-mon[31460]: pgmap v1709: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:08.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:08 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:04:08.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:08 smithi059 ceph-mon[36190]: pgmap v1709: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:08.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:08 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:04:08.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:08 smithi170 ceph-mon[36280]: pgmap v1709: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:08.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:08 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:04:09.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:09 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:04:09.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:09 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:04:09.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:09 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:04:09.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:09 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:04:09.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:09 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:04:09.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:09 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:04:09.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:09 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:04:09.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:09 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:04:09.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:09 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:04:10.619 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:10 smithi037 ceph-mon[31460]: pgmap v1710: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:10.620 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:10 smithi037 ceph-mon[31460]: pgmap v1711: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:10.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:10 smithi059 ceph-mon[36190]: pgmap v1710: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:10.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:10 smithi059 ceph-mon[36190]: pgmap v1711: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:10.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:10 smithi170 ceph-mon[36280]: pgmap v1710: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:10.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:10 smithi170 ceph-mon[36280]: pgmap v1711: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2246.143448 secs 2024-03-28T19:04:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:11 smithi037 ceph-mon[31460]: pgmap v1712: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2246.143448 secs 2024-03-28T19:04:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:11 smithi059 ceph-mon[36190]: pgmap v1712: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:11.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2246.143448 secs 2024-03-28T19:04:11.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:11 smithi170 ceph-mon[36280]: pgmap v1712: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:13 smithi037 ceph-mon[31460]: pgmap v1713: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:14.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:13 smithi059 ceph-mon[36190]: pgmap v1713: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:14.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:13 smithi170 ceph-mon[36280]: pgmap v1713: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:16.087 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:15 smithi170 ceph-mon[36280]: pgmap v1714: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:16.087 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:15 smithi059 ceph-mon[36190]: pgmap v1714: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:16.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:15 smithi037 ceph-mon[31460]: pgmap v1714: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:17.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2251.143549 secs 2024-03-28T19:04:17.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2251.143549 secs 2024-03-28T19:04:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2251.143549 secs 2024-03-28T19:04:18.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:17 smithi037 ceph-mon[31460]: pgmap v1715: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:18.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:17 smithi059 ceph-mon[36190]: pgmap v1715: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:18.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:17 smithi170 ceph-mon[36280]: pgmap v1715: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:20.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:19 smithi037 ceph-mon[31460]: pgmap v1716: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:04:20.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:19 smithi059 ceph-mon[36190]: pgmap v1716: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:04:20.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:19 smithi170 ceph-mon[36280]: pgmap v1716: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:04:22.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2256.143650 secs 2024-03-28T19:04:22.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:21 smithi037 ceph-mon[31460]: pgmap v1717: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:22.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2256.143650 secs 2024-03-28T19:04:22.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:21 smithi059 ceph-mon[36190]: pgmap v1717: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2256.143650 secs 2024-03-28T19:04:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:21 smithi170 ceph-mon[36280]: pgmap v1717: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:24.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:23 smithi037 ceph-mon[31460]: pgmap v1718: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:24.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:23 smithi059 ceph-mon[36190]: pgmap v1718: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:24.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:23 smithi170 ceph-mon[36280]: pgmap v1718: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:26.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:25 smithi037 ceph-mon[31460]: pgmap v1719: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:26.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:25 smithi059 ceph-mon[36190]: pgmap v1719: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:25 smithi170 ceph-mon[36280]: pgmap v1719: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:27.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2261.143768 secs 2024-03-28T19:04:27.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2261.143768 secs 2024-03-28T19:04:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2261.143768 secs 2024-03-28T19:04:28.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:27 smithi037 ceph-mon[31460]: pgmap v1720: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:28.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:27 smithi059 ceph-mon[36190]: pgmap v1720: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:28.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:27 smithi170 ceph-mon[36280]: pgmap v1720: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:30.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:29 smithi037 ceph-mon[31460]: pgmap v1721: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:30.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:29 smithi059 ceph-mon[36190]: pgmap v1721: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:30.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:29 smithi170 ceph-mon[36280]: pgmap v1721: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:32.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2266.143888 secs 2024-03-28T19:04:32.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:31 smithi037 ceph-mon[31460]: pgmap v1722: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:32.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2266.143888 secs 2024-03-28T19:04:32.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:31 smithi059 ceph-mon[36190]: pgmap v1722: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2266.143888 secs 2024-03-28T19:04:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:31 smithi170 ceph-mon[36280]: pgmap v1722: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:34.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:33 smithi037 ceph-mon[31460]: pgmap v1723: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:34.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:33 smithi059 ceph-mon[36190]: pgmap v1723: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:34.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:33 smithi170 ceph-mon[36280]: pgmap v1723: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:36.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:35 smithi059 ceph-mon[36190]: pgmap v1724: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:36.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:35 smithi170 ceph-mon[36280]: pgmap v1724: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:36.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:35 smithi037 ceph-mon[31460]: pgmap v1724: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:37.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2271.144010 secs 2024-03-28T19:04:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2271.144010 secs 2024-03-28T19:04:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2271.144010 secs 2024-03-28T19:04:38.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:37 smithi059 ceph-mon[36190]: pgmap v1725: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:38.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:37 smithi170 ceph-mon[36280]: pgmap v1725: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:38.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:37 smithi037 ceph-mon[31460]: pgmap v1725: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:40.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:39 smithi059 ceph-mon[36190]: pgmap v1726: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:40.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:39 smithi170 ceph-mon[36280]: pgmap v1726: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:40.376 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:39 smithi037 ceph-mon[31460]: pgmap v1726: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:42.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2276.144130 secs 2024-03-28T19:04:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:41 smithi059 ceph-mon[36190]: pgmap v1727: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:42.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2276.144130 secs 2024-03-28T19:04:42.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:41 smithi170 ceph-mon[36280]: pgmap v1727: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:42.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2276.144130 secs 2024-03-28T19:04:42.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:41 smithi037 ceph-mon[31460]: pgmap v1727: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:44.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:43 smithi059 ceph-mon[36190]: pgmap v1728: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:44.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:43 smithi170 ceph-mon[36280]: pgmap v1728: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:44.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:43 smithi037 ceph-mon[31460]: pgmap v1728: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:46.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:45 smithi059 ceph-mon[36190]: pgmap v1729: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:45 smithi170 ceph-mon[36280]: pgmap v1729: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:46.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:45 smithi037 ceph-mon[31460]: pgmap v1729: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:47.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2281.144250 secs 2024-03-28T19:04:47.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2281.144250 secs 2024-03-28T19:04:47.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2281.144250 secs 2024-03-28T19:04:48.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:47 smithi059 ceph-mon[36190]: pgmap v1730: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:48.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:47 smithi170 ceph-mon[36280]: pgmap v1730: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:48.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:47 smithi037 ceph-mon[31460]: pgmap v1730: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:50.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:49 smithi059 ceph-mon[36190]: pgmap v1731: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:50.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:49 smithi170 ceph-mon[36280]: pgmap v1731: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:50.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:49 smithi037 ceph-mon[31460]: pgmap v1731: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:52.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2286.144344 secs 2024-03-28T19:04:52.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:51 smithi059 ceph-mon[36190]: pgmap v1732: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2286.144344 secs 2024-03-28T19:04:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:51 smithi170 ceph-mon[36280]: pgmap v1732: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:52.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2286.144344 secs 2024-03-28T19:04:52.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:51 smithi037 ceph-mon[31460]: pgmap v1732: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:54.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:53 smithi059 ceph-mon[36190]: pgmap v1733: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:54.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:53 smithi170 ceph-mon[36280]: pgmap v1733: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:54.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:53 smithi037 ceph-mon[31460]: pgmap v1733: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:56.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:55 smithi059 ceph-mon[36190]: pgmap v1734: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:56.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:55 smithi170 ceph-mon[36280]: pgmap v1734: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:56.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:55 smithi037 ceph-mon[31460]: pgmap v1734: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:57.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2291.144412 secs 2024-03-28T19:04:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2291.144412 secs 2024-03-28T19:04:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2291.144412 secs 2024-03-28T19:04:58.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:57 smithi059 ceph-mon[36190]: pgmap v1735: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:58.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:57 smithi170 ceph-mon[36280]: pgmap v1735: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:58.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:57 smithi037 ceph-mon[31460]: pgmap v1735: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:00.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:04:59 smithi059 ceph-mon[36190]: pgmap v1736: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:00.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:04:59 smithi170 ceph-mon[36280]: pgmap v1736: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:00.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:04:59 smithi037 ceph-mon[31460]: pgmap v1736: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:02.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2296.144483 secs 2024-03-28T19:05:02.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:01 smithi059 ceph-mon[36190]: pgmap v1737: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:02.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2296.144483 secs 2024-03-28T19:05:02.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:01 smithi170 ceph-mon[36280]: pgmap v1737: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:02.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2296.144483 secs 2024-03-28T19:05:02.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:01 smithi037 ceph-mon[31460]: pgmap v1737: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:04.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:03 smithi059 ceph-mon[36190]: pgmap v1738: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:04.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:03 smithi170 ceph-mon[36280]: pgmap v1738: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:04.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:03 smithi037 ceph-mon[31460]: pgmap v1738: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:06.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:05 smithi059 ceph-mon[36190]: pgmap v1739: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:06.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:05 smithi170 ceph-mon[36280]: pgmap v1739: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:06.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:05 smithi037 ceph-mon[31460]: pgmap v1739: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:07.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2301.144590 secs 2024-03-28T19:05:07.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2301.144590 secs 2024-03-28T19:05:07.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2301.144590 secs 2024-03-28T19:05:08.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:07 smithi059 ceph-mon[36190]: pgmap v1740: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:08.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:07 smithi170 ceph-mon[36280]: pgmap v1740: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:08.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:07 smithi037 ceph-mon[31460]: pgmap v1740: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:10.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:09 smithi059 ceph-mon[36190]: pgmap v1741: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:10.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:09 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:05:10.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:09 smithi170 ceph-mon[36280]: pgmap v1741: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:10.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:09 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:05:10.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:09 smithi037 ceph-mon[31460]: pgmap v1741: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:10.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:09 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:05:11.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:05:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:10 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:05:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:10 smithi059 ceph-mon[36190]: pgmap v1742: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:10 smithi059 ceph-mon[36190]: pgmap v1743: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:11.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:10 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:05:11.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:05:11.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:10 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:05:11.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:10 smithi170 ceph-mon[36280]: pgmap v1742: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:11.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:10 smithi170 ceph-mon[36280]: pgmap v1743: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:11.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:10 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:05:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:05:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:10 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:05:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:10 smithi037 ceph-mon[31460]: pgmap v1742: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:10 smithi037 ceph-mon[31460]: pgmap v1743: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:10 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:05:12.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2306.144703 secs 2024-03-28T19:05:12.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2306.144703 secs 2024-03-28T19:05:12.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2306.144703 secs 2024-03-28T19:05:13.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:13 smithi059 ceph-mon[36190]: pgmap v1744: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:13.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:13 smithi170 ceph-mon[36280]: pgmap v1744: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:13.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:13 smithi037 ceph-mon[31460]: pgmap v1744: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:15.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:15 smithi059 ceph-mon[36190]: pgmap v1745: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:15.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:15 smithi170 ceph-mon[36280]: pgmap v1745: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:15.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:15 smithi037 ceph-mon[31460]: pgmap v1745: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:16.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2311.144823 secs 2024-03-28T19:05:16.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2311.144823 secs 2024-03-28T19:05:16.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2311.144823 secs 2024-03-28T19:05:17.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:17 smithi059 ceph-mon[36190]: pgmap v1746: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:05:17.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:17 smithi170 ceph-mon[36280]: pgmap v1746: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:05:17.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:17 smithi037 ceph-mon[31460]: pgmap v1746: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:05:19.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:19 smithi170 ceph-mon[36280]: pgmap v1747: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:19.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:19 smithi037 ceph-mon[31460]: pgmap v1747: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:19.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:19 smithi059 ceph-mon[36190]: pgmap v1747: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:21.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:21 smithi037 ceph-mon[31460]: pgmap v1748: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:05:21.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2316.144941 secs 2024-03-28T19:05:21.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:21 smithi059 ceph-mon[36190]: pgmap v1748: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:05:21.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2316.144941 secs 2024-03-28T19:05:21.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:21 smithi170 ceph-mon[36280]: pgmap v1748: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:05:21.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2316.144941 secs 2024-03-28T19:05:23.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:23 smithi037 ceph-mon[31460]: pgmap v1749: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:23.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:23 smithi059 ceph-mon[36190]: pgmap v1749: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:23.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:23 smithi170 ceph-mon[36280]: pgmap v1749: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:25.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:25 smithi037 ceph-mon[31460]: pgmap v1750: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:25.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:25 smithi059 ceph-mon[36190]: pgmap v1750: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:25.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:25 smithi170 ceph-mon[36280]: pgmap v1750: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:26.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2321.145058 secs 2024-03-28T19:05:26.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2321.145058 secs 2024-03-28T19:05:26.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2321.145058 secs 2024-03-28T19:05:27.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:27 smithi037 ceph-mon[31460]: pgmap v1751: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:27.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:27 smithi059 ceph-mon[36190]: pgmap v1751: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:27.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:27 smithi170 ceph-mon[36280]: pgmap v1751: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:29.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:29 smithi037 ceph-mon[31460]: pgmap v1752: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:29.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:29 smithi059 ceph-mon[36190]: pgmap v1752: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:29.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:29 smithi170 ceph-mon[36280]: pgmap v1752: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:31 smithi037 ceph-mon[31460]: pgmap v1753: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2326.145178 secs 2024-03-28T19:05:31.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:31 smithi059 ceph-mon[36190]: pgmap v1753: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:31.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2326.145178 secs 2024-03-28T19:05:31.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:31 smithi170 ceph-mon[36280]: pgmap v1753: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:31.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2326.145178 secs 2024-03-28T19:05:33.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:33 smithi037 ceph-mon[31460]: pgmap v1754: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:33.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:33 smithi059 ceph-mon[36190]: pgmap v1754: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:33.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:33 smithi170 ceph-mon[36280]: pgmap v1754: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:35.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:35 smithi037 ceph-mon[31460]: pgmap v1755: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:35.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:35 smithi059 ceph-mon[36190]: pgmap v1755: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:35.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:35 smithi170 ceph-mon[36280]: pgmap v1755: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:36.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2331.145298 secs 2024-03-28T19:05:36.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2331.145298 secs 2024-03-28T19:05:36.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2331.145298 secs 2024-03-28T19:05:37.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:37 smithi037 ceph-mon[31460]: pgmap v1756: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:37.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:37 smithi059 ceph-mon[36190]: pgmap v1756: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:37.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:37 smithi170 ceph-mon[36280]: pgmap v1756: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:39.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:39 smithi037 ceph-mon[31460]: pgmap v1757: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:39.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:39 smithi059 ceph-mon[36190]: pgmap v1757: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:39.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:39 smithi170 ceph-mon[36280]: pgmap v1757: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:41.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:41 smithi037 ceph-mon[31460]: pgmap v1758: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:41.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2336.145390 secs 2024-03-28T19:05:41.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:41 smithi059 ceph-mon[36190]: pgmap v1758: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:41.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2336.145390 secs 2024-03-28T19:05:41.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:41 smithi170 ceph-mon[36280]: pgmap v1758: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:41.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2336.145390 secs 2024-03-28T19:05:43.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:43 smithi037 ceph-mon[31460]: pgmap v1759: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:43.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:43 smithi059 ceph-mon[36190]: pgmap v1759: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:43.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:43 smithi170 ceph-mon[36280]: pgmap v1759: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:45.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:45 smithi037 ceph-mon[31460]: pgmap v1760: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:45.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:45 smithi059 ceph-mon[36190]: pgmap v1760: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:45.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:45 smithi170 ceph-mon[36280]: pgmap v1760: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:46.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2341.145511 secs 2024-03-28T19:05:46.686 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2341.145511 secs 2024-03-28T19:05:46.686 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2341.145511 secs 2024-03-28T19:05:47.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:47 smithi037 ceph-mon[31460]: pgmap v1761: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:47.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:47 smithi059 ceph-mon[36190]: pgmap v1761: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:47.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:47 smithi170 ceph-mon[36280]: pgmap v1761: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:49.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:49 smithi037 ceph-mon[31460]: pgmap v1762: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:49.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:49 smithi059 ceph-mon[36190]: pgmap v1762: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:49.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:49 smithi170 ceph-mon[36280]: pgmap v1762: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:50.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:50 smithi059 ceph-mon[36190]: pgmap v1763: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:50.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:50 smithi170 ceph-mon[36280]: pgmap v1763: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:50.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:50 smithi037 ceph-mon[31460]: pgmap v1763: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:51.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2346.145631 secs 2024-03-28T19:05:51.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2346.145631 secs 2024-03-28T19:05:51.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2346.145631 secs 2024-03-28T19:05:52.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:52 smithi059 ceph-mon[36190]: pgmap v1764: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:52.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:52 smithi170 ceph-mon[36280]: pgmap v1764: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:52.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:52 smithi037 ceph-mon[31460]: pgmap v1764: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:55.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:55 smithi170 ceph-mon[36280]: pgmap v1765: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:55.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:55 smithi037 ceph-mon[31460]: pgmap v1765: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:55.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:55 smithi059 ceph-mon[36190]: pgmap v1765: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2351.145752 secs 2024-03-28T19:05:56.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2351.145752 secs 2024-03-28T19:05:56.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2351.145752 secs 2024-03-28T19:05:57.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:57 smithi037 ceph-mon[31460]: pgmap v1766: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:57.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:57 smithi059 ceph-mon[36190]: pgmap v1766: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:57.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:57 smithi170 ceph-mon[36280]: pgmap v1766: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:59.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:05:59 smithi037 ceph-mon[31460]: pgmap v1767: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:59.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:05:59 smithi059 ceph-mon[36190]: pgmap v1767: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:59.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:05:59 smithi170 ceph-mon[36280]: pgmap v1767: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:01.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:01 smithi037 ceph-mon[31460]: pgmap v1768: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:01.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2356.145870 secs 2024-03-28T19:06:01.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:01 smithi059 ceph-mon[36190]: pgmap v1768: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:01.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2356.145870 secs 2024-03-28T19:06:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:01 smithi170 ceph-mon[36280]: pgmap v1768: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2356.145870 secs 2024-03-28T19:06:03.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:03 smithi037 ceph-mon[31460]: pgmap v1769: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:03.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:03 smithi059 ceph-mon[36190]: pgmap v1769: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:03.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:03 smithi170 ceph-mon[36280]: pgmap v1769: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:05.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:05 smithi037 ceph-mon[31460]: pgmap v1770: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:05.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:05 smithi059 ceph-mon[36190]: pgmap v1770: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:05.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:05 smithi170 ceph-mon[36280]: pgmap v1770: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:06.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2361.145993 secs 2024-03-28T19:06:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2361.145993 secs 2024-03-28T19:06:06.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2361.145993 secs 2024-03-28T19:06:07.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:07 smithi037 ceph-mon[31460]: pgmap v1771: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:07.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:07 smithi059 ceph-mon[36190]: pgmap v1771: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:07.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:07 smithi170 ceph-mon[36280]: pgmap v1771: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:09.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:09 smithi037 ceph-mon[31460]: pgmap v1772: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:09.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:09 smithi059 ceph-mon[36190]: pgmap v1772: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:09.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:09 smithi170 ceph-mon[36280]: pgmap v1772: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:11 smithi037 ceph-mon[31460]: pgmap v1773: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:11 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:06:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2366.146079 secs 2024-03-28T19:06:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:11 smithi059 ceph-mon[36190]: pgmap v1773: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:11 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:06:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2366.146079 secs 2024-03-28T19:06:11.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:11 smithi170 ceph-mon[36280]: pgmap v1773: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:11.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:11 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:06:11.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2366.146079 secs 2024-03-28T19:06:12.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:12 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:06:12.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:12 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:06:12.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:12 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:06:12.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:12 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:06:12.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:12 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:06:12.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:12 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:06:12.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:12 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:06:12.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:12 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:06:12.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:12 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:06:13.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:13 smithi037 ceph-mon[31460]: pgmap v1774: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:06:13.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:13 smithi037 ceph-mon[31460]: pgmap v1775: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:13.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:13 smithi059 ceph-mon[36190]: pgmap v1774: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:06:13.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:13 smithi059 ceph-mon[36190]: pgmap v1775: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:13.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:13 smithi170 ceph-mon[36280]: pgmap v1774: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:06:13.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:13 smithi170 ceph-mon[36280]: pgmap v1775: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:15.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:15 smithi037 ceph-mon[31460]: pgmap v1776: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:15.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:15 smithi059 ceph-mon[36190]: pgmap v1776: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:15.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:15 smithi170 ceph-mon[36280]: pgmap v1776: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:16.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2371.146164 secs 2024-03-28T19:06:16.685 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2371.146164 secs 2024-03-28T19:06:16.686 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2371.146164 secs 2024-03-28T19:06:17.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:17 smithi037 ceph-mon[31460]: pgmap v1777: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:17.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:17 smithi059 ceph-mon[36190]: pgmap v1777: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:17.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:17 smithi170 ceph-mon[36280]: pgmap v1777: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:19.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:19 smithi037 ceph-mon[31460]: pgmap v1778: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:19.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:19 smithi059 ceph-mon[36190]: pgmap v1778: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:19.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:19 smithi170 ceph-mon[36280]: pgmap v1778: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:21.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:21 smithi037 ceph-mon[31460]: pgmap v1779: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:21.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2376.146263 secs 2024-03-28T19:06:21.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:21 smithi059 ceph-mon[36190]: pgmap v1779: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:21.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2376.146263 secs 2024-03-28T19:06:21.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:21 smithi170 ceph-mon[36280]: pgmap v1779: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:21.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2376.146263 secs 2024-03-28T19:06:23.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:23 smithi037 ceph-mon[31460]: pgmap v1780: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:06:23.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:23 smithi059 ceph-mon[36190]: pgmap v1780: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:06:23.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:23 smithi170 ceph-mon[36280]: pgmap v1780: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:06:25.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:25 smithi037 ceph-mon[31460]: pgmap v1781: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:25.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:25 smithi059 ceph-mon[36190]: pgmap v1781: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:25.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:25 smithi170 ceph-mon[36280]: pgmap v1781: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:26.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2381.146384 secs 2024-03-28T19:06:26.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2381.146384 secs 2024-03-28T19:06:26.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2381.146384 secs 2024-03-28T19:06:27.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:27 smithi059 ceph-mon[36190]: pgmap v1782: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:27.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:27 smithi170 ceph-mon[36280]: pgmap v1782: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:27.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:27 smithi037 ceph-mon[31460]: pgmap v1782: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:28.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:28 smithi059 ceph-mon[36190]: pgmap v1783: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:28.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:28 smithi170 ceph-mon[36280]: pgmap v1783: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:28.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:28 smithi037 ceph-mon[31460]: pgmap v1783: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:31.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:30 smithi059 ceph-mon[36190]: pgmap v1784: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:31.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:30 smithi170 ceph-mon[36280]: pgmap v1784: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:31.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:30 smithi037 ceph-mon[31460]: pgmap v1784: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:32.027 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2386.146506 secs 2024-03-28T19:06:32.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2386.146506 secs 2024-03-28T19:06:32.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2386.146506 secs 2024-03-28T19:06:33.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:32 smithi059 ceph-mon[36190]: pgmap v1785: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:33.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:32 smithi170 ceph-mon[36280]: pgmap v1785: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:33.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:32 smithi037 ceph-mon[31460]: pgmap v1785: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:35.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:34 smithi059 ceph-mon[36190]: pgmap v1786: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:35.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:34 smithi170 ceph-mon[36280]: pgmap v1786: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:35.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:34 smithi037 ceph-mon[31460]: pgmap v1786: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:37.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2391.146627 secs 2024-03-28T19:06:37.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:36 smithi037 ceph-mon[31460]: pgmap v1787: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:37.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2391.146627 secs 2024-03-28T19:06:37.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:36 smithi059 ceph-mon[36190]: pgmap v1787: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2391.146627 secs 2024-03-28T19:06:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:36 smithi170 ceph-mon[36280]: pgmap v1787: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:39.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:38 smithi037 ceph-mon[31460]: pgmap v1788: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:39.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:38 smithi059 ceph-mon[36190]: pgmap v1788: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:39.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:38 smithi170 ceph-mon[36280]: pgmap v1788: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:41.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:40 smithi037 ceph-mon[31460]: pgmap v1789: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:41.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:40 smithi059 ceph-mon[36190]: pgmap v1789: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:41.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:40 smithi170 ceph-mon[36280]: pgmap v1789: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:42.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2396.146747 secs 2024-03-28T19:06:42.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2396.146747 secs 2024-03-28T19:06:42.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2396.146747 secs 2024-03-28T19:06:43.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:42 smithi037 ceph-mon[31460]: pgmap v1790: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:43.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:42 smithi059 ceph-mon[36190]: pgmap v1790: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:43.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:42 smithi170 ceph-mon[36280]: pgmap v1790: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:45.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:44 smithi037 ceph-mon[31460]: pgmap v1791: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:45.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:44 smithi059 ceph-mon[36190]: pgmap v1791: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:45.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:44 smithi170 ceph-mon[36280]: pgmap v1791: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:47.087 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2401.146869 secs 2024-03-28T19:06:47.087 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:46 smithi059 ceph-mon[36190]: pgmap v1792: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:47.087 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2401.146869 secs 2024-03-28T19:06:47.087 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:46 smithi170 ceph-mon[36280]: pgmap v1792: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2401.146869 secs 2024-03-28T19:06:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:46 smithi037 ceph-mon[31460]: pgmap v1792: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:49.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:48 smithi037 ceph-mon[31460]: pgmap v1793: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:49.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:48 smithi059 ceph-mon[36190]: pgmap v1793: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:49.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:48 smithi170 ceph-mon[36280]: pgmap v1793: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:51.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:50 smithi037 ceph-mon[31460]: pgmap v1794: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:51.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:50 smithi059 ceph-mon[36190]: pgmap v1794: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:50 smithi170 ceph-mon[36280]: pgmap v1794: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2406.146989 secs 2024-03-28T19:06:52.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2406.146989 secs 2024-03-28T19:06:52.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2406.146989 secs 2024-03-28T19:06:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:52 smithi037 ceph-mon[31460]: pgmap v1795: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:53.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:52 smithi059 ceph-mon[36190]: pgmap v1795: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:53.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:52 smithi170 ceph-mon[36280]: pgmap v1795: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:55.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:54 smithi037 ceph-mon[31460]: pgmap v1796: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:55.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:54 smithi059 ceph-mon[36190]: pgmap v1796: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:55.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:54 smithi170 ceph-mon[36280]: pgmap v1796: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2411.147109 secs 2024-03-28T19:06:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:56 smithi037 ceph-mon[31460]: pgmap v1797: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2411.147109 secs 2024-03-28T19:06:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:56 smithi059 ceph-mon[36190]: pgmap v1797: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:57.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2411.147109 secs 2024-03-28T19:06:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:56 smithi170 ceph-mon[36280]: pgmap v1797: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:59.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:06:58 smithi037 ceph-mon[31460]: pgmap v1798: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:59.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:06:58 smithi059 ceph-mon[36190]: pgmap v1798: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:59.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:06:58 smithi170 ceph-mon[36280]: pgmap v1798: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:01.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:00 smithi037 ceph-mon[31460]: pgmap v1799: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:01.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:00 smithi059 ceph-mon[36190]: pgmap v1799: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:00 smithi170 ceph-mon[36280]: pgmap v1799: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:02.025 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2416.147176 secs 2024-03-28T19:07:02.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2416.147176 secs 2024-03-28T19:07:02.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2416.147176 secs 2024-03-28T19:07:03.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:02 smithi037 ceph-mon[31460]: pgmap v1800: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:03.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:02 smithi059 ceph-mon[36190]: pgmap v1800: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:03.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:02 smithi170 ceph-mon[36280]: pgmap v1800: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:05.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:04 smithi037 ceph-mon[31460]: pgmap v1801: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:05.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:04 smithi059 ceph-mon[36190]: pgmap v1801: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:05.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:04 smithi170 ceph-mon[36280]: pgmap v1801: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2421.147244 secs 2024-03-28T19:07:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:06 smithi037 ceph-mon[31460]: pgmap v1802: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:07.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2421.147244 secs 2024-03-28T19:07:07.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:06 smithi059 ceph-mon[36190]: pgmap v1802: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:07.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2421.147244 secs 2024-03-28T19:07:07.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:06 smithi170 ceph-mon[36280]: pgmap v1802: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:09.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:08 smithi059 ceph-mon[36190]: pgmap v1803: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:09.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:08 smithi170 ceph-mon[36280]: pgmap v1803: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:09.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:08 smithi037 ceph-mon[31460]: pgmap v1803: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:11 smithi037 ceph-mon[31460]: pgmap v1804: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:11.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2426.147309 secs 2024-03-28T19:07:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:11 smithi059 ceph-mon[36190]: pgmap v1804: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2426.147309 secs 2024-03-28T19:07:11.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:11 smithi170 ceph-mon[36280]: pgmap v1804: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:11.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2426.147309 secs 2024-03-28T19:07:12.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:12 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:07:12.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:12 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:07:12.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:12 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:07:13.615 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:13 smithi059 ceph-mon[36190]: pgmap v1805: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:13.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:13 smithi037 ceph-mon[31460]: pgmap v1805: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:13.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:13 smithi170 ceph-mon[36280]: pgmap v1805: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:15.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:15 smithi037 ceph-mon[31460]: pgmap v1806: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:15.670 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:15 smithi059 ceph-mon[36190]: pgmap v1806: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:15.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:15 smithi170 ceph-mon[36280]: pgmap v1806: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:16.475 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2431.147408 secs 2024-03-28T19:07:16.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2431.147408 secs 2024-03-28T19:07:16.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2431.147408 secs 2024-03-28T19:07:17.656 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:17 smithi037 ceph-mon[31460]: pgmap v1807: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:17.656 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:17 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:07:17.656 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:17 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:07:17.686 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:17 smithi170 ceph-mon[36280]: pgmap v1807: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:17.686 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:17 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:07:17.686 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:17 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:07:17.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:17 smithi059 ceph-mon[36190]: pgmap v1807: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:17.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:17 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:07:17.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:17 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:07:19.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:19 smithi059 ceph-mon[36190]: pgmap v1808: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:19.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:19 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:07:19.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:19 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:07:19.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:19 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:07:19.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:19 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:07:19.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:19 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:07:19.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:19 smithi170 ceph-mon[36280]: pgmap v1808: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:19.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:19 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:07:19.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:19 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:07:19.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:19 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:07:19.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:19 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:07:19.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:19 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:07:19.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:19 smithi037 ceph-mon[31460]: pgmap v1808: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:19.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:19 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:07:19.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:19 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:07:19.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:19 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:07:19.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:19 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:07:19.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:19 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:07:20.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:20 smithi059 ceph-mon[36190]: pgmap v1809: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:07:20.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:20 smithi059 ceph-mon[36190]: pgmap v1810: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:07:20.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:20 smithi170 ceph-mon[36280]: pgmap v1809: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:07:20.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:20 smithi170 ceph-mon[36280]: pgmap v1810: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:07:20.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:20 smithi037 ceph-mon[31460]: pgmap v1809: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:07:20.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:20 smithi037 ceph-mon[31460]: pgmap v1810: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:07:21.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:21 smithi059 ceph-mon[36190]: pgmap v1811: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:07:21.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2436.147525 secs 2024-03-28T19:07:21.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:21 smithi170 ceph-mon[36280]: pgmap v1811: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:07:21.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2436.147525 secs 2024-03-28T19:07:21.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:21 smithi037 ceph-mon[31460]: pgmap v1811: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:07:21.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2436.147525 secs 2024-03-28T19:07:23.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:23 smithi037 ceph-mon[31460]: pgmap v1812: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:07:24.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:23 smithi059 ceph-mon[36190]: pgmap v1812: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:07:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:23 smithi170 ceph-mon[36280]: pgmap v1812: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:07:25.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:25 smithi037 ceph-mon[31460]: pgmap v1813: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:07:26.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:25 smithi059 ceph-mon[36190]: pgmap v1813: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:07:26.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:25 smithi170 ceph-mon[36280]: pgmap v1813: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:07:26.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2441.147628 secs 2024-03-28T19:07:26.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2441.147628 secs 2024-03-28T19:07:27.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2441.147628 secs 2024-03-28T19:07:27.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:27 smithi037 ceph-mon[31460]: pgmap v1814: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:07:28.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:27 smithi059 ceph-mon[36190]: pgmap v1814: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:07:28.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:27 smithi170 ceph-mon[36280]: pgmap v1814: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:07:29.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:29 smithi037 ceph-mon[31460]: pgmap v1815: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:07:30.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:29 smithi059 ceph-mon[36190]: pgmap v1815: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:07:30.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:29 smithi170 ceph-mon[36280]: pgmap v1815: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:07:31.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:31 smithi037 ceph-mon[31460]: pgmap v1816: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2446.147756 secs 2024-03-28T19:07:32.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:31 smithi059 ceph-mon[36190]: pgmap v1816: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:32.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2446.147756 secs 2024-03-28T19:07:32.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:31 smithi170 ceph-mon[36280]: pgmap v1816: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:32.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2446.147756 secs 2024-03-28T19:07:33.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:33 smithi037 ceph-mon[31460]: pgmap v1817: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:34.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:33 smithi059 ceph-mon[36190]: pgmap v1817: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:33 smithi170 ceph-mon[36280]: pgmap v1817: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:35.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:35 smithi037 ceph-mon[31460]: pgmap v1818: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:36.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:35 smithi059 ceph-mon[36190]: pgmap v1818: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:36.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:35 smithi170 ceph-mon[36280]: pgmap v1818: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:37.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2451.147877 secs 2024-03-28T19:07:37.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2451.147877 secs 2024-03-28T19:07:37.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2451.147877 secs 2024-03-28T19:07:38.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:37 smithi059 ceph-mon[36190]: pgmap v1819: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:38.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:37 smithi170 ceph-mon[36280]: pgmap v1819: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:38.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:37 smithi037 ceph-mon[31460]: pgmap v1819: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:40.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:39 smithi059 ceph-mon[36190]: pgmap v1820: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:40.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:39 smithi170 ceph-mon[36280]: pgmap v1820: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:40.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:39 smithi037 ceph-mon[31460]: pgmap v1820: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:42.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:41 smithi059 ceph-mon[36190]: pgmap v1821: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:42.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2456.147994 secs 2024-03-28T19:07:42.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:41 smithi170 ceph-mon[36280]: pgmap v1821: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:42.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2456.147994 secs 2024-03-28T19:07:42.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:41 smithi037 ceph-mon[31460]: pgmap v1821: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:42.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2456.147994 secs 2024-03-28T19:07:44.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:43 smithi059 ceph-mon[36190]: pgmap v1822: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:44.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:43 smithi170 ceph-mon[36280]: pgmap v1822: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:44.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:43 smithi037 ceph-mon[31460]: pgmap v1822: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:46.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:45 smithi059 ceph-mon[36190]: pgmap v1823: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:46.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:45 smithi170 ceph-mon[36280]: pgmap v1823: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:46.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:45 smithi037 ceph-mon[31460]: pgmap v1823: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:47.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2461.148114 secs 2024-03-28T19:07:47.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2461.148114 secs 2024-03-28T19:07:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2461.148114 secs 2024-03-28T19:07:47.935 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:47 smithi059 ceph-mon[36190]: pgmap v1824: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:47.936 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:47 smithi170 ceph-mon[36280]: pgmap v1824: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:48.025 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:47 smithi037 ceph-mon[31460]: pgmap v1824: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:50.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:49 smithi059 ceph-mon[36190]: pgmap v1825: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:49 smithi170 ceph-mon[36280]: pgmap v1825: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:49 smithi037 ceph-mon[31460]: pgmap v1825: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:52.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:51 smithi059 ceph-mon[36190]: pgmap v1826: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:52.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2466.148233 secs 2024-03-28T19:07:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:51 smithi170 ceph-mon[36280]: pgmap v1826: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2466.148233 secs 2024-03-28T19:07:52.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:51 smithi037 ceph-mon[31460]: pgmap v1826: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:52.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2466.148233 secs 2024-03-28T19:07:54.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:53 smithi059 ceph-mon[36190]: pgmap v1827: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:54.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:53 smithi170 ceph-mon[36280]: pgmap v1827: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:54.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:53 smithi037 ceph-mon[31460]: pgmap v1827: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:56.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:55 smithi059 ceph-mon[36190]: pgmap v1828: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:56.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:55 smithi170 ceph-mon[36280]: pgmap v1828: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:56.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:55 smithi037 ceph-mon[31460]: pgmap v1828: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:57.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2471.148320 secs 2024-03-28T19:07:57.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2471.148320 secs 2024-03-28T19:07:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2471.148320 secs 2024-03-28T19:07:58.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:57 smithi059 ceph-mon[36190]: pgmap v1829: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:58.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:57 smithi170 ceph-mon[36280]: pgmap v1829: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:58.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:57 smithi037 ceph-mon[31460]: pgmap v1829: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:00.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:07:59 smithi059 ceph-mon[36190]: pgmap v1830: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:00.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:07:59 smithi170 ceph-mon[36280]: pgmap v1830: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:00.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:07:59 smithi037 ceph-mon[31460]: pgmap v1830: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:02.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:01 smithi059 ceph-mon[36190]: pgmap v1831: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:02.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2476.148441 secs 2024-03-28T19:08:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:01 smithi170 ceph-mon[36280]: pgmap v1831: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:02.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2476.148441 secs 2024-03-28T19:08:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:01 smithi037 ceph-mon[31460]: pgmap v1831: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2476.148441 secs 2024-03-28T19:08:03.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:03 smithi037 ceph-mon[31460]: pgmap v1832: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:04.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:03 smithi059 ceph-mon[36190]: pgmap v1832: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:04.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:03 smithi170 ceph-mon[36280]: pgmap v1832: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:06.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:05 smithi059 ceph-mon[36190]: pgmap v1833: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:06.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:05 smithi170 ceph-mon[36280]: pgmap v1833: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:06.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:05 smithi037 ceph-mon[31460]: pgmap v1833: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:07.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2481.148541 secs 2024-03-28T19:08:07.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2481.148541 secs 2024-03-28T19:08:07.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2481.148541 secs 2024-03-28T19:08:08.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:07 smithi059 ceph-mon[36190]: pgmap v1834: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:08.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:07 smithi170 ceph-mon[36280]: pgmap v1834: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:08.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:07 smithi037 ceph-mon[31460]: pgmap v1834: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:10.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:09 smithi059 ceph-mon[36190]: pgmap v1835: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:10.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:09 smithi170 ceph-mon[36280]: pgmap v1835: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:10.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:09 smithi037 ceph-mon[31460]: pgmap v1835: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:12.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:11 smithi059 ceph-mon[36190]: pgmap v1836: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:12.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2486.148665 secs 2024-03-28T19:08:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:11 smithi170 ceph-mon[36280]: pgmap v1836: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2486.148665 secs 2024-03-28T19:08:12.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:11 smithi037 ceph-mon[31460]: pgmap v1836: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2486.148665 secs 2024-03-28T19:08:14.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:13 smithi059 ceph-mon[36190]: pgmap v1837: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:14.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:13 smithi170 ceph-mon[36280]: pgmap v1837: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:13 smithi037 ceph-mon[31460]: pgmap v1837: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:16.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:15 smithi059 ceph-mon[36190]: pgmap v1838: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:16.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:15 smithi170 ceph-mon[36280]: pgmap v1838: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:16.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:15 smithi037 ceph-mon[31460]: pgmap v1838: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:17.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2491.148780 secs 2024-03-28T19:08:17.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2491.148780 secs 2024-03-28T19:08:17.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2491.148780 secs 2024-03-28T19:08:18.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:17 smithi059 ceph-mon[36190]: pgmap v1839: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:18.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:17 smithi170 ceph-mon[36280]: pgmap v1839: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:18.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:17 smithi037 ceph-mon[31460]: pgmap v1839: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:20.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:19 smithi059 ceph-mon[36190]: pgmap v1840: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:20.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:19 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:08:20.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:19 smithi170 ceph-mon[36280]: pgmap v1840: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:20.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:19 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:08:20.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:19 smithi037 ceph-mon[31460]: pgmap v1840: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:20.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:19 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:08:22.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:21 smithi059 ceph-mon[36190]: pgmap v1841: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:22.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2496.148909 secs 2024-03-28T19:08:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:21 smithi170 ceph-mon[36280]: pgmap v1841: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:22.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2496.148909 secs 2024-03-28T19:08:22.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:21 smithi037 ceph-mon[31460]: pgmap v1841: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:22.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2496.148909 secs 2024-03-28T19:08:24.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:24 smithi059 ceph-mon[36190]: pgmap v1842: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:24.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:24 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:08:24.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:24 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:08:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:24 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:08:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:24 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:08:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:24 smithi059 ceph-mon[36190]: pgmap v1843: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:08:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:24 smithi059 ceph-mon[36190]: pgmap v1844: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:08:24.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:24 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:08:24.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:24 smithi170 ceph-mon[36280]: pgmap v1842: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:24.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:24 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:08:24.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:24 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:08:24.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:24 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:08:24.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:24 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:08:24.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:24 smithi170 ceph-mon[36280]: pgmap v1843: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:08:24.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:24 smithi170 ceph-mon[36280]: pgmap v1844: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:08:24.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:24 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:08:24.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:24 smithi037 ceph-mon[31460]: pgmap v1842: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:24.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:24 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:08:24.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:24 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:08:24.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:24 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:08:24.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:24 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:08:24.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:24 smithi037 ceph-mon[31460]: pgmap v1843: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:08:24.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:24 smithi037 ceph-mon[31460]: pgmap v1844: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:08:24.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:24 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:08:26.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:26 smithi059 ceph-mon[36190]: pgmap v1845: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:08:26.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2501.149042 secs 2024-03-28T19:08:26.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:26 smithi170 ceph-mon[36280]: pgmap v1845: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:08:26.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2501.149042 secs 2024-03-28T19:08:26.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:26 smithi037 ceph-mon[31460]: pgmap v1845: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:08:26.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2501.149042 secs 2024-03-28T19:08:28.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:28 smithi059 ceph-mon[36190]: pgmap v1846: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:08:28.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:28 smithi170 ceph-mon[36280]: pgmap v1846: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:08:28.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:28 smithi037 ceph-mon[31460]: pgmap v1846: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:08:30.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:30 smithi059 ceph-mon[36190]: pgmap v1847: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:08:30.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:30 smithi170 ceph-mon[36280]: pgmap v1847: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:08:30.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:30 smithi037 ceph-mon[31460]: pgmap v1847: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:08:31.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2506.149134 secs 2024-03-28T19:08:31.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2506.149134 secs 2024-03-28T19:08:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2506.149134 secs 2024-03-28T19:08:32.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:32 smithi059 ceph-mon[36190]: pgmap v1848: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:08:32.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:32 smithi170 ceph-mon[36280]: pgmap v1848: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:08:32.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:32 smithi037 ceph-mon[31460]: pgmap v1848: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:08:34.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:34 smithi059 ceph-mon[36190]: pgmap v1849: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:08:34.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:34 smithi170 ceph-mon[36280]: pgmap v1849: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:08:34.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:34 smithi037 ceph-mon[31460]: pgmap v1849: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:08:36.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:36 smithi059 ceph-mon[36190]: pgmap v1850: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:08:36.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2511.149225 secs 2024-03-28T19:08:36.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:36 smithi170 ceph-mon[36280]: pgmap v1850: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:08:36.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2511.149225 secs 2024-03-28T19:08:36.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:36 smithi037 ceph-mon[31460]: pgmap v1850: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:08:36.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2511.149225 secs 2024-03-28T19:08:38.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:38 smithi059 ceph-mon[36190]: pgmap v1851: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:08:38.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:38 smithi170 ceph-mon[36280]: pgmap v1851: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:08:38.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:38 smithi037 ceph-mon[31460]: pgmap v1851: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:08:40.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:40 smithi059 ceph-mon[36190]: pgmap v1852: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:40.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:40 smithi170 ceph-mon[36280]: pgmap v1852: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:40.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:40 smithi037 ceph-mon[31460]: pgmap v1852: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:41.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2516.149308 secs 2024-03-28T19:08:41.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2516.149308 secs 2024-03-28T19:08:41.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2516.149308 secs 2024-03-28T19:08:42.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:42 smithi037 ceph-mon[31460]: pgmap v1853: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:42.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:42 smithi059 ceph-mon[36190]: pgmap v1853: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:42.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:42 smithi170 ceph-mon[36280]: pgmap v1853: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:44.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:44 smithi037 ceph-mon[31460]: pgmap v1854: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:44.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:44 smithi059 ceph-mon[36190]: pgmap v1854: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:44.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:44 smithi170 ceph-mon[36280]: pgmap v1854: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:46.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:46 smithi037 ceph-mon[31460]: pgmap v1855: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:46.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2521.149392 secs 2024-03-28T19:08:46.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:46 smithi059 ceph-mon[36190]: pgmap v1855: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:46.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2521.149392 secs 2024-03-28T19:08:46.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:46 smithi170 ceph-mon[36280]: pgmap v1855: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:46.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2521.149392 secs 2024-03-28T19:08:48.587 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:48 smithi059 ceph-mon[36190]: pgmap v1856: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:48.587 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:48 smithi170 ceph-mon[36280]: pgmap v1856: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:48.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:48 smithi037 ceph-mon[31460]: pgmap v1856: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:50.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:50 smithi037 ceph-mon[31460]: pgmap v1857: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:50.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:50 smithi059 ceph-mon[36190]: pgmap v1857: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:50.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:50 smithi170 ceph-mon[36280]: pgmap v1857: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:51.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2526.149494 secs 2024-03-28T19:08:51.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2526.149494 secs 2024-03-28T19:08:51.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2526.149494 secs 2024-03-28T19:08:52.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:52 smithi037 ceph-mon[31460]: pgmap v1858: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:52.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:52 smithi059 ceph-mon[36190]: pgmap v1858: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:52.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:52 smithi170 ceph-mon[36280]: pgmap v1858: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:54.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:54 smithi037 ceph-mon[31460]: pgmap v1859: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:54.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:54 smithi059 ceph-mon[36190]: pgmap v1859: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:54.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:54 smithi170 ceph-mon[36280]: pgmap v1859: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:56 smithi037 ceph-mon[31460]: pgmap v1860: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2531.149643 secs 2024-03-28T19:08:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:56 smithi059 ceph-mon[36190]: pgmap v1860: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2531.149643 secs 2024-03-28T19:08:56.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:56 smithi170 ceph-mon[36280]: pgmap v1860: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:56.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2531.149643 secs 2024-03-28T19:08:58.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:08:58 smithi037 ceph-mon[31460]: pgmap v1861: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:58.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:08:58 smithi059 ceph-mon[36190]: pgmap v1861: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:58.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:08:58 smithi170 ceph-mon[36280]: pgmap v1861: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:00.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:00 smithi037 ceph-mon[31460]: pgmap v1862: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:00.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:00 smithi059 ceph-mon[36190]: pgmap v1862: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:00.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:00 smithi170 ceph-mon[36280]: pgmap v1862: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:01.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2536.149762 secs 2024-03-28T19:09:01.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2536.149762 secs 2024-03-28T19:09:01.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2536.149762 secs 2024-03-28T19:09:02.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:02 smithi037 ceph-mon[31460]: pgmap v1863: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:02.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:02 smithi059 ceph-mon[36190]: pgmap v1863: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:02.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:02 smithi170 ceph-mon[36280]: pgmap v1863: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:04.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:04 smithi037 ceph-mon[31460]: pgmap v1864: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:04.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:04 smithi059 ceph-mon[36190]: pgmap v1864: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:04.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:04 smithi170 ceph-mon[36280]: pgmap v1864: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:06.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:06 smithi037 ceph-mon[31460]: pgmap v1865: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:06.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2541.149883 secs 2024-03-28T19:09:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:06 smithi059 ceph-mon[36190]: pgmap v1865: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2541.149883 secs 2024-03-28T19:09:06.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:06 smithi170 ceph-mon[36280]: pgmap v1865: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:06.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2541.149883 secs 2024-03-28T19:09:08.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:08 smithi037 ceph-mon[31460]: pgmap v1866: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:08.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:08 smithi059 ceph-mon[36190]: pgmap v1866: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:08.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:08 smithi170 ceph-mon[36280]: pgmap v1866: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:10.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:10 smithi037 ceph-mon[31460]: pgmap v1867: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:10.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:10 smithi059 ceph-mon[36190]: pgmap v1867: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:10.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:10 smithi170 ceph-mon[36280]: pgmap v1867: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2546.149996 secs 2024-03-28T19:09:11.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2546.149996 secs 2024-03-28T19:09:11.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2546.149996 secs 2024-03-28T19:09:12.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:12 smithi059 ceph-mon[36190]: pgmap v1868: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:12.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:12 smithi170 ceph-mon[36280]: pgmap v1868: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:12.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:12 smithi037 ceph-mon[31460]: pgmap v1868: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:14.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:14 smithi059 ceph-mon[36190]: pgmap v1869: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:14.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:14 smithi170 ceph-mon[36280]: pgmap v1869: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:14.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:14 smithi037 ceph-mon[31460]: pgmap v1869: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:16.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:16 smithi059 ceph-mon[36190]: pgmap v1870: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:16.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2551.150097 secs 2024-03-28T19:09:16.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:16 smithi170 ceph-mon[36280]: pgmap v1870: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:16.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2551.150097 secs 2024-03-28T19:09:16.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:16 smithi037 ceph-mon[31460]: pgmap v1870: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:16.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2551.150097 secs 2024-03-28T19:09:18.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:18 smithi059 ceph-mon[36190]: pgmap v1871: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:18.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:18 smithi170 ceph-mon[36280]: pgmap v1871: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:18.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:18 smithi037 ceph-mon[31460]: pgmap v1871: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:20.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:20 smithi059 ceph-mon[36190]: pgmap v1872: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:20.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:20 smithi170 ceph-mon[36280]: pgmap v1872: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:20.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:20 smithi037 ceph-mon[31460]: pgmap v1872: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:21.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2556.150209 secs 2024-03-28T19:09:21.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2556.150209 secs 2024-03-28T19:09:21.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2556.150209 secs 2024-03-28T19:09:22.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:22 smithi059 ceph-mon[36190]: pgmap v1873: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:22.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:22 smithi170 ceph-mon[36280]: pgmap v1873: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:22.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:22 smithi037 ceph-mon[31460]: pgmap v1873: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:23.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:23 smithi059 ceph-mon[36190]: pgmap v1874: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:23.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:23 smithi170 ceph-mon[36280]: pgmap v1874: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:23.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:23 smithi037 ceph-mon[31460]: pgmap v1874: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:24.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:24 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:09:24.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:24 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:09:24.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:24 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:09:25.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:25 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:09:25.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:25 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:09:25.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:25 smithi059 ceph-mon[36190]: pgmap v1875: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T19:09:25.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:25 smithi059 ceph-mon[36190]: pgmap v1876: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:25.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:25 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:09:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:25 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:09:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:25 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:09:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:25 smithi170 ceph-mon[36280]: pgmap v1875: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T19:09:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:25 smithi170 ceph-mon[36280]: pgmap v1876: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:25 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:09:25.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:25 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:09:25.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:25 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:09:25.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:25 smithi037 ceph-mon[31460]: pgmap v1875: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T19:09:25.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:25 smithi037 ceph-mon[31460]: pgmap v1876: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:25.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:25 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:09:26.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2561.150298 secs 2024-03-28T19:09:26.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2561.150298 secs 2024-03-28T19:09:26.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2561.150298 secs 2024-03-28T19:09:27.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:27 smithi059 ceph-mon[36190]: pgmap v1877: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:27.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:27 smithi170 ceph-mon[36280]: pgmap v1877: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:27.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:27 smithi037 ceph-mon[31460]: pgmap v1877: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:30.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:29 smithi059 ceph-mon[36190]: pgmap v1878: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:30.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:29 smithi170 ceph-mon[36280]: pgmap v1878: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:30.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:29 smithi037 ceph-mon[31460]: pgmap v1878: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:32.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:31 smithi059 ceph-mon[36190]: pgmap v1879: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:32.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2566.150384 secs 2024-03-28T19:09:32.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:31 smithi170 ceph-mon[36280]: pgmap v1879: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:32.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2566.150384 secs 2024-03-28T19:09:32.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:31 smithi037 ceph-mon[31460]: pgmap v1879: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:32.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2566.150384 secs 2024-03-28T19:09:34.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:33 smithi059 ceph-mon[36190]: pgmap v1880: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:34.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:33 smithi170 ceph-mon[36280]: pgmap v1880: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:34.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:33 smithi037 ceph-mon[31460]: pgmap v1880: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:36.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:35 smithi059 ceph-mon[36190]: pgmap v1881: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:09:36.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:35 smithi170 ceph-mon[36280]: pgmap v1881: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:09:36.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:35 smithi037 ceph-mon[31460]: pgmap v1881: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:09:37.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2571.150484 secs 2024-03-28T19:09:37.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2571.150484 secs 2024-03-28T19:09:37.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2571.150484 secs 2024-03-28T19:09:38.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:37 smithi059 ceph-mon[36190]: pgmap v1882: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:38.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:37 smithi170 ceph-mon[36280]: pgmap v1882: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:38.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:37 smithi037 ceph-mon[31460]: pgmap v1882: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:40.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:39 smithi059 ceph-mon[36190]: pgmap v1883: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:40.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:39 smithi170 ceph-mon[36280]: pgmap v1883: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:40.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:39 smithi037 ceph-mon[31460]: pgmap v1883: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:42.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:41 smithi059 ceph-mon[36190]: pgmap v1884: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:42.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2576.150605 secs 2024-03-28T19:09:42.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:41 smithi170 ceph-mon[36280]: pgmap v1884: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:42.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2576.150605 secs 2024-03-28T19:09:42.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:41 smithi037 ceph-mon[31460]: pgmap v1884: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:42.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2576.150605 secs 2024-03-28T19:09:44.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:43 smithi059 ceph-mon[36190]: pgmap v1885: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:44.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:43 smithi170 ceph-mon[36280]: pgmap v1885: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:44.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:43 smithi037 ceph-mon[31460]: pgmap v1885: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:46.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:45 smithi059 ceph-mon[36190]: pgmap v1886: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:46.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:45 smithi170 ceph-mon[36280]: pgmap v1886: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:46.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:45 smithi037 ceph-mon[31460]: pgmap v1886: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:47.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2581.150726 secs 2024-03-28T19:09:47.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2581.150726 secs 2024-03-28T19:09:47.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2581.150726 secs 2024-03-28T19:09:48.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:47 smithi059 ceph-mon[36190]: pgmap v1887: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:48.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:47 smithi170 ceph-mon[36280]: pgmap v1887: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:48.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:47 smithi037 ceph-mon[31460]: pgmap v1887: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:50.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:49 smithi059 ceph-mon[36190]: pgmap v1888: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:50.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:49 smithi170 ceph-mon[36280]: pgmap v1888: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:50.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:49 smithi037 ceph-mon[31460]: pgmap v1888: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:52.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:51 smithi059 ceph-mon[36190]: pgmap v1889: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:52.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2586.150844 secs 2024-03-28T19:09:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:51 smithi170 ceph-mon[36280]: pgmap v1889: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2586.150844 secs 2024-03-28T19:09:52.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:51 smithi037 ceph-mon[31460]: pgmap v1889: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:52.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2586.150844 secs 2024-03-28T19:09:54.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:53 smithi059 ceph-mon[36190]: pgmap v1890: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:54.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:53 smithi170 ceph-mon[36280]: pgmap v1890: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:54.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:53 smithi037 ceph-mon[31460]: pgmap v1890: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:56.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:55 smithi059 ceph-mon[36190]: pgmap v1891: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:56.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:55 smithi170 ceph-mon[36280]: pgmap v1891: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:56.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:55 smithi037 ceph-mon[31460]: pgmap v1891: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:57.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2591.150913 secs 2024-03-28T19:09:57.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2591.150913 secs 2024-03-28T19:09:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2591.150913 secs 2024-03-28T19:09:58.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:57 smithi037 ceph-mon[31460]: pgmap v1892: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:58.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:57 smithi059 ceph-mon[36190]: pgmap v1892: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:58.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:57 smithi170 ceph-mon[36280]: pgmap v1892: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:00.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:09:59 smithi037 ceph-mon[31460]: pgmap v1893: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:00.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:09:59 smithi059 ceph-mon[36190]: pgmap v1893: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:00.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:09:59 smithi170 ceph-mon[36280]: pgmap v1893: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:01.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:00 smithi170 ceph-mon[36280]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T19:10:01.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:00 smithi037 ceph-mon[31460]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T19:10:01.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:00 smithi059 ceph-mon[36190]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T19:10:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:01 smithi037 ceph-mon[31460]: pgmap v1894: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2596.150982 secs 2024-03-28T19:10:02.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:01 smithi059 ceph-mon[36190]: pgmap v1894: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:02.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2596.150982 secs 2024-03-28T19:10:02.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:01 smithi170 ceph-mon[36280]: pgmap v1894: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:02.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2596.150982 secs 2024-03-28T19:10:04.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:03 smithi037 ceph-mon[31460]: pgmap v1895: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:04.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:03 smithi059 ceph-mon[36190]: pgmap v1895: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:04.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:03 smithi170 ceph-mon[36280]: pgmap v1895: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:05 smithi037 ceph-mon[31460]: pgmap v1896: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:06.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:05 smithi059 ceph-mon[36190]: pgmap v1896: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:06.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:05 smithi170 ceph-mon[36280]: pgmap v1896: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:07.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2601.151044 secs 2024-03-28T19:10:07.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2601.151044 secs 2024-03-28T19:10:07.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2601.151044 secs 2024-03-28T19:10:08.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:07 smithi037 ceph-mon[31460]: pgmap v1897: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:08.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:07 smithi059 ceph-mon[36190]: pgmap v1897: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:08.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:07 smithi170 ceph-mon[36280]: pgmap v1897: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:10.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:09 smithi037 ceph-mon[31460]: pgmap v1898: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:10.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:09 smithi059 ceph-mon[36190]: pgmap v1898: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:10.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:09 smithi170 ceph-mon[36280]: pgmap v1898: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:12.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:11 smithi037 ceph-mon[31460]: pgmap v1899: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2606.151111 secs 2024-03-28T19:10:12.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:11 smithi059 ceph-mon[36190]: pgmap v1899: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:12.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2606.151111 secs 2024-03-28T19:10:12.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:11 smithi170 ceph-mon[36280]: pgmap v1899: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:12.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2606.151111 secs 2024-03-28T19:10:14.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:13 smithi037 ceph-mon[31460]: pgmap v1900: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:14.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:13 smithi059 ceph-mon[36190]: pgmap v1900: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:14.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:13 smithi170 ceph-mon[36280]: pgmap v1900: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:16.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:15 smithi037 ceph-mon[31460]: pgmap v1901: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:16.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:15 smithi059 ceph-mon[36190]: pgmap v1901: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:16.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:15 smithi170 ceph-mon[36280]: pgmap v1901: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:17.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2611.151177 secs 2024-03-28T19:10:17.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2611.151177 secs 2024-03-28T19:10:17.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2611.151177 secs 2024-03-28T19:10:18.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:17 smithi037 ceph-mon[31460]: pgmap v1902: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:10:18.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:17 smithi059 ceph-mon[36190]: pgmap v1902: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:10:18.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:17 smithi170 ceph-mon[36280]: pgmap v1902: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:10:20.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:19 smithi037 ceph-mon[31460]: pgmap v1903: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:10:20.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:19 smithi059 ceph-mon[36190]: pgmap v1903: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:10:20.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:19 smithi170 ceph-mon[36280]: pgmap v1903: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:10:22.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:21 smithi037 ceph-mon[31460]: pgmap v1904: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:10:22.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2616.151245 secs 2024-03-28T19:10:22.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:21 smithi059 ceph-mon[36190]: pgmap v1904: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:10:22.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2616.151245 secs 2024-03-28T19:10:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:21 smithi170 ceph-mon[36280]: pgmap v1904: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:10:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2616.151245 secs 2024-03-28T19:10:24.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:23 smithi037 ceph-mon[31460]: pgmap v1905: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:10:24.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:23 smithi059 ceph-mon[36190]: pgmap v1905: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:10:24.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:23 smithi170 ceph-mon[36280]: pgmap v1905: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:10:26.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:25 smithi059 ceph-mon[36190]: pgmap v1906: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:10:26.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:25 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:10:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:25 smithi170 ceph-mon[36280]: pgmap v1906: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:10:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:25 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:10:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:25 smithi037 ceph-mon[31460]: pgmap v1906: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:10:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:25 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:10:27.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2621.151317 secs 2024-03-28T19:10:27.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:26 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:10:27.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:26 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:10:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:26 smithi059 ceph-mon[36190]: pgmap v1907: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:26 smithi059 ceph-mon[36190]: pgmap v1908: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:10:27.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:26 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:10:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2621.151317 secs 2024-03-28T19:10:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:26 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:10:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:26 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:10:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:26 smithi170 ceph-mon[36280]: pgmap v1907: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:26 smithi170 ceph-mon[36280]: pgmap v1908: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:10:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:26 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:10:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2621.151317 secs 2024-03-28T19:10:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:26 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:10:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:26 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:10:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:26 smithi037 ceph-mon[31460]: pgmap v1907: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:26 smithi037 ceph-mon[31460]: pgmap v1908: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:10:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:26 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:10:29.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:29 smithi059 ceph-mon[36190]: pgmap v1909: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:29.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:29 smithi170 ceph-mon[36280]: pgmap v1909: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:29.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:29 smithi037 ceph-mon[31460]: pgmap v1909: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:31.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:31 smithi059 ceph-mon[36190]: pgmap v1910: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2626.151410 secs 2024-03-28T19:10:31.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:31 smithi170 ceph-mon[36280]: pgmap v1910: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:31.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2626.151410 secs 2024-03-28T19:10:31.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:31 smithi037 ceph-mon[31460]: pgmap v1910: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2626.151410 secs 2024-03-28T19:10:33.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:33 smithi059 ceph-mon[36190]: pgmap v1911: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:33.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:33 smithi170 ceph-mon[36280]: pgmap v1911: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:33.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:33 smithi037 ceph-mon[31460]: pgmap v1911: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:35.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:35 smithi059 ceph-mon[36190]: pgmap v1912: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:35.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:35 smithi170 ceph-mon[36280]: pgmap v1912: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:35.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:35 smithi037 ceph-mon[31460]: pgmap v1912: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:36.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2631.151512 secs 2024-03-28T19:10:36.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2631.151512 secs 2024-03-28T19:10:36.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2631.151512 secs 2024-03-28T19:10:37.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:37 smithi059 ceph-mon[36190]: pgmap v1913: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:10:37.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:37 smithi170 ceph-mon[36280]: pgmap v1913: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:10:37.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:37 smithi037 ceph-mon[31460]: pgmap v1913: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:10:39.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:39 smithi059 ceph-mon[36190]: pgmap v1914: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:39.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:39 smithi170 ceph-mon[36280]: pgmap v1914: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:39.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:39 smithi037 ceph-mon[31460]: pgmap v1914: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:41.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:41 smithi059 ceph-mon[36190]: pgmap v1915: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:41.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2636.151629 secs 2024-03-28T19:10:41.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:41 smithi170 ceph-mon[36280]: pgmap v1915: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:41.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2636.151629 secs 2024-03-28T19:10:41.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:41 smithi037 ceph-mon[31460]: pgmap v1915: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:41.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2636.151629 secs 2024-03-28T19:10:43.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:43 smithi059 ceph-mon[36190]: pgmap v1916: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:43.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:43 smithi170 ceph-mon[36280]: pgmap v1916: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:43.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:43 smithi037 ceph-mon[31460]: pgmap v1916: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:45.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:45 smithi059 ceph-mon[36190]: pgmap v1917: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:45.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:45 smithi170 ceph-mon[36280]: pgmap v1917: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:45.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:45 smithi037 ceph-mon[31460]: pgmap v1917: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:46.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2641.151745 secs 2024-03-28T19:10:46.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2641.151745 secs 2024-03-28T19:10:46.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2641.151745 secs 2024-03-28T19:10:47.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:47 smithi059 ceph-mon[36190]: pgmap v1918: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:47.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:47 smithi170 ceph-mon[36280]: pgmap v1918: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:47.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:47 smithi037 ceph-mon[31460]: pgmap v1918: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:49.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:49 smithi059 ceph-mon[36190]: pgmap v1919: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:49.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:49 smithi170 ceph-mon[36280]: pgmap v1919: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:49.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:49 smithi037 ceph-mon[31460]: pgmap v1919: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:51.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:51 smithi059 ceph-mon[36190]: pgmap v1920: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:51.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2646.151862 secs 2024-03-28T19:10:51.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:51 smithi170 ceph-mon[36280]: pgmap v1920: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:51.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2646.151862 secs 2024-03-28T19:10:51.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:51 smithi037 ceph-mon[31460]: pgmap v1920: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:51.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2646.151862 secs 2024-03-28T19:10:53.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:53 smithi059 ceph-mon[36190]: pgmap v1921: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:53.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:53 smithi170 ceph-mon[36280]: pgmap v1921: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:53.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:53 smithi037 ceph-mon[31460]: pgmap v1921: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:55.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:55 smithi059 ceph-mon[36190]: pgmap v1922: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:55.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:55 smithi170 ceph-mon[36280]: pgmap v1922: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:55.611 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:55 smithi037 ceph-mon[31460]: pgmap v1922: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:56.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2651.151977 secs 2024-03-28T19:10:56.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2651.151977 secs 2024-03-28T19:10:56.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2651.151977 secs 2024-03-28T19:10:57.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:57 smithi059 ceph-mon[36190]: pgmap v1923: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:57.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:57 smithi170 ceph-mon[36280]: pgmap v1923: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:57.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:57 smithi037 ceph-mon[31460]: pgmap v1923: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:59.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:10:59 smithi037 ceph-mon[31460]: pgmap v1924: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:59.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:10:59 smithi059 ceph-mon[36190]: pgmap v1924: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:59.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:10:59 smithi170 ceph-mon[36280]: pgmap v1924: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:01.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:01 smithi037 ceph-mon[31460]: pgmap v1925: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:01.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2656.152096 secs 2024-03-28T19:11:01.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:01 smithi059 ceph-mon[36190]: pgmap v1925: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:01.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2656.152096 secs 2024-03-28T19:11:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:01 smithi170 ceph-mon[36280]: pgmap v1925: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2656.152096 secs 2024-03-28T19:11:03.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:03 smithi037 ceph-mon[31460]: pgmap v1926: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:03.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:03 smithi059 ceph-mon[36190]: pgmap v1926: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:03.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:03 smithi170 ceph-mon[36280]: pgmap v1926: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:05.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:05 smithi037 ceph-mon[31460]: pgmap v1927: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:05.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:05 smithi059 ceph-mon[36190]: pgmap v1927: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:05.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:05 smithi170 ceph-mon[36280]: pgmap v1927: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:06.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2661.152183 secs 2024-03-28T19:11:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2661.152183 secs 2024-03-28T19:11:06.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2661.152183 secs 2024-03-28T19:11:07.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:07 smithi037 ceph-mon[31460]: pgmap v1928: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:07.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:07 smithi059 ceph-mon[36190]: pgmap v1928: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:07.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:07 smithi170 ceph-mon[36280]: pgmap v1928: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:09.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:09 smithi037 ceph-mon[31460]: pgmap v1929: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:09.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:09 smithi059 ceph-mon[36190]: pgmap v1929: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:09.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:09 smithi170 ceph-mon[36280]: pgmap v1929: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:11 smithi059 ceph-mon[36190]: pgmap v1930: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2666.152262 secs 2024-03-28T19:11:11.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:11 smithi170 ceph-mon[36280]: pgmap v1930: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:11.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2666.152262 secs 2024-03-28T19:11:11.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:11 smithi037 ceph-mon[31460]: pgmap v1930: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:11.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2666.152262 secs 2024-03-28T19:11:13.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:13 smithi059 ceph-mon[36190]: pgmap v1931: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:13.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:13 smithi170 ceph-mon[36280]: pgmap v1931: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:13.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:13 smithi037 ceph-mon[31460]: pgmap v1931: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:15.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:15 smithi059 ceph-mon[36190]: pgmap v1932: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:15.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:15 smithi170 ceph-mon[36280]: pgmap v1932: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:15.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:15 smithi037 ceph-mon[31460]: pgmap v1932: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:16.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2671.152347 secs 2024-03-28T19:11:16.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2671.152347 secs 2024-03-28T19:11:16.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2671.152347 secs 2024-03-28T19:11:17.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:17 smithi059 ceph-mon[36190]: pgmap v1933: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:17.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:17 smithi170 ceph-mon[36280]: pgmap v1933: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:17.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:17 smithi037 ceph-mon[31460]: pgmap v1933: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:18.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:18 smithi059 ceph-mon[36190]: pgmap v1934: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:18.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:18 smithi170 ceph-mon[36280]: pgmap v1934: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:18.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:18 smithi037 ceph-mon[31460]: pgmap v1934: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:21.488 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:21 smithi059 ceph-mon[36190]: pgmap v1935: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:21.488 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2676.152468 secs 2024-03-28T19:11:21.488 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:21 smithi170 ceph-mon[36280]: pgmap v1935: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:21.489 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2676.152468 secs 2024-03-28T19:11:21.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:21 smithi037 ceph-mon[31460]: pgmap v1935: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:21.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2676.152468 secs 2024-03-28T19:11:23.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:23 smithi059 ceph-mon[36190]: pgmap v1936: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:23.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:23 smithi170 ceph-mon[36280]: pgmap v1936: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:23.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:23 smithi037 ceph-mon[31460]: pgmap v1936: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:25.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:25 smithi059 ceph-mon[36190]: pgmap v1937: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:25.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:25 smithi170 ceph-mon[36280]: pgmap v1937: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:25.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:25 smithi037 ceph-mon[31460]: pgmap v1937: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:26.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2681.152589 secs 2024-03-28T19:11:26.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2681.152589 secs 2024-03-28T19:11:26.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2681.152589 secs 2024-03-28T19:11:27.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:27 smithi059 ceph-mon[36190]: pgmap v1938: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:27.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:27 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:11:27.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:27 smithi170 ceph-mon[36280]: pgmap v1938: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:27.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:27 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:11:27.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:27 smithi037 ceph-mon[31460]: pgmap v1938: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:27.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:27 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:11:29.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:28 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:29.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:28 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:29.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:28 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:29.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:28 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:29.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:28 smithi059 ceph-mon[36190]: pgmap v1939: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:29.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:28 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:29.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:28 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:29.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:28 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:11:29.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:28 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:11:29.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:28 smithi059 ceph-mon[36190]: pgmap v1940: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:11:29.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:28 smithi059 ceph-mon[36190]: pgmap v1941: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:11:29.035 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:28 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:29.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:28 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:29.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:28 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:29.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:28 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:29.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:28 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:29.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:28 smithi170 ceph-mon[36280]: pgmap v1939: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:29.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:28 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:29.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:28 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:29.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:28 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:11:29.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:28 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:11:29.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:28 smithi170 ceph-mon[36280]: pgmap v1940: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:11:29.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:28 smithi170 ceph-mon[36280]: pgmap v1941: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:11:29.036 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:28 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:29.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:28 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:29.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:28 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:29.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:28 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:29.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:28 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:29.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:28 smithi037 ceph-mon[31460]: pgmap v1939: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:29.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:28 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:29.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:28 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:29.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:28 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:11:29.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:28 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:11:29.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:28 smithi037 ceph-mon[31460]: pgmap v1940: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:11:29.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:28 smithi037 ceph-mon[31460]: pgmap v1941: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:11:29.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:28 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:11:30.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:30 smithi059 ceph-mon[36190]: pgmap v1942: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:11:30.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:30 smithi170 ceph-mon[36280]: pgmap v1942: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:11:30.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:30 smithi037 ceph-mon[31460]: pgmap v1942: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:11:31.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2686.152710 secs 2024-03-28T19:11:31.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2686.152710 secs 2024-03-28T19:11:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2686.152710 secs 2024-03-28T19:11:32.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:32 smithi059 ceph-mon[36190]: pgmap v1943: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:11:32.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:32 smithi170 ceph-mon[36280]: pgmap v1943: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:11:32.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:32 smithi037 ceph-mon[31460]: pgmap v1943: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:11:34.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:34 smithi059 ceph-mon[36190]: pgmap v1944: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:11:34.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:34 smithi170 ceph-mon[36280]: pgmap v1944: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:11:34.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:34 smithi037 ceph-mon[31460]: pgmap v1944: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:11:36.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2691.152830 secs 2024-03-28T19:11:36.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2691.152830 secs 2024-03-28T19:11:36.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2691.152830 secs 2024-03-28T19:11:37.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:37 smithi059 ceph-mon[36190]: pgmap v1945: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:11:37.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:37 smithi170 ceph-mon[36280]: pgmap v1945: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:11:37.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:37 smithi037 ceph-mon[31460]: pgmap v1945: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:11:38.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:38 smithi059 ceph-mon[36190]: pgmap v1946: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:11:38.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:38 smithi170 ceph-mon[36280]: pgmap v1946: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:11:38.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:38 smithi037 ceph-mon[31460]: pgmap v1946: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:11:40.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:40 smithi059 ceph-mon[36190]: pgmap v1947: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:40.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:40 smithi170 ceph-mon[36280]: pgmap v1947: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:40.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:40 smithi037 ceph-mon[31460]: pgmap v1947: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:41.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2696.152951 secs 2024-03-28T19:11:41.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2696.152951 secs 2024-03-28T19:11:41.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2696.152951 secs 2024-03-28T19:11:42.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:42 smithi059 ceph-mon[36190]: pgmap v1948: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:42.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:42 smithi170 ceph-mon[36280]: pgmap v1948: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:42.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:42 smithi037 ceph-mon[31460]: pgmap v1948: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:44.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:44 smithi059 ceph-mon[36190]: pgmap v1949: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:44.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:44 smithi170 ceph-mon[36280]: pgmap v1949: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:44.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:44 smithi037 ceph-mon[31460]: pgmap v1949: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:46.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2701.153068 secs 2024-03-28T19:11:46.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2701.153068 secs 2024-03-28T19:11:46.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2701.153068 secs 2024-03-28T19:11:47.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:47 smithi059 ceph-mon[36190]: pgmap v1950: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:47.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:47 smithi170 ceph-mon[36280]: pgmap v1950: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:47.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:47 smithi037 ceph-mon[31460]: pgmap v1950: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:48.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:48 smithi059 ceph-mon[36190]: pgmap v1951: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:48.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:48 smithi170 ceph-mon[36280]: pgmap v1951: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:48.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:48 smithi037 ceph-mon[31460]: pgmap v1951: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:50.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:50 smithi059 ceph-mon[36190]: pgmap v1952: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:50.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:50 smithi170 ceph-mon[36280]: pgmap v1952: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:50.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:50 smithi037 ceph-mon[31460]: pgmap v1952: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:51.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2706.153189 secs 2024-03-28T19:11:51.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2706.153189 secs 2024-03-28T19:11:51.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2706.153189 secs 2024-03-28T19:11:52.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:52 smithi059 ceph-mon[36190]: pgmap v1953: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:52.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:52 smithi170 ceph-mon[36280]: pgmap v1953: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:52.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:52 smithi037 ceph-mon[31460]: pgmap v1953: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:54.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:54 smithi059 ceph-mon[36190]: pgmap v1954: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:54.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:54 smithi170 ceph-mon[36280]: pgmap v1954: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:54.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:54 smithi037 ceph-mon[31460]: pgmap v1954: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:56.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2711.153271 secs 2024-03-28T19:11:56.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2711.153271 secs 2024-03-28T19:11:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2711.153271 secs 2024-03-28T19:11:57.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:57 smithi059 ceph-mon[36190]: pgmap v1955: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:57.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:57 smithi170 ceph-mon[36280]: pgmap v1955: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:57.564 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:57 smithi037 ceph-mon[31460]: pgmap v1955: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:58.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:11:58 smithi059 ceph-mon[36190]: pgmap v1956: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:58.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:11:58 smithi170 ceph-mon[36280]: pgmap v1956: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:58.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:11:58 smithi037 ceph-mon[31460]: pgmap v1956: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:00.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:00 smithi059 ceph-mon[36190]: pgmap v1957: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:00.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:00 smithi170 ceph-mon[36280]: pgmap v1957: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:00.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:00 smithi037 ceph-mon[31460]: pgmap v1957: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:01.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2716.153389 secs 2024-03-28T19:12:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2716.153389 secs 2024-03-28T19:12:01.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2716.153389 secs 2024-03-28T19:12:02.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:02 smithi059 ceph-mon[36190]: pgmap v1958: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:02.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:02 smithi170 ceph-mon[36280]: pgmap v1958: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:02.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:02 smithi037 ceph-mon[31460]: pgmap v1958: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:04.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:04 smithi059 ceph-mon[36190]: pgmap v1959: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:04.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:04 smithi170 ceph-mon[36280]: pgmap v1959: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:04.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:04 smithi037 ceph-mon[31460]: pgmap v1959: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:06.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2721.153510 secs 2024-03-28T19:12:06.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2721.153510 secs 2024-03-28T19:12:06.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2721.153510 secs 2024-03-28T19:12:07.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:07 smithi059 ceph-mon[36190]: pgmap v1960: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:07.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:07 smithi170 ceph-mon[36280]: pgmap v1960: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:07.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:07 smithi037 ceph-mon[31460]: pgmap v1960: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:08.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:08 smithi059 ceph-mon[36190]: pgmap v1961: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:08.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:08 smithi170 ceph-mon[36280]: pgmap v1961: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:08.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:08 smithi037 ceph-mon[31460]: pgmap v1961: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:10.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:10 smithi059 ceph-mon[36190]: pgmap v1962: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:10.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:10 smithi170 ceph-mon[36280]: pgmap v1962: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:10.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:10 smithi037 ceph-mon[31460]: pgmap v1962: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2726.153631 secs 2024-03-28T19:12:11.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2726.153631 secs 2024-03-28T19:12:11.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2726.153631 secs 2024-03-28T19:12:12.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:12 smithi059 ceph-mon[36190]: pgmap v1963: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:12.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:12 smithi170 ceph-mon[36280]: pgmap v1963: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:12.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:12 smithi037 ceph-mon[31460]: pgmap v1963: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:14.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:14 smithi059 ceph-mon[36190]: pgmap v1964: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:14.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:14 smithi170 ceph-mon[36280]: pgmap v1964: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:14.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:14 smithi037 ceph-mon[31460]: pgmap v1964: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:16.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2731.153750 secs 2024-03-28T19:12:16.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2731.153750 secs 2024-03-28T19:12:16.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2731.153750 secs 2024-03-28T19:12:17.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:17 smithi059 ceph-mon[36190]: pgmap v1965: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:17.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:17 smithi170 ceph-mon[36280]: pgmap v1965: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:17.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:17 smithi037 ceph-mon[31460]: pgmap v1965: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:18.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:18 smithi059 ceph-mon[36190]: pgmap v1966: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:18.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:18 smithi170 ceph-mon[36280]: pgmap v1966: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:18.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:18 smithi037 ceph-mon[31460]: pgmap v1966: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:20.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:20 smithi059 ceph-mon[36190]: pgmap v1967: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:20.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:20 smithi170 ceph-mon[36280]: pgmap v1967: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:20.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:20 smithi037 ceph-mon[31460]: pgmap v1967: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:21.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2736.153869 secs 2024-03-28T19:12:21.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2736.153869 secs 2024-03-28T19:12:21.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2736.153869 secs 2024-03-28T19:12:22.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:22 smithi059 ceph-mon[36190]: pgmap v1968: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:22.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:22 smithi170 ceph-mon[36280]: pgmap v1968: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:22.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:22 smithi037 ceph-mon[31460]: pgmap v1968: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:24.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:24 smithi059 ceph-mon[36190]: pgmap v1969: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:24.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:24 smithi170 ceph-mon[36280]: pgmap v1969: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:24.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:24 smithi037 ceph-mon[31460]: pgmap v1969: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:26.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2741.153989 secs 2024-03-28T19:12:26.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2741.153989 secs 2024-03-28T19:12:26.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2741.153989 secs 2024-03-28T19:12:27.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:27 smithi059 ceph-mon[36190]: pgmap v1970: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:27.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:27 smithi170 ceph-mon[36280]: pgmap v1970: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:27.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:27 smithi037 ceph-mon[31460]: pgmap v1970: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:28.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:28 smithi059 ceph-mon[36190]: pgmap v1971: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:28.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:28 smithi170 ceph-mon[36280]: pgmap v1971: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:28.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:28 smithi037 ceph-mon[31460]: pgmap v1971: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:29.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:29 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:12:29.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:29 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:12:29.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:29 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:12:30.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:30 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:12:30.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:30 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:12:30.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:30 smithi059 ceph-mon[36190]: pgmap v1972: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:30.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:30 smithi059 ceph-mon[36190]: pgmap v1973: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:12:30.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:30 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:12:30.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:30 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:12:30.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:30 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:12:30.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:30 smithi170 ceph-mon[36280]: pgmap v1972: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:30.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:30 smithi170 ceph-mon[36280]: pgmap v1973: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:12:30.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:30 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:12:30.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:30 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:12:30.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:30 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:12:30.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:30 smithi037 ceph-mon[31460]: pgmap v1972: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:30.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:30 smithi037 ceph-mon[31460]: pgmap v1973: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:12:30.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:30 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:12:31.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2746.154111 secs 2024-03-28T19:12:31.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2746.154111 secs 2024-03-28T19:12:31.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2746.154111 secs 2024-03-28T19:12:32.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:32 smithi059 ceph-mon[36190]: pgmap v1974: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:12:32.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:32 smithi170 ceph-mon[36280]: pgmap v1974: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:12:32.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:32 smithi037 ceph-mon[31460]: pgmap v1974: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:12:35.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:34 smithi059 ceph-mon[36190]: pgmap v1975: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:12:35.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:34 smithi170 ceph-mon[36280]: pgmap v1975: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:12:35.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:34 smithi037 ceph-mon[31460]: pgmap v1975: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:12:37.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2751.154230 secs 2024-03-28T19:12:37.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:36 smithi059 ceph-mon[36190]: pgmap v1976: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:12:37.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2751.154230 secs 2024-03-28T19:12:37.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:36 smithi170 ceph-mon[36280]: pgmap v1976: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:12:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2751.154230 secs 2024-03-28T19:12:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:36 smithi037 ceph-mon[31460]: pgmap v1976: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:12:39.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:38 smithi059 ceph-mon[36190]: pgmap v1977: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:12:39.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:38 smithi170 ceph-mon[36280]: pgmap v1977: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:12:39.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:38 smithi037 ceph-mon[31460]: pgmap v1977: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:12:41.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:40 smithi059 ceph-mon[36190]: pgmap v1978: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:12:41.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:40 smithi170 ceph-mon[36280]: pgmap v1978: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:12:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:40 smithi037 ceph-mon[31460]: pgmap v1978: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:12:42.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2756.154350 secs 2024-03-28T19:12:42.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2756.154350 secs 2024-03-28T19:12:42.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2756.154350 secs 2024-03-28T19:12:43.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:42 smithi059 ceph-mon[36190]: pgmap v1979: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:43.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:42 smithi170 ceph-mon[36280]: pgmap v1979: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:43.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:42 smithi037 ceph-mon[31460]: pgmap v1979: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:45.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:44 smithi059 ceph-mon[36190]: pgmap v1980: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:45.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:44 smithi170 ceph-mon[36280]: pgmap v1980: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:45.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:44 smithi037 ceph-mon[31460]: pgmap v1980: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:47.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:47 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2761.154470 secs 2024-03-28T19:12:47.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:47 smithi059 ceph-mon[36190]: pgmap v1981: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:47 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2761.154470 secs 2024-03-28T19:12:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:47 smithi170 ceph-mon[36280]: pgmap v1981: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:47.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:47 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2761.154470 secs 2024-03-28T19:12:47.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:47 smithi037 ceph-mon[31460]: pgmap v1981: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:49.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:49 smithi059 ceph-mon[36190]: pgmap v1982: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:49 smithi170 ceph-mon[36280]: pgmap v1982: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:49.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:49 smithi037 ceph-mon[31460]: pgmap v1982: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:51.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:51 smithi059 ceph-mon[36190]: pgmap v1983: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:51.297 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:51 smithi170 ceph-mon[36280]: pgmap v1983: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:51.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:51 smithi037 ceph-mon[31460]: pgmap v1983: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:52.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:52 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2766.154566 secs 2024-03-28T19:12:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:52 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2766.154566 secs 2024-03-28T19:12:52.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:52 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2766.154566 secs 2024-03-28T19:12:53.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:53 smithi059 ceph-mon[36190]: pgmap v1984: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:53.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:53 smithi170 ceph-mon[36280]: pgmap v1984: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:53.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:53 smithi037 ceph-mon[31460]: pgmap v1984: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:55.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:55 smithi059 ceph-mon[36190]: pgmap v1985: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:55.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:55 smithi170 ceph-mon[36280]: pgmap v1985: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:55.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:55 smithi037 ceph-mon[31460]: pgmap v1985: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:57 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2771.154654 secs 2024-03-28T19:12:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:57 smithi059 ceph-mon[36190]: pgmap v1986: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:57.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:57 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2771.154654 secs 2024-03-28T19:12:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:57 smithi170 ceph-mon[36280]: pgmap v1986: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:57.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:57 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2771.154654 secs 2024-03-28T19:12:57.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:57 smithi037 ceph-mon[31460]: pgmap v1986: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:59.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:12:59 smithi037 ceph-mon[31460]: pgmap v1987: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:59.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:12:59 smithi059 ceph-mon[36190]: pgmap v1987: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:59.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:12:59 smithi170 ceph-mon[36280]: pgmap v1987: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:01 smithi037 ceph-mon[31460]: pgmap v1988: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:01.515 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:01 smithi059 ceph-mon[36190]: pgmap v1988: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:01.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:01 smithi170 ceph-mon[36280]: pgmap v1988: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:02.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:02 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2776.154771 secs 2024-03-28T19:13:02.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:02 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2776.154771 secs 2024-03-28T19:13:02.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:02 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2776.154771 secs 2024-03-28T19:13:03.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:03 smithi037 ceph-mon[31460]: pgmap v1989: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:03.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:03 smithi059 ceph-mon[36190]: pgmap v1989: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:03.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:03 smithi170 ceph-mon[36280]: pgmap v1989: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:05.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:05 smithi037 ceph-mon[31460]: pgmap v1990: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:05.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:05 smithi059 ceph-mon[36190]: pgmap v1990: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:05.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:05 smithi170 ceph-mon[36280]: pgmap v1990: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:07.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:07 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2781.154893 secs 2024-03-28T19:13:07.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:07 smithi037 ceph-mon[31460]: pgmap v1991: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:07.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:07 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2781.154893 secs 2024-03-28T19:13:07.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:07 smithi059 ceph-mon[36190]: pgmap v1991: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:07.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:07 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2781.154893 secs 2024-03-28T19:13:07.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:07 smithi170 ceph-mon[36280]: pgmap v1991: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:09.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:09 smithi037 ceph-mon[31460]: pgmap v1992: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:09.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:09 smithi059 ceph-mon[36190]: pgmap v1992: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:09.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:09 smithi170 ceph-mon[36280]: pgmap v1992: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:11.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:11 smithi037 ceph-mon[31460]: pgmap v1993: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:11.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:11 smithi059 ceph-mon[36190]: pgmap v1993: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:11.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:11 smithi170 ceph-mon[36280]: pgmap v1993: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:12.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:12 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2786.155013 secs 2024-03-28T19:13:12.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:12 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2786.155013 secs 2024-03-28T19:13:12.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:12 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2786.155013 secs 2024-03-28T19:13:13.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:13 smithi037 ceph-mon[31460]: pgmap v1994: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:13.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:13 smithi059 ceph-mon[36190]: pgmap v1994: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:13.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:13 smithi170 ceph-mon[36280]: pgmap v1994: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:15.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:15 smithi037 ceph-mon[31460]: pgmap v1995: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:15.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:15 smithi059 ceph-mon[36190]: pgmap v1995: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:15.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:15 smithi170 ceph-mon[36280]: pgmap v1995: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:17.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:17 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2791.155063 secs 2024-03-28T19:13:17.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:17 smithi037 ceph-mon[31460]: pgmap v1996: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:17.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:17 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2791.155063 secs 2024-03-28T19:13:17.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:17 smithi059 ceph-mon[36190]: pgmap v1996: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:17.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:17 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2791.155063 secs 2024-03-28T19:13:17.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:17 smithi170 ceph-mon[36280]: pgmap v1996: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:19.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:19 smithi037 ceph-mon[31460]: pgmap v1997: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:19.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:19 smithi059 ceph-mon[36190]: pgmap v1997: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:19.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:19 smithi170 ceph-mon[36280]: pgmap v1997: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:21.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:21 smithi037 ceph-mon[31460]: pgmap v1998: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:21.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:21 smithi059 ceph-mon[36190]: pgmap v1998: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:21.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:21 smithi170 ceph-mon[36280]: pgmap v1998: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:22.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:22 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2796.155149 secs 2024-03-28T19:13:22.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:22 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2796.155149 secs 2024-03-28T19:13:22.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:22 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2796.155149 secs 2024-03-28T19:13:23.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:23 smithi037 ceph-mon[31460]: pgmap v1999: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:23.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:23 smithi059 ceph-mon[36190]: pgmap v1999: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:23.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:23 smithi170 ceph-mon[36280]: pgmap v1999: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:25.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:25 smithi037 ceph-mon[31460]: pgmap v2000: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:25.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:25 smithi059 ceph-mon[36190]: pgmap v2000: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:25.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:25 smithi170 ceph-mon[36280]: pgmap v2000: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:27.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:27 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2801.155233 secs 2024-03-28T19:13:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:27 smithi037 ceph-mon[31460]: pgmap v2001: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:27.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:27 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2801.155233 secs 2024-03-28T19:13:27.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:27 smithi059 ceph-mon[36190]: pgmap v2001: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:27.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:27 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2801.155233 secs 2024-03-28T19:13:27.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:27 smithi170 ceph-mon[36280]: pgmap v2001: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:29.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:29 smithi037 ceph-mon[31460]: pgmap v2002: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:29.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:29 smithi059 ceph-mon[36190]: pgmap v2002: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:29.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:29 smithi170 ceph-mon[36280]: pgmap v2002: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:31.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:31 smithi059 ceph-mon[36190]: pgmap v2003: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:31 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:13:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:31 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:31 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:31 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:31 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:31 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:31.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:31 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:31 smithi170 ceph-mon[36280]: pgmap v2003: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:31 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:13:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:31 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:31 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:31 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:31 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:31 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:31.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:31 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:31.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:31 smithi037 ceph-mon[31460]: pgmap v2003: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:31 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:13:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:31 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:31 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:31 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:31 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:31 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:31.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:31 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:32.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:32 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2806.155310 secs 2024-03-28T19:13:32.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:32 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2806.155310 secs 2024-03-28T19:13:32.536 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:32 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2806.155310 secs 2024-03-28T19:13:33.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:33 smithi059 ceph-mon[36190]: pgmap v2004: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:33.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:33 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:33.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:33 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:33.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:33 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:33.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:33 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:33.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:33 smithi170 ceph-mon[36280]: pgmap v2004: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:33.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:33 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:33.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:33 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:33.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:33 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:33.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:33 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:33.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:33 smithi037 ceph-mon[31460]: pgmap v2004: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:33.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:33 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:33.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:33 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:33.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:33 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:33.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:33 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:34.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:34 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:34 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:34 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:34 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:34 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:34 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:34 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:34 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:34 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:34 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:34 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:34 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:34 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:34 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:34 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:34 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:34 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:34 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:34 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:34 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:34 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:34 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:34 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:34 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:34 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:34 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:34 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:34 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:34 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:34 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:34 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:34 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:34 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:34 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:34 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:34 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:34 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:34 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:34 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:34 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:34 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:34 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:34 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:34 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:34 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:34 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:34 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:34.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:34 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:35.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:35 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T19:13:35.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:35 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T19:13:35.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:35 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T19:13:35.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:35 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T19:13:35.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:35 smithi059 ceph-mon[36190]: pgmap v2005: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:35.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:35 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:35.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:35 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:35.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:35 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T19:13:35.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:35 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T19:13:35.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:35 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T19:13:35.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:35 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T19:13:35.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:35 smithi170 ceph-mon[36280]: pgmap v2005: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:35.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:35 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:35.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:35 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:35.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:35 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T19:13:35.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:35 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T19:13:35.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:35 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T19:13:35.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:35 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T19:13:35.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:35 smithi037 ceph-mon[31460]: pgmap v2005: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:35.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:35 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:35.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:35 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:37.427 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:37 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2811.155393 secs 2024-03-28T19:13:37.427 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:37 smithi037 ceph-mon[31460]: pgmap v2006: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:37.427 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:37 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.427 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:37 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.427 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:37 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.427 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:37 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.427 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:37 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.428 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:37 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.428 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:37 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.428 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:37 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.428 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:37 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T19:13:37.428 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:37 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T19:13:37.428 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:37 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:13:37.428 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:37 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:13:37.428 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:37 smithi037 ceph-mon[31460]: pgmap v2007: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:13:37.428 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:37 smithi037 ceph-mon[31460]: pgmap v2008: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:13:37.428 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:37 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:37.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:37 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2811.155393 secs 2024-03-28T19:13:37.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:37 smithi059 ceph-mon[36190]: pgmap v2006: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:37 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:37 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:37 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:37 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:37 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:37 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:37 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:37 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:37 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T19:13:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:37 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T19:13:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:37 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:13:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:37 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:13:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:37 smithi059 ceph-mon[36190]: pgmap v2007: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:13:37.535 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:37 smithi059 ceph-mon[36190]: pgmap v2008: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:13:37.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:37 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:37.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:37 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2811.155393 secs 2024-03-28T19:13:37.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:37 smithi170 ceph-mon[36280]: pgmap v2006: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:37.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:37 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:37 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:37 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:37 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:37 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.536 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:37 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:37 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:37 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:13:37.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:37 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T19:13:37.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:37 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T19:13:37.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:37 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:13:37.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:37 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:13:37.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:37 smithi170 ceph-mon[36280]: pgmap v2007: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:13:37.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:37 smithi170 ceph-mon[36280]: pgmap v2008: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:13:37.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:37 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:13:39.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:39 smithi059 ceph-mon[36190]: pgmap v2009: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:13:39.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:39 smithi170 ceph-mon[36280]: pgmap v2009: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:13:39.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:39 smithi037 ceph-mon[31460]: pgmap v2009: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:13:41.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:41 smithi059 ceph-mon[36190]: pgmap v2010: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:13:41.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2816.155511 secs 2024-03-28T19:13:41.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:41 smithi170 ceph-mon[36280]: pgmap v2010: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:13:41.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2816.155511 secs 2024-03-28T19:13:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:41 smithi037 ceph-mon[31460]: pgmap v2010: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:13:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2816.155511 secs 2024-03-28T19:13:43.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:43 smithi059 ceph-mon[36190]: pgmap v2011: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:13:43.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:43 smithi170 ceph-mon[36280]: pgmap v2011: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:13:43.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:43 smithi037 ceph-mon[31460]: pgmap v2011: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:13:45.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:45 smithi059 ceph-mon[36190]: pgmap v2012: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:13:45.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:45 smithi170 ceph-mon[36280]: pgmap v2012: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:13:45.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:45 smithi037 ceph-mon[31460]: pgmap v2012: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:13:46.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2821.155632 secs 2024-03-28T19:13:46.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2821.155632 secs 2024-03-28T19:13:46.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2821.155632 secs 2024-03-28T19:13:47.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:47 smithi059 ceph-mon[36190]: pgmap v2013: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:13:47.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:47 smithi170 ceph-mon[36280]: pgmap v2013: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:13:47.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:47 smithi037 ceph-mon[31460]: pgmap v2013: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:13:49.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:49 smithi059 ceph-mon[36190]: pgmap v2014: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:49.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:49 smithi170 ceph-mon[36280]: pgmap v2014: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:49.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:49 smithi037 ceph-mon[31460]: pgmap v2014: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:51.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:51 smithi059 ceph-mon[36190]: pgmap v2015: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:51.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2826.155750 secs 2024-03-28T19:13:51.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:51 smithi170 ceph-mon[36280]: pgmap v2015: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:51.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2826.155750 secs 2024-03-28T19:13:51.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:51 smithi037 ceph-mon[31460]: pgmap v2015: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:51.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2826.155750 secs 2024-03-28T19:13:53.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:53 smithi059 ceph-mon[36190]: pgmap v2016: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:53.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:53 smithi170 ceph-mon[36280]: pgmap v2016: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:53.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:53 smithi037 ceph-mon[31460]: pgmap v2016: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:55.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:55 smithi059 ceph-mon[36190]: pgmap v2017: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:55.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:55 smithi170 ceph-mon[36280]: pgmap v2017: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:55.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:55 smithi037 ceph-mon[31460]: pgmap v2017: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:56.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2831.155847 secs 2024-03-28T19:13:56.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2831.155847 secs 2024-03-28T19:13:56.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2831.155847 secs 2024-03-28T19:13:57.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:57 smithi059 ceph-mon[36190]: pgmap v2018: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:57.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:57 smithi170 ceph-mon[36280]: pgmap v2018: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:57.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:57 smithi037 ceph-mon[31460]: pgmap v2018: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:59.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:13:59 smithi059 ceph-mon[36190]: pgmap v2019: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:59.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:13:59 smithi170 ceph-mon[36280]: pgmap v2019: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:59.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:13:59 smithi037 ceph-mon[31460]: pgmap v2019: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:01.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:01 smithi059 ceph-mon[36190]: pgmap v2020: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:01.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2836.155913 secs 2024-03-28T19:14:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:01 smithi170 ceph-mon[36280]: pgmap v2020: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2836.155913 secs 2024-03-28T19:14:01.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:01 smithi037 ceph-mon[31460]: pgmap v2020: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:01.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2836.155913 secs 2024-03-28T19:14:03.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:03 smithi170 ceph-mon[36280]: pgmap v2021: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:03.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:03 smithi059 ceph-mon[36190]: pgmap v2021: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:03.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:03 smithi037 ceph-mon[31460]: pgmap v2021: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:05.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:05 smithi059 ceph-mon[36190]: pgmap v2022: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:05.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:05 smithi170 ceph-mon[36280]: pgmap v2022: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:05.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:05 smithi037 ceph-mon[31460]: pgmap v2022: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2841.155980 secs 2024-03-28T19:14:06.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2841.155980 secs 2024-03-28T19:14:06.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2841.155980 secs 2024-03-28T19:14:07.775 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:07 smithi037 ceph-mon[31460]: pgmap v2023: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:07.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:07 smithi059 ceph-mon[36190]: pgmap v2023: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:07.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:07 smithi170 ceph-mon[36280]: pgmap v2023: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:08.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:08 smithi059 ceph-mon[36190]: pgmap v2024: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:08.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:08 smithi170 ceph-mon[36280]: pgmap v2024: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:08.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:08 smithi037 ceph-mon[31460]: pgmap v2024: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:11 smithi059 ceph-mon[36190]: pgmap v2025: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2846.156096 secs 2024-03-28T19:14:11.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:11 smithi170 ceph-mon[36280]: pgmap v2025: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:11.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2846.156096 secs 2024-03-28T19:14:11.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:11 smithi037 ceph-mon[31460]: pgmap v2025: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:11.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2846.156096 secs 2024-03-28T19:14:13.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:13 smithi059 ceph-mon[36190]: pgmap v2026: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:13.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:13 smithi170 ceph-mon[36280]: pgmap v2026: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:13.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:13 smithi037 ceph-mon[31460]: pgmap v2026: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:15.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:15 smithi059 ceph-mon[36190]: pgmap v2027: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:15.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:15 smithi170 ceph-mon[36280]: pgmap v2027: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:15.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:15 smithi037 ceph-mon[31460]: pgmap v2027: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:16.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2851.156215 secs 2024-03-28T19:14:16.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2851.156215 secs 2024-03-28T19:14:16.827 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2851.156215 secs 2024-03-28T19:14:17.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:17 smithi059 ceph-mon[36190]: pgmap v2028: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:17.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:17 smithi170 ceph-mon[36280]: pgmap v2028: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:17.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:17 smithi037 ceph-mon[31460]: pgmap v2028: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:19.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:19 smithi059 ceph-mon[36190]: pgmap v2029: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:19.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:19 smithi170 ceph-mon[36280]: pgmap v2029: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:19.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:19 smithi037 ceph-mon[31460]: pgmap v2029: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:21.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:21 smithi059 ceph-mon[36190]: pgmap v2030: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:21.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2856.156335 secs 2024-03-28T19:14:21.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:21 smithi170 ceph-mon[36280]: pgmap v2030: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:21.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2856.156335 secs 2024-03-28T19:14:21.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:21 smithi037 ceph-mon[31460]: pgmap v2030: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:21.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2856.156335 secs 2024-03-28T19:14:23.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:23 smithi059 ceph-mon[36190]: pgmap v2031: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:23.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:23 smithi170 ceph-mon[36280]: pgmap v2031: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:23.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:23 smithi037 ceph-mon[31460]: pgmap v2031: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:25.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:25 smithi059 ceph-mon[36190]: pgmap v2032: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:25 smithi170 ceph-mon[36280]: pgmap v2032: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:25.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:25 smithi037 ceph-mon[31460]: pgmap v2032: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:26.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2861.156453 secs 2024-03-28T19:14:26.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2861.156453 secs 2024-03-28T19:14:26.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2861.156453 secs 2024-03-28T19:14:27.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:27 smithi059 ceph-mon[36190]: pgmap v2033: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:27.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:27 smithi170 ceph-mon[36280]: pgmap v2033: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:27.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:27 smithi037 ceph-mon[31460]: pgmap v2033: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:29.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:29 smithi059 ceph-mon[36190]: pgmap v2034: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:29.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:29 smithi170 ceph-mon[36280]: pgmap v2034: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:29.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:29 smithi037 ceph-mon[31460]: pgmap v2034: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:31.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:31 smithi059 ceph-mon[36190]: pgmap v2035: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:31.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2866.156557 secs 2024-03-28T19:14:31.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:31 smithi170 ceph-mon[36280]: pgmap v2035: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:31.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2866.156557 secs 2024-03-28T19:14:31.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:31 smithi037 ceph-mon[31460]: pgmap v2035: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2866.156557 secs 2024-03-28T19:14:33.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:33 smithi059 ceph-mon[36190]: pgmap v2036: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:33.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:33 smithi170 ceph-mon[36280]: pgmap v2036: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:33.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:33 smithi037 ceph-mon[31460]: pgmap v2036: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:35.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:35 smithi059 ceph-mon[36190]: pgmap v2037: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:35.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:35 smithi170 ceph-mon[36280]: pgmap v2037: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:35.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:35 smithi037 ceph-mon[31460]: pgmap v2037: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:36.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2871.156644 secs 2024-03-28T19:14:36.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:36 smithi059 ceph-mon[36190]: pgmap v2038: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:36.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2871.156644 secs 2024-03-28T19:14:36.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:36 smithi170 ceph-mon[36280]: pgmap v2038: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:36.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2871.156644 secs 2024-03-28T19:14:36.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:36 smithi037 ceph-mon[31460]: pgmap v2038: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:37.775 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:37 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:14:37.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:37 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:14:37.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:37 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:14:38.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:38 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:14:38.820 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:38 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:14:38.820 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:38 smithi059 ceph-mon[36190]: pgmap v2039: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:14:38.820 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:38 smithi059 ceph-mon[36190]: pgmap v2040: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:38.820 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:38 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:14:38.821 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:38 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:14:38.821 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:38 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:14:38.821 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:38 smithi170 ceph-mon[36280]: pgmap v2039: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:14:38.821 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:38 smithi170 ceph-mon[36280]: pgmap v2040: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:38.821 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:38 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:14:38.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:38 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:14:38.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:38 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:14:38.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:38 smithi037 ceph-mon[31460]: pgmap v2039: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:14:38.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:38 smithi037 ceph-mon[31460]: pgmap v2040: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:38.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:38 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:14:41.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:40 smithi037 ceph-mon[31460]: pgmap v2041: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:41.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:40 smithi059 ceph-mon[36190]: pgmap v2041: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:41.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:40 smithi170 ceph-mon[36280]: pgmap v2041: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:42.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2876.156763 secs 2024-03-28T19:14:42.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2876.156763 secs 2024-03-28T19:14:42.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2876.156763 secs 2024-03-28T19:14:43.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:42 smithi037 ceph-mon[31460]: pgmap v2042: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:43.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:42 smithi059 ceph-mon[36190]: pgmap v2042: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:43.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:42 smithi170 ceph-mon[36280]: pgmap v2042: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:45.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:44 smithi037 ceph-mon[31460]: pgmap v2043: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:45.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:44 smithi059 ceph-mon[36190]: pgmap v2043: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:45.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:44 smithi170 ceph-mon[36280]: pgmap v2043: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:46 smithi037 ceph-mon[31460]: 2 slow requests, 1 included below; oldest blocked for > 2881.156882 secs 2024-03-28T19:14:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:46 smithi037 ceph-mon[31460]: slow request 2881.156881 seconds old, received at 2024-03-28T18:26:44.515856+0000: peer_request:client.25216:13687 currently acquired locks 2024-03-28T19:14:47.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:46 smithi037 ceph-mon[31460]: pgmap v2044: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:47.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:46 smithi059 ceph-mon[36190]: 2 slow requests, 1 included below; oldest blocked for > 2881.156882 secs 2024-03-28T19:14:47.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:46 smithi059 ceph-mon[36190]: slow request 2881.156881 seconds old, received at 2024-03-28T18:26:44.515856+0000: peer_request:client.25216:13687 currently acquired locks 2024-03-28T19:14:47.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:46 smithi059 ceph-mon[36190]: pgmap v2044: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:46 smithi170 ceph-mon[36280]: 2 slow requests, 1 included below; oldest blocked for > 2881.156882 secs 2024-03-28T19:14:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:46 smithi170 ceph-mon[36280]: slow request 2881.156881 seconds old, received at 2024-03-28T18:26:44.515856+0000: peer_request:client.25216:13687 currently acquired locks 2024-03-28T19:14:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:46 smithi170 ceph-mon[36280]: pgmap v2044: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:49.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:48 smithi059 ceph-mon[36190]: pgmap v2045: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:14:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:48 smithi170 ceph-mon[36280]: pgmap v2045: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:14:49.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:48 smithi037 ceph-mon[31460]: pgmap v2045: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:14:51.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:50 smithi059 ceph-mon[36190]: pgmap v2046: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:50 smithi170 ceph-mon[36280]: pgmap v2046: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:51.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:50 smithi037 ceph-mon[31460]: pgmap v2046: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:52.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2886.157003 secs 2024-03-28T19:14:52.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2886.157003 secs 2024-03-28T19:14:52.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2886.157003 secs 2024-03-28T19:14:53.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:52 smithi059 ceph-mon[36190]: pgmap v2047: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:53.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:52 smithi170 ceph-mon[36280]: pgmap v2047: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:53.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:52 smithi037 ceph-mon[31460]: pgmap v2047: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:55.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:54 smithi059 ceph-mon[36190]: pgmap v2048: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:55.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:54 smithi170 ceph-mon[36280]: pgmap v2048: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:55.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:54 smithi037 ceph-mon[31460]: pgmap v2048: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2891.157102 secs 2024-03-28T19:14:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:56 smithi059 ceph-mon[36190]: pgmap v2049: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2891.157102 secs 2024-03-28T19:14:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:56 smithi170 ceph-mon[36280]: pgmap v2049: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:57.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2891.157102 secs 2024-03-28T19:14:57.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:56 smithi037 ceph-mon[31460]: pgmap v2049: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:59.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:14:58 smithi059 ceph-mon[36190]: pgmap v2050: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:59.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:14:58 smithi170 ceph-mon[36280]: pgmap v2050: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:59.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:14:58 smithi037 ceph-mon[31460]: pgmap v2050: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:01.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:00 smithi059 ceph-mon[36190]: pgmap v2051: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:00 smithi170 ceph-mon[36280]: pgmap v2051: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:00 smithi037 ceph-mon[31460]: pgmap v2051: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:02.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:01 smithi059 ceph-mon[36190]: 2 slow requests, 1 included below; oldest blocked for > 2896.157221 secs 2024-03-28T19:15:02.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:01 smithi059 ceph-mon[36190]: slow request 2880.576895 seconds old, received at 2024-03-28T18:27:00.096181+0000: client_request(client.25553:180 mksnap #0x1//scheduled-2024-03-28-18_27_00_UTC 2024-03-28T18:27:00.095960+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T19:15:02.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:01 smithi170 ceph-mon[36280]: 2 slow requests, 1 included below; oldest blocked for > 2896.157221 secs 2024-03-28T19:15:02.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:01 smithi170 ceph-mon[36280]: slow request 2880.576895 seconds old, received at 2024-03-28T18:27:00.096181+0000: client_request(client.25553:180 mksnap #0x1//scheduled-2024-03-28-18_27_00_UTC 2024-03-28T18:27:00.095960+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T19:15:02.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:01 smithi037 ceph-mon[31460]: 2 slow requests, 1 included below; oldest blocked for > 2896.157221 secs 2024-03-28T19:15:02.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:01 smithi037 ceph-mon[31460]: slow request 2880.576895 seconds old, received at 2024-03-28T18:27:00.096181+0000: client_request(client.25553:180 mksnap #0x1//scheduled-2024-03-28-18_27_00_UTC 2024-03-28T18:27:00.095960+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T19:15:03.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:02 smithi059 ceph-mon[36190]: pgmap v2052: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:03.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:02 smithi170 ceph-mon[36280]: pgmap v2052: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:03.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:02 smithi037 ceph-mon[31460]: pgmap v2052: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:05.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:04 smithi059 ceph-mon[36190]: pgmap v2053: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:05.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:04 smithi170 ceph-mon[36280]: pgmap v2053: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:05.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:04 smithi037 ceph-mon[31460]: pgmap v2053: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:07.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2901.157340 secs 2024-03-28T19:15:07.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:06 smithi059 ceph-mon[36190]: pgmap v2054: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:07.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2901.157340 secs 2024-03-28T19:15:07.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:06 smithi170 ceph-mon[36280]: pgmap v2054: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:07.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2901.157340 secs 2024-03-28T19:15:07.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:06 smithi037 ceph-mon[31460]: pgmap v2054: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:09.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:08 smithi059 ceph-mon[36190]: pgmap v2055: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:09.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:08 smithi170 ceph-mon[36280]: pgmap v2055: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:09.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:08 smithi037 ceph-mon[31460]: pgmap v2055: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:11.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:10 smithi059 ceph-mon[36190]: pgmap v2056: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:11.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:10 smithi170 ceph-mon[36280]: pgmap v2056: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:11.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:10 smithi037 ceph-mon[31460]: pgmap v2056: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:12.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2906.157494 secs 2024-03-28T19:15:12.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2906.157494 secs 2024-03-28T19:15:12.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2906.157494 secs 2024-03-28T19:15:13.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:12 smithi059 ceph-mon[36190]: pgmap v2057: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:13.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:12 smithi170 ceph-mon[36280]: pgmap v2057: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:13.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:12 smithi037 ceph-mon[31460]: pgmap v2057: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:15.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:14 smithi059 ceph-mon[36190]: pgmap v2058: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:15.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:14 smithi170 ceph-mon[36280]: pgmap v2058: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:15.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:14 smithi037 ceph-mon[31460]: pgmap v2058: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:17.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2911.157596 secs 2024-03-28T19:15:17.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:16 smithi059 ceph-mon[36190]: pgmap v2059: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2911.157596 secs 2024-03-28T19:15:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:16 smithi170 ceph-mon[36280]: pgmap v2059: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:17.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2911.157596 secs 2024-03-28T19:15:17.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:16 smithi037 ceph-mon[31460]: pgmap v2059: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:19.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:18 smithi037 ceph-mon[31460]: pgmap v2060: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:19.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:18 smithi059 ceph-mon[36190]: pgmap v2060: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:19.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:18 smithi170 ceph-mon[36280]: pgmap v2060: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:21.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:20 smithi059 ceph-mon[36190]: pgmap v2061: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:21.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:20 smithi170 ceph-mon[36280]: pgmap v2061: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:21.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:20 smithi037 ceph-mon[31460]: pgmap v2061: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:22.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2916.157709 secs 2024-03-28T19:15:22.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2916.157709 secs 2024-03-28T19:15:22.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2916.157709 secs 2024-03-28T19:15:23.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:22 smithi059 ceph-mon[36190]: pgmap v2062: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:23.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:22 smithi170 ceph-mon[36280]: pgmap v2062: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:23.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:22 smithi037 ceph-mon[31460]: pgmap v2062: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:25.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:24 smithi059 ceph-mon[36190]: pgmap v2063: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:25.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:24 smithi170 ceph-mon[36280]: pgmap v2063: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:25.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:24 smithi037 ceph-mon[31460]: pgmap v2063: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:27.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2921.157828 secs 2024-03-28T19:15:27.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:26 smithi059 ceph-mon[36190]: pgmap v2064: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2921.157828 secs 2024-03-28T19:15:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:26 smithi170 ceph-mon[36280]: pgmap v2064: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:27.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2921.157828 secs 2024-03-28T19:15:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:26 smithi037 ceph-mon[31460]: pgmap v2064: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:29.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:28 smithi059 ceph-mon[36190]: pgmap v2065: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:29.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:28 smithi170 ceph-mon[36280]: pgmap v2065: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:29.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:28 smithi037 ceph-mon[31460]: pgmap v2065: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:31.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:31 smithi059 ceph-mon[36190]: pgmap v2066: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:31.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:30 smithi170 ceph-mon[36280]: pgmap v2066: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:31.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:30 smithi037 ceph-mon[31460]: pgmap v2066: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:32.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:32 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2926.157944 secs 2024-03-28T19:15:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:32 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2926.157944 secs 2024-03-28T19:15:32.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:32 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2926.157944 secs 2024-03-28T19:15:33.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:33 smithi059 ceph-mon[36190]: pgmap v2067: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:33.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:33 smithi170 ceph-mon[36280]: pgmap v2067: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:33.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:33 smithi037 ceph-mon[31460]: pgmap v2067: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:35.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:35 smithi059 ceph-mon[36190]: pgmap v2068: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:35.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:35 smithi170 ceph-mon[36280]: pgmap v2068: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:35.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:35 smithi037 ceph-mon[31460]: pgmap v2068: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:36.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2931.158074 secs 2024-03-28T19:15:36.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:36 smithi037 ceph-mon[31460]: pgmap v2069: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:37.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2931.158074 secs 2024-03-28T19:15:37.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:36 smithi059 ceph-mon[36190]: pgmap v2069: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:37.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2931.158074 secs 2024-03-28T19:15:37.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:36 smithi170 ceph-mon[36280]: pgmap v2069: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:39.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:38 smithi059 ceph-mon[36190]: pgmap v2070: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:39.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:38 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:15:39.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:38 smithi170 ceph-mon[36280]: pgmap v2070: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:39.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:38 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:15:39.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:38 smithi037 ceph-mon[31460]: pgmap v2070: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:39.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:38 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:15:40.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:39 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:15:40.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:39 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:15:40.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:39 smithi059 ceph-mon[36190]: pgmap v2071: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:40.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:39 smithi059 ceph-mon[36190]: pgmap v2072: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:40.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:39 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:15:40.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:39 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:15:40.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:39 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:15:40.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:39 smithi170 ceph-mon[36280]: pgmap v2071: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:40.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:39 smithi170 ceph-mon[36280]: pgmap v2072: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:40.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:39 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:15:40.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:39 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:15:40.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:39 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:15:40.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:39 smithi037 ceph-mon[31460]: pgmap v2071: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:40.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:39 smithi037 ceph-mon[31460]: pgmap v2072: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:40.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:39 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:15:41.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2936.158161 secs 2024-03-28T19:15:41.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2936.158161 secs 2024-03-28T19:15:41.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2936.158161 secs 2024-03-28T19:15:42.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:42 smithi059 ceph-mon[36190]: pgmap v2073: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:15:42.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:42 smithi170 ceph-mon[36280]: pgmap v2073: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:15:42.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:42 smithi037 ceph-mon[31460]: pgmap v2073: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:15:44.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:44 smithi037 ceph-mon[31460]: pgmap v2074: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:44.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:44 smithi059 ceph-mon[36190]: pgmap v2074: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:44.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:44 smithi170 ceph-mon[36280]: pgmap v2074: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:46.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:46 smithi037 ceph-mon[31460]: pgmap v2075: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:46.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2941.158281 secs 2024-03-28T19:15:46.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:46 smithi059 ceph-mon[36190]: pgmap v2075: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:46.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2941.158281 secs 2024-03-28T19:15:46.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:46 smithi170 ceph-mon[36280]: pgmap v2075: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:46.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2941.158281 secs 2024-03-28T19:15:48.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:48 smithi037 ceph-mon[31460]: pgmap v2076: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:48.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:48 smithi059 ceph-mon[36190]: pgmap v2076: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:48.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:48 smithi170 ceph-mon[36280]: pgmap v2076: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:50.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:50 smithi059 ceph-mon[36190]: pgmap v2077: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:15:50.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:50 smithi170 ceph-mon[36280]: pgmap v2077: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:15:50.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:50 smithi037 ceph-mon[31460]: pgmap v2077: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:15:51.685 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2946.158399 secs 2024-03-28T19:15:51.686 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2946.158399 secs 2024-03-28T19:15:51.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2946.158399 secs 2024-03-28T19:15:52.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:52 smithi059 ceph-mon[36190]: pgmap v2078: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:52.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:52 smithi170 ceph-mon[36280]: pgmap v2078: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:52.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:52 smithi037 ceph-mon[31460]: pgmap v2078: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:54.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:54 smithi059 ceph-mon[36190]: pgmap v2079: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:54.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:54 smithi170 ceph-mon[36280]: pgmap v2079: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:54.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:54 smithi037 ceph-mon[31460]: pgmap v2079: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:56.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:56 smithi059 ceph-mon[36190]: pgmap v2080: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2951.158499 secs 2024-03-28T19:15:56.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:56 smithi170 ceph-mon[36280]: pgmap v2080: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:56.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2951.158499 secs 2024-03-28T19:15:56.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:56 smithi037 ceph-mon[31460]: pgmap v2080: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:56.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2951.158499 secs 2024-03-28T19:15:58.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:15:58 smithi059 ceph-mon[36190]: pgmap v2081: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:58.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:15:58 smithi170 ceph-mon[36280]: pgmap v2081: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:58.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:15:58 smithi037 ceph-mon[31460]: pgmap v2081: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:00.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:00 smithi059 ceph-mon[36190]: pgmap v2082: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:00.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:00 smithi170 ceph-mon[36280]: pgmap v2082: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:00.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:00 smithi037 ceph-mon[31460]: pgmap v2082: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:01.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2956.158615 secs 2024-03-28T19:16:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2956.158615 secs 2024-03-28T19:16:01.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2956.158615 secs 2024-03-28T19:16:02.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:02 smithi059 ceph-mon[36190]: pgmap v2083: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:02.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:02 smithi170 ceph-mon[36280]: pgmap v2083: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:02.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:02 smithi037 ceph-mon[31460]: pgmap v2083: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:04.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:04 smithi059 ceph-mon[36190]: pgmap v2084: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:04.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:04 smithi170 ceph-mon[36280]: pgmap v2084: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:04.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:04 smithi037 ceph-mon[31460]: pgmap v2084: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:06 smithi059 ceph-mon[36190]: pgmap v2085: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2961.158736 secs 2024-03-28T19:16:06.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:06 smithi170 ceph-mon[36280]: pgmap v2085: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:06.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2961.158736 secs 2024-03-28T19:16:06.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:06 smithi037 ceph-mon[31460]: pgmap v2085: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:06.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2961.158736 secs 2024-03-28T19:16:08.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:08 smithi059 ceph-mon[36190]: pgmap v2086: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:08.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:08 smithi170 ceph-mon[36280]: pgmap v2086: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:08.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:08 smithi037 ceph-mon[31460]: pgmap v2086: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:10.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:10 smithi059 ceph-mon[36190]: pgmap v2087: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:10.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:10 smithi170 ceph-mon[36280]: pgmap v2087: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:10.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:10 smithi037 ceph-mon[31460]: pgmap v2087: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:11.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2966.158855 secs 2024-03-28T19:16:11.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:11 smithi037 ceph-mon[31460]: pgmap v2088: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:12.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2966.158855 secs 2024-03-28T19:16:12.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:11 smithi059 ceph-mon[36190]: pgmap v2088: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2966.158855 secs 2024-03-28T19:16:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:11 smithi170 ceph-mon[36280]: pgmap v2088: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:14.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:14 smithi059 ceph-mon[36190]: pgmap v2089: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:14.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:14 smithi170 ceph-mon[36280]: pgmap v2089: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:14.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:14 smithi037 ceph-mon[31460]: pgmap v2089: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:16.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:16 smithi059 ceph-mon[36190]: pgmap v2090: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:16.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2971.158969 secs 2024-03-28T19:16:16.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:16 smithi170 ceph-mon[36280]: pgmap v2090: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:16.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2971.158969 secs 2024-03-28T19:16:16.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:16 smithi037 ceph-mon[31460]: pgmap v2090: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:16.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2971.158969 secs 2024-03-28T19:16:18.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:18 smithi059 ceph-mon[36190]: pgmap v2091: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:18.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:18 smithi170 ceph-mon[36280]: pgmap v2091: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:18.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:18 smithi037 ceph-mon[31460]: pgmap v2091: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:20.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:20 smithi059 ceph-mon[36190]: pgmap v2092: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:20.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:20 smithi170 ceph-mon[36280]: pgmap v2092: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:20.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:20 smithi037 ceph-mon[31460]: pgmap v2092: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:21.686 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2976.159089 secs 2024-03-28T19:16:21.686 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2976.159089 secs 2024-03-28T19:16:21.755 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2976.159089 secs 2024-03-28T19:16:22.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:22 smithi059 ceph-mon[36190]: pgmap v2093: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:22.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:22 smithi170 ceph-mon[36280]: pgmap v2093: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:22.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:22 smithi037 ceph-mon[31460]: pgmap v2093: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:24.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:24 smithi059 ceph-mon[36190]: pgmap v2094: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:24.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:24 smithi170 ceph-mon[36280]: pgmap v2094: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:24.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:24 smithi037 ceph-mon[31460]: pgmap v2094: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:26.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:26 smithi059 ceph-mon[36190]: pgmap v2095: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:26.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2981.159210 secs 2024-03-28T19:16:26.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:26 smithi170 ceph-mon[36280]: pgmap v2095: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:26.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2981.159210 secs 2024-03-28T19:16:26.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:26 smithi037 ceph-mon[31460]: pgmap v2095: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:26.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2981.159210 secs 2024-03-28T19:16:28.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:28 smithi059 ceph-mon[36190]: pgmap v2096: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:16:28.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:28 smithi170 ceph-mon[36280]: pgmap v2096: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:16:28.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:28 smithi037 ceph-mon[31460]: pgmap v2096: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:16:30.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:30 smithi059 ceph-mon[36190]: pgmap v2097: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:16:30.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:30 smithi170 ceph-mon[36280]: pgmap v2097: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:16:30.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:30 smithi037 ceph-mon[31460]: pgmap v2097: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:16:31.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2986.159295 secs 2024-03-28T19:16:31.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2986.159295 secs 2024-03-28T19:16:31.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2986.159295 secs 2024-03-28T19:16:32.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:32 smithi059 ceph-mon[36190]: pgmap v2098: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:16:32.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:32 smithi170 ceph-mon[36280]: pgmap v2098: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:16:32.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:32 smithi037 ceph-mon[31460]: pgmap v2098: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:16:34.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:34 smithi059 ceph-mon[36190]: pgmap v2099: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:16:34.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:34 smithi170 ceph-mon[36280]: pgmap v2099: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:16:34.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:34 smithi037 ceph-mon[31460]: pgmap v2099: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:16:36.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:36 smithi059 ceph-mon[36190]: pgmap v2100: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:16:36.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2991.159383 secs 2024-03-28T19:16:36.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:36 smithi170 ceph-mon[36280]: pgmap v2100: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:16:36.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2991.159383 secs 2024-03-28T19:16:36.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:36 smithi037 ceph-mon[31460]: pgmap v2100: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:16:36.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2991.159383 secs 2024-03-28T19:16:38.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:38 smithi059 ceph-mon[36190]: pgmap v2101: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:16:38.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:38 smithi170 ceph-mon[36280]: pgmap v2101: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:16:38.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:38 smithi037 ceph-mon[31460]: pgmap v2101: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:16:40.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:40 smithi059 ceph-mon[36190]: pgmap v2102: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:40.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:40 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:16:40.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:40 smithi170 ceph-mon[36280]: pgmap v2102: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:40.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:40 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:16:40.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:40 smithi037 ceph-mon[31460]: pgmap v2102: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:40.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:40 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:16:41.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 2996.159459 secs 2024-03-28T19:16:41.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:41 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:16:41.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:41 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:16:41.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:41 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:16:41.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 2996.159459 secs 2024-03-28T19:16:41.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:41 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:16:41.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:41 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:16:41.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:41 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:16:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 2996.159459 secs 2024-03-28T19:16:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:41 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:16:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:41 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:16:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:41 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:16:42.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:42 smithi059 ceph-mon[36190]: pgmap v2103: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:42.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:42 smithi059 ceph-mon[36190]: pgmap v2104: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:16:42.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:42 smithi170 ceph-mon[36280]: pgmap v2103: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:42.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:42 smithi170 ceph-mon[36280]: pgmap v2104: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:16:42.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:42 smithi037 ceph-mon[31460]: pgmap v2103: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:42.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:42 smithi037 ceph-mon[31460]: pgmap v2104: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:16:44.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:44 smithi059 ceph-mon[36190]: pgmap v2105: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:16:44.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:44 smithi170 ceph-mon[36280]: pgmap v2105: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:16:44.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:44 smithi037 ceph-mon[31460]: pgmap v2105: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:16:46.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:46 smithi059 ceph-mon[36190]: pgmap v2106: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:16:46.790 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3001.159552 secs 2024-03-28T19:16:46.790 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:46 smithi170 ceph-mon[36280]: pgmap v2106: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:16:46.791 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3001.159552 secs 2024-03-28T19:16:46.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:46 smithi037 ceph-mon[31460]: pgmap v2106: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:16:46.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3001.159552 secs 2024-03-28T19:16:48.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:48 smithi059 ceph-mon[36190]: pgmap v2107: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:16:48.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:48 smithi170 ceph-mon[36280]: pgmap v2107: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:16:48.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:48 smithi037 ceph-mon[31460]: pgmap v2107: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:16:50.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:50 smithi059 ceph-mon[36190]: pgmap v2108: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:16:50.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:50 smithi170 ceph-mon[36280]: pgmap v2108: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:16:50.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:50 smithi037 ceph-mon[31460]: pgmap v2108: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:16:51.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3006.159646 secs 2024-03-28T19:16:51.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3006.159646 secs 2024-03-28T19:16:51.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3006.159646 secs 2024-03-28T19:16:52.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:52 smithi059 ceph-mon[36190]: pgmap v2109: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:16:52.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:52 smithi170 ceph-mon[36280]: pgmap v2109: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:16:52.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:52 smithi037 ceph-mon[31460]: pgmap v2109: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:16:54.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:54 smithi059 ceph-mon[36190]: pgmap v2110: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:54.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:54 smithi170 ceph-mon[36280]: pgmap v2110: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:54.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:54 smithi037 ceph-mon[31460]: pgmap v2110: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:56 smithi059 ceph-mon[36190]: pgmap v2111: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:56.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3011.159743 secs 2024-03-28T19:16:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:56 smithi170 ceph-mon[36280]: pgmap v2111: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3011.159743 secs 2024-03-28T19:16:56.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:56 smithi037 ceph-mon[31460]: pgmap v2111: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:56.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3011.159743 secs 2024-03-28T19:16:57.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:57 smithi059 ceph-mon[36190]: pgmap v2112: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:57.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:57 smithi170 ceph-mon[36280]: pgmap v2112: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:57.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:57 smithi037 ceph-mon[31460]: pgmap v2112: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:00.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:16:59 smithi037 ceph-mon[31460]: pgmap v2113: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:00.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:16:59 smithi059 ceph-mon[36190]: pgmap v2113: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:00.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:16:59 smithi170 ceph-mon[36280]: pgmap v2113: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:02.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3016.159857 secs 2024-03-28T19:17:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:01 smithi037 ceph-mon[31460]: pgmap v2114: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:02.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3016.159857 secs 2024-03-28T19:17:02.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:01 smithi059 ceph-mon[36190]: pgmap v2114: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:02.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3016.159857 secs 2024-03-28T19:17:02.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:01 smithi170 ceph-mon[36280]: pgmap v2114: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:04.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:03 smithi037 ceph-mon[31460]: pgmap v2115: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:04.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:03 smithi059 ceph-mon[36190]: pgmap v2115: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:04.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:03 smithi170 ceph-mon[36280]: pgmap v2115: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:06.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:05 smithi037 ceph-mon[31460]: pgmap v2116: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:06.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:05 smithi059 ceph-mon[36190]: pgmap v2116: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:06.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:05 smithi170 ceph-mon[36280]: pgmap v2116: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3021.159974 secs 2024-03-28T19:17:07.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3021.159974 secs 2024-03-28T19:17:07.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3021.159974 secs 2024-03-28T19:17:08.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:07 smithi037 ceph-mon[31460]: pgmap v2117: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:17:08.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:07 smithi059 ceph-mon[36190]: pgmap v2117: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:17:08.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:07 smithi170 ceph-mon[36280]: pgmap v2117: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:17:10.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:09 smithi037 ceph-mon[31460]: pgmap v2118: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:17:10.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:09 smithi059 ceph-mon[36190]: pgmap v2118: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:17:10.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:09 smithi170 ceph-mon[36280]: pgmap v2118: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:17:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3026.160095 secs 2024-03-28T19:17:12.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:11 smithi037 ceph-mon[31460]: pgmap v2119: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:17:12.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3026.160095 secs 2024-03-28T19:17:12.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:11 smithi059 ceph-mon[36190]: pgmap v2119: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:17:12.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3026.160095 secs 2024-03-28T19:17:12.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:11 smithi170 ceph-mon[36280]: pgmap v2119: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:17:14.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:13 smithi037 ceph-mon[31460]: pgmap v2120: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:17:14.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:13 smithi059 ceph-mon[36190]: pgmap v2120: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:17:14.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:13 smithi170 ceph-mon[36280]: pgmap v2120: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:17:16.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:15 smithi059 ceph-mon[36190]: pgmap v2121: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:17:16.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:15 smithi170 ceph-mon[36280]: pgmap v2121: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:17:16.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:15 smithi037 ceph-mon[31460]: pgmap v2121: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:17:17.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3031.160181 secs 2024-03-28T19:17:17.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3031.160181 secs 2024-03-28T19:17:17.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3031.160181 secs 2024-03-28T19:17:18.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:17 smithi059 ceph-mon[36190]: pgmap v2122: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:17:18.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:17 smithi170 ceph-mon[36280]: pgmap v2122: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:17:18.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:17 smithi037 ceph-mon[31460]: pgmap v2122: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:17:20.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:19 smithi059 ceph-mon[36190]: pgmap v2123: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:20.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:19 smithi170 ceph-mon[36280]: pgmap v2123: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:20.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:19 smithi037 ceph-mon[31460]: pgmap v2123: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:22.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3036.160265 secs 2024-03-28T19:17:22.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:21 smithi059 ceph-mon[36190]: pgmap v2124: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3036.160265 secs 2024-03-28T19:17:22.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:21 smithi170 ceph-mon[36280]: pgmap v2124: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:22.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3036.160265 secs 2024-03-28T19:17:22.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:21 smithi037 ceph-mon[31460]: pgmap v2124: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:24.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:23 smithi037 ceph-mon[31460]: pgmap v2125: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:24.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:23 smithi059 ceph-mon[36190]: pgmap v2125: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:24.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:23 smithi170 ceph-mon[36280]: pgmap v2125: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:26.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:25 smithi059 ceph-mon[36190]: pgmap v2126: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:26.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:25 smithi170 ceph-mon[36280]: pgmap v2126: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:26.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:25 smithi037 ceph-mon[31460]: pgmap v2126: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:27.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3041.160382 secs 2024-03-28T19:17:27.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3041.160382 secs 2024-03-28T19:17:27.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3041.160382 secs 2024-03-28T19:17:28.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:27 smithi059 ceph-mon[36190]: pgmap v2127: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:28.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:27 smithi170 ceph-mon[36280]: pgmap v2127: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:28.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:27 smithi037 ceph-mon[31460]: pgmap v2127: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:30.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:29 smithi059 ceph-mon[36190]: pgmap v2128: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:30.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:29 smithi170 ceph-mon[36280]: pgmap v2128: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:30.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:29 smithi037 ceph-mon[31460]: pgmap v2128: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:32.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3046.160489 secs 2024-03-28T19:17:32.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:31 smithi059 ceph-mon[36190]: pgmap v2129: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:32.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3046.160489 secs 2024-03-28T19:17:32.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:31 smithi170 ceph-mon[36280]: pgmap v2129: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:32.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3046.160489 secs 2024-03-28T19:17:32.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:31 smithi037 ceph-mon[31460]: pgmap v2129: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:34.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:33 smithi059 ceph-mon[36190]: pgmap v2130: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:34.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:33 smithi170 ceph-mon[36280]: pgmap v2130: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:34.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:33 smithi037 ceph-mon[31460]: pgmap v2130: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:36.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:35 smithi059 ceph-mon[36190]: pgmap v2131: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:36.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:35 smithi170 ceph-mon[36280]: pgmap v2131: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:36.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:35 smithi037 ceph-mon[31460]: pgmap v2131: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:37.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3051.160569 secs 2024-03-28T19:17:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3051.160569 secs 2024-03-28T19:17:37.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3051.160569 secs 2024-03-28T19:17:38.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:37 smithi059 ceph-mon[36190]: pgmap v2132: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:38.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:37 smithi170 ceph-mon[36280]: pgmap v2132: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:38.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:37 smithi037 ceph-mon[31460]: pgmap v2132: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:40.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:39 smithi059 ceph-mon[36190]: pgmap v2133: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:40.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:39 smithi170 ceph-mon[36280]: pgmap v2133: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:40.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:39 smithi037 ceph-mon[31460]: pgmap v2133: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:42.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3056.160689 secs 2024-03-28T19:17:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:41 smithi059 ceph-mon[36190]: pgmap v2134: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:42.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:41 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:17:42.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3056.160689 secs 2024-03-28T19:17:42.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:41 smithi170 ceph-mon[36280]: pgmap v2134: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:42.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:41 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:17:42.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3056.160689 secs 2024-03-28T19:17:42.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:41 smithi037 ceph-mon[31460]: pgmap v2134: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:42.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:41 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:17:43.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:42 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:17:43.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:42 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:17:43.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:42 smithi059 ceph-mon[36190]: pgmap v2135: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:17:43.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:42 smithi059 ceph-mon[36190]: pgmap v2136: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:43.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:42 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:17:43.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:42 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:17:43.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:42 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:17:43.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:42 smithi170 ceph-mon[36280]: pgmap v2135: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:17:43.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:42 smithi170 ceph-mon[36280]: pgmap v2136: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:43.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:42 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:17:43.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:42 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:17:43.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:42 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:17:43.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:42 smithi037 ceph-mon[31460]: pgmap v2135: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:17:43.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:42 smithi037 ceph-mon[31460]: pgmap v2136: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:43.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:42 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:17:45.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:45 smithi037 ceph-mon[31460]: pgmap v2137: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:45.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:45 smithi059 ceph-mon[36190]: pgmap v2137: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:45.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:45 smithi170 ceph-mon[36280]: pgmap v2137: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:46.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3061.160803 secs 2024-03-28T19:17:46.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3061.160803 secs 2024-03-28T19:17:46.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3061.160803 secs 2024-03-28T19:17:47.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:47 smithi037 ceph-mon[31460]: pgmap v2138: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:17:47.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:47 smithi059 ceph-mon[36190]: pgmap v2138: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:17:47.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:47 smithi170 ceph-mon[36280]: pgmap v2138: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:17:49.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:49 smithi037 ceph-mon[31460]: pgmap v2139: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:49.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:49 smithi059 ceph-mon[36190]: pgmap v2139: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:49.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:49 smithi170 ceph-mon[36280]: pgmap v2139: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:51.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:51 smithi037 ceph-mon[31460]: pgmap v2140: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:51.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3066.160922 secs 2024-03-28T19:17:51.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:51 smithi059 ceph-mon[36190]: pgmap v2140: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:51.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3066.160922 secs 2024-03-28T19:17:51.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:51 smithi170 ceph-mon[36280]: pgmap v2140: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:51.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3066.160922 secs 2024-03-28T19:17:53.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:53 smithi037 ceph-mon[31460]: pgmap v2141: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:17:53.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:53 smithi059 ceph-mon[36190]: pgmap v2141: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:17:53.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:53 smithi170 ceph-mon[36280]: pgmap v2141: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:17:55.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:55 smithi037 ceph-mon[31460]: pgmap v2142: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:55.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:55 smithi059 ceph-mon[36190]: pgmap v2142: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:55.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:55 smithi170 ceph-mon[36280]: pgmap v2142: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:56.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3071.161053 secs 2024-03-28T19:17:56.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3071.161053 secs 2024-03-28T19:17:56.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3071.161053 secs 2024-03-28T19:17:57.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:57 smithi037 ceph-mon[31460]: pgmap v2143: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:57.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:57 smithi059 ceph-mon[36190]: pgmap v2143: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:57.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:57 smithi170 ceph-mon[36280]: pgmap v2143: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:59.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:17:59 smithi059 ceph-mon[36190]: pgmap v2144: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:59.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:17:59 smithi170 ceph-mon[36280]: pgmap v2144: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:59.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:17:59 smithi037 ceph-mon[31460]: pgmap v2144: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:01.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:01 smithi059 ceph-mon[36190]: pgmap v2145: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:01.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3076.161141 secs 2024-03-28T19:18:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:01 smithi170 ceph-mon[36280]: pgmap v2145: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:01.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3076.161141 secs 2024-03-28T19:18:01.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:01 smithi037 ceph-mon[31460]: pgmap v2145: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:01.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3076.161141 secs 2024-03-28T19:18:03.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:03 smithi059 ceph-mon[36190]: pgmap v2146: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:03.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:03 smithi170 ceph-mon[36280]: pgmap v2146: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:03.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:03 smithi037 ceph-mon[31460]: pgmap v2146: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:05.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:05 smithi059 ceph-mon[36190]: pgmap v2147: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:05.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:05 smithi170 ceph-mon[36280]: pgmap v2147: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:05.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:05 smithi037 ceph-mon[31460]: pgmap v2147: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3081.161228 secs 2024-03-28T19:18:06.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3081.161228 secs 2024-03-28T19:18:06.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3081.161228 secs 2024-03-28T19:18:07.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:07 smithi059 ceph-mon[36190]: pgmap v2148: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:07.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:07 smithi170 ceph-mon[36280]: pgmap v2148: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:07.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:07 smithi037 ceph-mon[31460]: pgmap v2148: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:09.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:09 smithi059 ceph-mon[36190]: pgmap v2149: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:09.788 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:09 smithi037 ceph-mon[31460]: pgmap v2149: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:09.789 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:09 smithi170 ceph-mon[36280]: pgmap v2149: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:11.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:11 smithi059 ceph-mon[36190]: pgmap v2150: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:11.787 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3086.161316 secs 2024-03-28T19:18:11.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:11 smithi170 ceph-mon[36280]: pgmap v2150: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:11.788 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3086.161316 secs 2024-03-28T19:18:11.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:11 smithi037 ceph-mon[31460]: pgmap v2150: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:11.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3086.161316 secs 2024-03-28T19:18:13.675 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:13 smithi037 ceph-mon[31460]: pgmap v2151: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:13.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:13 smithi059 ceph-mon[36190]: pgmap v2151: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:13.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:13 smithi170 ceph-mon[36280]: pgmap v2151: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:15.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:15 smithi059 ceph-mon[36190]: pgmap v2152: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:15.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:15 smithi170 ceph-mon[36280]: pgmap v2152: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:15.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:15 smithi037 ceph-mon[31460]: pgmap v2152: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:16.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3091.161402 secs 2024-03-28T19:18:16.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3091.161402 secs 2024-03-28T19:18:16.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3091.161402 secs 2024-03-28T19:18:17.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:17 smithi059 ceph-mon[36190]: pgmap v2153: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:17.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:17 smithi170 ceph-mon[36280]: pgmap v2153: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:17.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:17 smithi037 ceph-mon[31460]: pgmap v2153: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:19.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:19 smithi059 ceph-mon[36190]: pgmap v2154: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:19.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:19 smithi170 ceph-mon[36280]: pgmap v2154: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:19.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:19 smithi037 ceph-mon[31460]: pgmap v2154: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:21.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:21 smithi059 ceph-mon[36190]: pgmap v2155: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:21.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3096.161505 secs 2024-03-28T19:18:21.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:21 smithi170 ceph-mon[36280]: pgmap v2155: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:21.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3096.161505 secs 2024-03-28T19:18:21.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:21 smithi037 ceph-mon[31460]: pgmap v2155: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:21.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3096.161505 secs 2024-03-28T19:18:23.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:23 smithi059 ceph-mon[36190]: pgmap v2156: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:23.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:23 smithi170 ceph-mon[36280]: pgmap v2156: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:23.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:23 smithi037 ceph-mon[31460]: pgmap v2156: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:25.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:25 smithi059 ceph-mon[36190]: pgmap v2157: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:25.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:25 smithi170 ceph-mon[36280]: pgmap v2157: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:25.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:25 smithi037 ceph-mon[31460]: pgmap v2157: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:26.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3101.161587 secs 2024-03-28T19:18:26.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3101.161587 secs 2024-03-28T19:18:26.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3101.161587 secs 2024-03-28T19:18:27.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:27 smithi059 ceph-mon[36190]: pgmap v2158: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:27.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:27 smithi170 ceph-mon[36280]: pgmap v2158: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:27.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:27 smithi037 ceph-mon[31460]: pgmap v2158: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:29.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:29 smithi059 ceph-mon[36190]: pgmap v2159: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:29.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:29 smithi170 ceph-mon[36280]: pgmap v2159: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:29.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:29 smithi037 ceph-mon[31460]: pgmap v2159: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:31.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:31 smithi059 ceph-mon[36190]: pgmap v2160: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:31.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3106.161702 secs 2024-03-28T19:18:31.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:31 smithi170 ceph-mon[36280]: pgmap v2160: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:31.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3106.161702 secs 2024-03-28T19:18:31.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:31 smithi037 ceph-mon[31460]: pgmap v2160: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3106.161702 secs 2024-03-28T19:18:33.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:33 smithi059 ceph-mon[36190]: pgmap v2161: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:33.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:33 smithi170 ceph-mon[36280]: pgmap v2161: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:33.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:33 smithi037 ceph-mon[31460]: pgmap v2161: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:35.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:35 smithi059 ceph-mon[36190]: pgmap v2162: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:35.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:35 smithi170 ceph-mon[36280]: pgmap v2162: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:35.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:35 smithi037 ceph-mon[31460]: pgmap v2162: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:36.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3111.161788 secs 2024-03-28T19:18:36.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3111.161788 secs 2024-03-28T19:18:36.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3111.161788 secs 2024-03-28T19:18:37.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:37 smithi059 ceph-mon[36190]: pgmap v2163: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:37.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:37 smithi170 ceph-mon[36280]: pgmap v2163: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:37.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:37 smithi037 ceph-mon[31460]: pgmap v2163: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:39.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:39 smithi059 ceph-mon[36190]: pgmap v2164: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:39.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:39 smithi170 ceph-mon[36280]: pgmap v2164: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:39.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:39 smithi037 ceph-mon[31460]: pgmap v2164: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:41.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:41 smithi059 ceph-mon[36190]: pgmap v2165: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:41.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3116.161879 secs 2024-03-28T19:18:41.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:41 smithi170 ceph-mon[36280]: pgmap v2165: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:41.787 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3116.161879 secs 2024-03-28T19:18:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:41 smithi037 ceph-mon[31460]: pgmap v2165: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3116.161879 secs 2024-03-28T19:18:43.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:43 smithi059 ceph-mon[36190]: pgmap v2166: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:43.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:43 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:18:43.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:43 smithi170 ceph-mon[36280]: pgmap v2166: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:43.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:43 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:18:43.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:43 smithi037 ceph-mon[31460]: pgmap v2166: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:43.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:43 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:18:44.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:44 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:18:44.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:44 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:18:44.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:44 smithi059 ceph-mon[36190]: pgmap v2167: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:44.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:44 smithi059 ceph-mon[36190]: pgmap v2168: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:44.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:44 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:18:44.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:44 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:18:44.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:44 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:18:44.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:44 smithi170 ceph-mon[36280]: pgmap v2167: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:44.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:44 smithi170 ceph-mon[36280]: pgmap v2168: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:44.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:44 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:18:44.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:44 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:18:44.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:44 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:18:44.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:44 smithi037 ceph-mon[31460]: pgmap v2167: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:44.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:44 smithi037 ceph-mon[31460]: pgmap v2168: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:44.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:44 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:18:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3121.161994 secs 2024-03-28T19:18:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:46 smithi037 ceph-mon[31460]: pgmap v2169: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:47.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3121.161994 secs 2024-03-28T19:18:47.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:46 smithi059 ceph-mon[36190]: pgmap v2169: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3121.161994 secs 2024-03-28T19:18:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:46 smithi170 ceph-mon[36280]: pgmap v2169: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:49.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:48 smithi037 ceph-mon[31460]: pgmap v2170: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:49.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:48 smithi059 ceph-mon[36190]: pgmap v2170: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:49.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:48 smithi170 ceph-mon[36280]: pgmap v2170: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:51.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:50 smithi037 ceph-mon[31460]: pgmap v2171: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:51.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:50 smithi059 ceph-mon[36190]: pgmap v2171: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:51.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:50 smithi170 ceph-mon[36280]: pgmap v2171: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:52.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3126.162130 secs 2024-03-28T19:18:52.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3126.162130 secs 2024-03-28T19:18:52.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3126.162130 secs 2024-03-28T19:18:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:52 smithi037 ceph-mon[31460]: pgmap v2172: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:53.186 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:52 smithi059 ceph-mon[36190]: pgmap v2172: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:53.186 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:52 smithi170 ceph-mon[36280]: pgmap v2172: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:55.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:54 smithi037 ceph-mon[31460]: pgmap v2173: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:18:55.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:54 smithi059 ceph-mon[36190]: pgmap v2173: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:18:55.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:54 smithi170 ceph-mon[36280]: pgmap v2173: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:18:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3131.162218 secs 2024-03-28T19:18:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:56 smithi037 ceph-mon[31460]: pgmap v2174: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3131.162218 secs 2024-03-28T19:18:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:56 smithi059 ceph-mon[36190]: pgmap v2174: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3131.162218 secs 2024-03-28T19:18:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:56 smithi170 ceph-mon[36280]: pgmap v2174: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:59.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:18:58 smithi037 ceph-mon[31460]: pgmap v2175: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:59.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:18:58 smithi059 ceph-mon[36190]: pgmap v2175: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:59.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:18:58 smithi170 ceph-mon[36280]: pgmap v2175: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:01.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:00 smithi037 ceph-mon[31460]: pgmap v2176: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:01.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:00 smithi059 ceph-mon[36190]: pgmap v2176: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:00 smithi170 ceph-mon[36280]: pgmap v2176: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:02.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3136.162304 secs 2024-03-28T19:19:02.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3136.162304 secs 2024-03-28T19:19:02.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3136.162304 secs 2024-03-28T19:19:03.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:02 smithi037 ceph-mon[31460]: pgmap v2177: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:03.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:02 smithi059 ceph-mon[36190]: pgmap v2177: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:03.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:02 smithi170 ceph-mon[36280]: pgmap v2177: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:05.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:04 smithi037 ceph-mon[31460]: pgmap v2178: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:05.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:04 smithi059 ceph-mon[36190]: pgmap v2178: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:05.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:04 smithi170 ceph-mon[36280]: pgmap v2178: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3141.162389 secs 2024-03-28T19:19:07.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:06 smithi037 ceph-mon[31460]: pgmap v2179: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:07.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3141.162389 secs 2024-03-28T19:19:07.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:06 smithi059 ceph-mon[36190]: pgmap v2179: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:07.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3141.162389 secs 2024-03-28T19:19:07.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:06 smithi170 ceph-mon[36280]: pgmap v2179: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:09.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:08 smithi037 ceph-mon[31460]: pgmap v2180: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:09.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:08 smithi059 ceph-mon[36190]: pgmap v2180: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:09.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:08 smithi170 ceph-mon[36280]: pgmap v2180: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:11.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:10 smithi059 ceph-mon[36190]: pgmap v2181: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:11.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:10 smithi170 ceph-mon[36280]: pgmap v2181: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:11.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:10 smithi037 ceph-mon[31460]: pgmap v2181: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:12.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3146.162494 secs 2024-03-28T19:19:12.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3146.162494 secs 2024-03-28T19:19:12.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3146.162494 secs 2024-03-28T19:19:13.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:12 smithi037 ceph-mon[31460]: pgmap v2182: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:13.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:12 smithi059 ceph-mon[36190]: pgmap v2182: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:13.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:12 smithi170 ceph-mon[36280]: pgmap v2182: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:15.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:14 smithi059 ceph-mon[36190]: pgmap v2183: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:15.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:14 smithi170 ceph-mon[36280]: pgmap v2183: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:15.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:14 smithi037 ceph-mon[31460]: pgmap v2183: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:17.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3151.162638 secs 2024-03-28T19:19:17.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:16 smithi059 ceph-mon[36190]: pgmap v2184: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3151.162638 secs 2024-03-28T19:19:17.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:16 smithi170 ceph-mon[36280]: pgmap v2184: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:17.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3151.162638 secs 2024-03-28T19:19:17.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:16 smithi037 ceph-mon[31460]: pgmap v2184: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:19.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:18 smithi059 ceph-mon[36190]: pgmap v2185: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:19.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:18 smithi170 ceph-mon[36280]: pgmap v2185: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:19.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:18 smithi037 ceph-mon[31460]: pgmap v2185: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:21.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:20 smithi059 ceph-mon[36190]: pgmap v2186: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:21.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:20 smithi170 ceph-mon[36280]: pgmap v2186: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:21.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:20 smithi037 ceph-mon[31460]: pgmap v2186: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:22.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3156.162721 secs 2024-03-28T19:19:22.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3156.162721 secs 2024-03-28T19:19:22.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3156.162721 secs 2024-03-28T19:19:23.185 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:22 smithi059 ceph-mon[36190]: pgmap v2187: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:23.186 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:22 smithi170 ceph-mon[36280]: pgmap v2187: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:23.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:22 smithi037 ceph-mon[31460]: pgmap v2187: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:25.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:24 smithi059 ceph-mon[36190]: pgmap v2188: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:25.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:24 smithi170 ceph-mon[36280]: pgmap v2188: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:25.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:24 smithi037 ceph-mon[31460]: pgmap v2188: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:27.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3161.162840 secs 2024-03-28T19:19:27.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:26 smithi059 ceph-mon[36190]: pgmap v2189: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3161.162840 secs 2024-03-28T19:19:27.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:26 smithi170 ceph-mon[36280]: pgmap v2189: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:27.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3161.162840 secs 2024-03-28T19:19:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:26 smithi037 ceph-mon[31460]: pgmap v2189: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:29.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:28 smithi059 ceph-mon[36190]: pgmap v2190: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:29.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:28 smithi170 ceph-mon[36280]: pgmap v2190: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:29.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:28 smithi037 ceph-mon[31460]: pgmap v2190: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:31.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:30 smithi059 ceph-mon[36190]: pgmap v2191: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:31.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:30 smithi170 ceph-mon[36280]: pgmap v2191: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:31.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:30 smithi037 ceph-mon[31460]: pgmap v2191: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:32.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3166.162953 secs 2024-03-28T19:19:32.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3166.162953 secs 2024-03-28T19:19:32.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3166.162953 secs 2024-03-28T19:19:33.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:32 smithi059 ceph-mon[36190]: pgmap v2192: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:33.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:32 smithi170 ceph-mon[36280]: pgmap v2192: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:33.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:32 smithi037 ceph-mon[31460]: pgmap v2192: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:35.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:34 smithi059 ceph-mon[36190]: pgmap v2193: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:35.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:34 smithi170 ceph-mon[36280]: pgmap v2193: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:35.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:34 smithi037 ceph-mon[31460]: pgmap v2193: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:37.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3171.163060 secs 2024-03-28T19:19:37.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:36 smithi059 ceph-mon[36190]: pgmap v2194: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3171.163060 secs 2024-03-28T19:19:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:36 smithi170 ceph-mon[36280]: pgmap v2194: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3171.163060 secs 2024-03-28T19:19:37.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:36 smithi037 ceph-mon[31460]: pgmap v2194: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:39.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:38 smithi059 ceph-mon[36190]: pgmap v2195: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:39.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:38 smithi170 ceph-mon[36280]: pgmap v2195: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:39.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:38 smithi037 ceph-mon[31460]: pgmap v2195: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:41.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:40 smithi059 ceph-mon[36190]: pgmap v2196: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:41.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:40 smithi170 ceph-mon[36280]: pgmap v2196: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:41.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:40 smithi037 ceph-mon[31460]: pgmap v2196: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:42.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3176.163179 secs 2024-03-28T19:19:42.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3176.163179 secs 2024-03-28T19:19:42.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3176.163179 secs 2024-03-28T19:19:43.275 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:42 smithi037 ceph-mon[31460]: pgmap v2197: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:43.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:42 smithi059 ceph-mon[36190]: pgmap v2197: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:43.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:42 smithi170 ceph-mon[36280]: pgmap v2197: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:45.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:44 smithi059 ceph-mon[36190]: pgmap v2198: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:45.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:44 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:19:45.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:44 smithi170 ceph-mon[36280]: pgmap v2198: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:45.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:44 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:19:45.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:44 smithi037 ceph-mon[31460]: pgmap v2198: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:45.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:44 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:19:46.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:45 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:19:46.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:45 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:19:46.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:45 smithi059 ceph-mon[36190]: pgmap v2199: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:46.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:45 smithi059 ceph-mon[36190]: pgmap v2200: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:19:46.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:45 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:19:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:45 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:19:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:45 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:19:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:45 smithi170 ceph-mon[36280]: pgmap v2199: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:45 smithi170 ceph-mon[36280]: pgmap v2200: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:19:46.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:45 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:19:46.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:45 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:19:46.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:45 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:19:46.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:45 smithi037 ceph-mon[31460]: pgmap v2199: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:46.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:45 smithi037 ceph-mon[31460]: pgmap v2200: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:19:46.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:45 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:19:47.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3181.163269 secs 2024-03-28T19:19:47.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3181.163269 secs 2024-03-28T19:19:47.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3181.163269 secs 2024-03-28T19:19:48.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:48 smithi059 ceph-mon[36190]: pgmap v2201: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:19:48.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:48 smithi170 ceph-mon[36280]: pgmap v2201: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:19:48.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:48 smithi037 ceph-mon[31460]: pgmap v2201: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:19:50.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:50 smithi170 ceph-mon[36280]: pgmap v2202: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:19:50.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:50 smithi037 ceph-mon[31460]: pgmap v2202: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:19:50.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:50 smithi059 ceph-mon[36190]: pgmap v2202: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:19:51.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3186.163388 secs 2024-03-28T19:19:51.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3186.163388 secs 2024-03-28T19:19:51.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3186.163388 secs 2024-03-28T19:19:52.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:52 smithi037 ceph-mon[31460]: pgmap v2203: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:19:52.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:52 smithi059 ceph-mon[36190]: pgmap v2203: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:19:52.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:52 smithi170 ceph-mon[36280]: pgmap v2203: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:19:54.585 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:54 smithi170 ceph-mon[36280]: pgmap v2204: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:19:54.586 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:54 smithi059 ceph-mon[36190]: pgmap v2204: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:19:54.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:54 smithi037 ceph-mon[31460]: pgmap v2204: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:19:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:56 smithi037 ceph-mon[31460]: pgmap v2205: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:19:56.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3191.163465 secs 2024-03-28T19:19:56.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:56 smithi059 ceph-mon[36190]: pgmap v2205: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:19:56.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3191.163465 secs 2024-03-28T19:19:56.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:56 smithi170 ceph-mon[36280]: pgmap v2205: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:19:56.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3191.163465 secs 2024-03-28T19:19:58.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:19:58 smithi037 ceph-mon[31460]: pgmap v2206: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:58.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:19:58 smithi059 ceph-mon[36190]: pgmap v2206: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:58.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:19:58 smithi170 ceph-mon[36280]: pgmap v2206: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:00.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:00 smithi037 ceph-mon[31460]: pgmap v2207: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:00.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:00 smithi037 ceph-mon[31460]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T19:20:00.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:00 smithi059 ceph-mon[36190]: pgmap v2207: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:00.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:00 smithi059 ceph-mon[36190]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T19:20:00.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:00 smithi170 ceph-mon[36280]: pgmap v2207: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:00.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:00 smithi170 ceph-mon[36280]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T19:20:01.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3196.163620 secs 2024-03-28T19:20:01.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3196.163620 secs 2024-03-28T19:20:01.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3196.163620 secs 2024-03-28T19:20:02.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:02 smithi037 ceph-mon[31460]: pgmap v2208: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:02.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:02 smithi059 ceph-mon[36190]: pgmap v2208: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:02.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:02 smithi170 ceph-mon[36280]: pgmap v2208: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:04.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:04 smithi037 ceph-mon[31460]: pgmap v2209: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:04.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:04 smithi059 ceph-mon[36190]: pgmap v2209: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:04.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:04 smithi170 ceph-mon[36280]: pgmap v2209: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:06.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:06 smithi037 ceph-mon[31460]: pgmap v2210: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:06.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3201.163738 secs 2024-03-28T19:20:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:06 smithi059 ceph-mon[36190]: pgmap v2210: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3201.163738 secs 2024-03-28T19:20:06.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:06 smithi170 ceph-mon[36280]: pgmap v2210: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:06.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3201.163738 secs 2024-03-28T19:20:08.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:08 smithi037 ceph-mon[31460]: pgmap v2211: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:08.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:08 smithi059 ceph-mon[36190]: pgmap v2211: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:08.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:08 smithi170 ceph-mon[36280]: pgmap v2211: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:10.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:10 smithi037 ceph-mon[31460]: pgmap v2212: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:10.783 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:10 smithi059 ceph-mon[36190]: pgmap v2212: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:10.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:10 smithi170 ceph-mon[36280]: pgmap v2212: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:11.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3206.163856 secs 2024-03-28T19:20:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3206.163856 secs 2024-03-28T19:20:11.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3206.163856 secs 2024-03-28T19:20:12.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:12 smithi037 ceph-mon[31460]: pgmap v2213: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:12.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:12 smithi059 ceph-mon[36190]: pgmap v2213: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:12.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:12 smithi170 ceph-mon[36280]: pgmap v2213: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:14.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:14 smithi037 ceph-mon[31460]: pgmap v2214: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:14.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:14 smithi059 ceph-mon[36190]: pgmap v2214: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:14.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:14 smithi170 ceph-mon[36280]: pgmap v2214: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:16.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:16 smithi037 ceph-mon[31460]: pgmap v2215: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:16.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3211.164010 secs 2024-03-28T19:20:16.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:16 smithi059 ceph-mon[36190]: pgmap v2215: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:16.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3211.164010 secs 2024-03-28T19:20:16.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:16 smithi170 ceph-mon[36280]: pgmap v2215: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:16.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3211.164010 secs 2024-03-28T19:20:18.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:18 smithi037 ceph-mon[31460]: pgmap v2216: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:18.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:18 smithi059 ceph-mon[36190]: pgmap v2216: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:18.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:18 smithi170 ceph-mon[36280]: pgmap v2216: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:20.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:20 smithi037 ceph-mon[31460]: pgmap v2217: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:20.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:20 smithi059 ceph-mon[36190]: pgmap v2217: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:20.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:20 smithi170 ceph-mon[36280]: pgmap v2217: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:21.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3216.164129 secs 2024-03-28T19:20:21.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3216.164129 secs 2024-03-28T19:20:21.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3216.164129 secs 2024-03-28T19:20:22.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:22 smithi059 ceph-mon[36190]: pgmap v2218: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:22.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:22 smithi170 ceph-mon[36280]: pgmap v2218: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:22.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:22 smithi037 ceph-mon[31460]: pgmap v2218: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:24.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:24 smithi059 ceph-mon[36190]: pgmap v2219: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:24.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:24 smithi170 ceph-mon[36280]: pgmap v2219: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:24.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:24 smithi037 ceph-mon[31460]: pgmap v2219: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:26.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:26 smithi059 ceph-mon[36190]: pgmap v2220: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:26.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3221.164248 secs 2024-03-28T19:20:26.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:26 smithi170 ceph-mon[36280]: pgmap v2220: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:26.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3221.164248 secs 2024-03-28T19:20:26.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:26 smithi037 ceph-mon[31460]: pgmap v2220: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:26.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3221.164248 secs 2024-03-28T19:20:28.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:28 smithi059 ceph-mon[36190]: pgmap v2221: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:28.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:28 smithi170 ceph-mon[36280]: pgmap v2221: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:28.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:28 smithi037 ceph-mon[31460]: pgmap v2221: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:30.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:30 smithi059 ceph-mon[36190]: pgmap v2222: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:30.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:30 smithi170 ceph-mon[36280]: pgmap v2222: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:30.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:30 smithi037 ceph-mon[31460]: pgmap v2222: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:31.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3226.164331 secs 2024-03-28T19:20:31.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3226.164331 secs 2024-03-28T19:20:31.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3226.164331 secs 2024-03-28T19:20:32.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:32 smithi059 ceph-mon[36190]: pgmap v2223: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:32.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:32 smithi170 ceph-mon[36280]: pgmap v2223: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:32.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:32 smithi037 ceph-mon[31460]: pgmap v2223: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:34.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:34 smithi037 ceph-mon[31460]: pgmap v2224: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:34.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:34 smithi059 ceph-mon[36190]: pgmap v2224: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:34.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:34 smithi170 ceph-mon[36280]: pgmap v2224: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:36.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:36 smithi059 ceph-mon[36190]: pgmap v2225: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:36.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3231.164453 secs 2024-03-28T19:20:36.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:36 smithi170 ceph-mon[36280]: pgmap v2225: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:36.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3231.164453 secs 2024-03-28T19:20:36.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:36 smithi037 ceph-mon[31460]: pgmap v2225: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:36.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3231.164453 secs 2024-03-28T19:20:38.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:38 smithi059 ceph-mon[36190]: pgmap v2226: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:38.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:38 smithi170 ceph-mon[36280]: pgmap v2226: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:38.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:38 smithi037 ceph-mon[31460]: pgmap v2226: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:40.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:40 smithi059 ceph-mon[36190]: pgmap v2227: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:40.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:40 smithi170 ceph-mon[36280]: pgmap v2227: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:40.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:40 smithi037 ceph-mon[31460]: pgmap v2227: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:41.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3236.164573 secs 2024-03-28T19:20:41.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3236.164573 secs 2024-03-28T19:20:41.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3236.164573 secs 2024-03-28T19:20:42.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:42 smithi059 ceph-mon[36190]: pgmap v2228: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:42.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:42 smithi170 ceph-mon[36280]: pgmap v2228: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:42.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:42 smithi037 ceph-mon[31460]: pgmap v2228: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:44.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:44 smithi059 ceph-mon[36190]: pgmap v2229: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:44.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:44 smithi170 ceph-mon[36280]: pgmap v2229: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:44.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:44 smithi037 ceph-mon[31460]: pgmap v2229: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:46.706 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:46 smithi037 ceph-mon[31460]: pgmap v2230: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:46.707 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:46 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:20:46.707 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3241.164651 secs 2024-03-28T19:20:46.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:46 smithi059 ceph-mon[36190]: pgmap v2230: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:46.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:46 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:20:46.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3241.164651 secs 2024-03-28T19:20:46.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:46 smithi170 ceph-mon[36280]: pgmap v2230: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:46.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:46 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:20:46.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3241.164651 secs 2024-03-28T19:20:47.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:47 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:20:47.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:47 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:20:47.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:47 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:20:47.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:47 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:20:47.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:47 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:20:47.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:47 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:20:47.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:47 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:20:47.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:47 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:20:47.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:47 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:20:48.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:48 smithi037 ceph-mon[31460]: pgmap v2231: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:20:48.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:48 smithi037 ceph-mon[31460]: pgmap v2232: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:48.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:48 smithi059 ceph-mon[36190]: pgmap v2231: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:20:48.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:48 smithi059 ceph-mon[36190]: pgmap v2232: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:48.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:48 smithi170 ceph-mon[36280]: pgmap v2231: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:20:48.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:48 smithi170 ceph-mon[36280]: pgmap v2232: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:50.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:50 smithi059 ceph-mon[36190]: pgmap v2233: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:50.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:50 smithi170 ceph-mon[36280]: pgmap v2233: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:50.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:50 smithi037 ceph-mon[31460]: pgmap v2233: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:51.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3246.164771 secs 2024-03-28T19:20:51.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3246.164771 secs 2024-03-28T19:20:51.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3246.164771 secs 2024-03-28T19:20:52.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:52 smithi059 ceph-mon[36190]: pgmap v2234: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:52.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:52 smithi170 ceph-mon[36280]: pgmap v2234: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:52.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:52 smithi037 ceph-mon[31460]: pgmap v2234: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:54.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:54 smithi059 ceph-mon[36190]: pgmap v2235: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:54.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:54 smithi170 ceph-mon[36280]: pgmap v2235: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:54.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:54 smithi037 ceph-mon[31460]: pgmap v2235: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:56.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:56 smithi059 ceph-mon[36190]: pgmap v2236: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:56.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3251.164892 secs 2024-03-28T19:20:56.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:56 smithi170 ceph-mon[36280]: pgmap v2236: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:56.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3251.164892 secs 2024-03-28T19:20:56.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:56 smithi037 ceph-mon[31460]: pgmap v2236: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:56.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3251.164892 secs 2024-03-28T19:20:58.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:20:58 smithi059 ceph-mon[36190]: pgmap v2237: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:20:58.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:20:58 smithi170 ceph-mon[36280]: pgmap v2237: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:20:58.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:20:58 smithi037 ceph-mon[31460]: pgmap v2237: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:21:00.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:00 smithi059 ceph-mon[36190]: pgmap v2238: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:00.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:00 smithi170 ceph-mon[36280]: pgmap v2238: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:00.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:00 smithi037 ceph-mon[31460]: pgmap v2238: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:01.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3256.164970 secs 2024-03-28T19:21:01.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3256.164970 secs 2024-03-28T19:21:01.819 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3256.164970 secs 2024-03-28T19:21:02.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:02 smithi059 ceph-mon[36190]: pgmap v2239: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:02.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:02 smithi170 ceph-mon[36280]: pgmap v2239: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:02.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:02 smithi037 ceph-mon[31460]: pgmap v2239: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:04.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:04 smithi059 ceph-mon[36190]: pgmap v2240: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:04.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:04 smithi170 ceph-mon[36280]: pgmap v2240: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:04.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:04 smithi037 ceph-mon[31460]: pgmap v2240: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:06 smithi059 ceph-mon[36190]: pgmap v2241: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3261.165093 secs 2024-03-28T19:21:06.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:06 smithi170 ceph-mon[36280]: pgmap v2241: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:06.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3261.165093 secs 2024-03-28T19:21:06.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:06 smithi037 ceph-mon[31460]: pgmap v2241: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:06.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3261.165093 secs 2024-03-28T19:21:08.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:08 smithi059 ceph-mon[36190]: pgmap v2242: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:08.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:08 smithi170 ceph-mon[36280]: pgmap v2242: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:08.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:08 smithi037 ceph-mon[31460]: pgmap v2242: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:10.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:10 smithi059 ceph-mon[36190]: pgmap v2243: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:10.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:10 smithi170 ceph-mon[36280]: pgmap v2243: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:10.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:10 smithi037 ceph-mon[31460]: pgmap v2243: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3266.165215 secs 2024-03-28T19:21:11.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3266.165215 secs 2024-03-28T19:21:11.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3266.165215 secs 2024-03-28T19:21:12.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:12 smithi059 ceph-mon[36190]: pgmap v2244: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:12.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:12 smithi170 ceph-mon[36280]: pgmap v2244: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:12.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:12 smithi037 ceph-mon[31460]: pgmap v2244: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:14.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:14 smithi037 ceph-mon[31460]: pgmap v2245: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:15.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:14 smithi059 ceph-mon[36190]: pgmap v2245: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:15.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:14 smithi170 ceph-mon[36280]: pgmap v2245: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:16.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:16 smithi037 ceph-mon[31460]: pgmap v2246: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:16.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3271.165336 secs 2024-03-28T19:21:17.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:16 smithi059 ceph-mon[36190]: pgmap v2246: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:17.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3271.165336 secs 2024-03-28T19:21:17.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:16 smithi170 ceph-mon[36280]: pgmap v2246: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:17.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3271.165336 secs 2024-03-28T19:21:17.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:17 smithi037 ceph-mon[31460]: pgmap v2247: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:18.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:17 smithi059 ceph-mon[36190]: pgmap v2247: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:18.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:17 smithi170 ceph-mon[36280]: pgmap v2247: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:20.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:19 smithi037 ceph-mon[31460]: pgmap v2248: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:20.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:19 smithi059 ceph-mon[36190]: pgmap v2248: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:20.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:19 smithi170 ceph-mon[36280]: pgmap v2248: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:22.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3276.165456 secs 2024-03-28T19:21:22.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:21 smithi037 ceph-mon[31460]: pgmap v2249: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:22.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3276.165456 secs 2024-03-28T19:21:22.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:21 smithi059 ceph-mon[36190]: pgmap v2249: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:22.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3276.165456 secs 2024-03-28T19:21:22.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:21 smithi170 ceph-mon[36280]: pgmap v2249: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:24.085 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:23 smithi059 ceph-mon[36190]: pgmap v2250: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:24.086 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:23 smithi170 ceph-mon[36280]: pgmap v2250: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:24.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:23 smithi037 ceph-mon[31460]: pgmap v2250: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:26.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:25 smithi037 ceph-mon[31460]: pgmap v2251: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:26.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:25 smithi059 ceph-mon[36190]: pgmap v2251: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:26.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:25 smithi170 ceph-mon[36280]: pgmap v2251: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:27.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3281.165576 secs 2024-03-28T19:21:27.283 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3281.165576 secs 2024-03-28T19:21:27.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3281.165576 secs 2024-03-28T19:21:28.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:27 smithi037 ceph-mon[31460]: pgmap v2252: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:28.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:27 smithi059 ceph-mon[36190]: pgmap v2252: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:28.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:27 smithi170 ceph-mon[36280]: pgmap v2252: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:30.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:29 smithi037 ceph-mon[31460]: pgmap v2253: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:30.283 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:29 smithi059 ceph-mon[36190]: pgmap v2253: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:30.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:29 smithi170 ceph-mon[36280]: pgmap v2253: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:32.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3286.165697 secs 2024-03-28T19:21:32.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:31 smithi037 ceph-mon[31460]: pgmap v2254: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:32.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3286.165697 secs 2024-03-28T19:21:32.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:31 smithi059 ceph-mon[36190]: pgmap v2254: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:32.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3286.165697 secs 2024-03-28T19:21:32.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:31 smithi170 ceph-mon[36280]: pgmap v2254: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:34.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:33 smithi037 ceph-mon[31460]: pgmap v2255: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:34.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:33 smithi059 ceph-mon[36190]: pgmap v2255: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:34.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:33 smithi170 ceph-mon[36280]: pgmap v2255: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:36.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:35 smithi037 ceph-mon[31460]: pgmap v2256: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:36.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:35 smithi059 ceph-mon[36190]: pgmap v2256: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:36.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:35 smithi170 ceph-mon[36280]: pgmap v2256: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:37.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3291.165816 secs 2024-03-28T19:21:37.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3291.165816 secs 2024-03-28T19:21:37.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3291.165816 secs 2024-03-28T19:21:38.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:37 smithi037 ceph-mon[31460]: pgmap v2257: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:38.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:37 smithi059 ceph-mon[36190]: pgmap v2257: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:38.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:37 smithi170 ceph-mon[36280]: pgmap v2257: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:40.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:39 smithi037 ceph-mon[31460]: pgmap v2258: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:40.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:39 smithi059 ceph-mon[36190]: pgmap v2258: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:40.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:39 smithi170 ceph-mon[36280]: pgmap v2258: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:42.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3296.165937 secs 2024-03-28T19:21:42.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:41 smithi037 ceph-mon[31460]: pgmap v2259: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:42.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3296.165937 secs 2024-03-28T19:21:42.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:41 smithi059 ceph-mon[36190]: pgmap v2259: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:42.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3296.165937 secs 2024-03-28T19:21:42.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:41 smithi170 ceph-mon[36280]: pgmap v2259: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:44.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:43 smithi037 ceph-mon[31460]: pgmap v2260: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:44.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:43 smithi059 ceph-mon[36190]: pgmap v2260: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:44.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:43 smithi170 ceph-mon[36280]: pgmap v2260: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:46.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:45 smithi037 ceph-mon[31460]: pgmap v2261: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:46.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:45 smithi059 ceph-mon[36190]: pgmap v2261: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:46.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:45 smithi170 ceph-mon[36280]: pgmap v2261: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:47.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3301.166057 secs 2024-03-28T19:21:47.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3301.166057 secs 2024-03-28T19:21:47.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3301.166057 secs 2024-03-28T19:21:48.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:47 smithi059 ceph-mon[36190]: pgmap v2262: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:48.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:47 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:21:48.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:47 smithi170 ceph-mon[36280]: pgmap v2262: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:48.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:47 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:21:48.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:47 smithi037 ceph-mon[31460]: pgmap v2262: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:48.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:47 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:21:49.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:48 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:21:49.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:48 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:21:49.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:48 smithi059 ceph-mon[36190]: pgmap v2263: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:49.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:48 smithi059 ceph-mon[36190]: pgmap v2264: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:49.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:48 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:21:49.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:48 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:21:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:48 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:21:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:48 smithi170 ceph-mon[36280]: pgmap v2263: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:48 smithi170 ceph-mon[36280]: pgmap v2264: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:49.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:48 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:21:49.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:48 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:21:49.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:48 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:21:49.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:48 smithi037 ceph-mon[31460]: pgmap v2263: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:49.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:48 smithi037 ceph-mon[31460]: pgmap v2264: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:49.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:48 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:21:51.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:51 smithi170 ceph-mon[36280]: pgmap v2265: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:51.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3306.166181 secs 2024-03-28T19:21:51.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:51 smithi037 ceph-mon[31460]: pgmap v2265: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:51.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3306.166181 secs 2024-03-28T19:21:51.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:51 smithi059 ceph-mon[36190]: pgmap v2265: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:51.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3306.166181 secs 2024-03-28T19:21:53.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:53 smithi037 ceph-mon[31460]: pgmap v2266: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:53.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:53 smithi059 ceph-mon[36190]: pgmap v2266: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:53.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:53 smithi170 ceph-mon[36280]: pgmap v2266: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:55.586 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:55 smithi059 ceph-mon[36190]: pgmap v2267: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:55.587 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:55 smithi170 ceph-mon[36280]: pgmap v2267: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:55.625 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:55 smithi037 ceph-mon[31460]: pgmap v2267: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:56.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3311.166269 secs 2024-03-28T19:21:56.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3311.166269 secs 2024-03-28T19:21:56.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3311.166269 secs 2024-03-28T19:21:57.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:57 smithi037 ceph-mon[31460]: pgmap v2268: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:21:57.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:57 smithi059 ceph-mon[36190]: pgmap v2268: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:21:57.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:57 smithi170 ceph-mon[36280]: pgmap v2268: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:21:59.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:21:59 smithi037 ceph-mon[31460]: pgmap v2269: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:21:59.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:21:59 smithi059 ceph-mon[36190]: pgmap v2269: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:21:59.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:21:59 smithi170 ceph-mon[36280]: pgmap v2269: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:22:01.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:01 smithi037 ceph-mon[31460]: pgmap v2270: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:01.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3316.166351 secs 2024-03-28T19:22:01.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:01 smithi059 ceph-mon[36190]: pgmap v2270: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:01.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3316.166351 secs 2024-03-28T19:22:01.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:01 smithi170 ceph-mon[36280]: pgmap v2270: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:01.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3316.166351 secs 2024-03-28T19:22:03.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:03 smithi037 ceph-mon[31460]: pgmap v2271: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:03.786 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:03 smithi059 ceph-mon[36190]: pgmap v2271: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:03.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:03 smithi170 ceph-mon[36280]: pgmap v2271: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:05.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:05 smithi037 ceph-mon[31460]: pgmap v2272: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:05.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:05 smithi059 ceph-mon[36190]: pgmap v2272: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:05.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:05 smithi170 ceph-mon[36280]: pgmap v2272: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:06.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3321.166470 secs 2024-03-28T19:22:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3321.166470 secs 2024-03-28T19:22:06.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3321.166470 secs 2024-03-28T19:22:07.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:07 smithi037 ceph-mon[31460]: pgmap v2273: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:07.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:07 smithi059 ceph-mon[36190]: pgmap v2273: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:07.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:07 smithi170 ceph-mon[36280]: pgmap v2273: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:09.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:09 smithi037 ceph-mon[31460]: pgmap v2274: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:09.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:09 smithi059 ceph-mon[36190]: pgmap v2274: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:09.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:09 smithi170 ceph-mon[36280]: pgmap v2274: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:11.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:11 smithi037 ceph-mon[31460]: pgmap v2275: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:11.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3326.166573 secs 2024-03-28T19:22:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:11 smithi059 ceph-mon[36190]: pgmap v2275: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3326.166573 secs 2024-03-28T19:22:11.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:11 smithi170 ceph-mon[36280]: pgmap v2275: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:11.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3326.166573 secs 2024-03-28T19:22:13.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:13 smithi037 ceph-mon[31460]: pgmap v2276: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:13.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:13 smithi059 ceph-mon[36190]: pgmap v2276: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:13.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:13 smithi170 ceph-mon[36280]: pgmap v2276: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:15.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:15 smithi037 ceph-mon[31460]: pgmap v2277: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:15.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:15 smithi059 ceph-mon[36190]: pgmap v2277: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:15.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:15 smithi170 ceph-mon[36280]: pgmap v2277: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:16.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3331.166694 secs 2024-03-28T19:22:16.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3331.166694 secs 2024-03-28T19:22:16.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3331.166694 secs 2024-03-28T19:22:17.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:17 smithi037 ceph-mon[31460]: pgmap v2278: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:17.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:17 smithi059 ceph-mon[36190]: pgmap v2278: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:17.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:17 smithi170 ceph-mon[36280]: pgmap v2278: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:19.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:19 smithi037 ceph-mon[31460]: pgmap v2279: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:19.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:19 smithi059 ceph-mon[36190]: pgmap v2279: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:19.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:19 smithi170 ceph-mon[36280]: pgmap v2279: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:21.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:21 smithi059 ceph-mon[36190]: pgmap v2280: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:21.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3336.166813 secs 2024-03-28T19:22:21.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:21 smithi170 ceph-mon[36280]: pgmap v2280: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:21.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3336.166813 secs 2024-03-28T19:22:21.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:21 smithi037 ceph-mon[31460]: pgmap v2280: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:21.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3336.166813 secs 2024-03-28T19:22:23.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:23 smithi059 ceph-mon[36190]: pgmap v2281: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:23.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:23 smithi170 ceph-mon[36280]: pgmap v2281: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:23.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:23 smithi037 ceph-mon[31460]: pgmap v2281: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:25.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:25 smithi059 ceph-mon[36190]: pgmap v2282: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:25.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:25 smithi170 ceph-mon[36280]: pgmap v2282: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:25.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:25 smithi037 ceph-mon[31460]: pgmap v2282: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:26.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3341.166929 secs 2024-03-28T19:22:26.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3341.166929 secs 2024-03-28T19:22:26.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3341.166929 secs 2024-03-28T19:22:27.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:27 smithi059 ceph-mon[36190]: pgmap v2283: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:27.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:27 smithi170 ceph-mon[36280]: pgmap v2283: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:27.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:27 smithi037 ceph-mon[31460]: pgmap v2283: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:29.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:29 smithi059 ceph-mon[36190]: pgmap v2284: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:29.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:29 smithi170 ceph-mon[36280]: pgmap v2284: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:29.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:29 smithi037 ceph-mon[31460]: pgmap v2284: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:31.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:31 smithi059 ceph-mon[36190]: pgmap v2285: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:31.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3346.167072 secs 2024-03-28T19:22:31.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:31 smithi170 ceph-mon[36280]: pgmap v2285: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:31.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3346.167072 secs 2024-03-28T19:22:31.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:31 smithi037 ceph-mon[31460]: pgmap v2285: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:31.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3346.167072 secs 2024-03-28T19:22:33.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:33 smithi059 ceph-mon[36190]: pgmap v2286: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:33.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:33 smithi170 ceph-mon[36280]: pgmap v2286: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:33.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:33 smithi037 ceph-mon[31460]: pgmap v2286: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:35.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:35 smithi059 ceph-mon[36190]: pgmap v2287: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:35.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:35 smithi170 ceph-mon[36280]: pgmap v2287: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:35.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:35 smithi037 ceph-mon[31460]: pgmap v2287: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:36.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3351.167157 secs 2024-03-28T19:22:36.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3351.167157 secs 2024-03-28T19:22:36.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3351.167157 secs 2024-03-28T19:22:37.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:37 smithi059 ceph-mon[36190]: pgmap v2288: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:37.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:37 smithi170 ceph-mon[36280]: pgmap v2288: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:37.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:37 smithi037 ceph-mon[31460]: pgmap v2288: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:39.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:39 smithi059 ceph-mon[36190]: pgmap v2289: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:39.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:39 smithi170 ceph-mon[36280]: pgmap v2289: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:39.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:39 smithi037 ceph-mon[31460]: pgmap v2289: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:41.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:41 smithi059 ceph-mon[36190]: pgmap v2290: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:41.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3356.167245 secs 2024-03-28T19:22:41.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:41 smithi170 ceph-mon[36280]: pgmap v2290: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:41.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3356.167245 secs 2024-03-28T19:22:41.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:41 smithi037 ceph-mon[31460]: pgmap v2290: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:41.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3356.167245 secs 2024-03-28T19:22:43.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:43 smithi059 ceph-mon[36190]: pgmap v2291: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:43.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:43 smithi170 ceph-mon[36280]: pgmap v2291: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:43.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:43 smithi037 ceph-mon[31460]: pgmap v2291: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:45.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:45 smithi059 ceph-mon[36190]: pgmap v2292: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:45.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:45 smithi170 ceph-mon[36280]: pgmap v2292: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:45.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:45 smithi037 ceph-mon[31460]: pgmap v2292: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:46.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3361.167331 secs 2024-03-28T19:22:46.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3361.167331 secs 2024-03-28T19:22:46.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3361.167331 secs 2024-03-28T19:22:47.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:47 smithi059 ceph-mon[36190]: pgmap v2293: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:47.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:47 smithi170 ceph-mon[36280]: pgmap v2293: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:47.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:47 smithi037 ceph-mon[31460]: pgmap v2293: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:49.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:49 smithi059 ceph-mon[36190]: pgmap v2294: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:49.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:49 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:22:49.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:49 smithi170 ceph-mon[36280]: pgmap v2294: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:49.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:49 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:22:49.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:49 smithi037 ceph-mon[31460]: pgmap v2294: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:49.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:49 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:22:50.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:50 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:22:50.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:50 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:22:50.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:50 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:22:50.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:50 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:22:50.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:50 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:22:50.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:50 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:22:50.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:50 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:22:50.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:50 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:22:50.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:50 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:22:51.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:51 smithi059 ceph-mon[36190]: pgmap v2295: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:51.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:51 smithi059 ceph-mon[36190]: pgmap v2296: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:22:51.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3366.167453 secs 2024-03-28T19:22:51.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:51 smithi170 ceph-mon[36280]: pgmap v2295: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:51.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:51 smithi170 ceph-mon[36280]: pgmap v2296: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:22:51.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3366.167453 secs 2024-03-28T19:22:51.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:51 smithi037 ceph-mon[31460]: pgmap v2295: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:51.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:51 smithi037 ceph-mon[31460]: pgmap v2296: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:22:51.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3366.167453 secs 2024-03-28T19:22:53.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:53 smithi059 ceph-mon[36190]: pgmap v2297: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:22:53.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:53 smithi170 ceph-mon[36280]: pgmap v2297: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:22:53.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:53 smithi037 ceph-mon[31460]: pgmap v2297: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:22:55.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:55 smithi059 ceph-mon[36190]: pgmap v2298: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:22:55.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:55 smithi170 ceph-mon[36280]: pgmap v2298: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:22:55.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:55 smithi037 ceph-mon[31460]: pgmap v2298: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:22:56.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3371.167573 secs 2024-03-28T19:22:56.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3371.167573 secs 2024-03-28T19:22:56.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3371.167573 secs 2024-03-28T19:22:57.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:57 smithi059 ceph-mon[36190]: pgmap v2299: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:22:57.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:57 smithi170 ceph-mon[36280]: pgmap v2299: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:22:57.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:57 smithi037 ceph-mon[31460]: pgmap v2299: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:22:59.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:22:59 smithi059 ceph-mon[36190]: pgmap v2300: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:22:59.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:22:59 smithi170 ceph-mon[36280]: pgmap v2300: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:22:59.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:22:59 smithi037 ceph-mon[31460]: pgmap v2300: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:23:01.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:01 smithi059 ceph-mon[36190]: pgmap v2301: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:23:01.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3376.167689 secs 2024-03-28T19:23:01.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:01 smithi170 ceph-mon[36280]: pgmap v2301: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:23:01.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3376.167689 secs 2024-03-28T19:23:01.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:01 smithi037 ceph-mon[31460]: pgmap v2301: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:23:01.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3376.167689 secs 2024-03-28T19:23:03.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:03 smithi059 ceph-mon[36190]: pgmap v2302: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:03.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:03 smithi170 ceph-mon[36280]: pgmap v2302: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:03.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:03 smithi037 ceph-mon[31460]: pgmap v2302: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:05.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:05 smithi059 ceph-mon[36190]: pgmap v2303: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:05.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:05 smithi170 ceph-mon[36280]: pgmap v2303: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:05.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:05 smithi037 ceph-mon[31460]: pgmap v2303: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3381.167773 secs 2024-03-28T19:23:06.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3381.167773 secs 2024-03-28T19:23:06.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3381.167773 secs 2024-03-28T19:23:07.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:07 smithi059 ceph-mon[36190]: pgmap v2304: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:07.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:07 smithi170 ceph-mon[36280]: pgmap v2304: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:07.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:07 smithi037 ceph-mon[31460]: pgmap v2304: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:09.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:09 smithi059 ceph-mon[36190]: pgmap v2305: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:09.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:09 smithi170 ceph-mon[36280]: pgmap v2305: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:09.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:09 smithi037 ceph-mon[31460]: pgmap v2305: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:11 smithi059 ceph-mon[36190]: pgmap v2306: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3386.167890 secs 2024-03-28T19:23:11.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:11 smithi170 ceph-mon[36280]: pgmap v2306: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:11.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3386.167890 secs 2024-03-28T19:23:11.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:11 smithi037 ceph-mon[31460]: pgmap v2306: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:11.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3386.167890 secs 2024-03-28T19:23:13.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:13 smithi059 ceph-mon[36190]: pgmap v2307: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:13.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:13 smithi170 ceph-mon[36280]: pgmap v2307: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:13.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:13 smithi037 ceph-mon[31460]: pgmap v2307: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:15.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:15 smithi059 ceph-mon[36190]: pgmap v2308: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:15.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:15 smithi170 ceph-mon[36280]: pgmap v2308: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:15.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:15 smithi037 ceph-mon[31460]: pgmap v2308: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:16.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3391.167979 secs 2024-03-28T19:23:17.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3391.167979 secs 2024-03-28T19:23:17.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3391.167979 secs 2024-03-28T19:23:17.877 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:17 smithi037 ceph-mon[31460]: pgmap v2309: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:18.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:17 smithi059 ceph-mon[36190]: pgmap v2309: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:18.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:17 smithi170 ceph-mon[36280]: pgmap v2309: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:19.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:19 smithi037 ceph-mon[31460]: pgmap v2310: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:20.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:19 smithi059 ceph-mon[36190]: pgmap v2310: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:20.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:19 smithi170 ceph-mon[36280]: pgmap v2310: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:21.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:21 smithi037 ceph-mon[31460]: pgmap v2311: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:21.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3396.168108 secs 2024-03-28T19:23:22.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:21 smithi059 ceph-mon[36190]: pgmap v2311: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:22.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3396.168108 secs 2024-03-28T19:23:22.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:21 smithi170 ceph-mon[36280]: pgmap v2311: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:22.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3396.168108 secs 2024-03-28T19:23:23.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:23 smithi037 ceph-mon[31460]: pgmap v2312: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:24.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:23 smithi059 ceph-mon[36190]: pgmap v2312: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:24.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:23 smithi170 ceph-mon[36280]: pgmap v2312: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:25.836 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:25 smithi059 ceph-mon[36190]: pgmap v2313: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:25.837 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:25 smithi170 ceph-mon[36280]: pgmap v2313: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:25.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:25 smithi037 ceph-mon[31460]: pgmap v2313: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:26.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3401.168189 secs 2024-03-28T19:23:27.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3401.168189 secs 2024-03-28T19:23:27.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3401.168189 secs 2024-03-28T19:23:27.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:27 smithi037 ceph-mon[31460]: pgmap v2314: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:28.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:27 smithi059 ceph-mon[36190]: pgmap v2314: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:28.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:27 smithi170 ceph-mon[36280]: pgmap v2314: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:28.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:28 smithi037 ceph-mon[31460]: pgmap v2315: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:29.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:28 smithi059 ceph-mon[36190]: pgmap v2315: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:29.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:28 smithi170 ceph-mon[36280]: pgmap v2315: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:31.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:30 smithi059 ceph-mon[36190]: pgmap v2316: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:31.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:30 smithi170 ceph-mon[36280]: pgmap v2316: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:31.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:30 smithi037 ceph-mon[31460]: pgmap v2316: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:32.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3406.168308 secs 2024-03-28T19:23:32.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3406.168308 secs 2024-03-28T19:23:32.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3406.168308 secs 2024-03-28T19:23:33.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:32 smithi059 ceph-mon[36190]: pgmap v2317: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:33.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:32 smithi170 ceph-mon[36280]: pgmap v2317: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:33.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:32 smithi037 ceph-mon[31460]: pgmap v2317: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:35.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:34 smithi059 ceph-mon[36190]: pgmap v2318: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:35.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:34 smithi170 ceph-mon[36280]: pgmap v2318: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:35.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:34 smithi037 ceph-mon[31460]: pgmap v2318: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:37.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3411.168426 secs 2024-03-28T19:23:37.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:36 smithi059 ceph-mon[36190]: pgmap v2319: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:37.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3411.168426 secs 2024-03-28T19:23:37.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:36 smithi170 ceph-mon[36280]: pgmap v2319: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:37.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3411.168426 secs 2024-03-28T19:23:37.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:36 smithi037 ceph-mon[31460]: pgmap v2319: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:39.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:38 smithi059 ceph-mon[36190]: pgmap v2320: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:39.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:38 smithi170 ceph-mon[36280]: pgmap v2320: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:39.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:38 smithi037 ceph-mon[31460]: pgmap v2320: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:41.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:40 smithi059 ceph-mon[36190]: pgmap v2321: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:41.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:40 smithi170 ceph-mon[36280]: pgmap v2321: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:41.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:40 smithi037 ceph-mon[31460]: pgmap v2321: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:42.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3416.168547 secs 2024-03-28T19:23:42.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3416.168547 secs 2024-03-28T19:23:42.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3416.168547 secs 2024-03-28T19:23:43.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:42 smithi037 ceph-mon[31460]: pgmap v2322: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:43.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:42 smithi059 ceph-mon[36190]: pgmap v2322: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:43.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:42 smithi170 ceph-mon[36280]: pgmap v2322: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:45.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:44 smithi037 ceph-mon[31460]: pgmap v2323: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:45.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:44 smithi059 ceph-mon[36190]: pgmap v2323: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:45.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:44 smithi170 ceph-mon[36280]: pgmap v2323: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:47.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3421.168666 secs 2024-03-28T19:23:47.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:46 smithi037 ceph-mon[31460]: pgmap v2324: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:47.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3421.168666 secs 2024-03-28T19:23:47.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:46 smithi059 ceph-mon[36190]: pgmap v2324: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:47.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3421.168666 secs 2024-03-28T19:23:47.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:46 smithi170 ceph-mon[36280]: pgmap v2324: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:49.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:48 smithi037 ceph-mon[31460]: pgmap v2325: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:49.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:48 smithi059 ceph-mon[36190]: pgmap v2325: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:49.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:48 smithi170 ceph-mon[36280]: pgmap v2325: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:51.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:50 smithi037 ceph-mon[31460]: pgmap v2326: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:51.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:50 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:23:51.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:50 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:51.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:50 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:51.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:50 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:51.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:50 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:51.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:50 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:51.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:50 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:51.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:50 smithi059 ceph-mon[36190]: pgmap v2326: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:51.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:50 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:23:51.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:50 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:51.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:50 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:50 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:50 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:50 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:51.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:50 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:50 smithi170 ceph-mon[36280]: pgmap v2326: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:50 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:23:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:50 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:50 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:50 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:50 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:50 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:51.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:50 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:52.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3426.168768 secs 2024-03-28T19:23:52.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3426.168768 secs 2024-03-28T19:23:52.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3426.168768 secs 2024-03-28T19:23:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:52 smithi037 ceph-mon[31460]: pgmap v2327: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:52 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:52 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:52 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:53.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:52 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:53.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:52 smithi059 ceph-mon[36190]: pgmap v2327: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:53.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:52 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:53.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:52 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:53.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:52 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:53.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:52 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:53.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:52 smithi170 ceph-mon[36280]: pgmap v2327: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:53.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:52 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:52 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:52 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:53.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:52 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:54.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:53 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:53 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:53 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:53 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:53 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:53 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:53 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:53 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:53 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T19:23:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:53 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T19:23:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:53 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:53 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:53 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:53 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:53 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:53 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:53 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:53 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:53 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T19:23:54.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:53 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T19:23:54.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:53 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:53 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:53 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:53 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:53 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:53 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:53 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:53 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:53 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T19:23:54.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:53 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T19:23:54.286 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:53 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.287 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:53 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.287 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:53 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.287 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:53 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.287 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:53 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.287 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:53 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.287 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:53 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.287 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:53 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.287 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:53 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T19:23:54.287 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:53 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T19:23:54.288 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:53 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.288 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:53 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.288 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:53 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.288 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:53 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.288 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:53 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.288 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:53 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.288 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:53 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.288 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:53 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.288 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:53 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-28T19:23:54.288 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:53 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi170 to 182640640: error parsing value: Value '182640640' is below minimum 939524096 2024-03-28T19:23:54.288 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:53 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.288 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:53 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.288 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:53 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.289 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:53 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.289 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:53 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.289 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:53 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.289 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:53 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.289 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:53 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:54.289 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:53 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi059 to 175.2M 2024-03-28T19:23:54.289 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:53 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi059 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-28T19:23:55.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:54 smithi037 ceph-mon[31460]: pgmap v2328: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:55.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:54 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:55.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:54 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:55.185 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:54 smithi059 ceph-mon[36190]: pgmap v2328: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:55.185 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:54 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:55.185 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:54 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:55.185 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:54 smithi170 ceph-mon[36280]: pgmap v2328: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:55.186 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:54 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:55.186 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:54 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3431.168853 secs 2024-03-28T19:23:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:56 smithi037 ceph-mon[31460]: pgmap v2329: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:56 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:56 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:56 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:56 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:56 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:56 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:56 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:56 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:56 smithi037 ceph-mon[31460]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T19:23:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:56 smithi037 ceph-mon[31460]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T19:23:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:56 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:23:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:56 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:23:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:56 smithi037 ceph-mon[31460]: pgmap v2330: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:23:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:56 smithi037 ceph-mon[31460]: pgmap v2331: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:23:57.125 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:56 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3431.168853 secs 2024-03-28T19:23:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:56 smithi059 ceph-mon[36190]: pgmap v2329: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:56 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:56 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:56 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:56 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:56 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:56 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:56 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:56 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:56 smithi059 ceph-mon[36190]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T19:23:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:56 smithi059 ceph-mon[36190]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T19:23:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:56 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:23:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:56 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:23:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:56 smithi059 ceph-mon[36190]: pgmap v2330: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:23:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:56 smithi059 ceph-mon[36190]: pgmap v2331: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:23:57.285 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:56 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3431.168853 secs 2024-03-28T19:23:57.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:56 smithi170 ceph-mon[36280]: pgmap v2329: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:56 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:56 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:56 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:56 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:56 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:56 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:56 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:56 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:23:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:56 smithi170 ceph-mon[36280]: Adjusting osd_memory_target on smithi037 to 177.8M 2024-03-28T19:23:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:56 smithi170 ceph-mon[36280]: Unable to set osd_memory_target on smithi037 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T19:23:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:56 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:23:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:56 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:23:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:56 smithi170 ceph-mon[36280]: pgmap v2330: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:23:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:56 smithi170 ceph-mon[36280]: pgmap v2331: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:23:57.286 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:56 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:23:59.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:23:58 smithi059 ceph-mon[36190]: pgmap v2332: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:23:59.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:23:58 smithi170 ceph-mon[36280]: pgmap v2332: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:23:59.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:23:58 smithi037 ceph-mon[31460]: pgmap v2332: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:24:01.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:00 smithi059 ceph-mon[36190]: pgmap v2333: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:24:01.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:00 smithi170 ceph-mon[36280]: pgmap v2333: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:24:01.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:00 smithi037 ceph-mon[31460]: pgmap v2333: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:24:02.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:02 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3436.168968 secs 2024-03-28T19:24:02.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:02 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3436.168968 secs 2024-03-28T19:24:02.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:02 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3436.168968 secs 2024-03-28T19:24:03.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:03 smithi059 ceph-mon[36190]: pgmap v2334: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:24:03.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:03 smithi170 ceph-mon[36280]: pgmap v2334: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:24:03.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:03 smithi037 ceph-mon[31460]: pgmap v2334: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:24:05.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:05 smithi059 ceph-mon[36190]: pgmap v2335: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:24:05.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:05 smithi170 ceph-mon[36280]: pgmap v2335: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:24:05.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:05 smithi037 ceph-mon[31460]: pgmap v2335: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:24:07.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:07 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3441.169090 secs 2024-03-28T19:24:07.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:07 smithi059 ceph-mon[36190]: pgmap v2336: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:24:07.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:07 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3441.169090 secs 2024-03-28T19:24:07.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:07 smithi170 ceph-mon[36280]: pgmap v2336: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:24:07.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:07 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3441.169090 secs 2024-03-28T19:24:07.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:07 smithi037 ceph-mon[31460]: pgmap v2336: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:24:09.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:09 smithi059 ceph-mon[36190]: pgmap v2337: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:09.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:09 smithi170 ceph-mon[36280]: pgmap v2337: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:09.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:09 smithi037 ceph-mon[31460]: pgmap v2337: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:11.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:11 smithi059 ceph-mon[36190]: pgmap v2338: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:11.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:11 smithi170 ceph-mon[36280]: pgmap v2338: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:11.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:11 smithi037 ceph-mon[31460]: pgmap v2338: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:12.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:12 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3446.169211 secs 2024-03-28T19:24:12.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:12 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3446.169211 secs 2024-03-28T19:24:12.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:12 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3446.169211 secs 2024-03-28T19:24:13.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:13 smithi037 ceph-mon[31460]: pgmap v2339: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:13.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:13 smithi059 ceph-mon[36190]: pgmap v2339: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:13.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:13 smithi170 ceph-mon[36280]: pgmap v2339: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:15.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:15 smithi037 ceph-mon[31460]: pgmap v2340: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:15.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:15 smithi059 ceph-mon[36190]: pgmap v2340: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:15.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:15 smithi170 ceph-mon[36280]: pgmap v2340: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:17.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:17 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3451.169332 secs 2024-03-28T19:24:17.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:17 smithi037 ceph-mon[31460]: pgmap v2341: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:17.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:17 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3451.169332 secs 2024-03-28T19:24:17.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:17 smithi059 ceph-mon[36190]: pgmap v2341: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:17.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:17 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3451.169332 secs 2024-03-28T19:24:17.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:17 smithi170 ceph-mon[36280]: pgmap v2341: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:19.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:19 smithi037 ceph-mon[31460]: pgmap v2342: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:19.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:19 smithi059 ceph-mon[36190]: pgmap v2342: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:19.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:19 smithi170 ceph-mon[36280]: pgmap v2342: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:21.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:21 smithi037 ceph-mon[31460]: pgmap v2343: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:21.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:21 smithi059 ceph-mon[36190]: pgmap v2343: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:21.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:21 smithi170 ceph-mon[36280]: pgmap v2343: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:22.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:22 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3456.169423 secs 2024-03-28T19:24:22.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:22 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3456.169423 secs 2024-03-28T19:24:22.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:22 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3456.169423 secs 2024-03-28T19:24:23.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:23 smithi037 ceph-mon[31460]: pgmap v2344: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:23.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:23 smithi059 ceph-mon[36190]: pgmap v2344: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:23.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:23 smithi170 ceph-mon[36280]: pgmap v2344: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:25.336 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:25 smithi059 ceph-mon[36190]: pgmap v2345: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:25.337 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:25 smithi170 ceph-mon[36280]: pgmap v2345: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:25.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:25 smithi037 ceph-mon[31460]: pgmap v2345: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:27.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:27 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3461.169542 secs 2024-03-28T19:24:27.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:27 smithi037 ceph-mon[31460]: pgmap v2346: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:27.533 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:27 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3461.169542 secs 2024-03-28T19:24:27.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:27 smithi059 ceph-mon[36190]: pgmap v2346: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:27.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:27 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3461.169542 secs 2024-03-28T19:24:27.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:27 smithi170 ceph-mon[36280]: pgmap v2346: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:29.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:29 smithi037 ceph-mon[31460]: pgmap v2347: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:29.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:29 smithi059 ceph-mon[36190]: pgmap v2347: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:29.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:29 smithi170 ceph-mon[36280]: pgmap v2347: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:31.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:31 smithi037 ceph-mon[31460]: pgmap v2348: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:31.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:31 smithi059 ceph-mon[36190]: pgmap v2348: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:31.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:31 smithi170 ceph-mon[36280]: pgmap v2348: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:32.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:32 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3466.169636 secs 2024-03-28T19:24:32.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:32 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3466.169636 secs 2024-03-28T19:24:32.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:32 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3466.169636 secs 2024-03-28T19:24:33.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:33 smithi037 ceph-mon[31460]: pgmap v2349: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:33.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:33 smithi059 ceph-mon[36190]: pgmap v2349: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:33.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:33 smithi170 ceph-mon[36280]: pgmap v2349: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:35.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:35 smithi037 ceph-mon[31460]: pgmap v2350: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:35.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:35 smithi059 ceph-mon[36190]: pgmap v2350: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:35.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:35 smithi170 ceph-mon[36280]: pgmap v2350: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:37.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:37 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3471.169754 secs 2024-03-28T19:24:37.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:37 smithi037 ceph-mon[31460]: pgmap v2351: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:37.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:37 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3471.169754 secs 2024-03-28T19:24:37.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:37 smithi059 ceph-mon[36190]: pgmap v2351: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:37.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:37 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3471.169754 secs 2024-03-28T19:24:37.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:37 smithi170 ceph-mon[36280]: pgmap v2351: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:39.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:39 smithi037 ceph-mon[31460]: pgmap v2352: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:39.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:39 smithi170 ceph-mon[36280]: pgmap v2352: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:39.536 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:39 smithi059 ceph-mon[36190]: pgmap v2352: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:41.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:41 smithi037 ceph-mon[31460]: pgmap v2353: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:41.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:41 smithi059 ceph-mon[36190]: pgmap v2353: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:41.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:41 smithi170 ceph-mon[36280]: pgmap v2353: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:42.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:42 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3476.169876 secs 2024-03-28T19:24:42.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:42 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3476.169876 secs 2024-03-28T19:24:42.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:42 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3476.169876 secs 2024-03-28T19:24:43.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:43 smithi059 ceph-mon[36190]: pgmap v2354: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:43.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:43 smithi170 ceph-mon[36280]: pgmap v2354: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:43.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:43 smithi037 ceph-mon[31460]: pgmap v2354: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:45.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:45 smithi059 ceph-mon[36190]: pgmap v2355: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:45.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:45 smithi170 ceph-mon[36280]: pgmap v2355: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:45.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:45 smithi037 ceph-mon[31460]: pgmap v2355: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:47.425 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:47 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3481.169958 secs 2024-03-28T19:24:47.425 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:47 smithi037 ceph-mon[31460]: pgmap v2356: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:47.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:47 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3481.169958 secs 2024-03-28T19:24:47.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:47 smithi059 ceph-mon[36190]: pgmap v2356: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:47.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:47 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3481.169958 secs 2024-03-28T19:24:47.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:47 smithi170 ceph-mon[36280]: pgmap v2356: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:49.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:49 smithi059 ceph-mon[36190]: pgmap v2357: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:49.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:49 smithi170 ceph-mon[36280]: pgmap v2357: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:49.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:49 smithi037 ceph-mon[31460]: pgmap v2357: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:51.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:51 smithi059 ceph-mon[36190]: pgmap v2358: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:51.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:51 smithi170 ceph-mon[36280]: pgmap v2358: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:51.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:51 smithi037 ceph-mon[31460]: pgmap v2358: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:52.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:52 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3486.170079 secs 2024-03-28T19:24:52.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:52 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3486.170079 secs 2024-03-28T19:24:52.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:52 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3486.170079 secs 2024-03-28T19:24:53.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:53 smithi059 ceph-mon[36190]: pgmap v2359: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:24:53.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:53 smithi170 ceph-mon[36280]: pgmap v2359: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:24:53.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:53 smithi037 ceph-mon[31460]: pgmap v2359: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:24:55.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:55 smithi059 ceph-mon[36190]: pgmap v2360: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:24:55.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:55 smithi170 ceph-mon[36280]: pgmap v2360: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:24:55.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:55 smithi037 ceph-mon[31460]: pgmap v2360: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:24:57.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:57 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3491.170200 secs 2024-03-28T19:24:57.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:57 smithi059 ceph-mon[36190]: pgmap v2361: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:24:57.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:57 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:24:57.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:57 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3491.170200 secs 2024-03-28T19:24:57.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:57 smithi170 ceph-mon[36280]: pgmap v2361: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:24:57.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:57 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:24:57.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:57 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3491.170200 secs 2024-03-28T19:24:57.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:57 smithi037 ceph-mon[31460]: pgmap v2361: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:24:57.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:57 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:24:58.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:58 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:24:58.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:58 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:24:58.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:58 smithi059 ceph-mon[36190]: pgmap v2362: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T19:24:58.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:58 smithi059 ceph-mon[36190]: pgmap v2363: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T19:24:58.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:24:58 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:24:58.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:58 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:24:58.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:58 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:24:58.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:58 smithi170 ceph-mon[36280]: pgmap v2362: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T19:24:58.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:58 smithi170 ceph-mon[36280]: pgmap v2363: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T19:24:58.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:24:58 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:24:58.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:58 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:24:58.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:58 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:24:58.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:58 smithi037 ceph-mon[31460]: pgmap v2362: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T19:24:58.623 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:58 smithi037 ceph-mon[31460]: pgmap v2363: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T19:24:58.624 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:24:58 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:25:00.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:00 smithi059 ceph-mon[36190]: pgmap v2364: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T19:25:00.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:00 smithi170 ceph-mon[36280]: pgmap v2364: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T19:25:00.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:00 smithi037 ceph-mon[31460]: pgmap v2364: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T19:25:01.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3496.170293 secs 2024-03-28T19:25:01.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3496.170293 secs 2024-03-28T19:25:01.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3496.170293 secs 2024-03-28T19:25:02.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:02 smithi059 ceph-mon[36190]: pgmap v2365: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:25:02.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:02 smithi170 ceph-mon[36280]: pgmap v2365: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:25:02.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:02 smithi037 ceph-mon[31460]: pgmap v2365: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:25:04.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:04 smithi059 ceph-mon[36190]: pgmap v2366: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:25:04.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:04 smithi170 ceph-mon[36280]: pgmap v2366: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:25:04.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:04 smithi037 ceph-mon[31460]: pgmap v2366: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:25:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:06 smithi059 ceph-mon[36190]: pgmap v2367: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:25:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3501.170412 secs 2024-03-28T19:25:06.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:06 smithi170 ceph-mon[36280]: pgmap v2367: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:25:06.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3501.170412 secs 2024-03-28T19:25:06.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:06 smithi037 ceph-mon[31460]: pgmap v2367: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:25:06.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3501.170412 secs 2024-03-28T19:25:08.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:08 smithi059 ceph-mon[36190]: pgmap v2368: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:25:08.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:08 smithi170 ceph-mon[36280]: pgmap v2368: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:25:08.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:08 smithi037 ceph-mon[31460]: pgmap v2368: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:25:10.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:10 smithi059 ceph-mon[36190]: pgmap v2369: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:10.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:10 smithi170 ceph-mon[36280]: pgmap v2369: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:10.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:10 smithi037 ceph-mon[31460]: pgmap v2369: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3506.170516 secs 2024-03-28T19:25:11.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3506.170516 secs 2024-03-28T19:25:11.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3506.170516 secs 2024-03-28T19:25:12.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:12 smithi059 ceph-mon[36190]: pgmap v2370: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:12.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:12 smithi170 ceph-mon[36280]: pgmap v2370: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:12.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:12 smithi037 ceph-mon[31460]: pgmap v2370: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:14.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:14 smithi059 ceph-mon[36190]: pgmap v2371: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:14.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:14 smithi170 ceph-mon[36280]: pgmap v2371: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:14.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:14 smithi037 ceph-mon[31460]: pgmap v2371: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:16.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:16 smithi059 ceph-mon[36190]: pgmap v2372: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:16.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3511.170604 secs 2024-03-28T19:25:16.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:16 smithi170 ceph-mon[36280]: pgmap v2372: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:16.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3511.170604 secs 2024-03-28T19:25:16.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:16 smithi037 ceph-mon[31460]: pgmap v2372: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:16.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3511.170604 secs 2024-03-28T19:25:18.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:18 smithi059 ceph-mon[36190]: pgmap v2373: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:18.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:18 smithi170 ceph-mon[36280]: pgmap v2373: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:18.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:18 smithi037 ceph-mon[31460]: pgmap v2373: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:20.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:20 smithi059 ceph-mon[36190]: pgmap v2374: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:20.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:20 smithi170 ceph-mon[36280]: pgmap v2374: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:20.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:20 smithi037 ceph-mon[31460]: pgmap v2374: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:21.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3516.170724 secs 2024-03-28T19:25:21.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3516.170724 secs 2024-03-28T19:25:21.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3516.170724 secs 2024-03-28T19:25:22.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:22 smithi059 ceph-mon[36190]: pgmap v2375: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:22.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:22 smithi170 ceph-mon[36280]: pgmap v2375: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:22.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:22 smithi037 ceph-mon[31460]: pgmap v2375: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:24.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:24 smithi059 ceph-mon[36190]: pgmap v2376: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:24.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:24 smithi170 ceph-mon[36280]: pgmap v2376: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:24.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:24 smithi037 ceph-mon[31460]: pgmap v2376: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:26.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:26 smithi059 ceph-mon[36190]: pgmap v2377: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:26.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3521.170819 secs 2024-03-28T19:25:26.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:26 smithi170 ceph-mon[36280]: pgmap v2377: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:26.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3521.170819 secs 2024-03-28T19:25:26.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:26 smithi037 ceph-mon[31460]: pgmap v2377: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:26.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3521.170819 secs 2024-03-28T19:25:28.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:28 smithi170 ceph-mon[36280]: pgmap v2378: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:28.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:28 smithi037 ceph-mon[31460]: pgmap v2378: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:29.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:28 smithi059 ceph-mon[36190]: pgmap v2378: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:30.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:30 smithi037 ceph-mon[31460]: pgmap v2379: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:31.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:30 smithi059 ceph-mon[36190]: pgmap v2379: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:31.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:30 smithi170 ceph-mon[36280]: pgmap v2379: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:31.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3526.170940 secs 2024-03-28T19:25:32.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3526.170940 secs 2024-03-28T19:25:32.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3526.170940 secs 2024-03-28T19:25:32.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:32 smithi037 ceph-mon[31460]: pgmap v2380: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:33.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:32 smithi059 ceph-mon[36190]: pgmap v2380: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:33.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:32 smithi170 ceph-mon[36280]: pgmap v2380: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:34.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:34 smithi037 ceph-mon[31460]: pgmap v2381: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:35.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:34 smithi059 ceph-mon[36190]: pgmap v2381: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:35.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:34 smithi170 ceph-mon[36280]: pgmap v2381: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:36.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:36 smithi037 ceph-mon[31460]: pgmap v2382: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:36.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3531.171057 secs 2024-03-28T19:25:37.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:36 smithi059 ceph-mon[36190]: pgmap v2382: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:37.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3531.171057 secs 2024-03-28T19:25:37.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:36 smithi170 ceph-mon[36280]: pgmap v2382: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:37.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3531.171057 secs 2024-03-28T19:25:38.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:38 smithi037 ceph-mon[31460]: pgmap v2383: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:39.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:38 smithi059 ceph-mon[36190]: pgmap v2383: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:39.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:38 smithi170 ceph-mon[36280]: pgmap v2383: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:40.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:40 smithi037 ceph-mon[31460]: pgmap v2384: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:41.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:40 smithi059 ceph-mon[36190]: pgmap v2384: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:41.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:40 smithi170 ceph-mon[36280]: pgmap v2384: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:42.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3536.171178 secs 2024-03-28T19:25:42.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:41 smithi059 ceph-mon[36190]: pgmap v2385: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:42.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3536.171178 secs 2024-03-28T19:25:42.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:41 smithi170 ceph-mon[36280]: pgmap v2385: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:42.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3536.171178 secs 2024-03-28T19:25:42.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:41 smithi037 ceph-mon[31460]: pgmap v2385: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:44.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:44 smithi059 ceph-mon[36190]: pgmap v2386: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:44.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:44 smithi170 ceph-mon[36280]: pgmap v2386: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:44.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:44 smithi037 ceph-mon[31460]: pgmap v2386: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:46.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:46 smithi059 ceph-mon[36190]: pgmap v2387: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:46.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3541.171299 secs 2024-03-28T19:25:46.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:46 smithi170 ceph-mon[36280]: pgmap v2387: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:46.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3541.171299 secs 2024-03-28T19:25:46.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:46 smithi037 ceph-mon[31460]: pgmap v2387: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:46.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3541.171299 secs 2024-03-28T19:25:48.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:48 smithi059 ceph-mon[36190]: pgmap v2388: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:48.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:48 smithi170 ceph-mon[36280]: pgmap v2388: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:48.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:48 smithi037 ceph-mon[31460]: pgmap v2388: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:50.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:50 smithi059 ceph-mon[36190]: pgmap v2389: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:50.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:50 smithi170 ceph-mon[36280]: pgmap v2389: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:50.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:50 smithi037 ceph-mon[31460]: pgmap v2389: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:51.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3546.171384 secs 2024-03-28T19:25:51.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3546.171384 secs 2024-03-28T19:25:51.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3546.171384 secs 2024-03-28T19:25:52.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:52 smithi059 ceph-mon[36190]: pgmap v2390: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:52.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:52 smithi170 ceph-mon[36280]: pgmap v2390: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:52.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:52 smithi037 ceph-mon[31460]: pgmap v2390: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:54.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:54 smithi059 ceph-mon[36190]: pgmap v2391: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:54.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:54 smithi170 ceph-mon[36280]: pgmap v2391: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:54.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:54 smithi037 ceph-mon[31460]: pgmap v2391: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:56.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:56 smithi059 ceph-mon[36190]: pgmap v2392: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:56.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3551.171503 secs 2024-03-28T19:25:56.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:56 smithi170 ceph-mon[36280]: pgmap v2392: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:56.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3551.171503 secs 2024-03-28T19:25:56.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:56 smithi037 ceph-mon[31460]: pgmap v2392: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:56.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3551.171503 secs 2024-03-28T19:25:58.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:58 smithi059 ceph-mon[36190]: pgmap v2393: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:58.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:58 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:25:58.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:58 smithi170 ceph-mon[36280]: pgmap v2393: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:58.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:58 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:25:58.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:58 smithi037 ceph-mon[31460]: pgmap v2393: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:58.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:58 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:25:59.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:59 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:25:59.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:59 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:25:59.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:25:59 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:25:59.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:59 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:25:59.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:59 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:25:59.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:25:59 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:25:59.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:59 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:25:59.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:59 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:25:59.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:25:59 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:26:00.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:00 smithi059 ceph-mon[36190]: pgmap v2394: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:00.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:00 smithi170 ceph-mon[36280]: pgmap v2394: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:00.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:00 smithi037 ceph-mon[31460]: pgmap v2394: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:01.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3556.171622 secs 2024-03-28T19:26:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3556.171622 secs 2024-03-28T19:26:01.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3556.171622 secs 2024-03-28T19:26:02.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:02 smithi059 ceph-mon[36190]: pgmap v2395: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:02.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:02 smithi170 ceph-mon[36280]: pgmap v2395: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:02.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:02 smithi037 ceph-mon[31460]: pgmap v2395: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:04.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:04 smithi059 ceph-mon[36190]: pgmap v2396: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:26:04.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:04 smithi170 ceph-mon[36280]: pgmap v2396: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:26:04.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:04 smithi037 ceph-mon[31460]: pgmap v2396: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:26:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:06 smithi059 ceph-mon[36190]: pgmap v2397: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:06.785 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3561.171740 secs 2024-03-28T19:26:06.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:06 smithi170 ceph-mon[36280]: pgmap v2397: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:06.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3561.171740 secs 2024-03-28T19:26:06.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:06 smithi037 ceph-mon[31460]: pgmap v2397: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:06.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3561.171740 secs 2024-03-28T19:26:08.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:08 smithi037 ceph-mon[31460]: pgmap v2398: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:26:09.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:08 smithi059 ceph-mon[36190]: pgmap v2398: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:26:09.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:08 smithi170 ceph-mon[36280]: pgmap v2398: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:26:10.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:10 smithi037 ceph-mon[31460]: pgmap v2399: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:11.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:10 smithi059 ceph-mon[36190]: pgmap v2399: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:11.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:10 smithi170 ceph-mon[36280]: pgmap v2399: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:11.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3566.171840 secs 2024-03-28T19:26:12.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3566.171840 secs 2024-03-28T19:26:12.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3566.171840 secs 2024-03-28T19:26:12.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:12 smithi037 ceph-mon[31460]: pgmap v2400: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:13.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:12 smithi059 ceph-mon[36190]: pgmap v2400: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:13.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:12 smithi170 ceph-mon[36280]: pgmap v2400: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:14.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:14 smithi037 ceph-mon[31460]: pgmap v2401: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:15.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:14 smithi059 ceph-mon[36190]: pgmap v2401: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:15.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:14 smithi170 ceph-mon[36280]: pgmap v2401: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:16.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:16 smithi037 ceph-mon[31460]: pgmap v2402: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:16.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3571.171943 secs 2024-03-28T19:26:17.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:16 smithi059 ceph-mon[36190]: pgmap v2402: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:17.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3571.171943 secs 2024-03-28T19:26:17.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:16 smithi170 ceph-mon[36280]: pgmap v2402: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:17.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3571.171943 secs 2024-03-28T19:26:18.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:18 smithi037 ceph-mon[31460]: pgmap v2403: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:19.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:18 smithi059 ceph-mon[36190]: pgmap v2403: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:19.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:18 smithi170 ceph-mon[36280]: pgmap v2403: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:20.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:20 smithi037 ceph-mon[31460]: pgmap v2404: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:21.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:20 smithi059 ceph-mon[36190]: pgmap v2404: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:21.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:20 smithi170 ceph-mon[36280]: pgmap v2404: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:21.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3576.172075 secs 2024-03-28T19:26:22.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3576.172075 secs 2024-03-28T19:26:22.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3576.172075 secs 2024-03-28T19:26:22.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:22 smithi037 ceph-mon[31460]: pgmap v2405: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:23.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:22 smithi059 ceph-mon[36190]: pgmap v2405: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:23.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:22 smithi170 ceph-mon[36280]: pgmap v2405: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:24.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:24 smithi037 ceph-mon[31460]: pgmap v2406: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:25.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:24 smithi059 ceph-mon[36190]: pgmap v2406: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:25.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:24 smithi170 ceph-mon[36280]: pgmap v2406: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:26.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:26 smithi037 ceph-mon[31460]: pgmap v2407: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:26.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3581.172168 secs 2024-03-28T19:26:27.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:26 smithi059 ceph-mon[36190]: pgmap v2407: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:27.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3581.172168 secs 2024-03-28T19:26:27.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:26 smithi170 ceph-mon[36280]: pgmap v2407: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:27.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3581.172168 secs 2024-03-28T19:26:28.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:28 smithi037 ceph-mon[31460]: pgmap v2408: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:29.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:28 smithi059 ceph-mon[36190]: pgmap v2408: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:29.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:28 smithi170 ceph-mon[36280]: pgmap v2408: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:29.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:29 smithi037 ceph-mon[31460]: pgmap v2409: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:30.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:29 smithi059 ceph-mon[36190]: pgmap v2409: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:30.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:29 smithi170 ceph-mon[36280]: pgmap v2409: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:32.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3586.172263 secs 2024-03-28T19:26:32.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:31 smithi059 ceph-mon[36190]: pgmap v2410: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3586.172263 secs 2024-03-28T19:26:32.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:31 smithi170 ceph-mon[36280]: pgmap v2410: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:32.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3586.172263 secs 2024-03-28T19:26:32.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:31 smithi037 ceph-mon[31460]: pgmap v2410: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:34.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:33 smithi059 ceph-mon[36190]: pgmap v2411: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:34.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:33 smithi170 ceph-mon[36280]: pgmap v2411: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:34.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:33 smithi037 ceph-mon[31460]: pgmap v2411: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:36.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:35 smithi059 ceph-mon[36190]: pgmap v2412: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:36.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:35 smithi170 ceph-mon[36280]: pgmap v2412: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:36.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:35 smithi037 ceph-mon[31460]: pgmap v2412: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:37.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3591.172363 secs 2024-03-28T19:26:37.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3591.172363 secs 2024-03-28T19:26:37.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3591.172363 secs 2024-03-28T19:26:38.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:37 smithi059 ceph-mon[36190]: pgmap v2413: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:38.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:37 smithi170 ceph-mon[36280]: pgmap v2413: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:38.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:37 smithi037 ceph-mon[31460]: pgmap v2413: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:40.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:39 smithi059 ceph-mon[36190]: pgmap v2414: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:40.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:39 smithi170 ceph-mon[36280]: pgmap v2414: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:40.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:39 smithi037 ceph-mon[31460]: pgmap v2414: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:42.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:42 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3596.172481 secs 2024-03-28T19:26:42.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:42 smithi059 ceph-mon[36190]: pgmap v2415: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:42.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:42 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3596.172481 secs 2024-03-28T19:26:42.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:42 smithi170 ceph-mon[36280]: pgmap v2415: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:42.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:42 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3596.172481 secs 2024-03-28T19:26:42.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:42 smithi037 ceph-mon[31460]: pgmap v2415: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:44.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:44 smithi037 ceph-mon[31460]: pgmap v2416: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:44.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:44 smithi059 ceph-mon[36190]: pgmap v2416: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:44.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:44 smithi170 ceph-mon[36280]: pgmap v2416: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:46.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:46 smithi059 ceph-mon[36190]: pgmap v2417: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:46.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:46 smithi170 ceph-mon[36280]: pgmap v2417: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:46.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:46 smithi037 ceph-mon[31460]: pgmap v2417: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:47.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:47 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3601.172603 secs 2024-03-28T19:26:47.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:47 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3601.172603 secs 2024-03-28T19:26:47.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:47 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3601.172603 secs 2024-03-28T19:26:48.274 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:48 smithi037 ceph-mon[31460]: pgmap v2418: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:48.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:48 smithi059 ceph-mon[36190]: pgmap v2418: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:48.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:48 smithi170 ceph-mon[36280]: pgmap v2418: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:50.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:50 smithi059 ceph-mon[36190]: pgmap v2419: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:50.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:50 smithi170 ceph-mon[36280]: pgmap v2419: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:50.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:50 smithi037 ceph-mon[31460]: pgmap v2419: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:52.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:52 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3606.172696 secs 2024-03-28T19:26:52.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:52 smithi059 ceph-mon[36190]: pgmap v2420: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:52.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:52 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3606.172696 secs 2024-03-28T19:26:52.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:52 smithi170 ceph-mon[36280]: pgmap v2420: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:52.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:52 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3606.172696 secs 2024-03-28T19:26:52.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:52 smithi037 ceph-mon[31460]: pgmap v2420: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:54.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:54 smithi059 ceph-mon[36190]: pgmap v2421: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:54.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:54 smithi170 ceph-mon[36280]: pgmap v2421: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:54.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:54 smithi037 ceph-mon[31460]: pgmap v2421: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:56.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:56 smithi037 ceph-mon[31460]: pgmap v2422: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:56.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:56 smithi059 ceph-mon[36190]: pgmap v2422: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:56.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:56 smithi170 ceph-mon[36280]: pgmap v2422: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:57.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:57 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3611.172814 secs 2024-03-28T19:26:57.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:57 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3611.172814 secs 2024-03-28T19:26:57.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:57 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3611.172814 secs 2024-03-28T19:26:58.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:26:58 smithi037 ceph-mon[31460]: pgmap v2423: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:58.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:26:58 smithi059 ceph-mon[36190]: pgmap v2423: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:58.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:26:58 smithi170 ceph-mon[36280]: pgmap v2423: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:00.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:00 smithi037 ceph-mon[31460]: pgmap v2424: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:00.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:00 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:27:00.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:00 smithi059 ceph-mon[36190]: pgmap v2424: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:00.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:00 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:27:00.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:00 smithi170 ceph-mon[36280]: pgmap v2424: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:00.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:00 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:27:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:01 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:27:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:01 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:27:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:01 smithi037 ceph-mon[31460]: pgmap v2425: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:01 smithi037 ceph-mon[31460]: pgmap v2426: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:01.374 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:01 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:27:01.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:01 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:27:01.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:01 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:27:01.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:01 smithi059 ceph-mon[36190]: pgmap v2425: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:01.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:01 smithi059 ceph-mon[36190]: pgmap v2426: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:01.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:01 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:27:01.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:01 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:27:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:01 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:27:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:01 smithi170 ceph-mon[36280]: pgmap v2425: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:01 smithi170 ceph-mon[36280]: pgmap v2426: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:01.535 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:01 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:27:02.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:02 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3616.172932 secs 2024-03-28T19:27:02.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:02 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3616.172932 secs 2024-03-28T19:27:02.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:02 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3616.172932 secs 2024-03-28T19:27:03.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:03 smithi037 ceph-mon[31460]: pgmap v2427: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:03.534 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:03 smithi059 ceph-mon[36190]: pgmap v2427: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:03.534 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:03 smithi170 ceph-mon[36280]: pgmap v2427: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:05.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:05 smithi059 ceph-mon[36190]: pgmap v2428: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:05.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:05 smithi170 ceph-mon[36280]: pgmap v2428: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:05.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:05 smithi037 ceph-mon[31460]: pgmap v2428: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3621.173055 secs 2024-03-28T19:27:06.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3621.173055 secs 2024-03-28T19:27:06.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3621.173055 secs 2024-03-28T19:27:07.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:07 smithi059 ceph-mon[36190]: pgmap v2429: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:27:07.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:07 smithi170 ceph-mon[36280]: pgmap v2429: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:27:07.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:07 smithi037 ceph-mon[31460]: pgmap v2429: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:27:09.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:09 smithi059 ceph-mon[36190]: pgmap v2430: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:09.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:09 smithi170 ceph-mon[36280]: pgmap v2430: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:09.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:09 smithi037 ceph-mon[31460]: pgmap v2430: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:11 smithi059 ceph-mon[36190]: pgmap v2431: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:27:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3626.173152 secs 2024-03-28T19:27:11.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:11 smithi170 ceph-mon[36280]: pgmap v2431: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:27:11.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3626.173152 secs 2024-03-28T19:27:11.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:11 smithi037 ceph-mon[31460]: pgmap v2431: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:27:11.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3626.173152 secs 2024-03-28T19:27:13.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:13 smithi059 ceph-mon[36190]: pgmap v2432: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:13.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:13 smithi170 ceph-mon[36280]: pgmap v2432: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:13.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:13 smithi037 ceph-mon[31460]: pgmap v2432: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:15.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:15 smithi059 ceph-mon[36190]: pgmap v2433: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:15.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:15 smithi170 ceph-mon[36280]: pgmap v2433: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:15.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:15 smithi037 ceph-mon[31460]: pgmap v2433: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:16.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3631.173247 secs 2024-03-28T19:27:16.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3631.173247 secs 2024-03-28T19:27:16.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3631.173247 secs 2024-03-28T19:27:17.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:17 smithi059 ceph-mon[36190]: pgmap v2434: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:17.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:17 smithi170 ceph-mon[36280]: pgmap v2434: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:17.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:17 smithi037 ceph-mon[31460]: pgmap v2434: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:19.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:19 smithi059 ceph-mon[36190]: pgmap v2435: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:19.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:19 smithi170 ceph-mon[36280]: pgmap v2435: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:19.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:19 smithi037 ceph-mon[31460]: pgmap v2435: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:21.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:21 smithi059 ceph-mon[36190]: pgmap v2436: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:21.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3636.173341 secs 2024-03-28T19:27:21.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:21 smithi170 ceph-mon[36280]: pgmap v2436: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:21.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3636.173341 secs 2024-03-28T19:27:21.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:21 smithi037 ceph-mon[31460]: pgmap v2436: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:21.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3636.173341 secs 2024-03-28T19:27:23.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:23 smithi059 ceph-mon[36190]: pgmap v2437: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:23.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:23 smithi170 ceph-mon[36280]: pgmap v2437: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:23.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:23 smithi037 ceph-mon[31460]: pgmap v2437: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:25.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:25 smithi059 ceph-mon[36190]: pgmap v2438: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:25.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:25 smithi170 ceph-mon[36280]: pgmap v2438: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:25.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:25 smithi037 ceph-mon[31460]: pgmap v2438: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:26.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3641.173440 secs 2024-03-28T19:27:26.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3641.173440 secs 2024-03-28T19:27:27.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3641.173440 secs 2024-03-28T19:27:27.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:27 smithi037 ceph-mon[31460]: pgmap v2439: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:28.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:27 smithi059 ceph-mon[36190]: pgmap v2439: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:28.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:27 smithi170 ceph-mon[36280]: pgmap v2439: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:29.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:29 smithi037 ceph-mon[31460]: pgmap v2440: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:30.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:29 smithi059 ceph-mon[36190]: pgmap v2440: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:30.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:29 smithi170 ceph-mon[36280]: pgmap v2440: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:31 smithi037 ceph-mon[31460]: pgmap v2441: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3646.173558 secs 2024-03-28T19:27:32.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:31 smithi059 ceph-mon[36190]: pgmap v2441: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:32.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3646.173558 secs 2024-03-28T19:27:32.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:31 smithi170 ceph-mon[36280]: pgmap v2441: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:32.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3646.173558 secs 2024-03-28T19:27:33.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:33 smithi037 ceph-mon[31460]: pgmap v2442: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:34.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:33 smithi059 ceph-mon[36190]: pgmap v2442: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:34.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:33 smithi170 ceph-mon[36280]: pgmap v2442: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:35.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:35 smithi037 ceph-mon[31460]: pgmap v2443: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:36.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:35 smithi059 ceph-mon[36190]: pgmap v2443: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:36.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:35 smithi170 ceph-mon[36280]: pgmap v2443: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:36.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3651.173655 secs 2024-03-28T19:27:37.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3651.173655 secs 2024-03-28T19:27:37.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3651.173655 secs 2024-03-28T19:27:37.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:37 smithi037 ceph-mon[31460]: pgmap v2444: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:38.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:37 smithi059 ceph-mon[36190]: pgmap v2444: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:38.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:37 smithi170 ceph-mon[36280]: pgmap v2444: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:39.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:39 smithi037 ceph-mon[31460]: pgmap v2445: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:27:40.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:39 smithi059 ceph-mon[36190]: pgmap v2445: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:27:40.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:39 smithi170 ceph-mon[36280]: pgmap v2445: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:27:41.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:41 smithi037 ceph-mon[31460]: pgmap v2446: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:27:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3656.173774 secs 2024-03-28T19:27:42.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:41 smithi059 ceph-mon[36190]: pgmap v2446: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:27:42.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3656.173774 secs 2024-03-28T19:27:42.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:41 smithi170 ceph-mon[36280]: pgmap v2446: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:27:42.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3656.173774 secs 2024-03-28T19:27:43.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:43 smithi037 ceph-mon[31460]: pgmap v2447: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:27:44.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:43 smithi059 ceph-mon[36190]: pgmap v2447: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:27:44.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:43 smithi170 ceph-mon[36280]: pgmap v2447: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:27:45.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:45 smithi037 ceph-mon[31460]: pgmap v2448: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:27:46.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:45 smithi059 ceph-mon[36190]: pgmap v2448: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:27:46.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:45 smithi170 ceph-mon[36280]: pgmap v2448: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:27:46.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3661.173892 secs 2024-03-28T19:27:47.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3661.173892 secs 2024-03-28T19:27:47.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3661.173892 secs 2024-03-28T19:27:47.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:47 smithi037 ceph-mon[31460]: pgmap v2449: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:27:48.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:47 smithi059 ceph-mon[36190]: pgmap v2449: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:27:48.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:47 smithi170 ceph-mon[36280]: pgmap v2449: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T19:27:49.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:49 smithi037 ceph-mon[31460]: pgmap v2450: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:27:50.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:49 smithi059 ceph-mon[36190]: pgmap v2450: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:27:50.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:49 smithi170 ceph-mon[36280]: pgmap v2450: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:27:51.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:51 smithi037 ceph-mon[31460]: pgmap v2451: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:51.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:51 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3666.174012 secs 2024-03-28T19:27:52.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:51 smithi059 ceph-mon[36190]: pgmap v2451: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:52.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:51 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3666.174012 secs 2024-03-28T19:27:52.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:51 smithi170 ceph-mon[36280]: pgmap v2451: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:52.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:51 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3666.174012 secs 2024-03-28T19:27:53.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:53 smithi037 ceph-mon[31460]: pgmap v2452: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:54.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:53 smithi059 ceph-mon[36190]: pgmap v2452: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:54.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:53 smithi170 ceph-mon[36280]: pgmap v2452: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:55.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:55 smithi037 ceph-mon[31460]: pgmap v2453: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:56.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:55 smithi059 ceph-mon[36190]: pgmap v2453: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:56.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:55 smithi170 ceph-mon[36280]: pgmap v2453: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:56.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:56 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3671.174085 secs 2024-03-28T19:27:56.935 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:56 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3671.174085 secs 2024-03-28T19:27:56.936 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:56 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3671.174085 secs 2024-03-28T19:27:57.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:57 smithi037 ceph-mon[31460]: pgmap v2454: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:58.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:57 smithi059 ceph-mon[36190]: pgmap v2454: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:58.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:57 smithi170 ceph-mon[36280]: pgmap v2454: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:59.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:27:58 smithi059 ceph-mon[36190]: pgmap v2455: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:59.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:27:58 smithi170 ceph-mon[36280]: pgmap v2455: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:59.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:27:58 smithi037 ceph-mon[31460]: pgmap v2455: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:01.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:01 smithi059 ceph-mon[36190]: pgmap v2456: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:01.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:01 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:28:01.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:01 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3676.174156 secs 2024-03-28T19:28:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:01 smithi170 ceph-mon[36280]: pgmap v2456: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:01 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:28:01.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:01 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3676.174156 secs 2024-03-28T19:28:01.827 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:01 smithi037 ceph-mon[31460]: pgmap v2456: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:01.827 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:01 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:28:01.827 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:01 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3676.174156 secs 2024-03-28T19:28:02.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:02 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:28:02.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:02 smithi059 ceph-mon[36190]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:28:02.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:02 smithi059 ceph-mon[36190]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:28:02.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:02 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:28:02.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:02 smithi170 ceph-mon[36280]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:28:02.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:02 smithi170 ceph-mon[36280]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:28:02.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:02 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:28:02.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:02 smithi037 ceph-mon[31460]: from='mgr.15207 172.21.15.37:0/1575909284' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:28:02.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:02 smithi037 ceph-mon[31460]: from='mgr.15207 ' entity='mgr.x' 2024-03-28T19:28:03.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:03 smithi059 ceph-mon[36190]: pgmap v2457: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:28:03.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:03 smithi059 ceph-mon[36190]: pgmap v2458: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:03.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:03 smithi170 ceph-mon[36280]: pgmap v2457: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:28:03.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:03 smithi170 ceph-mon[36280]: pgmap v2458: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:03.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:03 smithi037 ceph-mon[31460]: pgmap v2457: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:28:03.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:03 smithi037 ceph-mon[31460]: pgmap v2458: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:05.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:05 smithi059 ceph-mon[36190]: pgmap v2459: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:05.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:05 smithi170 ceph-mon[36280]: pgmap v2459: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:05.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:05 smithi037 ceph-mon[31460]: pgmap v2459: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:06.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:06 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3681.174278 secs 2024-03-28T19:28:06.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:06 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3681.174278 secs 2024-03-28T19:28:06.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:06 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3681.174278 secs 2024-03-28T19:28:07.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:07 smithi059 ceph-mon[36190]: pgmap v2460: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:07.784 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:07 smithi170 ceph-mon[36280]: pgmap v2460: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:07.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:07 smithi037 ceph-mon[31460]: pgmap v2460: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:09.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:09 smithi059 ceph-mon[36190]: pgmap v2461: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:09.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:09 smithi170 ceph-mon[36280]: pgmap v2461: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:09.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:09 smithi037 ceph-mon[31460]: pgmap v2461: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:11 smithi059 ceph-mon[36190]: pgmap v2462: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:11.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:11 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3686.174398 secs 2024-03-28T19:28:11.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:11 smithi170 ceph-mon[36280]: pgmap v2462: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:11.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:11 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3686.174398 secs 2024-03-28T19:28:11.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:11 smithi037 ceph-mon[31460]: pgmap v2462: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:11.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:11 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3686.174398 secs 2024-03-28T19:28:13.784 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:13 smithi059 ceph-mon[36190]: pgmap v2463: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:28:13.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:13 smithi170 ceph-mon[36280]: pgmap v2463: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:28:13.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:13 smithi037 ceph-mon[31460]: pgmap v2463: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:28:15.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:15 smithi037 ceph-mon[31460]: pgmap v2464: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:16.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:15 smithi059 ceph-mon[36190]: pgmap v2464: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:16.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:15 smithi170 ceph-mon[36280]: pgmap v2464: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:16.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:16 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3691.174520 secs 2024-03-28T19:28:17.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:16 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3691.174520 secs 2024-03-28T19:28:17.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:16 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3691.174520 secs 2024-03-28T19:28:17.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:17 smithi037 ceph-mon[31460]: pgmap v2465: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:18.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:17 smithi059 ceph-mon[36190]: pgmap v2465: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:18.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:17 smithi170 ceph-mon[36280]: pgmap v2465: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:19.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:19 smithi037 ceph-mon[31460]: pgmap v2466: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:20.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:19 smithi059 ceph-mon[36190]: pgmap v2466: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:20.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:19 smithi170 ceph-mon[36280]: pgmap v2466: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:21.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:21 smithi037 ceph-mon[31460]: pgmap v2467: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:21.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:21 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3696.174640 secs 2024-03-28T19:28:22.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:21 smithi059 ceph-mon[36190]: pgmap v2467: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:22.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:21 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3696.174640 secs 2024-03-28T19:28:22.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:21 smithi170 ceph-mon[36280]: pgmap v2467: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:22.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:21 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3696.174640 secs 2024-03-28T19:28:23.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:23 smithi037 ceph-mon[31460]: pgmap v2468: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:24.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:23 smithi059 ceph-mon[36190]: pgmap v2468: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:24.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:23 smithi170 ceph-mon[36280]: pgmap v2468: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:25.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:25 smithi037 ceph-mon[31460]: pgmap v2469: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:26.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:25 smithi059 ceph-mon[36190]: pgmap v2469: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:26.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:25 smithi170 ceph-mon[36280]: pgmap v2469: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:26.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:26 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3701.174756 secs 2024-03-28T19:28:26.935 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:26 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3701.174756 secs 2024-03-28T19:28:26.935 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:26 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3701.174756 secs 2024-03-28T19:28:27.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:27 smithi037 ceph-mon[31460]: pgmap v2470: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:28.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:27 smithi059 ceph-mon[36190]: pgmap v2470: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:28.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:27 smithi170 ceph-mon[36280]: pgmap v2470: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:29.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:29 smithi037 ceph-mon[31460]: pgmap v2471: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:30.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:29 smithi059 ceph-mon[36190]: pgmap v2471: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:30.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:29 smithi170 ceph-mon[36280]: pgmap v2471: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:31 smithi037 ceph-mon[31460]: pgmap v2472: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:31.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:31 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3706.174875 secs 2024-03-28T19:28:32.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:31 smithi059 ceph-mon[36190]: pgmap v2472: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:32.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:31 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3706.174875 secs 2024-03-28T19:28:32.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:31 smithi170 ceph-mon[36280]: pgmap v2472: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:32.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:31 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3706.174875 secs 2024-03-28T19:28:33.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:33 smithi037 ceph-mon[31460]: pgmap v2473: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:34.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:33 smithi059 ceph-mon[36190]: pgmap v2473: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:34.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:33 smithi170 ceph-mon[36280]: pgmap v2473: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:35.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:35 smithi037 ceph-mon[31460]: pgmap v2474: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:36.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:35 smithi059 ceph-mon[36190]: pgmap v2474: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:36.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:35 smithi170 ceph-mon[36280]: pgmap v2474: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:36.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:36 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3711.174993 secs 2024-03-28T19:28:37.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:36 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3711.174993 secs 2024-03-28T19:28:37.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:36 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3711.174993 secs 2024-03-28T19:28:37.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:37 smithi037 ceph-mon[31460]: pgmap v2475: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:38.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:37 smithi059 ceph-mon[36190]: pgmap v2475: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:38.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:37 smithi170 ceph-mon[36280]: pgmap v2475: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:39.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:39 smithi037 ceph-mon[31460]: pgmap v2476: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:40.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:39 smithi059 ceph-mon[36190]: pgmap v2476: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:40.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:39 smithi170 ceph-mon[36280]: pgmap v2476: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:41.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:41 smithi037 ceph-mon[31460]: pgmap v2477: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:41.874 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:41 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3716.175110 secs 2024-03-28T19:28:42.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:41 smithi059 ceph-mon[36190]: pgmap v2477: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:42.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:41 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3716.175110 secs 2024-03-28T19:28:42.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:41 smithi170 ceph-mon[36280]: pgmap v2477: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:42.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:41 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3716.175110 secs 2024-03-28T19:28:43.873 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:43 smithi037 ceph-mon[31460]: pgmap v2478: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:44.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:43 smithi059 ceph-mon[36190]: pgmap v2478: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:44.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:43 smithi170 ceph-mon[36280]: pgmap v2478: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:46.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:45 smithi059 ceph-mon[36190]: pgmap v2479: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:46.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:45 smithi170 ceph-mon[36280]: pgmap v2479: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:46.123 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:45 smithi037 ceph-mon[31460]: pgmap v2479: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:47.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:46 smithi059 ceph-mon[36190]: 2 slow requests, 0 included below; oldest blocked for > 3721.175229 secs 2024-03-28T19:28:47.034 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:46 smithi059 ceph-mon[36190]: pgmap v2480: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:47.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:46 smithi170 ceph-mon[36280]: 2 slow requests, 0 included below; oldest blocked for > 3721.175229 secs 2024-03-28T19:28:47.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:46 smithi170 ceph-mon[36280]: pgmap v2480: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:46 smithi037 ceph-mon[31460]: 2 slow requests, 0 included below; oldest blocked for > 3721.175229 secs 2024-03-28T19:28:47.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:46 smithi037 ceph-mon[31460]: pgmap v2480: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:49.124 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:49 smithi037 ceph-mon[31460]: pgmap v2481: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:49.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:49 smithi059 ceph-mon[36190]: pgmap v2481: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:49.284 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:49 smithi170 ceph-mon[36280]: pgmap v2481: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:51.284 INFO:journalctl@ceph.mon.b.smithi059.stdout:Mar 28 19:28:51 smithi059 ceph-mon[36190]: pgmap v2482: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:51.285 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 28 19:28:51 smithi170 ceph-mon[36280]: pgmap v2482: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:51.373 INFO:journalctl@ceph.mon.a.smithi037.stdout:Mar 28 19:28:51 smithi037 ceph-mon[31460]: pgmap v2482: 97 pgs: 97 active+clean; 45 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:51.775 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-03-28T19:28:51.776 DEBUG:teuthology.task.console_log:Killing console logger for smithi037 2024-03-28T19:28:51.777 DEBUG:teuthology.task.console_log:Killing console logger for smithi059 2024-03-28T19:28:51.777 DEBUG:teuthology.task.console_log:Killing console logger for smithi170 2024-03-28T19:28:51.777 DEBUG:teuthology.exit:Finished running handlers